diff --git a/CHANGELOG.md b/CHANGELOG.md index f6424c7..6fffb35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,20 @@ All notable changes to BraggTrack will be documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.3.0] - 2026-11-17 + +### Added +- **Multi-scan beam-dropout gap closing (`braggtrack.tracking.gap_closing`)**: Added Jaqaman-style augmented linear assignment (`GapClosingConfig`, `GapCloseRecord`, `resolve_track_gaps`) with diffusive displacement scaling (`pos_d2 / delta_scans`), bidirectional spurious jump-edge severing, semantic cosine gating, and chronological track ID propagation (`--max-gap` across `track_dataset`, `detect_events`, and `run_pipeline`). +- **Chunked & lazy HDF5 streaming (`braggtrack.io.lazy`)**: Added context-managed `LazyVolume` (`open_lazy_volume`, `read_roi`, `iter_z_chunks`) and exact two-pass `otsu_threshold_chunked`, wired into `segment_dataset` Pass 1, `embed_dataset`, `detect_events`, and `crop_spot_cube` to avoid full-volume `float64` allocations on gigabyte-scale detector stacks. +- **Crystallographic scattering geometry & downstream exports (`braggtrack.io.crystallography`, `braggtrack.io.export`)**: Added `DetectorGeometry` (`spot_to_scattering`, `annotate_crystallographic_columns`) converting spot centroids into `(omega_deg, two_theta_deg, eta_deg, d_spacing_angstrom, qx/qy/qz/q_mag_inv_angstrom)` via Bragg's law, plus direct `ImageD11` columnfile export/parsing (`export_imaged11_flt`, `read_imaged11_flt`) and self-describing NeXus `NXprocess` HDF5 archival export (`export_nexus_process`). +- **Publication visualization package & interactive HTML 4D Explorer (`braggtrack.viz`, `braggtrack-report`)**: Added `plot_spot_mip_card`, `plot_scan_projection_overlay`, `plot_lineage_dag`, `plot_kinematics_dashboard`, `plot_precursor_timeline`, and `generate_html_report` (zero-CDN offline `report.html` with stdlib-encoded base64 3-view MIP thumbnails, pan/zoom SVG Lineage DAG, filter pills, and synchronized strain/precursor inspector), plus the 9th CLI entry point `braggtrack-report` and `--report` flag on `braggtrack-run`. +- **Quantitative 5-scenario real DINOv2 benchmark suite (`scripts/benchmark_suite.py`, `docs/benchmarks.md`)**: Added multi-trial bootstrap-CI benchmark harness comparing **Geometry-Only**, **Mock-Hash**, and **Real `facebook/dinov2-small`** across crossing tilted reflections, beam-flux jitter, mosaic rocking broadening, staggered beam dropouts (`max_gap=2`), and 4D chemomechanical fracture/precursor contrast, alongside publication figures (`docs/figures/`) and an ESRF-ID03 `facebook/dinov2-small` case study. +- **9th CI acceptance gate (`scripts/check_report_and_export_acceptance.py`)**: Wired automated verification of `--max-gap`, `report.html` + static figures, `ImageD11` `.flt` round-trip parsing, and NeXus `NXprocess` HDF5 schema into `scripts/ci_report.py`. + +### Fixed +- Fixed patch-grid shape mismatch and far-edge clipping in `segment_dino` when HuggingFace ViT processors resize non-multiple slices (e.g. `250x250 -> 224x224`, `16x16` tokens vs `250 // 14 = 17`), and honored `BRAGGTRACK_DINO_BACKEND` when `backend="auto"` in `braggtrack.semantic.dino`. +- Fixed `@Finegan2017` citation key/year mismatch (`2017`, DOI `10.1039/C7EE00385D`) and added `@Jaqaman2008` in `paper.bib` and `paper.md`; synced `CITATION.cff`, `CONTRIBUTING.md`, `docs/architecture.md`, and `README.md` to `v0.3.0`. + ## [0.2.0] - 2026-11-17 ### Added diff --git a/CITATION.cff b/CITATION.cff index 8b445ef..8d1e56b 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -2,22 +2,25 @@ cff-version: 1.2.0 message: "If you use this software, please cite it as below." title: "BraggTrack" type: software -version: 0.1.0 +version: 0.3.0 date-released: "2026-05-16" license: MIT repository-code: "https://github.com/BASE-Laboratory/BraggTrack" url: "https://github.com/BASE-Laboratory/BraggTrack" abstract: >- - Semantic 4D kinematics and fracture tracking for operando - X-ray diffraction. BraggTrack segments Bragg reflections in - 3D reciprocal-space volumes and tracks them across time using - classical image processing and self-supervised vision-transformer - embeddings. + Semantic 4D kinematics, multi-scan gap closing, and chemomechanical + fracture tracking for operando X-ray diffraction. BraggTrack segments + Bragg reflections in 3D reciprocal-space volumes, tracks trajectories + and split/merge lineage using classical crystallography and + vision-transformer embeddings, and exports ImageD11, NeXus HDF5, and + interactive HTML 4D lineage reports. keywords: - X-ray diffraction - operando crystallography - 3D segmentation - object tracking + - grain fracture + - synchrotron - materials science - Python authors: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1833647..f9b3376 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,13 +4,15 @@ Thanks for helping improve BraggTrack. ## Before you open a PR -1. Install the package in editable mode with dev/runtime dependencies you need (`pip install -e .` plus `numpy`, `scipy`, `scikit-image`, `networkx`; add `h5py` / `torch` / `transformers` as required). -2. Run **`python scripts/pre_pr_check.py`** (conflict markers and optional sync hint against `origin/main`). -3. Run **`python scripts/ci_report.py`** (unit tests + Week 1–4 acceptance scripts). +1. Install the package in editable mode with developer dependencies (`pip install -e ".[dev]"` for full PyTorch, notebook, pytest, and Ruff tooling, or `pip install -e .` for the lightweight core). +2. Run **`ruff check braggtrack/ tests/ scripts/`** and **`ruff format --check braggtrack/ tests/ scripts/`**. +3. Run **`python scripts/pre_pr_check.py`** (conflict markers and optional sync hint against `origin/main`). +4. Run **`python scripts/ci_report.py`** (unit tests + all 9 acceptance gates covering IO, smoke, segmentation, tracking, embedding, DINO, events/pipeline, and interactive report + ImageD11/NeXus exports). ## Where things live -- **[Architecture](docs/architecture.md)** — package layers, data flow, extension points. +- **[Architecture](docs/architecture.md)** — package layers (`io`, `segmentation`, `semantic`, `tracking`, `viz`, `cli`), data flow, extension points. +- **[Benchmarks](docs/benchmarks.md)** — 5-scenario quantitative evaluation (`scripts/benchmark_suite.py`) and ESRF-ID03 `facebook/dinov2-small` case study. - **[Coding practices](docs/coding-practices.md)** — style, testing, optional deps, CI expectations. - **Sample scan data** — versioned under [`data/sample_operando/`](data/sample_operando/README.md) (not the repo root). diff --git a/README.md b/README.md index 8f43b53..d61abd5 100644 --- a/README.md +++ b/README.md @@ -4,15 +4,24 @@ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb) -Semantic 4D kinematics, grain lineage (`1 -> 2+` fracture splits & `2+ -> 1` coalescence merges), and persistent fracture precursor tracking for operando X-ray diffraction using classical 3D crystallography and foundation vision transformers. +Semantic 4D kinematics, multi-scan beam-dropout gap closing, grain lineage (`1 -> 2+` fracture splits & `2+ -> 1` coalescence merges), and persistent fracture precursor tracking for operando X-ray diffraction using classical 3D crystallography and foundation vision transformers. + +

+ BraggTrack 4D lineage, precursor decomposition, and ESRF-ID03 strain kinematics +

## Key Capabilities -- **Beamline-ready NeXus/HDF5 I/O & calibration (`braggtrack.io`)** — Auto-discovers 3D detector cubes across standard NeXus paths, ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), and arbitrary HDF5 trees (`auto_discover=True`), with optional physical `(mu, chi, d)` coordinate calibration via `AxisCalibration`. -- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Intensity-domain Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`). +- **Beamline-scale NeXus/HDF5 I/O, lazy ROI streaming & crystallographic mapping (`braggtrack.io`)** — Auto-discovers 3D detector cubes across standard NeXus paths, ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), and arbitrary HDF5 trees (`auto_discover=True`). Streams `z`-slabs via `LazyVolume` (`otsu_threshold_chunked`, `read_roi`), calibrates `(mu, chi, d)` via `AxisCalibration`, maps transmission geometry `(2theta, eta, d, |Q|)` via `DetectorGeometry`, and exports directly to **`ImageD11` (`.flt`)** and **NeXus `NXprocess` HDF5 (`.nxs`)**. +- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Streaming Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`). - **Multi-view foundation vision descriptors (`braggtrack.semantic`)** — Orthogonal $(\mu, \chi, d)$ maximum-intensity projections (MIPs) encoded with ungated **`facebook/dinov2-small`** by default (zero HuggingFace login required), drop-in support for **DINOv3** (`facebook/dinov3-vitb16-pretrain-lvd1689m`), or a deterministic zero-GPU `mock` backend for CI. -- **4D grain kinematics, split/merge lineage & fracture precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association into a NetworkX trajectory DAG, per-grain lattice strain ($\Delta d / d_0$) and angular misorientation ($\Delta\mu$, $\Delta\chi$), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking combining 3D shape anisotropy, covariance spread growth, and spot ROI sub-peak / DINO patch-token dispersion. -- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, segmentation, semantic embedding, tracking, kinematics, and fracture event analysis end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`. +- **4D grain kinematics, multi-scan gap closing, split/merge lineage & precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association, Jaqaman-style multi-scan gap closing (`--max-gap`) across transient beam dropouts, per-grain lattice strain ($\Delta d / d_0$) and angular misorientation ($\Delta\mu$, $\Delta\chi$), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking combining 3D shape anisotropy, covariance spread growth, and spot ROI sub-peak / DINO patch-token dispersion. +- **Publication figures & offline interactive HTML 4D Explorer (`braggtrack.viz`, `braggtrack-report`)** — Generates Okabe–Ito styled lineage DAGs, strain/misorientation dashboards, precursor heatmaps, 3-view MIP cards, and a single-file interactive `report.html` with embedded base64 MIP thumbnails that opens offline on air-gapped beamline networks. +- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, segmentation, semantic embedding, tracking, kinematics, fracture event analysis, HTML reporting, and `ImageD11`/NeXus export end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`. + +

+ BraggTrack v0.3.0 Quantitative 4D Diffraction Benchmark Suite +

--- @@ -25,7 +34,7 @@ pip install -e . # With PyTorch + HuggingFace Transformers for real DINOv2 / DINOv3 backbones pip install -e ".[torch]" -# With Jupyter, Matplotlib, and Pandas for interactive notebooks +# With Jupyter, Matplotlib, and Pandas for interactive notebooks & static figures pip install -e ".[notebook]" # Full developer environment (torch + notebook + pytest + ruff) @@ -38,10 +47,14 @@ pip install -e ".[dev]" BraggTrack ships with three real ESRF ID03 operando Ni-rich cathode scans under [`data/sample_operando/`](data/sample_operando/README.md) (`100 x 250 x 250` `uint16` cubes). Omit the dataset path to run on the bundled scans automatically, or pass your own directory of `scanXXXX/` folders: -### One-command end-to-end pipeline +### One-command end-to-end pipeline (with interactive HTML report & NeXus/`ImageD11` exports) ```bash -braggtrack-run --outdir artifacts/pipeline +braggtrack-run --outdir artifacts/pipeline \ + --max-gap 1 \ + --report \ + --export-nexus artifacts/pipeline/results.nxs \ + --export-imaged11 artifacts/pipeline/peaks.flt ``` Produces: @@ -50,26 +63,29 @@ Produces: - `artifacts/pipeline/embedding/` (`embeddings.npz`, `embedding_manifest.json`) - `artifacts/pipeline/tracking/` (`tracks.csv`, `kinematics.csv`, `kinematics_summary.json`, `tracking_summary.json`, QC notebook) - `artifacts/pipeline/events/` (`events.csv`, `events.json`, `precursors.csv`, `events_summary.json`) +- `artifacts/pipeline/report/` (`report.html` interactive 4D Explorer + `figures/*.png`) +- `artifacts/pipeline/results.nxs` & `artifacts/pipeline/peaks.flt` - `artifacts/pipeline/run_manifest.json` (SHA-256 input checksums, git SHA, config, stage timings) -### Stage-by-stage CLI commands +### Stage-by-stage CLI commands (9 Entry Points) | CLI Entry Point | Module Equivalent | Purpose | |---|---|---| | `braggtrack-inspect` | `python -m braggtrack.cli.inspect_datasets` | Discover `scan*` folders, primary 3D HDF5 dataset paths, and metadata | | `braggtrack-validate` | `python -m braggtrack.cli.validate_dataset` | Validate sequence ordering, monotonic timestamps, and axis contracts | | `braggtrack-segment-synthetic` | `python -m braggtrack.cli.segment_synthetic` | Run fast synthetic 3D segmentation smoke test | -| `braggtrack-segment-dataset` | `python -m braggtrack.cli.segment_dataset` | Segment 3D Bragg spots across scans with smoothed thresholds & optional calibration | -| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Compute orthogonal MIP embeddings (`auto`, `mock`, or `torch` DINOv2/DINOv3) | -| `braggtrack-track-dataset` | `python -m braggtrack.cli.track_dataset` | Build trajectory DAG, export `tracks.csv` + `kinematics.csv` (+ optional `--detect-events`) | -| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and ROI-based fracture precursors | -| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config config.json` and `run_manifest.json` | +| `braggtrack-segment-dataset` | `python -m braggtrack.cli.segment_dataset` | Segment 3D Bragg spots via chunked Otsu + smoothed thresholds & optional calibration | +| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Stream lazy 3D spot ROIs & compute orthogonal MIP embeddings (`auto`, `mock`, `torch`) | +| `braggtrack-track-dataset` | `python -m braggtrack.cli.track_dataset` | Build trajectory DAG (`--max-gap`), export `tracks.csv`, `kinematics.csv`, `.flt`, `.nxs` | +| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and lazy-ROI fracture precursors | +| `braggtrack-report` | `python -m braggtrack.cli.report_dataset` | Render publication PNG/PDF figures and self-contained interactive `report.html` | +| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config config.json`, `--report`, and exports | --- -## Physical Axis Calibration (`mu`, `chi`, `d`) +## Physical Axis Calibration (`mu`, `chi`, `d`) & Crystallographic Geometry (`2theta`, `eta`, `|Q|`) -Cropped detector cubes store integer voxel indices `[0..N)`. For physically meaningful lattice strain $\varepsilon = (d(t) - d_0) / d_0$ and angular misorientation in degrees, supply an `AxisCalibration` JSON file (or CLI flags) when segmenting: +Cropped detector cubes store integer voxel indices `[0..N)`. For physically meaningful lattice strain $\varepsilon = (d(t) - d_0) / d_0$ and angular misorientation in degrees, supply an `AxisCalibration` JSON file when segmenting: ```json { @@ -82,30 +98,36 @@ Cropped detector cubes store integer voxel indices `[0..N)`. For physically mean } ``` -```bash -braggtrack-segment-dataset data/sample_operando \ - --calibration-json calibration.json \ - --outdir artifacts/segmentation -``` +For transmission 3DXRD / `ImageD11` indexing workflows, supply a `DetectorGeometry` JSON file (`--detector-json`) to annotate `omega_deg`, `two_theta_deg`, `eta_deg`, `d_spacing_angstrom`, and `(qx, qy, qz, q_mag_inv_angstrom)` and export `.flt` columnfiles: -If HDF5 scans already include 1D `mu`, `chi`, `d` coordinate datasets or `/entry/calibration` attributes, `extract_axis_calibration()` detects them automatically. +```json +{ + "distance_mm": 150.0, + "pixel_size_y_mm": 0.075, + "pixel_size_x_mm": 0.075, + "beam_center_y_px": 1024.0, + "beam_center_x_px": 1024.0, + "wavelength_angstrom": 0.2898, + "omega_origin_deg": 0.0, + "omega_step_deg": 0.05 +} +``` --- -## Vision Backbone Options (DINOv2 Default & Optional DINOv3) +## Vision Backbone Options & Quantitative Benchmarks - **Default (`facebook/dinov2-small`)**: Public and ungated on HuggingFace — works out of the box with `pip install -e ".[torch]"` without tokens or license gates. -- **DINOv3 (`facebook/dinov3-vitb16-pretrain-lvd1689m`)**: Supported via `--model` / `--dino-model` after accepting Meta's license on HuggingFace and setting `HF_TOKEN` (or running `huggingface-cli login`): - ```bash - braggtrack-embed-dataset --backend torch --model facebook/dinov3-vitb16-pretrain-lvd1689m - ``` +- **DINOv3 (`facebook/dinov3-vitb16-pretrain-lvd1689m`)**: Supported via `--model` / `--dino-model` after accepting Meta's license on HuggingFace and setting `HF_TOKEN`. - **Offline / CI (`mock`)**: Deterministic SHA-256 multi-view projection encoder requiring zero GPU or network access (`--backend mock` or `BRAGGTRACK_DINO_BACKEND=mock`). +See [`docs/benchmarks.md`](docs/benchmarks.md) for the full 5-scenario benchmark evaluation (`scripts/benchmark_suite.py`) comparing **Geometry-Only**, **Mock-Hash**, and **Real `facebook/dinov2-small`** alongside the ESRF-ID03 case study. + --- ## Notebooks -- [`notebooks/braggtrack_demo.ipynb`](notebooks/braggtrack_demo.ipynb) — End-to-end tutorial on the bundled ESRF-ID03 scans: discovery, stabilised 3D segmentation, physics & semantic tracking, per-grain kinematics, α/β ablation, and synthetic 4D grain fracture + precursor ranking. +- [`notebooks/braggtrack_demo.ipynb`](notebooks/braggtrack_demo.ipynb) — End-to-end tutorial on the bundled ESRF-ID03 scans: discovery, lazy/chunked segmentation, physics & semantic tracking, multi-scan gap closing, per-grain kinematics, `braggtrack.viz` lineage & precursor plots, and crystallographic geometry mapping. - [`notebooks/dino_segmentation_comparison.ipynb`](notebooks/dino_segmentation_comparison.ipynb) — Side-by-side comparison of stabilised classical 3D watershed vs. experimental DINO patch-clustering segmentation. --- @@ -120,9 +142,12 @@ ruff format --check braggtrack/ tests/ scripts/ # Unit test suite with coverage pytest tests/ -v --cov=braggtrack -# Full CI gate runner (unit tests + IO, smoke, segmentation, tracking, embedding, DINO, and events acceptance) +# Full CI gate runner (unit tests + all 9 acceptance gates) python scripts/ci_report.py +# Quantitative 5-scenario benchmark suite (add --include-torch for real DINOv2) +python scripts/benchmark_suite.py --trials 5 + # Pre-PR hygiene check python scripts/pre_pr_check.py ``` @@ -131,16 +156,16 @@ python scripts/pre_pr_check.py ## Troubleshooting & FAQ -- **`MissingH5DependencyError` or custom HDF5 dataset paths:** `h5py` is installed by default with `pip install -e .`. `load_primary_volume()` automatically inspects standard NeXus paths, `/entry_0000/ESRF-ID03/pco_nf/data`, and falls back to the largest 3D numeric dataset in the file. Pass `--allow-synthetic-fallback` only when testing schema plumbing on empty/metadata-only HDF5 stubs. +- **`MissingH5DependencyError` or custom HDF5 dataset paths:** `h5py` is installed by default with `pip install -e .`. `load_primary_volume()` and `LazyVolume` automatically inspect standard NeXus paths, `/entry_0000/ESRF-ID03/pco_nf/data`, and fall back to the largest 3D numeric dataset in the file. Pass `--allow-synthetic-fallback` only when testing schema plumbing on empty/metadata-only HDF5 stubs. - **`401 Unauthorized` / gated HuggingFace model error:** You passed a gated model ID (such as `facebook/dinov3-*`) without authenticating. Either omit `--model` to use the ungated default (`facebook/dinov2-small`), or accept the model license on HuggingFace and export `HF_TOKEN`. -- **Unphysical strain percentages in `kinematics.csv`:** Check `"calibrated_axes"` in `kinematics_summary.json`. When `false`, `centroid_d` is measured in cropped detector pixels (`y` index) rather than Angstroms — pass `--calibration-json` (or `--d-origin` / `--d-step`) to compute true crystallographic $\Delta d / d_0$. +- **Unphysical strain percentages in `kinematics.csv`:** Check `"calibrated_axes"` in `kinematics_summary.json`. When `false`, `centroid_d` is measured in cropped detector pixels (`y` index) rather than Angstroms — pass `--calibration-json` (or `--detector-json`) to compute physical crystallographic quantities. ## Repository Layout & Contributing -- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `cli`). +- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `viz`, `cli`). - **`data/sample_operando/`** — Bundled ESRF-ID03 operando test scans (`scan0001`–`scan0003`). -- **`docs/`** — [Architecture](docs/architecture.md) and [Coding Practices](docs/coding-practices.md). -- **`scripts/`** — Acceptance gates, semantic α/β ablation (`ablation_semantic.py`), and CI runner (`ci_report.py`). +- **`docs/`** — [Architecture](docs/architecture.md), [Benchmarks](docs/benchmarks.md), [Coding Practices](docs/coding-practices.md), and publication figures (`docs/figures/`). +- **`scripts/`** — 9 acceptance gates, 5-scenario benchmark suite (`benchmark_suite.py`), semantic α/β ablation (`ablation_semantic.py`), and CI runner (`ci_report.py`). - **`artifacts/`** — Generated pipeline outputs (gitignored; never commit). See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines and [paper.md](paper.md) for the manuscript overview. diff --git a/braggtrack/__init__.py b/braggtrack/__init__.py index 20123fa..14eeacb 100644 --- a/braggtrack/__init__.py +++ b/braggtrack/__init__.py @@ -5,6 +5,6 @@ try: __version__ = version("braggtrack") except PackageNotFoundError: - __version__ = "0.2.0.dev0" + __version__ = "0.3.0.dev0" -__all__ = ["io", "segmentation", "semantic", "tracking"] +__all__ = ["io", "segmentation", "semantic", "tracking", "viz"] diff --git a/braggtrack/cli/detect_events.py b/braggtrack/cli/detect_events.py index 321dfaf..9eb799d 100644 --- a/braggtrack/cli/detect_events.py +++ b/braggtrack/cli/detect_events.py @@ -4,6 +4,7 @@ import argparse import json +from contextlib import ExitStack from pathlib import Path from typing import Any @@ -13,14 +14,16 @@ from braggtrack.cli.track_dataset import _load_embeddings_npz, _merge_embeddings from braggtrack.io import ( BeamlineAdapter, + LazyVolume, MissingH5DependencyError, discover_operando_scans, - load_primary_volume, + open_lazy_volume, resolve_dataset_root, ) from braggtrack.semantic import DEFAULT_DINO_MODEL, make_patch_encoder from braggtrack.semantic.dino import _requested_backend, _resolve_backend from braggtrack.tracking import ( + GapClosingConfig, GeometrySemanticCost, PositionShapeCost, PrecursorConfig, @@ -66,6 +69,8 @@ def build_parser() -> argparse.ArgumentParser: parser.add_argument("--gate-chi", type=float, default=float("inf")) parser.add_argument("--gate-d", type=float, default=float("inf")) parser.add_argument("--max-cost", type=float, default=float("inf")) + parser.add_argument("--max-gap", type=int, default=0, help="Max missing intermediate scans to bridge (0 disables)") + parser.add_argument("--gap-max-cost", type=float, default=25.0, help="Max penalized cost for gap-closing bridges") parser.add_argument("--cost-alpha", type=float, default=1.0) parser.add_argument("--cost-beta", type=float, default=0.0) parser.add_argument("--split-max-distance", type=float, default=25.0) @@ -83,13 +88,14 @@ def build_parser() -> argparse.ArgumentParser: def _try_load_roi_inputs( + stack: ExitStack, root_arg: str | None, scan_dirs: list[Path], scan_names: list[str], *, allow_synthetic_fallback: bool, -) -> tuple[dict[int, np.ndarray], dict[int, np.ndarray], dict[int, str | None]]: - volumes: dict[int, np.ndarray] = {} +) -> tuple[dict[int, Any], dict[int, np.ndarray], dict[int, str | None]]: + volumes: dict[int, Any] = {} label_volumes: dict[int, np.ndarray] = {} timestamps: dict[int, str | None] = {} @@ -123,13 +129,13 @@ def _try_load_roi_inputs( if sf is None or idx not in label_volumes: continue try: - vol = load_primary_volume(sf.path) + vol: LazyVolume = stack.enter_context(open_lazy_volume(sf.path)) except MissingH5DependencyError: - vol = synth_volume_from_file(sf.path) + vol = LazyVolume.from_array(synth_volume_from_file(sf.path)) except (KeyError, ValueError): if not allow_synthetic_fallback: continue - vol = synth_volume_from_file(sf.path) + vol = LazyVolume.from_array(synth_volume_from_file(sf.path)) if vol.shape == label_volumes[idx].shape: volumes[idx] = vol @@ -175,63 +181,69 @@ def main(argv: list[str] | None = None) -> int: print(json.dumps({"error": "Missing embeddings.npz", "path": str(npz)})) return 1 - volumes, label_volumes, scan_timestamps = _try_load_roi_inputs( - args.root, - used_scan_dirs, - scan_names, - allow_synthetic_fallback=bool(args.allow_synthetic_fallback), - ) + with ExitStack() as stack: + volumes, label_volumes, scan_timestamps = _try_load_roi_inputs( + stack, + args.root, + used_scan_dirs, + scan_names, + allow_synthetic_fallback=bool(args.allow_synthetic_fallback), + ) - geo = PositionShapeCost( - position_weight=args.position_weight, - shape_weight=args.shape_weight, - gate_mu=args.gate_mu, - gate_chi=args.gate_chi, - gate_d=args.gate_d, - ) - cost_fn = ( - GeometrySemanticCost(geo, cost_alpha=args.cost_alpha, cost_beta=args.cost_beta) - if args.cost_beta != 0.0 - else geo - ) - sm_cfg = SplitMergeConfig( - max_distance=args.split_max_distance, - gate_mu=args.gate_mu, - gate_chi=args.gate_chi, - gate_d=args.gate_d, - min_intensity_ratio=args.min_intensity_ratio, - max_intensity_ratio=args.max_intensity_ratio, - min_fragment_share=args.min_fragment_share, - ) + geo = PositionShapeCost( + position_weight=args.position_weight, + shape_weight=args.shape_weight, + gate_mu=args.gate_mu, + gate_chi=args.gate_chi, + gate_d=args.gate_d, + ) + cost_fn = ( + GeometrySemanticCost(geo, cost_alpha=args.cost_alpha, cost_beta=args.cost_beta) + if args.cost_beta != 0.0 + else geo + ) + gap_cfg = GapClosingConfig(max_gap=args.max_gap, max_cost=args.gap_max_cost) if args.max_gap > 0 else None + sm_cfg = SplitMergeConfig( + max_distance=args.split_max_distance, + gate_mu=args.gate_mu, + gate_chi=args.gate_chi, + gate_d=args.gate_d, + min_intensity_ratio=args.min_intensity_ratio, + max_intensity_ratio=args.max_intensity_ratio, + min_fragment_share=args.min_fragment_share, + ) - G = build_tracks( - scan_tables, - cost_fn=cost_fn, - max_cost=args.max_cost, - split_merge_config=sm_cfg, - scan_timestamps=scan_timestamps or None, - ) - metrics = compute_tracking_metrics(G, n_scans=len(scan_tables)) - lineage_events = list(G.graph.get("lineage_events", [])) - track_rows = tracks_to_table(G) - for tr in track_rows: - tr.pop("embedding", None) + G = build_tracks( + scan_tables, + cost_fn=cost_fn, + max_cost=args.max_cost, + gap_closing_config=gap_cfg, + split_merge_config=sm_cfg, + scan_timestamps=scan_timestamps or None, + ) + metrics = compute_tracking_metrics(G, n_scans=len(scan_tables)) + lineage_events = list(G.graph.get("lineage_events", [])) + track_rows = tracks_to_table(G) + for tr in track_rows: + tr.pop("embedding", None) - resolved_backend = _resolve_backend(_requested_backend(args.backend)) # type: ignore[arg-type] - patch_encoder = make_patch_encoder(resolved_backend, model_name=args.model) if (volumes and label_volumes) else None + resolved_backend = _resolve_backend(_requested_backend(args.backend)) # type: ignore[arg-type] + patch_encoder = ( + make_patch_encoder(resolved_backend, model_name=args.model) if (volumes and label_volumes) else None + ) - prec_cfg = PrecursorConfig( - score_threshold=args.precursor_threshold, - min_persistence=max(1, args.precursor_persistence), - ) - candidates, obs_scores = compute_fracture_precursors( - track_rows, - prec_cfg, - volumes=volumes or None, - label_volumes=label_volumes or None, - patch_encoder=patch_encoder, - lineage_events=lineage_events, - ) + prec_cfg = PrecursorConfig( + score_threshold=args.precursor_threshold, + min_persistence=max(1, args.precursor_persistence), + ) + candidates, obs_scores = compute_fracture_precursors( + track_rows, + prec_cfg, + volumes=volumes or None, + label_volumes=label_volumes or None, + patch_encoder=patch_encoder, + lineage_events=lineage_events, + ) event_rows = events_to_table(lineage_events) prec_rows = precursors_to_table(candidates) @@ -262,6 +274,7 @@ def main(argv: list[str] | None = None) -> int: "scan_names": scan_names, "n_scans": len(scan_tables), "total_tracks": metrics["total_tracks"], + "gap_closed_count": metrics["gap_closed_count"], "split_count": metrics["split_count"], "merge_count": metrics["merge_count"], "event_count": len(event_rows), diff --git a/braggtrack/cli/embed_dataset.py b/braggtrack/cli/embed_dataset.py index b819c01..f1baec7 100644 --- a/braggtrack/cli/embed_dataset.py +++ b/braggtrack/cli/embed_dataset.py @@ -10,9 +10,10 @@ from braggtrack.cli._utils import load_feature_csv, synth_volume_from_file from braggtrack.io import ( + LazyVolume, MissingH5DependencyError, discover_operando_scans, - load_primary_volume, + open_lazy_volume, resolve_dataset_root, ) from braggtrack.semantic import DEFAULT_DINO_MODEL, crop_spot_cube, make_multiview_encoder, orthogonal_mips @@ -83,38 +84,39 @@ def main(argv: list[str] | None = None) -> int: return 1 try: - volume = load_primary_volume(scan_file.path) + lazy_vol = open_lazy_volume(scan_file.path) except MissingH5DependencyError: - volume = synth_volume_from_file(scan_file.path) + lazy_vol = LazyVolume.from_array(synth_volume_from_file(scan_file.path)) except (KeyError, ValueError): if not args.allow_synthetic_fallback: raise - volume = synth_volume_from_file(scan_file.path) - - if volume.shape != labels_full.shape: - print( - json.dumps( - { - "error": "Volume/labels shape mismatch", - "scan": name, - "volume": list(volume.shape), - "labels": list(labels_full.shape), - } + lazy_vol = LazyVolume.from_array(synth_volume_from_file(scan_file.path)) + + with lazy_vol: + if lazy_vol.shape != labels_full.shape: + print( + json.dumps( + { + "error": "Volume/labels shape mismatch", + "scan": name, + "volume": list(lazy_vol.shape), + "labels": list(labels_full.shape), + } + ) ) - ) - return 1 + return 1 + + label_ids: list[int] = [] + vectors: list[np.ndarray] = [] - label_ids: list[int] = [] - vectors: list[np.ndarray] = [] - - for row in rows: - lid = int(row["label"]) - mip_mu, mip_chi, mip_d = orthogonal_mips( - crop_spot_cube(volume, labels_full, lid, row, margin=max(0, args.margin))[0] - ) - vec = enc.embed(mip_mu, mip_chi, mip_d) - label_ids.append(lid) - vectors.append(vec) + for row in rows: + lid = int(row["label"]) + mip_mu, mip_chi, mip_d = orthogonal_mips( + crop_spot_cube(lazy_vol, labels_full, lid, row, margin=max(0, args.margin))[0] + ) + vec = enc.embed(mip_mu, mip_chi, mip_d) + label_ids.append(lid) + vectors.append(vec) if not vectors: dim = 384 diff --git a/braggtrack/cli/report_dataset.py b/braggtrack/cli/report_dataset.py new file mode 100644 index 0000000..f6ac4fc --- /dev/null +++ b/braggtrack/cli/report_dataset.py @@ -0,0 +1,140 @@ +"""Generate publication figures and an interactive HTML 4D beamtime explorer from BraggTrack outputs.""" + +from __future__ import annotations + +import argparse +import csv +import json +from pathlib import Path +from typing import Any + +from braggtrack.viz import ( + generate_html_report, + plot_kinematics_dashboard, + plot_lineage_dag, + plot_precursor_timeline, +) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "artifact_dir", + nargs="?", + default="artifacts/pipeline", + help="Pipeline or tracking output directory containing tracks.csv / events / manifest", + ) + parser.add_argument( + "--dataset-root", + default=None, + help="Optional raw HDF5 dataset root for embedding 3-axis MIP spot thumbnails", + ) + parser.add_argument( + "--outdir", + default=None, + help="Optional report output directory (defaults to /report)", + ) + parser.add_argument( + "--no-figures", + action="store_true", + help="Skip static Matplotlib PNG/PDF generation and emit only the self-contained HTML explorer", + ) + parser.add_argument( + "--max-mip-tracks", + type=int, + default=16, + help="Maximum priority tracks for lazy 3-view MIP thumbnail extraction", + ) + return parser + + +def _read_csv_rows(path: Path) -> list[dict[str, str]]: + if not path.exists() or path.stat().st_size == 0: + return [] + with path.open(newline="") as f: + return list(csv.DictReader(f)) + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + in_root = Path(args.artifact_dir) + if not in_root.is_dir(): + print(json.dumps({"error": "Artifact directory not found", "artifact_dir": str(in_root)})) + return 1 + + out_root = Path(args.outdir) if args.outdir else (in_root / "report") + out_root.mkdir(parents=True, exist_ok=True) + + trk_dir = in_root / "tracking" if (in_root / "tracking").is_dir() else in_root + evt_dir = in_root / "events" if (in_root / "events").is_dir() else trk_dir + track_rows = _read_csv_rows(evt_dir / "tracks.csv") or _read_csv_rows(trk_dir / "tracks.csv") + if not track_rows: + print(json.dumps({"error": "No tracks.csv found under artifact directory", "artifact_dir": str(in_root)})) + return 1 + + html_path = generate_html_report( + in_root, + out_root / "report.html", + dataset_root=args.dataset_root, + max_mip_tracks=max(0, args.max_mip_tracks), + ) + + figures: dict[str, str] = {} + if not args.no_figures: + try: + import matplotlib + + matplotlib.use("Agg", force=False) + import matplotlib.pyplot as plt + + fig_dir = out_root / "figures" + fig_dir.mkdir(parents=True, exist_ok=True) + event_rows = _read_csv_rows(evt_dir / "events.csv") or _read_csv_rows(trk_dir / "events.csv") + prec_rows = _read_csv_rows(evt_dir / "precursors.csv") or _read_csv_rows(trk_dir / "precursors.csv") + prec_obs_rows = _read_csv_rows(evt_dir / "precursor_observations.csv") + + calibrated = False + ts_path = trk_dir / "tracking_summary.json" + if ts_path.exists(): + try: + calibrated = bool(json.loads(ts_path.read_text()).get("calibrated_axes", False)) + except (OSError, json.JSONDecodeError): + calibrated = False + + fig_dag, _ = plot_lineage_dag(track_rows, events=event_rows, save_path=fig_dir / "lineage_dag.png") + plt.close(fig_dag) + figures["lineage_dag"] = str((fig_dir / "lineage_dag.png").resolve()) + + fig_kin, _ = plot_kinematics_dashboard( + track_rows, + events=event_rows, + calibrated=calibrated, + save_path=fig_dir / "kinematics_dashboard.png", + ) + plt.close(fig_kin) + figures["kinematics_dashboard"] = str((fig_dir / "kinematics_dashboard.png").resolve()) + + if prec_obs_rows: + fig_prec, _ = plot_precursor_timeline( + prec_obs_rows, + candidates=prec_rows, + save_path=fig_dir / "precursor_timeline.png", + ) + plt.close(fig_prec) + figures["precursor_timeline"] = str((fig_dir / "precursor_timeline.png").resolve()) + except ImportError: + pass + + summary: dict[str, Any] = { + "schema_version": "report.v1", + "html_report": str(html_path.resolve()), + "figures": figures, + "track_row_count": len(track_rows), + } + (out_root / "report_summary.json").write_text(json.dumps(summary, indent=2)) + print(json.dumps(summary, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/braggtrack/cli/run_pipeline.py b/braggtrack/cli/run_pipeline.py index 12ef93e..756f5b1 100644 --- a/braggtrack/cli/run_pipeline.py +++ b/braggtrack/cli/run_pipeline.py @@ -14,7 +14,7 @@ from typing import Any import braggtrack -from braggtrack.cli import detect_events, embed_dataset, segment_dataset, track_dataset +from braggtrack.cli import detect_events, embed_dataset, report_dataset, segment_dataset, track_dataset from braggtrack.io import BeamlineAdapter, discover_operando_scans, resolve_dataset_root, validate_sequence from braggtrack.semantic import DEFAULT_DINO_MODEL @@ -48,8 +48,12 @@ def build_parser() -> argparse.ArgumentParser: ) parser.add_argument("--smooth-window", type=int, default=5, help="Cross-scan rolling median threshold window") parser.add_argument("--calibration-json", default=None, help="Optional AxisCalibration JSON path") + parser.add_argument( + "--detector-json", default=None, help="Optional DetectorGeometry JSON path for (2theta, eta, Q)" + ) parser.add_argument("--cost-alpha", type=float, default=1.0, help="Geometry weight in semantic cost") parser.add_argument("--cost-beta", type=float, default=0.25, help="Semantic embedding weight in tracking cost") + parser.add_argument("--max-gap", type=int, default=0, help="Max missing intermediate scans to bridge (0 disables)") parser.add_argument( "--split-max-distance", type=float, default=25.0, help="Max centroid distance for splits/merges" ) @@ -61,6 +65,13 @@ def build_parser() -> argparse.ArgumentParser: default=True, help="Run split/merge lineage and fracture precursor detection stage (default: enabled)", ) + parser.add_argument( + "--report", + action="store_true", + help="Generate static publication figures and interactive HTML 4D explorer under /report", + ) + parser.add_argument("--export-imaged11", default=None, help="Optional ImageD11 .flt export path") + parser.add_argument("--export-nexus", default=None, help="Optional NeXus NXprocess .nxs/.h5 export path") parser.add_argument( "--allow-synthetic-fallback", action="store_true", @@ -102,12 +113,17 @@ def _merge_config_overrides(args: argparse.Namespace) -> dict[str, Any]: "method": args.method, "smooth_window": args.smooth_window, "calibration_json": args.calibration_json, + "detector_json": args.detector_json, "cost_alpha": args.cost_alpha, "cost_beta": args.cost_beta, + "max_gap": args.max_gap, "split_max_distance": args.split_max_distance, "precursor_threshold": args.precursor_threshold, "precursor_persistence": args.precursor_persistence, "detect_events": bool(args.detect_events), + "report": bool(args.report), + "export_imaged11": args.export_imaged11, + "export_nexus": args.export_nexus, "allow_synthetic_fallback": bool(args.allow_synthetic_fallback), "segmentation_extra_args": [], "tracking_extra_args": [], @@ -132,6 +148,7 @@ def main(argv: list[str] | None = None) -> int: emb_dir = outdir / "embedding" trk_dir = outdir / "tracking" evt_dir = outdir / "events" + rep_dir = outdir / "report" timings: dict[str, float] = {} t_total0 = time.perf_counter() @@ -190,7 +207,7 @@ def main(argv: list[str] | None = None) -> int: if rc_seg != 0: return rc_seg - # 3. Embed dataset (whenever cost_beta != 0 or detect_events is enabled) + # 3. Embed dataset (whenever cost_beta != 0) need_embed = float(cfg["cost_beta"]) != 0.0 if need_embed: t0 = time.perf_counter() @@ -222,6 +239,8 @@ def main(argv: list[str] | None = None) -> int: str(cfg["cost_alpha"]), "--cost-beta", str(cfg["cost_beta"]), + "--max-gap", + str(cfg["max_gap"]), "--split-max-distance", str(cfg["split_max_distance"]), "--precursor-threshold", @@ -233,6 +252,12 @@ def main(argv: list[str] | None = None) -> int: trk_argv.extend(["--embedding-dir", str(emb_dir)]) if cfg.get("detect_events"): trk_argv.append("--detect-events") + if cfg.get("detector_json"): + trk_argv.extend(["--detector-json", str(cfg["detector_json"])]) + if cfg.get("export_imaged11"): + trk_argv.extend(["--export-imaged11", str(cfg["export_imaged11"])]) + if cfg.get("export_nexus") and not cfg.get("detect_events"): + trk_argv.extend(["--export-nexus", str(cfg["export_nexus"])]) trk_argv.extend(str(x) for x in cfg.get("tracking_extra_args", [])) rc_trk = track_dataset.main(trk_argv) timings["track"] = round(time.perf_counter() - t0, 4) @@ -256,6 +281,8 @@ def main(argv: list[str] | None = None) -> int: str(cfg["cost_alpha"]), "--cost-beta", str(cfg["cost_beta"]), + "--max-gap", + str(cfg["max_gap"]), "--split-max-distance", str(cfg["split_max_distance"]), "--precursor-threshold", @@ -272,34 +299,57 @@ def main(argv: list[str] | None = None) -> int: if rc_evt != 0: return rc_evt - total_sec = round(time.perf_counter() - t_total0, 4) - manifest = { - "schema_version": "manifest.v1", - "created_at": datetime.now(timezone.utc).isoformat(), - "braggtrack_version": braggtrack.__version__, - "git_sha": _resolve_git_sha(), - "python_version": sys.version.split()[0], - "platform": platform.platform(), - "dataset_root": str(root.resolve()), - "outdir": str(outdir.resolve()), - "config": cfg, - "inputs": input_files, - "stage_timings_sec": timings, - "total_duration_sec": total_sec, - "artifacts": { - "validation_summary": str((outdir / "validation_summary.json").resolve()), - "segmentation_summary": str((seg_dir / "segmentation_summary.json").resolve()), - "embedding_summary": str((emb_dir / "embedding_summary.json").resolve()) if need_embed else None, - "tracking_summary": str((trk_dir / "tracking_summary.json").resolve()), - "kinematics_csv": str((trk_dir / "kinematics.csv").resolve()), - "kinematics_summary": str((trk_dir / "kinematics_summary.json").resolve()), - "events_summary": str((evt_dir / "events_summary.json").resolve()) if cfg.get("detect_events") else None, - "events_csv": str((evt_dir / "events.csv").resolve()) if cfg.get("detect_events") else None, - "precursors_csv": str((evt_dir / "precursors.csv").resolve()) if cfg.get("detect_events") else None, - }, - } + if cfg.get("export_nexus") and cfg.get("detect_events"): + from braggtrack.io import DetectorGeometry, export_nexus_process + + dgeom = DetectorGeometry.from_json(cfg["detector_json"]) if cfg.get("detector_json") else None + export_nexus_process(outdir, cfg["export_nexus"], geometry=dgeom) + manifest_path = outdir / "run_manifest.json" - manifest_path.write_text(json.dumps(manifest, indent=2)) + + def _build_manifest(cur_total_sec: float) -> dict[str, Any]: + return { + "schema_version": "manifest.v1", + "created_at": datetime.now(timezone.utc).isoformat(), + "braggtrack_version": braggtrack.__version__, + "git_sha": _resolve_git_sha(), + "python_version": sys.version.split()[0], + "platform": platform.platform(), + "dataset_root": str(root.resolve()), + "outdir": str(outdir.resolve()), + "config": cfg, + "inputs": input_files, + "stage_timings_sec": timings, + "total_duration_sec": cur_total_sec, + "artifacts": { + "validation_summary": str((outdir / "validation_summary.json").resolve()), + "segmentation_summary": str((seg_dir / "segmentation_summary.json").resolve()), + "embedding_summary": str((emb_dir / "embedding_summary.json").resolve()) if need_embed else None, + "tracking_summary": str((trk_dir / "tracking_summary.json").resolve()), + "kinematics_csv": str((trk_dir / "kinematics.csv").resolve()), + "kinematics_summary": str((trk_dir / "kinematics_summary.json").resolve()), + "events_summary": str((evt_dir / "events_summary.json").resolve()) + if cfg.get("detect_events") + else None, + "events_csv": str((evt_dir / "events.csv").resolve()) if cfg.get("detect_events") else None, + "precursors_csv": str((evt_dir / "precursors.csv").resolve()) if cfg.get("detect_events") else None, + "html_report": str((rep_dir / "report.html").resolve()) if cfg.get("report") else None, + }, + } + + total_sec = round(time.perf_counter() - t_total0, 4) + manifest_path.write_text(json.dumps(_build_manifest(total_sec), indent=2)) + + # 6. Optional interactive HTML & static figure report + if cfg.get("report"): + t0 = time.perf_counter() + rc_rep = report_dataset.main([str(outdir), "--dataset-root", str(root), "--outdir", str(rep_dir)]) + timings["report"] = round(time.perf_counter() - t0, 4) + if rc_rep != 0: + return rc_rep + total_sec = round(time.perf_counter() - t_total0, 4) + manifest_path.write_text(json.dumps(_build_manifest(total_sec), indent=2)) + print(json.dumps({"manifest": str(manifest_path.resolve()), "total_duration_sec": total_sec}, indent=2)) return 0 diff --git a/braggtrack/cli/segment_dataset.py b/braggtrack/cli/segment_dataset.py index aaaf79e..9f6ff29 100644 --- a/braggtrack/cli/segment_dataset.py +++ b/braggtrack/cli/segment_dataset.py @@ -16,6 +16,8 @@ discover_primary_volume_path, extract_axis_calibration, load_primary_volume, + open_lazy_volume, + otsu_threshold_chunked, resolve_dataset_root, ) from braggtrack.segmentation import ( @@ -191,12 +193,20 @@ def _load_one(scan_obj: object) -> tuple[np.ndarray, str, str | None, AxisCalibr src = "synthetic_fallback" return vol, src, dpath, scan_cal - # First pass: compute scalar per-scan Otsu thresholds in O(1) volume memory. + # First pass: compute scalar per-scan Otsu thresholds via streaming z-slabs. raw_thresholds: list[float] = [] for scan in scans: - vol_tmp, _, _, _ = _load_one(scan) - raw_thresholds.append(otsu_threshold(vol_tmp.ravel()) * float(args.threshold_fraction)) - del vol_tmp + scan_path = scan.path # type: ignore[attr-defined] + try: + with open_lazy_volume(scan_path) as lazy_tmp: + thr = otsu_threshold_chunked(lazy_tmp) + except MissingH5DependencyError: + thr = otsu_threshold(synth_volume_from_file(scan_path)) + except (KeyError, ValueError): + if not args.allow_synthetic_fallback: + raise + thr = otsu_threshold(synth_volume_from_file(scan_path)) + raw_thresholds.append(thr * float(args.threshold_fraction)) if args.method == "classical" and args.smooth_window > 1 and len(raw_thresholds) > 1: effective_thresholds = [float(x) for x in smooth_thresholds(raw_thresholds, window=args.smooth_window)] diff --git a/braggtrack/cli/track_dataset.py b/braggtrack/cli/track_dataset.py index 2472cf2..c73c107 100644 --- a/braggtrack/cli/track_dataset.py +++ b/braggtrack/cli/track_dataset.py @@ -10,7 +10,15 @@ import numpy as np from braggtrack.cli._utils import load_feature_csv, write_csv, write_qc_notebook +from braggtrack.io import ( + AxisCalibration, + DetectorGeometry, + annotate_crystallographic_columns, + export_imaged11_flt, + export_nexus_process, +) from braggtrack.tracking import ( + GapClosingConfig, GeometrySemanticCost, PositionShapeCost, PrecursorConfig, @@ -40,6 +48,18 @@ def build_parser() -> argparse.ArgumentParser: parser.add_argument("--gate-chi", type=float, default=float("inf")) parser.add_argument("--gate-d", type=float, default=float("inf")) parser.add_argument("--max-cost", type=float, default=float("inf")) + parser.add_argument( + "--max-gap", + type=int, + default=0, + help="Maximum missing intermediate scans to bridge via gap closing (0 disables)", + ) + parser.add_argument( + "--gap-max-cost", + type=float, + default=25.0, + help="Maximum penalized cost allowed when bridging missing scans", + ) parser.add_argument( "--embedding-dir", default=None, @@ -80,6 +100,21 @@ def build_parser() -> argparse.ArgumentParser: default=2, help="Minimum contiguous scans above threshold to rank as a fracture precursor", ) + parser.add_argument( + "--detector-json", + default=None, + help="Optional DetectorGeometry JSON path to annotate (omega, 2theta, eta, d, |Q|) columns", + ) + parser.add_argument( + "--export-imaged11", + default=None, + help="Optional path to write an ImageD11 .flt columnfile of tracked reflections", + ) + parser.add_argument( + "--export-nexus", + default=None, + help="Optional path to write a NeXus NXprocess HDF5 summary file", + ) return parser @@ -100,7 +135,7 @@ def _merge_embeddings(rows: list[dict[str, Any]], emb: dict[int, np.ndarray]) -> row["embedding"] = emb[lid] -def _detect_calibrated_axes(scan_dirs: list[Path]) -> bool: +def _detect_calibration(scan_dirs: list[Path]) -> AxisCalibration | None: for sd in scan_dirs: summary_path = sd / "summary.json" if not summary_path.exists(): @@ -111,8 +146,8 @@ def _detect_calibrated_axes(scan_dirs: list[Path]) -> bool: continue cal = meta.get("calibration") if isinstance(cal, dict) and bool(cal.get("calibrated", False)): - return True - return False + return AxisCalibration.from_dict(cal) + return None _TRACKING_QC_CODE = [ @@ -204,6 +239,7 @@ def main(argv: list[str] | None = None) -> int: else: cost_fn = geo + gap_cfg = GapClosingConfig(max_gap=args.max_gap, max_cost=args.gap_max_cost) if args.max_gap > 0 else None sm_cfg = ( SplitMergeConfig( max_distance=args.split_max_distance, @@ -214,17 +250,28 @@ def main(argv: list[str] | None = None) -> int: if args.detect_events else None ) - G = build_tracks(scan_tables, cost_fn=cost_fn, max_cost=args.max_cost, split_merge_config=sm_cfg) + G = build_tracks( + scan_tables, + cost_fn=cost_fn, + max_cost=args.max_cost, + gap_closing_config=gap_cfg, + split_merge_config=sm_cfg, + ) metrics = compute_tracking_metrics(G, n_scans=len(scan_tables)) + active_cal = _detect_calibration(used_scan_dirs) + calibrated_axes = active_cal is not None + detector_geom = DetectorGeometry.from_json(args.detector_json) if args.detector_json else None + track_rows = tracks_to_table(G) for tr in track_rows: tr.pop("embedding", None) + if detector_geom is not None: + track_rows = annotate_crystallographic_columns(track_rows, detector_geom, calibration=active_cal) write_csv(outdir / "tracks.csv", track_rows) (outdir / "tracking_metrics.json").write_text(json.dumps(metrics, indent=2)) # Per-grain kinematics export. - calibrated_axes = _detect_calibrated_axes(used_scan_dirs) gk = compute_grain_kinematics(track_rows) kin_summary = summarize_kinematics(gk, n_scans=len(scan_tables), calibrated_axes=calibrated_axes) write_csv(outdir / "kinematics.csv", kinematics_to_table(gk, include_delta_d=True)) @@ -257,12 +304,19 @@ def main(argv: list[str] | None = None) -> int: "schema_version": schema_version, "cost_alpha": args.cost_alpha, "cost_beta": args.cost_beta, + "max_gap": args.max_gap, "embedding_dir": str(emb_root) if emb_root else None, "detect_events": bool(args.detect_events), + "detector_geometry": detector_geom.to_dict() if detector_geom is not None else None, } (outdir / "tracking_summary.json").write_text(json.dumps(summary, indent=2)) write_qc_notebook(outdir / "qc" / "tracking_qc.ipynb", title="Tracking QC", code_source=_TRACKING_QC_CODE) + if args.export_imaged11: + export_imaged11_flt(track_rows, args.export_imaged11, geometry=detector_geom, calibration=active_cal) + if args.export_nexus: + export_nexus_process(outdir, args.export_nexus, geometry=detector_geom, calibration=active_cal) + print(json.dumps(summary, indent=2)) return 0 diff --git a/braggtrack/io/__init__.py b/braggtrack/io/__init__.py index c2b0577..9a37a54 100644 --- a/braggtrack/io/__init__.py +++ b/braggtrack/io/__init__.py @@ -1,7 +1,10 @@ """Data I/O helpers for BraggTrack.""" from .beamline import BeamlineAdapter +from .crystallography import DetectorGeometry, annotate_crystallographic_columns from .discovery import discover_operando_scans +from .export import IMAGED11_COLUMNS, export_imaged11_flt, export_nexus_process, read_imaged11_flt +from .lazy import LazyVolume, open_lazy_volume, otsu_threshold_chunked from .models import AxisCalibration, AxisSpec, ExperimentSequence, ScanVolumeMeta from .nexus import ( MissingH5DependencyError, @@ -15,21 +18,30 @@ from .validation import ValidationIssue, validate_sequence __all__ = [ + "AxisCalibration", + "AxisSpec", + "BeamlineAdapter", + "DetectorGeometry", + "ExperimentSequence", + "IMAGED11_COLUMNS", + "LazyVolume", + "MissingH5DependencyError", + "ScanVolumeMeta", + "ValidationIssue", + "annotate_crystallographic_columns", "default_dataset_root", - "resolve_dataset_root", - "sample_operando_root", "discover_operando_scans", "discover_primary_volume_path", + "export_imaged11_flt", + "export_nexus_process", "extract_axis_calibration", "extract_scan_metadata", - "summarize_hdf5_tree", "load_primary_volume", - "MissingH5DependencyError", - "AxisCalibration", - "AxisSpec", - "ScanVolumeMeta", - "ExperimentSequence", - "BeamlineAdapter", - "ValidationIssue", + "open_lazy_volume", + "otsu_threshold_chunked", + "read_imaged11_flt", + "resolve_dataset_root", + "sample_operando_root", + "summarize_hdf5_tree", "validate_sequence", ] diff --git a/braggtrack/io/crystallography.py b/braggtrack/io/crystallography.py new file mode 100644 index 0000000..d7169f8 --- /dev/null +++ b/braggtrack/io/crystallography.py @@ -0,0 +1,146 @@ +"""Transmission diffraction detector geometry and reciprocal-space (2theta, eta, |Q|) mapping.""" + +from __future__ import annotations + +import json +import math +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from .models import AxisCalibration + + +@dataclass(frozen=True) +class DetectorGeometry: + """Area-detector diffraction geometry for crystallographic angle & Q conversion. + + Follows standard transmission 3DXRD conventions (e.g. ESRF / ImageD11): + incident beam along ``+X_lab``, detector plane at ``X_lab = distance_mm``, + horizontal detector coordinate ``x_px`` (BraggTrack ``chi`` axis), + vertical detector coordinate ``y_px`` (BraggTrack ``d`` axis), and + rotation stack ``z_idx`` (BraggTrack ``mu`` / ``omega`` axis). + """ + + distance_mm: float = 150.0 + pixel_size_y_mm: float = 0.075 + pixel_size_x_mm: float = 0.075 + beam_center_y_px: float = 1024.0 + beam_center_x_px: float = 1024.0 + wavelength_angstrom: float = 0.2898 + omega_origin_deg: float = 0.0 + omega_step_deg: float = 0.05 + + def __post_init__(self) -> None: + if self.distance_mm <= 0.0: + raise ValueError("distance_mm must be positive.") + if self.pixel_size_y_mm <= 0.0 or self.pixel_size_x_mm <= 0.0: + raise ValueError("pixel_size_y_mm and pixel_size_x_mm must be positive.") + if self.wavelength_angstrom <= 0.0: + raise ValueError("wavelength_angstrom must be positive.") + + def to_dict(self) -> dict[str, float]: + return { + "distance_mm": float(self.distance_mm), + "pixel_size_y_mm": float(self.pixel_size_y_mm), + "pixel_size_x_mm": float(self.pixel_size_x_mm), + "beam_center_y_px": float(self.beam_center_y_px), + "beam_center_x_px": float(self.beam_center_x_px), + "wavelength_angstrom": float(self.wavelength_angstrom), + "omega_origin_deg": float(self.omega_origin_deg), + "omega_step_deg": float(self.omega_step_deg), + } + + @classmethod + def from_dict(cls, data: dict[str, Any]) -> DetectorGeometry: + return cls( + distance_mm=float(data.get("distance_mm", 150.0)), + pixel_size_y_mm=float(data.get("pixel_size_y_mm", 0.075)), + pixel_size_x_mm=float(data.get("pixel_size_x_mm", 0.075)), + beam_center_y_px=float(data.get("beam_center_y_px", 1024.0)), + beam_center_x_px=float(data.get("beam_center_x_px", 1024.0)), + wavelength_angstrom=float(data.get("wavelength_angstrom", 0.2898)), + omega_origin_deg=float(data.get("omega_origin_deg", 0.0)), + omega_step_deg=float(data.get("omega_step_deg", 0.05)), + ) + + @classmethod + def from_json(cls, path: str | Path) -> DetectorGeometry: + raw = json.loads(Path(path).read_text()) + if not isinstance(raw, dict): + raise ValueError("DetectorGeometry JSON must contain an object.") + return cls.from_dict(raw) + + def save_json(self, path: str | Path) -> None: + out = Path(path) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(self.to_dict(), indent=2), encoding="utf-8") + + def spot_to_scattering(self, z_idx: float, y_px: float, x_px: float) -> dict[str, float]: + """Convert detector voxel centroid ``(z_idx, y_px, x_px)`` to ``(omega, 2theta, eta, d, Q)``.""" + dx_mm = (float(x_px) - self.beam_center_x_px) * self.pixel_size_x_mm + dy_mm = (float(y_px) - self.beam_center_y_px) * self.pixel_size_y_mm + r_mm = math.hypot(dx_mm, dy_mm) + path_len = math.hypot(self.distance_mm, r_mm) + + two_theta_rad = math.atan2(r_mm, self.distance_mm) + two_theta_deg = math.degrees(two_theta_rad) + eta_deg = math.degrees(math.atan2(dy_mm, dx_mm)) + omega_deg = self.omega_origin_deg + float(z_idx) * self.omega_step_deg + omega_rad = math.radians(omega_deg) + + sin_theta = math.sin(0.5 * two_theta_rad) + if sin_theta > 1e-12: + d_angstrom = self.wavelength_angstrom / (2.0 * sin_theta) + else: + d_angstrom = float("inf") + + k_mag = (2.0 * math.pi) / self.wavelength_angstrom + qlab_x = k_mag * (self.distance_mm / path_len - 1.0) + qlab_y = k_mag * (dx_mm / path_len) + qlab_z = k_mag * (dy_mm / path_len) + + cos_w = math.cos(omega_rad) + sin_w = math.sin(omega_rad) + qx = qlab_x * cos_w + qlab_y * sin_w + qy = -qlab_x * sin_w + qlab_y * cos_w + qz = qlab_z + q_mag = math.sqrt(qx * qx + qy * qy + qz * qz) + + return { + "omega_deg": round(omega_deg, 6), + "two_theta_deg": round(two_theta_deg, 6), + "eta_deg": round(eta_deg, 6), + "d_spacing_angstrom": round(d_angstrom, 6) if math.isfinite(d_angstrom) else float("inf"), + "qx_inv_angstrom": round(qx, 6), + "qy_inv_angstrom": round(qy, 6), + "qz_inv_angstrom": round(qz, 6), + "q_mag_inv_angstrom": round(q_mag, 6), + } + + +def _recover_voxel_centroid(row: dict[str, Any], calibration: AxisCalibration | None) -> tuple[float, float, float]: + mu = float(row["centroid_mu"]) + chi = float(row["centroid_chi"]) + d_val = float(row["centroid_d"]) + if calibration is not None and calibration.calibrated: + z_idx = (mu - calibration.mu_origin) / calibration.mu_step + x_px = (chi - calibration.chi_origin) / calibration.chi_step + y_px = (d_val - calibration.d_origin) / calibration.d_step + return z_idx, y_px, x_px + return mu, d_val, chi + + +def annotate_crystallographic_columns( + rows: list[dict[str, Any]], + geometry: DetectorGeometry, + *, + calibration: AxisCalibration | None = None, +) -> list[dict[str, Any]]: + """Return shallow-copied rows augmented with ``(omega_deg, two_theta_deg, eta_deg, d_spacing_angstrom, q_*)``.""" + enriched: list[dict[str, Any]] = [] + for row in rows: + z_idx, y_px, x_px = _recover_voxel_centroid(row, calibration) + scat = geometry.spot_to_scattering(z_idx, y_px, x_px) + enriched.append({**row, **scat}) + return enriched diff --git a/braggtrack/io/export.py b/braggtrack/io/export.py new file mode 100644 index 0000000..5b67a2d --- /dev/null +++ b/braggtrack/io/export.py @@ -0,0 +1,256 @@ +"""Interoperability exporters for ImageD11 columnfiles (.flt) and NeXus NXprocess HDF5.""" + +from __future__ import annotations + +import csv +import json +import math +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import numpy as np + +import braggtrack + +from .crystallography import DetectorGeometry, _recover_voxel_centroid +from .models import AxisCalibration +from .nexus import _require_h5py + +IMAGED11_COLUMNS = ( + "sc", + "fc", + "omega", + "Number_of_pixels", + "avg_intensity", + "sum_intensity", + "tth", + "eta", + "d_spacing", + "gx", + "gy", + "gz", + "scan_idx", + "spot_id", + "track_id", +) + + +def export_imaged11_flt( + rows: list[dict[str, Any]], + out_path: str | Path, + *, + geometry: DetectorGeometry | None = None, + calibration: AxisCalibration | None = None, +) -> Path: + """Write segmented or tracked Bragg reflections as an ImageD11 ``.flt`` columnfile. + + In ImageD11 convention, ``sc`` is the slow (vertical, ``y``) pixel coordinate, + ``fc`` is the fast (horizontal, ``x``) pixel coordinate, and ``(gx, gy, gz)`` + are reciprocal-space scattering vectors in ``1/d`` (``Q / 2*pi``) units. + """ + dest = Path(out_path) + dest.parent.mkdir(parents=True, exist_ok=True) + geom = geometry or DetectorGeometry() + two_pi = 2.0 * math.pi + + lines: list[str] = [ + f"# ImageD11 columnfile exported by BraggTrack {braggtrack.__version__}", + f"# wavelength = {geom.wavelength_angstrom:.6f}", + f"# distance = {geom.distance_mm * 1000.0:.3f}", + f"# y_center = {geom.beam_center_y_px:.3f}", + f"# z_center = {geom.beam_center_x_px:.3f}", + f"# y_size = {geom.pixel_size_y_mm * 1000.0:.3f}", + f"# z_size = {geom.pixel_size_x_mm * 1000.0:.3f}", + "# " + " ".join(IMAGED11_COLUMNS), + ] + + for idx, row in enumerate(rows): + z_idx, y_px, x_px = _recover_voxel_centroid(row, calibration) + scat = geom.spot_to_scattering(z_idx, y_px, x_px) + vox = max(1, int(float(row.get("voxel_count", 1)))) + sum_int = float(row.get("integrated_intensity", 0.0)) + avg_int = sum_int / float(vox) + scan_idx = int(float(row.get("scan_idx", 0))) + spot_id = int(float(row.get("label", row.get("spot_idx", idx + 1)))) + track_id = int(float(row.get("track_id", -1))) + d_val = scat["d_spacing_angstrom"] if math.isfinite(scat["d_spacing_angstrom"]) else 0.0 + gx = scat["qx_inv_angstrom"] / two_pi + gy = scat["qy_inv_angstrom"] / two_pi + gz = scat["qz_inv_angstrom"] / two_pi + + lines.append( + f"{y_px:10.4f} {x_px:10.4f} {scat['omega_deg']:10.4f} " + f"{vox:8d} {avg_int:12.4f} {sum_int:14.4f} " + f"{scat['two_theta_deg']:10.5f} {scat['eta_deg']:10.4f} {d_val:10.5f} " + f"{gx:11.6f} {gy:11.6f} {gz:11.6f} " + f"{scan_idx:6d} {spot_id:6d} {track_id:6d}" + ) + + dest.write_text("\n".join(lines) + "\n") + return dest + + +def read_imaged11_flt(path: str | Path) -> tuple[dict[str, str], list[dict[str, float]]]: + """Parse an ImageD11 ``.flt`` columnfile back into header parameters and numeric row dicts.""" + params: dict[str, str] = {} + columns: list[str] = [] + rows: list[dict[str, float]] = [] + + for raw_line in Path(path).read_text().splitlines(): + line = raw_line.strip() + if not line: + continue + if line.startswith("#"): + body = line[1:].strip() + if "=" in body: + k, v = body.split("=", 1) + params[k.strip()] = v.strip() + elif body.startswith("sc ") or body.startswith("sc\t"): + columns = body.split() + continue + if not columns: + raise ValueError("Missing '# sc fc ...' column header in ImageD11 file.") + parts = line.split() + if len(parts) != len(columns): + raise ValueError(f"Expected {len(columns)} columns, got {len(parts)} in line: {line}") + rows.append({col: float(val) for col, val in zip(columns, parts)}) + + return params, rows + + +def _read_csv_rows(path: Path) -> list[dict[str, str]]: + if not path.exists() or path.stat().st_size == 0: + return [] + with path.open(newline="") as f: + return list(csv.DictReader(f)) + + +def _write_table_group(grp: Any, h5py: Any, rows: list[dict[str, Any]]) -> None: + if not rows: + grp.attrs["row_count"] = 0 + return + grp.attrs["row_count"] = len(rows) + str_dt = h5py.string_dtype(encoding="utf-8") + keys = list(rows[0].keys()) + for key in keys: + vals = [r.get(key, "") for r in rows] + # Attempt numeric conversion first (float64 or int64), else UTF-8 strings. + is_int = True + is_float = True + int_vals: list[int] = [] + float_vals: list[float] = [] + for v in vals: + s = str(v).strip() + if s == "" or s.lower() in ("true", "false") or ";" in s: + is_int = False + is_float = False + break + try: + iv = int(s) + int_vals.append(iv) + except ValueError: + is_int = False + try: + fv = float(s) + float_vals.append(fv) + except ValueError: + is_float = False + break + if is_int: + grp.create_dataset(key, data=np.asarray(int_vals, dtype=np.int64), compression="gzip") + elif is_float: + grp.create_dataset(key, data=np.asarray(float_vals, dtype=np.float64), compression="gzip") + else: + arr = np.asarray([str(v) for v in vals], dtype=object) + grp.create_dataset(key, data=arr, dtype=str_dt) + + +def export_nexus_process( + artifact_dir: str | Path, + out_h5: str | Path, + *, + geometry: DetectorGeometry | None = None, + calibration: AxisCalibration | None = None, +) -> Path: + """Package BraggTrack pipeline/tracking outputs into a NeXus ``NXentry`` / ``NXprocess`` HDF5 file.""" + h5py = _require_h5py() + root = Path(artifact_dir) + dest = Path(out_h5) + dest.parent.mkdir(parents=True, exist_ok=True) + + # Support either a pipeline root (containing tracking/ & events/) or a direct tracking output dir. + trk_dir = root / "tracking" if (root / "tracking").is_dir() else root + evt_dir = root / "events" if (root / "events").is_dir() else trk_dir + + track_rows: list[dict[str, Any]] = [dict(r) for r in _read_csv_rows(trk_dir / "tracks.csv")] + if geometry is not None and track_rows: + from .crystallography import annotate_crystallographic_columns + + track_rows = annotate_crystallographic_columns(track_rows, geometry, calibration=calibration) + + kin_rows = _read_csv_rows(trk_dir / "kinematics.csv") + event_rows = _read_csv_rows(evt_dir / "events.csv") + prec_rows = _read_csv_rows(evt_dir / "precursors.csv") + + provenance_json = "{}" + for cand in (root / "run_manifest.json", trk_dir / "tracking_summary.json"): + if cand.exists(): + provenance_json = cand.read_text() + break + + str_dt = h5py.string_dtype(encoding="utf-8") + with h5py.File(dest, "w") as h5: + h5.attrs["NX_class"] = "NXroot" + h5.attrs["default"] = "entry" + + entry = h5.create_group("entry") + entry.attrs["NX_class"] = "NXentry" + entry.attrs["default"] = "braggtrack" + entry.create_dataset("title", data="BraggTrack 4D Diffraction Tracking & Fracture Analysis", dtype=str_dt) + + proc = entry.create_group("braggtrack") + proc.attrs["NX_class"] = "NXprocess" + proc.create_dataset("program", data="braggtrack", dtype=str_dt) + proc.create_dataset("version", data=str(braggtrack.__version__), dtype=str_dt) + proc.create_dataset("date", data=datetime.now(timezone.utc).isoformat(), dtype=str_dt) + proc.create_dataset("provenance_json", data=provenance_json, dtype=str_dt) + + if geometry is not None: + geom_grp = proc.create_group("detector_geometry") + geom_grp.attrs["NX_class"] = "NXparameters" + for k, v in geometry.to_dict().items(): + geom_grp.create_dataset(k, data=float(v)) + + tracks_grp = proc.create_group("tracks") + tracks_grp.attrs["NX_class"] = "NXdata" + _write_table_group(tracks_grp, h5py, track_rows) + + kin_grp = proc.create_group("kinematics") + kin_grp.attrs["NX_class"] = "NXdata" + _write_table_group(kin_grp, h5py, kin_rows) + + evt_grp = proc.create_group("events") + evt_grp.attrs["NX_class"] = "NXcollection" + _write_table_group(evt_grp, h5py, event_rows) + + prec_grp = proc.create_group("precursors") + prec_grp.attrs["NX_class"] = "NXcollection" + _write_table_group(prec_grp, h5py, prec_rows) + + # Attach summary metrics if available + trk_summary_path = trk_dir / "tracking_summary.json" + if trk_summary_path.exists(): + try: + ts = json.loads(trk_summary_path.read_text()) + metrics_grp = proc.create_group("metrics") + metrics_grp.attrs["NX_class"] = "NXparameters" + for k, v in ts.items(): + if isinstance(v, bool): + metrics_grp.create_dataset(k, data=bool(v)) + elif isinstance(v, (int, float)): + metrics_grp.create_dataset(k, data=v) + except (OSError, json.JSONDecodeError): + pass + + return dest diff --git a/braggtrack/io/lazy.py b/braggtrack/io/lazy.py new file mode 100644 index 0000000..968a26e --- /dev/null +++ b/braggtrack/io/lazy.py @@ -0,0 +1,144 @@ +"""Chunked and ROI-lazy 3D volume access for multi-gigabyte HDF5 detector stacks.""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import numpy as np +from skimage.filters import threshold_otsu + +from .nexus import _require_h5py, _resolve_volume_dataset + + +class LazyVolume: + """Lazy 3D volume wrapper supporting bounded ROI reads and z-slab streaming. + + Works seamlessly over either an open ``h5py.Dataset`` or an in-memory + ``numpy.ndarray`` fallback so downstream stages (:func:`crop_spot_cube`, + chunked Otsu thresholding) never materialize full ``float64`` detector + cubes unless explicitly requested via ``vol[()]``. + """ + + def __init__( + self, + dataset: Any, + *, + handle: Any | None = None, + dataset_path: str | None = None, + ) -> None: + shape = tuple(int(x) for x in dataset.shape) + if len(shape) != 3: + if handle is not None: + handle.close() + raise ValueError(f"LazyVolume requires a 3D dataset, got shape={shape}") + self._dataset = dataset + self._handle = handle + self._shape: tuple[int, int, int] = (shape[0], shape[1], shape[2]) + self._dtype = np.dtype(dataset.dtype) + self._dataset_path = dataset_path + + @classmethod + def from_array(cls, array: np.ndarray, *, dataset_path: str | None = None) -> LazyVolume: + arr = np.asarray(array) + return cls(arr, handle=None, dataset_path=dataset_path) + + @property + def shape(self) -> tuple[int, int, int]: + return self._shape + + @property + def ndim(self) -> int: + return 3 + + @property + def dtype(self) -> np.dtype[Any]: + return self._dtype + + @property + def dataset_path(self) -> str | None: + return self._dataset_path + + def read_roi(self, z_slice: slice, y_slice: slice, x_slice: slice) -> np.ndarray: + """Read a bounded 3D sub-block into memory as ``float64``.""" + return np.asarray(self._dataset[z_slice, y_slice, x_slice], dtype=np.float64) + + def __getitem__(self, key: Any) -> np.ndarray: + return np.asarray(self._dataset[key], dtype=np.float64) + + def iter_z_chunks(self, chunk_z: int = 32) -> Iterator[np.ndarray]: + """Yield contiguous ``(dz, ny, nx)`` ``float64`` slabs along axis 0.""" + step = max(1, int(chunk_z)) + nz = self._shape[0] + for z0 in range(0, nz, step): + z1 = min(nz, z0 + step) + yield np.asarray(self._dataset[z0:z1, :, :], dtype=np.float64) + + def to_dense(self) -> np.ndarray: + """Materialize the full 3D volume as ``float64``.""" + return np.asarray(self._dataset[()], dtype=np.float64) + + def close(self) -> None: + if self._handle is not None: + self._handle.close() + self._handle = None + + def __enter__(self) -> LazyVolume: + return self + + def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None: + self.close() + + +def open_lazy_volume( + path: str | Path, + candidates: list[str] | None = None, + *, + auto_discover: bool = True, +) -> LazyVolume: + """Open an HDF5 file and return a :class:`LazyVolume` bound to its primary 3D dataset.""" + h5py = _require_h5py() + handle = h5py.File(path, "r") + try: + ds_path = _resolve_volume_dataset(handle, h5py, candidates=candidates, auto_discover=auto_discover) + ds = handle[ds_path] + except Exception: + handle.close() + raise + return LazyVolume(ds, handle=handle, dataset_path=ds_path) + + +def otsu_threshold_chunked( + volume: LazyVolume | np.ndarray, + *, + nbins: int = 256, + chunk_z: int = 32, +) -> float: + """Compute global Otsu threshold via streaming z-slabs in ``O(chunk)`` memory.""" + lazy = volume if isinstance(volume, LazyVolume) else LazyVolume.from_array(np.asarray(volume)) + nz, ny, nx = lazy.shape + if nz * ny * nx == 0: + raise ValueError("Otsu threshold requires at least one value.") + + vmin = float("inf") + vmax = float("-inf") + for slab in lazy.iter_z_chunks(chunk_z=chunk_z): + s_min = float(np.min(slab)) + s_max = float(np.max(slab)) + if s_min < vmin: + vmin = s_min + if s_max > vmax: + vmax = s_max + + if vmin == vmax: + return float(vmin) + + edges = np.linspace(vmin, vmax, max(2, int(nbins)) + 1, dtype=np.float64) + counts = np.zeros(len(edges) - 1, dtype=np.int64) + for slab in lazy.iter_z_chunks(chunk_z=chunk_z): + c, _ = np.histogram(slab, bins=edges) + counts += c + + bin_centers = 0.5 * (edges[:-1] + edges[1:]) + return float(threshold_otsu(hist=(counts, bin_centers))) diff --git a/braggtrack/segmentation/dino_segment.py b/braggtrack/segmentation/dino_segment.py index d5b48ef..1250b23 100644 --- a/braggtrack/segmentation/dino_segment.py +++ b/braggtrack/segmentation/dino_segment.py @@ -61,10 +61,13 @@ def _patch_foreground_masks( threshold: float, patch_size: int, axis: int = 0, + grid_hw: tuple[int, int] | None = None, ) -> list[np.ndarray]: """Boolean foreground mask at patch resolution for each slice. A patch is foreground if any of its pixels exceed *threshold*. + Handles both fixed-stride grids (``mock``) and resized ViT patch grids + (e.g. ``16x16`` ``facebook/dinov2-small`` tokens on ``250x250`` slices). """ n_slices = volume.shape[axis] masks: list[np.ndarray] = [] @@ -72,11 +75,22 @@ def _patch_foreground_masks( slc = np.take(volume, i, axis=axis) fg = (slc >= threshold).astype(np.float64) h, w = slc.shape - hp = max(1, h // patch_size) - wp = max(1, w // patch_size) - cropped = fg[: hp * patch_size, : wp * patch_size] - blocks = cropped.reshape(hp, patch_size, wp, patch_size) - masks.append(blocks.max(axis=(1, 3)) > 0) + default_hw = (max(1, h // patch_size), max(1, w // patch_size)) + hp, wp = grid_hw if grid_hw is not None else default_hw + if (hp, wp) == default_hw and hp * patch_size <= h and wp * patch_size <= w: + cropped = fg[: hp * patch_size, : wp * patch_size] + blocks = cropped.reshape(hp, patch_size, wp, patch_size) + masks.append(blocks.max(axis=(1, 3)) > 0) + else: + r_edges = np.linspace(0, h, hp + 1, dtype=int) + c_edges = np.linspace(0, w, wp + 1, dtype=int) + pooled = np.zeros((hp, wp), dtype=bool) + for r_i in range(hp): + r0, r1 = r_edges[r_i], max(r_edges[r_i] + 1, r_edges[r_i + 1]) + for c_i in range(wp): + c0, c1 = c_edges[c_i], max(c_edges[c_i] + 1, c_edges[c_i + 1]) + pooled[r_i, c_i] = bool(np.any(fg[r0:r1, c0:c1] > 0)) + masks.append(pooled) return masks @@ -159,6 +173,13 @@ def _upsample_labels( patch_size: int, ) -> np.ndarray: """Nearest-neighbor upsample patch-resolution labels to pixel resolution.""" + th, tw = target_shape + hp, wp = patch_labels.shape + default_hw = (max(1, th // patch_size), max(1, tw // patch_size)) + if (hp, wp) != default_hw and th > 0 and tw > 0 and hp > 0 and wp > 0: + r_idx = (np.arange(th) * hp) // th + c_idx = (np.arange(tw) * wp) // tw + return patch_labels[np.ix_(r_idx, c_idx)].astype(np.int32) expanded = np.repeat(np.repeat(patch_labels, patch_size, axis=0), patch_size, axis=1) out = np.zeros(target_shape, dtype=np.int32) h = min(expanded.shape[0], target_shape[0]) @@ -302,7 +323,13 @@ def segment_dino( features, slice_hw = _extract_slice_features(volume, encoder, axis=axis) # Foreground mask at patch resolution — exclude background from clustering. - fg_masks = _patch_foreground_masks(volume, threshold, encoder.patch_size, axis=axis) + fg_masks = _patch_foreground_masks( + volume, + threshold, + encoder.patch_size, + axis=axis, + grid_hw=(features.shape[1], features.shape[2]), + ) # Global PCA across all foreground patches for a consistent feature space # (per-slice PCA causes wildly different cluster counts). diff --git a/braggtrack/semantic/dino.py b/braggtrack/semantic/dino.py index 4430380..c636052 100644 --- a/braggtrack/semantic/dino.py +++ b/braggtrack/semantic/dino.py @@ -25,6 +25,9 @@ def _resolve_backend(requested: BackendName) -> Literal["mock", "torch"]: return "mock" if requested == "torch": return "torch" + env_override = os.environ.get("BRAGGTRACK_DINO_BACKEND", "").strip().lower() + if env_override in ("mock", "torch"): + return "mock" if env_override == "mock" else "torch" try: import torch # noqa: F401 except ImportError: diff --git a/braggtrack/semantic/mips.py b/braggtrack/semantic/mips.py index e3240c7..11e58af 100644 --- a/braggtrack/semantic/mips.py +++ b/braggtrack/semantic/mips.py @@ -5,14 +5,16 @@ from __future__ import annotations +from typing import Any + import numpy as np def crop_spot_cube( - volume: np.ndarray, + volume: np.ndarray | Any, labels: np.ndarray, label_id: int, - bbox: dict[str, int], + bbox: dict[str, Any], margin: int = 2, ) -> tuple[np.ndarray, np.ndarray]: """Crop a padded sub-volume and binary mask for one instance. diff --git a/braggtrack/tracking/__init__.py b/braggtrack/tracking/__init__.py index bc5d52d..68a1f62 100644 --- a/braggtrack/tracking/__init__.py +++ b/braggtrack/tracking/__init__.py @@ -3,6 +3,7 @@ from .assignment import associate_frames from .cost import CostFunction, GeometrySemanticCost, PositionShapeCost from .events import LineageEvent, SplitMergeConfig, events_to_table, resolve_splits_and_merges +from .gap_closing import GapCloseRecord, GapClosingConfig, resolve_track_gaps from .kinematics import ( GrainKinematics, KinematicsSummary, @@ -24,6 +25,8 @@ __all__ = [ "CostFunction", + "GapCloseRecord", + "GapClosingConfig", "GeometrySemanticCost", "GrainKinematics", "KinematicsSummary", @@ -45,6 +48,7 @@ "kinematics_to_table", "precursors_to_table", "resolve_splits_and_merges", + "resolve_track_gaps", "summarize_kinematics", "summary_to_dict", "tracks_to_table", diff --git a/braggtrack/tracking/gap_closing.py b/braggtrack/tracking/gap_closing.py new file mode 100644 index 0000000..c40ce8f --- /dev/null +++ b/braggtrack/tracking/gap_closing.py @@ -0,0 +1,275 @@ +"""Multi-scan gap closing for transient beam dropouts and rocking-clip recovery.""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import Any + +import networkx as nx +import numpy as np +from scipy.optimize import linear_sum_assignment + +from .cost import CostFunction, GeometrySemanticCost, PositionShapeCost +from .lifecycle import TrackEvent + + +@dataclass(frozen=True) +class GapClosingConfig: + """Configuration for reconnecting tracks across missing intermediate scans. + + Parameters + ---------- + max_gap : int + Maximum number of missing intermediate scans (``1 <= gap_scans <= max_gap``). + ``0`` disables gap closing. + gap_penalty_per_scan : float + Multiplicative penalty per missing scan: ``cost * (1 + gap_penalty_per_scan * gap_scans)``. + max_cost : float + Upper bound on the penalized association cost for accepting a gap bridge. + min_cosine_similarity : float + Minimum embedding cosine similarity when both endpoints carry ``embedding``. + """ + + max_gap: int = 2 + gap_penalty_per_scan: float = 0.15 + max_cost: float = 25.0 + min_cosine_similarity: float = 0.50 + + +@dataclass(frozen=True) +class GapCloseRecord: + """Metadata record for an accepted multi-scan gap closure.""" + + track_id: int + from_scan: int + to_scan: int + gap_scans: int + from_node: str + to_node: str + cost: float + + +def _cosine_sim(a_raw: Any, b_raw: Any) -> float | None: + if a_raw is None or b_raw is None: + return None + a = np.asarray(a_raw, dtype=np.float64).ravel() + b = np.asarray(b_raw, dtype=np.float64).ravel() + if a.size == 0 or b.size == 0 or a.shape != b.shape: + return None + na = float(np.linalg.norm(a)) + nb = float(np.linalg.norm(b)) + if na <= 0.0 or nb <= 0.0: + return None + return float(np.clip(np.dot(a, b) / (na * nb), -1.0, 1.0)) + + +def _eval_gap_cost( + cost_fn: CostFunction, + u_data: dict[str, Any], + v_data: dict[str, Any], + delta_scans: int, + min_cos_sim: float, +) -> float: + """Evaluate gap-normalized cost across ``delta_scans = to_scan - from_scan >= 2``.""" + sim = _cosine_sim(u_data.get("embedding"), v_data.get("embedding")) + if sim is not None and sim < min_cos_sim: + return math.inf + + geo: PositionShapeCost | None = None + alpha = 1.0 + beta = 0.0 + if isinstance(cost_fn, PositionShapeCost): + geo = cost_fn + elif isinstance(cost_fn, GeometrySemanticCost): + geo = cost_fn.geometry + alpha = cost_fn.cost_alpha + beta = cost_fn.cost_beta + + if geo is not None: + dmu = abs(float(u_data["centroid_mu"]) - float(v_data["centroid_mu"])) + dchi = abs(float(u_data["centroid_chi"]) - float(v_data["centroid_chi"])) + dd = abs(float(u_data["centroid_d"]) - float(v_data["centroid_d"])) + step_scale = float(max(1, delta_scans)) + if (dmu / step_scale) > geo.gate_mu or (dchi / step_scale) > geo.gate_chi or (dd / step_scale) > geo.gate_d: + return math.inf + + # Diffusive scaling (dx^2 / delta_t) keeps gap costs comparable to 1-step costs. + pos_dist2 = (dmu**2 + dchi**2 + dd**2) / step_scale + deig1 = float(u_data.get("eig_1", 0.0)) - float(v_data.get("eig_1", 0.0)) + deig2 = float(u_data.get("eig_2", 0.0)) - float(v_data.get("eig_2", 0.0)) + deig3 = float(u_data.get("eig_3", 0.0)) - float(v_data.get("eig_3", 0.0)) + shape_dist2 = deig1**2 + deig2**2 + deig3**2 + g_cost = geo.position_weight * pos_dist2 + geo.shape_weight * shape_dist2 + if beta == 0.0: + return alpha * g_cost + if sim is None: + return math.inf + return alpha * g_cost + beta * (1.0 - sim) + + return float(cost_fn(u_data, v_data)) + + +def _propagate_track_id_forward(G: nx.DiGraph, start_nid: str, old_tid: int, new_tid: int) -> None: + queue = [start_nid] + visited: set[str] = set() + while queue: + curr = queue.pop(0) + if curr in visited: + continue + visited.add(curr) + if G.nodes[curr].get("track_id") == old_tid: + G.nodes[curr]["track_id"] = new_tid + for _, succ in G.out_edges(curr): + queue.append(succ) + + +def resolve_track_gaps( + G: nx.DiGraph, + *, + n_scans: int, + cost_fn: CostFunction | None = None, + config: GapClosingConfig | None = None, +) -> list[GapCloseRecord]: + """Reconnect prematurely terminated tracks to matching births across ``1..max_gap`` missing scans.""" + cfg = config or GapClosingConfig() + if cfg.max_gap <= 0 or G.number_of_nodes() == 0 or n_scans < 3: + G.graph.setdefault("gap_closures", []) + return [] + + active_cost: CostFunction = cost_fn if cost_fn is not None else PositionShapeCost() + next_tid = max((int(G.nodes[n]["track_id"]) for n in G.nodes), default=0) + 1 + + # Pass 0: Sever forced cross-detector 1-to-1 jump edges created when max_cost=inf + # paired a dropping-out track with a distant stray spot on an intermediate frame. + nodes_by_scan: dict[int, list[str]] = {} + for nid in G.nodes: + nodes_by_scan.setdefault(int(G.nodes[nid]["scan_idx"]), []).append(nid) + + sever_threshold = max(cfg.max_cost * 4.0, 100.0) + + def _edge_is_jump(a: str, b: str) -> bool: + c_ab = float(active_cost(G.nodes[a], G.nodes[b])) + return not math.isfinite(c_ab) or c_ab > sever_threshold + + edges_to_sever: list[tuple[str, str]] = [] + for u, v, edata in list(G.edges(data=True)): + if edata.get("edge_type", "continuation") != "continuation": + continue + if not _edge_is_jump(u, v): + continue + su = int(G.nodes[u]["scan_idx"]) + sv = int(G.nodes[v]["scan_idx"]) + has_gap_partner = False + # Forward check: u can bridge over the jump to w at su + gap + 1 + for gap in range(1, cfg.max_gap + 1): + for w in nodes_by_scan.get(su + gap + 1, []): + if G.in_degree(w) == 0 or all(_edge_is_jump(p, w) for p, _ in G.in_edges(w)): + gc = _eval_gap_cost(active_cost, G.nodes[u], G.nodes[w], gap + 1, cfg.min_cosine_similarity) + if gc * (1.0 + cfg.gap_penalty_per_scan * gap) <= cfg.max_cost * 0.6: + has_gap_partner = True + break + if has_gap_partner: + break + # Backward check: v can receive a bridge from p at sv - gap - 1 + if not has_gap_partner: + for gap in range(1, cfg.max_gap + 1): + for p in nodes_by_scan.get(sv - gap - 1, []): + if G.out_degree(p) == 0 or all(_edge_is_jump(p, q) for _, q in G.out_edges(p)): + gc = _eval_gap_cost(active_cost, G.nodes[p], G.nodes[v], gap + 1, cfg.min_cosine_similarity) + if gc * (1.0 + cfg.gap_penalty_per_scan * gap) <= cfg.max_cost * 0.6: + has_gap_partner = True + break + if has_gap_partner: + break + if has_gap_partner: + edges_to_sever.append((u, v)) + + for u, v in edges_to_sever: + old_tid = int(G.nodes[u]["track_id"]) + G.remove_edge(u, v) + G.nodes[u]["event"] = TrackEvent.TERMINATED + G.nodes[v]["event"] = TrackEvent.BORN + _propagate_track_id_forward(G, v, old_tid, next_tid) + next_tid += 1 + + term_nodes = sorted( + [n for n in G.nodes if G.out_degree(n) == 0 and int(G.nodes[n]["scan_idx"]) < n_scans - 1], + key=lambda n: (int(G.nodes[n]["scan_idx"]), int(G.nodes[n]["spot_idx"])), + ) + born_nodes = sorted( + [n for n in G.nodes if G.in_degree(n) == 0 and int(G.nodes[n]["scan_idx"]) > 0], + key=lambda n: (int(G.nodes[n]["scan_idx"]), int(G.nodes[n]["spot_idx"])), + ) + + n_term = len(term_nodes) + n_born = len(born_nodes) + if n_term == 0 or n_born == 0: + G.graph["gap_closures"] = [] + return [] + + inadmissible = 1e12 + unmatched_cost = cfg.max_cost + 1e-6 + aug_matrix = np.full((n_term, n_born + n_term), inadmissible, dtype=np.float64) + for i in range(n_term): + aug_matrix[i, n_born + i] = unmatched_cost + + for i, u in enumerate(term_nodes): + u_data = G.nodes[u] + su = int(u_data["scan_idx"]) + for j, v in enumerate(born_nodes): + v_data = G.nodes[v] + sv = int(v_data["scan_idx"]) + gap_scans = sv - su - 1 + if gap_scans < 1 or gap_scans > cfg.max_gap: + continue + base = _eval_gap_cost(active_cost, u_data, v_data, gap_scans + 1, cfg.min_cosine_similarity) + if not math.isfinite(base): + continue + penalized = base * (1.0 + cfg.gap_penalty_per_scan * gap_scans) + if penalized <= cfg.max_cost: + aug_matrix[i, j] = penalized + + row_ind, col_ind = linear_sum_assignment(aug_matrix) + + accepted_pairs: list[tuple[int, int, str, str, float]] = [] + for r, c in zip(row_ind, col_ind): + if c < n_born and aug_matrix[r, c] <= cfg.max_cost: + u = term_nodes[int(r)] + v = born_nodes[int(c)] + su = int(G.nodes[u]["scan_idx"]) + sv = int(G.nodes[v]["scan_idx"]) + accepted_pairs.append((su, sv, u, v, float(aug_matrix[r, c]))) + + # Process chronologically so multi-gap chains (e.g., 0 -> 2 -> 4) propagate a single track_id. + accepted_pairs.sort(key=lambda item: (item[0], item[1], item[2])) + + records: list[GapCloseRecord] = [] + for su, sv, u, v, cost_val in accepted_pairs: + tid = int(G.nodes[u]["track_id"]) + old_v_tid = int(G.nodes[v]["track_id"]) + gap_scans = sv - su - 1 + + G.add_edge(u, v, edge_type="gap_close", gap_scans=gap_scans, cost=round(cost_val, 4)) + _propagate_track_id_forward(G, v, old_v_tid, tid) + + G.nodes[u]["event"] = TrackEvent.BORN if G.in_degree(u) == 0 else TrackEvent.CONTINUED + if G.out_degree(v) > 0 or sv == n_scans - 1: + G.nodes[v]["event"] = TrackEvent.CONTINUED + else: + G.nodes[v]["event"] = TrackEvent.TERMINATED + + records.append( + GapCloseRecord( + track_id=tid, + from_scan=su, + to_scan=sv, + gap_scans=gap_scans, + from_node=u, + to_node=v, + cost=round(cost_val, 4), + ) + ) + + G.graph["gap_closures"] = records + return records diff --git a/braggtrack/tracking/lifecycle.py b/braggtrack/tracking/lifecycle.py index d72089c..54b8c8c 100644 --- a/braggtrack/tracking/lifecycle.py +++ b/braggtrack/tracking/lifecycle.py @@ -12,6 +12,7 @@ if TYPE_CHECKING: from .events import SplitMergeConfig + from .gap_closing import GapClosingConfig class TrackEvent(str, enum.Enum): @@ -30,6 +31,7 @@ def build_tracks( cost_fn: CostFunction | None = None, max_cost: float = float("inf"), *, + gap_closing_config: GapClosingConfig | None = None, split_merge_config: SplitMergeConfig | None = None, scan_timestamps: dict[int, str | None] | None = None, ) -> nx.DiGraph: @@ -37,7 +39,8 @@ def build_tracks( Each node represents a spot observation ``(scan_index, spot_index)`` and carries the full feature dict plus a ``track_id`` and ``event`` - attribute. Edges connect matched spots across consecutive frames, and + attribute. Edges connect matched spots across consecutive frames, + optional *gap_closing_config* reconnects missing intermediate scans, and optional *split_merge_config* resolves ``1 -> 2+`` splits and ``2+ -> 1`` merges. """ @@ -106,6 +109,16 @@ def build_tracks( # Still alive at end — leave as CONTINUED (not terminated) pass + if gap_closing_config is not None and gap_closing_config.max_gap > 0: + from .gap_closing import resolve_track_gaps + + resolve_track_gaps( + G, + n_scans=n_scans, + cost_fn=active_cost, + config=gap_closing_config, + ) + if split_merge_config is not None: from .events import resolve_splits_and_merges diff --git a/braggtrack/tracking/metrics.py b/braggtrack/tracking/metrics.py index 6901b51..3b6ae49 100644 --- a/braggtrack/tracking/metrics.py +++ b/braggtrack/tracking/metrics.py @@ -53,7 +53,17 @@ def compute_tracking_metrics( terminated_count += 1 total_tracks = len(tracks) - full_length_tracks = sum(1 for obs in tracks.values() if len(obs) >= n_scans) + full_length_tracks = sum( + 1 + for obs in tracks.values() + if len(obs) >= n_scans + or ( + n_scans > 0 + and bool(G.graph.get("gap_closures")) + and min(int(r["scan_idx"]) for r in obs) == 0 + and max(int(r["scan_idx"]) for r in obs) >= n_scans - 1 + ) + ) fragmentation_ratio = 1.0 - full_length_tracks / total_tracks if total_tracks > 0 else 0.0 # ID-switch rate (requires ground truth). @@ -89,6 +99,7 @@ def compute_tracking_metrics( lineage_events = G.graph.get("lineage_events", []) split_count = sum(1 for ev in lineage_events if getattr(ev, "event_type", None) == "split") merge_count = sum(1 for ev in lineage_events if getattr(ev, "event_type", None) == "merge") + gap_closed_count = len(G.graph.get("gap_closures", [])) return { "total_tracks": total_tracks, @@ -97,6 +108,7 @@ def compute_tracking_metrics( "born_count": born_count, "continued_count": continued_count, "terminated_count": terminated_count, + "gap_closed_count": gap_closed_count, "split_count": split_count, "merge_count": merge_count, "id_switch_count": id_switch_count, diff --git a/braggtrack/viz/__init__.py b/braggtrack/viz/__init__.py new file mode 100644 index 0000000..52705e6 --- /dev/null +++ b/braggtrack/viz/__init__.py @@ -0,0 +1,17 @@ +"""Publication plotting and interactive HTML 4D beamtime explorer for BraggTrack.""" + +from .html_report import build_explorer_payload, generate_html_report +from .kinematics import plot_kinematics_dashboard +from .lineage import plot_lineage_dag +from .mips import plot_scan_projection_overlay, plot_spot_mip_card +from .precursors import plot_precursor_timeline + +__all__ = [ + "build_explorer_payload", + "generate_html_report", + "plot_kinematics_dashboard", + "plot_lineage_dag", + "plot_precursor_timeline", + "plot_scan_projection_overlay", + "plot_spot_mip_card", +] diff --git a/braggtrack/viz/_style.py b/braggtrack/viz/_style.py new file mode 100644 index 0000000..7306263 --- /dev/null +++ b/braggtrack/viz/_style.py @@ -0,0 +1,45 @@ +"""Shared styling and optional-dependency guards for braggtrack.viz.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +# Colorblind-safe Okabe-Ito inspired scientific palette +OKABE_ITO = ( + "#0072B2", # blue + "#D55E00", # vermilion / split highlight + "#009E73", # bluish green / merge highlight + "#CC79A7", # reddish purple + "#E69F00", # amber / gap-close highlight + "#56B4E9", # sky blue + "#F0E442", # yellow + "#333333", # charcoal +) + + +def _require_matplotlib() -> tuple[Any, Any]: + try: + import matplotlib + import matplotlib.pyplot as plt + except ImportError as exc: + raise ImportError( + "braggtrack.viz requires matplotlib. Install with `pip install braggtrack[notebook]`." + ) from exc + return matplotlib, plt + + +def apply_axis_style(ax: Any) -> None: + """Apply crisp publication-ready spine and grid styling to a Matplotlib Axes.""" + ax.spines["top"].set_visible(False) + ax.spines["right"].set_visible(False) + ax.grid(True, linestyle="--", linewidth=0.5, alpha=0.35) + + +def maybe_save(fig: Any, save_path: str | Path | None, *, dpi: int = 200) -> Path | None: + if save_path is None: + return None + out = Path(save_path) + out.parent.mkdir(parents=True, exist_ok=True) + fig.savefig(out, dpi=dpi, bbox_inches="tight") + return out diff --git a/braggtrack/viz/html_report.py b/braggtrack/viz/html_report.py new file mode 100644 index 0000000..49518ef --- /dev/null +++ b/braggtrack/viz/html_report.py @@ -0,0 +1,696 @@ +"""Self-contained offline interactive HTML 4D Lineage, Kinematics & Fracture Explorer.""" + +from __future__ import annotations + +import base64 +import csv +import html +import json +import struct +import zlib +from contextlib import ExitStack +from pathlib import Path +from typing import Any + +import numpy as np + +from braggtrack.io import ( + LazyVolume, + MissingH5DependencyError, + discover_operando_scans, + open_lazy_volume, + resolve_dataset_root, +) +from braggtrack.semantic.mips import crop_spot_cube, orthogonal_mips +from braggtrack.tracking.kinematics import compute_grain_kinematics + + +def _encode_rgb_png_base64(rgb: np.ndarray) -> str: + """Encode a ``(H, W, 3)`` ``uint8`` RGB array to a base64 PNG data URI using stdlib ``zlib``.""" + h, w, _ = rgb.shape + raw_rows = b"".join(b"\x00" + rgb[y].tobytes() for y in range(h)) + + def _chunk(tag: bytes, payload: bytes) -> bytes: + crc = zlib.crc32(tag + payload) & 0xFFFFFFFF + return struct.pack(">I", len(payload)) + tag + payload + struct.pack(">I", crc) + + ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0) + png = ( + b"\x89PNG\r\n\x1a\n" + + _chunk(b"IHDR", ihdr) + + _chunk(b"IDAT", zlib.compress(raw_rows, 6)) + + _chunk(b"IEND", b"") + ) + return "data:image/png;base64," + base64.b64encode(png).decode("ascii") + + +def _mip_to_magma_like_rgb(mip: np.ndarray, size: int = 44) -> np.ndarray: + """Nearest-neighbor resize a 2D MIP to ``(size, size)`` and apply a high-contrast magma-like LUT.""" + arr = np.asarray(mip, dtype=np.float64) + if arr.size == 0: + return np.zeros((size, size, 3), dtype=np.uint8) + yi = np.linspace(0, arr.shape[0] - 1, size).astype(np.intp) + xi = np.linspace(0, arr.shape[1] - 1, size).astype(np.intp) + resized = arr[np.ix_(yi, xi)] + lo = float(np.min(resized)) + hi = float(np.percentile(resized, 99.5)) if resized.size > 1 else float(np.max(resized)) + if hi <= lo: + hi = lo + 1.0 + t = np.clip((resized - lo) / (hi - lo), 0.0, 1.0) + # Smooth 4-stop magma-like ramp: #0b0924 -> #6b1d5c -> #dc493a -> #f9e784 + r = np.clip(11.0 + 235.0 * (t**0.75), 0.0, 255.0).astype(np.uint8) + g = np.clip(9.0 + 222.0 * (t**1.65), 0.0, 255.0).astype(np.uint8) + b = np.clip(36.0 + 140.0 * np.sin(np.pi * np.clip(t * 0.85, 0.0, 1.0)) + 80.0 * (t**3), 0.0, 255.0).astype(np.uint8) + return np.stack([r, g, b], axis=-1) + + +def _stitch_three_mips_b64(mip_mu: np.ndarray, mip_chi: np.ndarray, mip_d: np.ndarray, tile: int = 40) -> str: + """Stitch ``(d, chi) | (mu, d) | (mu, chi)`` MIPs horizontally with 2-px dividers into one base64 PNG.""" + t1 = _mip_to_magma_like_rgb(mip_mu, size=tile) + t2 = _mip_to_magma_like_rgb(mip_chi, size=tile) + t3 = _mip_to_magma_like_rgb(mip_d, size=tile) + sep = np.full((tile, 2, 3), 30, dtype=np.uint8) + strip = np.concatenate([t1, sep, t2, sep, t3], axis=1) + return _encode_rgb_png_base64(strip) + + +def _read_csv(path: Path) -> list[dict[str, str]]: + if not path.exists() or path.stat().st_size == 0: + return [] + with path.open(newline="") as f: + return list(csv.DictReader(f)) + + +def _read_json(path: Path) -> Any: + if not path.exists(): + return None + try: + return json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + return None + + +def _collect_mip_thumbnails( + seg_dir: Path, + dataset_root: Path | None, + track_rows: list[dict[str, Any]], + priority_tids: list[int], + *, + max_tracks: int, +) -> dict[str, str]: + """Extract compact 3-view MIP strips keyed by ``'{track_id}:{scan_idx}'`` via lazy HDF5 ROI reads.""" + if not seg_dir.is_dir() or dataset_root is None or not dataset_root.is_dir(): + return {} + try: + scans_fs = discover_operando_scans(dataset_root) + except Exception: + return {} + if not scans_fs: + return {} + + scan_dirs = sorted(d for d in seg_dir.iterdir() if d.is_dir() and d.name.startswith("scan")) + scan_by_name = {s.scan_name: s for s in scans_fs} + selected = set(priority_tids[:max_tracks]) + wanted_by_scan: dict[int, list[dict[str, Any]]] = {} + for r in track_rows: + tid = int(float(r["track_id"])) + if tid in selected and "bbox_min_z" in r and "label" in r: + s_idx = int(float(r["scan_idx"])) + wanted_by_scan.setdefault(s_idx, []).append(r) + + thumbs: dict[str, str] = {} + with ExitStack() as stack: + for s_idx, obs_list in wanted_by_scan.items(): + if s_idx >= len(scan_dirs): + continue + sd = scan_dirs[s_idx] + lab_path = sd / "labels.npz" + sf = scan_by_name.get(sd.name) + if not lab_path.exists() or sf is None: + continue + labels = np.load(lab_path)["labels"] + try: + lazy_vol: LazyVolume = stack.enter_context(open_lazy_volume(sf.path)) + except (MissingH5DependencyError, KeyError, ValueError, OSError): + continue + if lazy_vol.shape != labels.shape: + continue + for obs in obs_list: + lid = int(float(obs["label"])) + tid = int(float(obs["track_id"])) + masked_cube, _ = crop_spot_cube(lazy_vol, labels, lid, obs, margin=2) + m_mu, m_chi, m_d = orthogonal_mips(masked_cube) + thumbs[f"{tid}:{s_idx}"] = _stitch_three_mips_b64(m_mu, m_chi, m_d, tile=38) + return thumbs + + +def build_explorer_payload( + artifact_dir: str | Path, + *, + dataset_root: str | Path | None = None, + max_mip_tracks: int = 16, +) -> dict[str, Any]: + """Assemble structured JSON payload for the interactive 4D explorer from pipeline/tracking outputs.""" + root = Path(artifact_dir) + trk_dir = root / "tracking" if (root / "tracking").is_dir() else root + evt_dir = root / "events" if (root / "events").is_dir() else trk_dir + seg_dir = root / "segmentation" if (root / "segmentation").is_dir() else root.parent / "segmentation" + + manifest = _read_json(root / "run_manifest.json") or {} + trk_summary = _read_json(trk_dir / "tracking_summary.json") or {} + evt_summary = _read_json(evt_dir / "events_summary.json") or {} + + ds_root: Path | None = None + if dataset_root is not None: + ds_root = resolve_dataset_root(str(dataset_root)) + elif manifest.get("dataset_root"): + cand = Path(str(manifest["dataset_root"])) + if cand.is_dir(): + ds_root = cand + + raw_tracks = _read_csv(evt_dir / "tracks.csv") or _read_csv(trk_dir / "tracks.csv") + raw_events = _read_csv(evt_dir / "events.csv") or _read_csv(trk_dir / "events.csv") + raw_precursors = _read_csv(evt_dir / "precursors.csv") or _read_csv(trk_dir / "precursors.csv") + raw_prec_obs = _read_csv(evt_dir / "precursor_observations.csv") + + track_rows: list[dict[str, Any]] = [] + for r in raw_tracks: + parsed: dict[str, Any] = {} + for k, v in r.items(): + if k in ("event", "lineage_role", "parent_track_ids"): + parsed[k] = v + elif k in ("track_id", "scan_idx", "spot_idx", "label", "voxel_count"): + parsed[k] = int(float(v)) if v != "" else 0 + else: + try: + parsed[k] = float(v) + except (TypeError, ValueError): + parsed[k] = v + track_rows.append(parsed) + + gks = compute_grain_kinematics(track_rows) if track_rows else [] + gk_by_tid = {g.track_id: g for g in gks} + + prec_obs_by_key: dict[tuple[int, int], dict[str, float]] = {} + for r in raw_prec_obs: + key = (int(float(r["track_id"])), int(float(r["scan_idx"]))) + prec_obs_by_key[key] = { + "score": float(r.get("precursor_score", 0.0)), + "anisotropy_term": float(r.get("anisotropy_term", 0.0)), + "spread_term": float(r.get("spread_term", 0.0)), + "roi_dispersion_term": float(r.get("roi_dispersion_term", 0.0)), + "subpeaks": int(float(r.get("roi_subpeak_count", 1))), + } + + events_list: list[dict[str, Any]] = [] + split_tids: set[int] = set() + merge_tids: set[int] = set() + for r in raw_events: + ptids = [int(x) for x in str(r.get("parent_track_ids", "")).split(";") if x.strip()] + ctids = [int(x) for x in str(r.get("child_track_ids", "")).split(";") if x.strip()] + etype = str(r.get("event_type", "")) + if etype == "split": + split_tids.update(ptids) + split_tids.update(ctids) + elif etype == "merge": + merge_tids.update(ptids) + merge_tids.update(ctids) + events_list.append( + { + "event_type": etype, + "parent_scan": int(float(r.get("scan_idx_from", r.get("parent_scan", 0)))), + "child_scan": int(float(r.get("scan_idx_to", r.get("child_scan", 1)))), + "parent_track_ids": ptids, + "child_track_ids": ctids, + "intensity_ratio": round(float(r.get("intensity_ratio", 1.0)), 3), + "centroid_distance": round(float(r.get("mean_distance", r.get("centroid_distance", 0.0))), 3), + "confidence": round(float(r.get("confidence", 0.0)), 3), + } + ) + + precursors_list: list[dict[str, Any]] = [] + prec_tids: set[int] = set() + for r in raw_precursors: + tid = int(float(r["track_id"])) + prec_tids.add(tid) + precursors_list.append( + { + "rank": int(float(r.get("rank", 0))), + "track_id": tid, + "onset_scan": int(float(r.get("onset_scan", 0))), + "peak_scan": int(float(r.get("peak_scan", 0))), + "persistence_scans": int(float(r.get("persistence_scans", 0))), + "peak_score": round(float(r.get("peak_score", 0.0)), 4), + "max_anisotropy": round(float(r.get("max_anisotropy", 1.0)), 3), + "max_subpeak_count": int(float(r.get("max_subpeak_count", 1))), + "confidence": round(float(r.get("confidence", 0.0)), 4), + "subsequent_split": str(r.get("subsequent_split", "")).lower() == "true", + } + ) + + by_tid: dict[int, list[dict[str, Any]]] = {} + for r in track_rows: + by_tid.setdefault(int(r["track_id"]), []).append(r) + + n_scans = int(trk_summary.get("n_scans") or evt_summary.get("n_scans") or 0) + if n_scans == 0 and track_rows: + n_scans = max(int(r["scan_idx"]) for r in track_rows) + 1 + + priority_tids = sorted( + by_tid.keys(), + key=lambda tid: ( + 0 if (tid in split_tids or tid in merge_tids or tid in prec_tids) else 1, + -len(by_tid[tid]), + -max(float(x.get("integrated_intensity", 0.0)) for x in by_tid[tid]), + tid, + ), + ) + thumbs = _collect_mip_thumbnails(seg_dir, ds_root, track_rows, priority_tids, max_tracks=max_mip_tracks) + + tracks_payload: list[dict[str, Any]] = [] + for tid in sorted(by_tid.keys(), key=lambda t: (float(by_tid[t][0]["centroid_d"]), t)): + obs_sorted = sorted(by_tid[tid], key=lambda x: int(x["scan_idx"])) + gk = gk_by_tid.get(tid) + scans = [int(x["scan_idx"]) for x in obs_sorted] + has_gap = any(scans[i + 1] - scans[i] > 1 for i in range(len(scans) - 1)) + is_full = n_scans > 0 and min(scans) == 0 and max(scans) >= n_scans - 1 + + obs_items: list[dict[str, Any]] = [] + for idx_o, o in enumerate(obs_sorted): + s_i = int(o["scan_idx"]) + strain_pct = float(gk.strain[idx_o] * 100.0) if gk is not None and idx_o < len(gk.strain) else 0.0 + miso = ( + float(gk.misorientation_total[idx_o]) + if gk is not None and idx_o < len(gk.misorientation_total) + else 0.0 + ) + aniso = float(gk.anisotropy[idx_o]) if gk is not None and idx_o < len(gk.anisotropy) else 1.0 + po = prec_obs_by_key.get((tid, s_i), {}) + obs_items.append( + { + "scan_idx": s_i, + "label": int(o.get("label", 0)), + "mu": round(float(o["centroid_mu"]), 4), + "chi": round(float(o["centroid_chi"]), 4), + "d": round(float(o["centroid_d"]), 4), + "intensity": round(float(o.get("integrated_intensity", 0.0)), 2), + "voxels": int(o.get("voxel_count", 0)), + "strain_pct": round(strain_pct, 4), + "misorientation": round(miso, 4), + "anisotropy": round(aniso, 3), + "precursor_score": round(float(po.get("score", 0.0)), 4), + "roi_dispersion_term": round(float(po.get("roi_dispersion_term", 0.0)), 4), + "subpeaks": int(po.get("subpeaks", 1)), + "mip_b64": thumbs.get(f"{tid}:{s_i}"), + } + ) + + tracks_payload.append( + { + "track_id": tid, + "is_full": is_full, + "has_gap": has_gap, + "is_split": tid in split_tids, + "is_merge": tid in merge_tids, + "is_precursor": tid in prec_tids, + "max_abs_strain_pct": round(max((abs(x["strain_pct"]) for x in obs_items), default=0.0), 4), + "max_precursor": round(max((x["precursor_score"] for x in obs_items), default=0.0), 4), + "observations": obs_items, + } + ) + + scan_names = ( + trk_summary.get("scan_names") or evt_summary.get("scan_names") or [f"scan{i + 1:04d}" for i in range(n_scans)] + ) + return { + "title": f"BraggTrack 4D Explorer — {Path(manifest.get('dataset_root') or root).name}", + "schema_version": "report.v1", + "n_scans": n_scans, + "scan_names": scan_names, + "calibrated_axes": bool(trk_summary.get("calibrated_axes", False)), + "backend": evt_summary.get("backend") or manifest.get("config", {}).get("backend") or "n/a", + "model": evt_summary.get("model") or manifest.get("config", {}).get("model") or "n/a", + "braggtrack_version": manifest.get("braggtrack_version", "0.3.0"), + "total_duration_sec": manifest.get("total_duration_sec"), + "kpis": { + "total_tracks": len(tracks_payload), + "full_length_tracks": sum(1 for t in tracks_payload if t["is_full"]), + "gap_closed_count": int(trk_summary.get("gap_closed_count", evt_summary.get("gap_closed_count", 0))), + "split_count": sum(1 for e in events_list if e["event_type"] == "split"), + "merge_count": sum(1 for e in events_list if e["event_type"] == "merge"), + "precursor_count": len(precursors_list), + }, + "tracks": tracks_payload, + "events": events_list, + "precursors": precursors_list, + } + + +_HTML_TEMPLATE = """ + + + + +__TITLE__ + + + +
+
+

BraggTrack 4D Explorer

+
+
+ +
+
+
+
+

+ 4D Trajectory & Lineage DAG + Click any track node to inspect +

+
+
+
+
+

+ Track Inspector + +

+
+
+
+
+
+
+

Split & Merge Lineage Events

+
+
+
+

Persistent Fracture Precursor Candidates

+
+
+
+ + + + +""" + + +def generate_html_report( + artifact_dir: str | Path, + out_html: str | Path | None = None, + *, + dataset_root: str | Path | None = None, + max_mip_tracks: int = 16, +) -> Path: + """Render a self-contained interactive HTML 4D explorer from a BraggTrack output directory.""" + root = Path(artifact_dir) + dest = Path(out_html) if out_html is not None else (root / "report.html") + dest.parent.mkdir(parents=True, exist_ok=True) + + payload = build_explorer_payload(root, dataset_root=dataset_root, max_mip_tracks=max_mip_tracks) + safe_json = json.dumps(payload).replace(" tuple[Any, Any]: + """Plot 3-panel kinematic evolution: lattice strain, angular misorientation, and intensity/anisotropy.""" + _, plt = _require_matplotlib() + + if kinematics_or_tracks and isinstance(kinematics_or_tracks[0], GrainKinematics): + gks: list[GrainKinematics] = list(kinematics_or_tracks) # type: ignore[arg-type] + else: + rows = [dict(r) for r in kinematics_or_tracks] # type: ignore[arg-type] + if rows and "strain" in rows[0] and "misorientation_total" in rows[0] and "relative_intensity" in rows[0]: + # Reconstruct GrainKinematics from kinematics.csv rows or compute from tracks.csv rows + gks = compute_grain_kinematics(rows) + else: + gks = compute_grain_kinematics(rows) + + fig, axes = plt.subplots(1, 3, figsize=(13.5, 4.2)) + if not gks: + for ax in axes: + apply_axis_style(ax) + fig.suptitle(title or "Per-Grain Kinematics Dashboard (empty)") + maybe_save(fig, save_path) + return fig, axes + + ranked = sorted(gks, key=lambda g: (-len(g.scan_indices), -float(g.integrated_intensity[0]), g.track_id))[ + :max_tracks + ] + + for idx, gk in enumerate(ranked): + color = OKABE_ITO[idx % len(OKABE_ITO)] + lbl = f"T{gk.track_id}" + scans = gk.scan_indices + axes[0].plot(scans, 100.0 * gk.strain, "o-", color=color, linewidth=1.7, markersize=4.5, label=lbl) + axes[1].plot(scans, gk.misorientation_total, "o-", color=color, linewidth=1.7, markersize=4.5, label=lbl) + axes[2].plot(scans, gk.anisotropy, "s-", color=color, linewidth=1.6, markersize=4.0, label=lbl) + + for ev in events or []: + etype = ev.get("event_type") if isinstance(ev, dict) else getattr(ev, "event_type", "") + raw_scan = ( + ev.get("scan_idx_to", ev.get("child_scan", 0)) + if isinstance(ev, dict) + else getattr(ev, "scan_idx_to", getattr(ev, "child_scan", 0)) + ) + cscan = int(float(raw_scan)) + ev_col = OKABE_ITO[1] if etype == "split" else OKABE_ITO[2] + for ax in axes: + ax.axvline(cscan - 0.5, linestyle=":", linewidth=1.2, color=ev_col, alpha=0.8) + + strain_label = ( + r"Lattice strain $\varepsilon = \Delta d / d_0$ (%)" + if calibrated + else r"Relative $d$-shift $\Delta d / d_0$ (%)" + ) + miso_label = ( + r"Misorientation $\sqrt{\Delta\mu^2 + \Delta\chi^2}$ (deg)" + if calibrated + else r"Angular shift $\sqrt{\Delta\mu^2 + \Delta\chi^2}$ (px)" + ) + + axes[0].axhline(0.0, color="#94A3B8", linewidth=0.8) + axes[0].set_title("Strain / d-Spacing Evolution", fontsize=10, fontweight="bold") + axes[0].set_ylabel(strain_label, fontsize=8.8) + + axes[1].set_title("Lattice Rotation / Misorientation", fontsize=10, fontweight="bold") + axes[1].set_ylabel(miso_label, fontsize=8.8) + + axes[2].set_title(r"3D Shape Anisotropy ($\lambda_1 / \lambda_3$)", fontsize=10, fontweight="bold") + axes[2].set_ylabel(r"Eigenvalue ratio $\lambda_1 / \lambda_3$", fontsize=8.8) + + for ax in axes: + ax.set_xlabel("Scan index", fontsize=9) + apply_axis_style(ax) + + if len(ranked) <= 10: + axes[0].legend(fontsize=7.5, ncol=2, frameon=True) + + fig.suptitle( + title or f"Operando Grain Kinematics ({'Calibrated Physical Units' if calibrated else 'Detector Units'})", + fontsize=11, + fontweight="bold", + y=1.02, + ) + fig.tight_layout() + maybe_save(fig, save_path) + return fig, axes diff --git a/braggtrack/viz/lineage.py b/braggtrack/viz/lineage.py new file mode 100644 index 0000000..4dff72a --- /dev/null +++ b/braggtrack/viz/lineage.py @@ -0,0 +1,216 @@ +"""Publication-grade 4D trajectory and split/merge lineage DAG visualization.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import networkx as nx +import numpy as np + +from braggtrack.tracking.lifecycle import tracks_to_table + +from ._style import OKABE_ITO, _require_matplotlib, apply_axis_style, maybe_save + + +def _parse_int_list(val: Any) -> list[int]: + if val is None or val == "": + return [] + if isinstance(val, (list, tuple)): + return [int(x) for x in val] + return [int(tok) for tok in str(val).split(";") if tok.strip()] + + +def plot_lineage_dag( + tracks_or_graph: nx.DiGraph | list[dict[str, Any]], + *, + events: list[Any] | None = None, + max_tracks: int = 40, + title: str | None = None, + save_path: str | Path | None = None, +) -> tuple[Any, Any]: + """Plot a scan-layered 4D trajectory & lineage DAG highlighting splits, merges, and gap closures.""" + _, plt = _require_matplotlib() + + if isinstance(tracks_or_graph, nx.DiGraph): + rows = tracks_to_table(tracks_or_graph) + if events is None: + events = list(tracks_or_graph.graph.get("lineage_events", [])) + else: + rows = [dict(r) for r in tracks_or_graph] + + fig, ax = plt.subplots(figsize=(9.5, 5.2)) + if not rows: + ax.set_title(title or "4D Trajectory Lineage DAG (empty)") + apply_axis_style(ax) + maybe_save(fig, save_path) + return fig, ax + + by_track: dict[int, list[dict[str, Any]]] = {} + for r in rows: + by_track.setdefault(int(float(r["track_id"])), []).append(r) + for tid in by_track: + by_track[tid].sort(key=lambda r: int(float(r["scan_idx"]))) + + # Prioritize tracks involved in split/merge events, then longest & brightest tracks. + event_records: list[dict[str, Any]] = [] + involved_tids: set[int] = set() + for ev in events or []: + if isinstance(ev, dict): + etype = str(ev.get("event_type", "")) + pscan = int(float(ev.get("scan_idx_from", ev.get("parent_scan", 0)))) + cscan = int(float(ev.get("scan_idx_to", ev.get("child_scan", pscan + 1)))) + ptids = _parse_int_list(ev.get("parent_track_ids")) + ctids = _parse_int_list(ev.get("child_track_ids")) + else: + etype = str(getattr(ev, "event_type", "")) + pscan = int(getattr(ev, "scan_idx_from", getattr(ev, "parent_scan", 0))) + cscan = int(getattr(ev, "scan_idx_to", getattr(ev, "child_scan", pscan + 1))) + ptids = [int(x) for x in getattr(ev, "parent_track_ids", ())] + ctids = [int(x) for x in getattr(ev, "child_track_ids", ())] + event_records.append( + { + "event_type": etype, + "parent_scan": pscan, + "child_scan": cscan, + "parent_track_ids": ptids, + "child_track_ids": ctids, + } + ) + involved_tids.update(ptids) + involved_tids.update(ctids) + + ranked_tids = sorted( + by_track.keys(), + key=lambda tid: ( + 0 if tid in involved_tids else 1, + -len(by_track[tid]), + -max(float(r.get("integrated_intensity", 0.0)) for r in by_track[tid]), + tid, + ), + )[:max_tracks] + selected_set = set(ranked_tids) + + # Order vertical lanes by initial centroid_d so spatially nearby fragments sit adjacent. + ordered_tids = sorted(ranked_tids, key=lambda tid: (float(by_track[tid][0]["centroid_d"]), tid)) + lane_y = {tid: float(idx) for idx, tid in enumerate(ordered_tids)} + + node_xy: dict[tuple[int, int], tuple[float, float]] = {} + all_x: list[float] = [] + all_y: list[float] = [] + all_strain: list[float] = [] + + gap_labeled = False + cont_labeled = False + for tid in ordered_tids: + obs = by_track[tid] + y_val = lane_y[tid] + d0 = float(obs[0]["centroid_d"]) + scans = [int(float(r["scan_idx"])) for r in obs] + for r in obs: + s_idx = int(float(r["scan_idx"])) + d_cur = float(r["centroid_d"]) + strain_pct = 100.0 * (d_cur - d0) / d0 if abs(d0) > 1e-9 else 0.0 + node_xy[(tid, s_idx)] = (float(s_idx), y_val) + all_x.append(float(s_idx)) + all_y.append(y_val) + all_strain.append(strain_pct) + + for i in range(len(scans) - 1): + s0, s1 = scans[i], scans[i + 1] + if s1 - s0 > 1: + ax.plot( + [s0, s1], + [y_val, y_val], + linestyle="--", + linewidth=1.8, + color=OKABE_ITO[4], + label="Gap-closed bridge" if not gap_labeled else None, + zorder=2, + ) + gap_labeled = True + else: + ax.plot( + [s0, s1], + [y_val, y_val], + linestyle="-", + linewidth=1.8, + color="#64748B", + alpha=0.75, + label="Continuation" if not cont_labeled else None, + zorder=2, + ) + cont_labeled = True + + split_labeled = False + merge_labeled = False + for ev in event_records: + etype = ev["event_type"] + pscan = ev["parent_scan"] + cscan = ev["child_scan"] + for ptid in ev["parent_track_ids"]: + for ctid in ev["child_track_ids"]: + if ptid not in selected_set or ctid not in selected_set: + continue + p_xy = node_xy.get((ptid, pscan), (float(pscan), lane_y[ptid])) + c_xy = node_xy.get((ctid, cscan), (float(cscan), lane_y[ctid])) + color = OKABE_ITO[1] if etype == "split" else OKABE_ITO[2] + lbl = None + if etype == "split" and not split_labeled: + lbl = "1 -> 2+ Split" + split_labeled = True + elif etype == "merge" and not merge_labeled: + lbl = "2+ -> 1 Merge" + merge_labeled = True + ax.plot( + [p_xy[0], c_xy[0]], + [p_xy[1], c_xy[1]], + color=color, + linewidth=2.4, + alpha=0.92, + label=lbl, + zorder=3, + ) + ax.scatter( + [c_xy[0] if etype == "split" else p_xy[0]], + [c_xy[1] if etype == "split" else p_xy[1]], + marker="D" if etype == "split" else "s", + s=58, + color=color, + edgecolor="#FFFFFF", + linewidth=0.8, + zorder=5, + ) + + vmax = max(0.05, float(np.max(np.abs(all_strain)))) if all_strain else 0.1 + sc = ax.scatter( + all_x, + all_y, + c=all_strain, + cmap="coolwarm", + vmin=-vmax, + vmax=vmax, + s=42, + edgecolors="#1E293B", + linewidths=0.6, + zorder=4, + ) + cbar = fig.colorbar(sc, ax=ax, pad=0.02) + cbar.set_label(r"Relative $d$-shift $\Delta d / d_0$ (%)", fontsize=9) + + ax.set_yticks([lane_y[tid] for tid in ordered_tids]) + ax.set_yticklabels([f"T{tid}" for tid in ordered_tids], fontsize=8) + max_scan = int(max(all_x)) if all_x else 0 + ax.set_xticks(list(range(max_scan + 1))) + ax.set_xlabel("Scan index", fontsize=9.5) + ax.set_ylabel("Grain trajectory lane", fontsize=9.5) + ax.set_title( + title or f"4D Bragg Reflection Lineage DAG ({len(ordered_tids)} tracks)", fontsize=11, fontweight="bold" + ) + apply_axis_style(ax) + handles, labels_legend = ax.get_legend_handles_labels() + if handles: + ax.legend(loc="upper left", fontsize=8, frameon=True) + fig.tight_layout() + maybe_save(fig, save_path) + return fig, ax diff --git a/braggtrack/viz/mips.py b/braggtrack/viz/mips.py new file mode 100644 index 0000000..8328f37 --- /dev/null +++ b/braggtrack/viz/mips.py @@ -0,0 +1,104 @@ +"""Orthogonal 3-view MIP spot cards and full-scan projection overlays.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np + +from braggtrack.semantic.mips import crop_spot_cube, orthogonal_mips + +from ._style import _require_matplotlib, apply_axis_style, maybe_save + + +def plot_spot_mip_card( + volume: np.ndarray | Any, + labels: np.ndarray, + label_id: int, + bbox: dict[str, Any], + *, + margin: int = 3, + title: str | None = None, + save_path: str | Path | None = None, +) -> tuple[Any, Any]: + """Plot a 3-panel orthogonal MIP card ``(d, chi)``, ``(mu, d)``, ``(mu, chi)`` for one 3D reflection.""" + _, plt = _require_matplotlib() + masked_cube, mask = crop_spot_cube(volume, labels, int(label_id), bbox, margin=margin) + mip_mu, mip_chi, mip_d = orthogonal_mips(masked_cube) + mask_mu, mask_chi, mask_d = orthogonal_mips(mask) + + fig, axes = plt.subplots(1, 3, figsize=(10.5, 3.4)) + panels = [ + (mip_mu, mask_mu, r"$\mu$-collapsed $(d, \chi)$", r"$\chi$ (px)", r"$d$ (px)"), + (mip_chi, mask_chi, r"$\chi$-collapsed $(\mu, d)$", r"$d$ (px)", r"$\mu$ (frame)"), + (mip_d, mask_d, r"$d$-collapsed $(\mu, \chi)$", r"$\chi$ (px)", r"$\mu$ (frame)"), + ] + + for ax, (mip, msk, subtitle, xlabel, ylabel) in zip(axes, panels): + im = ax.imshow(mip, cmap="magma", origin="lower", aspect="auto") + if msk.shape[0] >= 2 and msk.shape[1] >= 2 and np.any(msk > 0) and np.any(msk == 0): + ax.contour(msk, levels=[0.5], colors=["#56B4E9"], linewidths=1.1) + ax.set_title(subtitle, fontsize=9.5, fontweight="bold") + ax.set_xlabel(xlabel, fontsize=8.5) + ax.set_ylabel(ylabel, fontsize=8.5) + fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04) + + vox = int(np.sum(mask > 0)) + tot_int = float(np.sum(masked_cube)) + header = title or f"Spot #{label_id} | {vox} voxels | I = {tot_int:.1f}" + fig.suptitle(header, fontsize=10.5, fontweight="bold", y=1.02) + fig.tight_layout() + maybe_save(fig, save_path) + return fig, axes + + +def plot_scan_projection_overlay( + volume: np.ndarray | Any, + labels: np.ndarray, + feature_rows: list[dict[str, Any]], + *, + max_labels: int = 25, + title: str | None = None, + save_path: str | Path | None = None, +) -> tuple[Any, Any]: + """Plot a full-scan ``(d, chi)`` MIP with instance contours and ranked spot IDs.""" + _, plt = _require_matplotlib() + vol_arr = volume[()] if not isinstance(volume, np.ndarray) else volume + mip_dc = np.max(np.asarray(vol_arr, dtype=np.float64), axis=0) + disp = np.log1p(np.maximum(0.0, mip_dc)) + fg_proj = np.max(labels > 0, axis=0).astype(np.float64) + + fig, ax = plt.subplots(figsize=(7.5, 6.0)) + im = ax.imshow(disp, cmap="inferno", origin="lower", aspect="auto") + if fg_proj.shape[0] >= 2 and fg_proj.shape[1] >= 2 and np.any(fg_proj > 0) and np.any(fg_proj == 0): + ax.contour(fg_proj, levels=[0.5], colors=["#56B4E9"], linewidths=0.8, alpha=0.8) + + ranked = sorted(feature_rows, key=lambda r: float(r.get("integrated_intensity", 0.0)), reverse=True)[:max_labels] + for row in ranked: + if "bbox_min_x" in row and "bbox_max_x" in row: + cx = 0.5 * (float(row["bbox_min_x"]) + float(row["bbox_max_x"])) + cy = 0.5 * (float(row["bbox_min_y"]) + float(row["bbox_max_y"])) + else: + cx = float(row["centroid_chi"]) + cy = float(row["centroid_d"]) + lid = int(float(row["label"])) + ax.plot(cx, cy, marker="+", color="#00E5FF", markersize=6, markeredgewidth=1.2) + ax.annotate( + f"#{lid}", + (cx, cy), + xytext=(4, 4), + textcoords="offset points", + fontsize=7.5, + color="#FFFFFF", + bbox={"boxstyle": "round,pad=0.15", "fc": "#111827", "ec": "none", "alpha": 0.65}, + ) + + ax.set_title(title or f"Detector Projection (log1p MIP) — {len(feature_rows)} segmented reflections", fontsize=10.5) + ax.set_xlabel(r"Detector $\chi$ / fast pixel", fontsize=9) + ax.set_ylabel(r"Detector $d$ / slow pixel", fontsize=9) + apply_axis_style(ax) + fig.colorbar(im, ax=ax, label="log(1 + max intensity)") + fig.tight_layout() + maybe_save(fig, save_path) + return fig, ax diff --git a/braggtrack/viz/precursors.py b/braggtrack/viz/precursors.py new file mode 100644 index 0000000..771fcad --- /dev/null +++ b/braggtrack/viz/precursors.py @@ -0,0 +1,149 @@ +"""Persistent 4D fracture precursor heatmap and component decomposition plots.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np + +from braggtrack.tracking.precursors import ObservationPrecursorScore, PrecursorCandidate + +from ._style import OKABE_ITO, _require_matplotlib, apply_axis_style, maybe_save + + +def _obs_to_dict(obs: ObservationPrecursorScore | dict[str, Any]) -> dict[str, Any]: + if isinstance(obs, dict): + return { + "track_id": int(float(obs["track_id"])), + "scan_idx": int(float(obs["scan_idx"])), + "anisotropy_term": float(obs.get("anisotropy_term", 0.0)), + "spread_term": float(obs.get("spread_term", 0.0)), + "roi_dispersion_term": float(obs.get("roi_dispersion_term", 0.0)), + "roi_subpeak_count": int(float(obs.get("roi_subpeak_count", 1))), + "precursor_score": float(obs.get("precursor_score", 0.0)), + } + return { + "track_id": obs.track_id, + "scan_idx": obs.scan_idx, + "anisotropy_term": obs.anisotropy_term, + "spread_term": obs.spread_term, + "roi_dispersion_term": obs.roi_dispersion_term, + "roi_subpeak_count": obs.roi_subpeak_count, + "precursor_score": obs.precursor_score, + } + + +def plot_precursor_timeline( + obs_scores: list[ObservationPrecursorScore] | list[dict[str, Any]], + *, + candidates: list[PrecursorCandidate] | list[dict[str, Any]] | None = None, + max_tracks: int = 15, + score_threshold: float = 0.40, + title: str | None = None, + save_path: str | Path | None = None, +) -> tuple[Any, Any]: + """Plot a track-by-scan fracture precursor heatmap and top-candidate component decomposition.""" + _, plt = _require_matplotlib() + rows = [_obs_to_dict(x) for x in obs_scores] + + fig, axes = plt.subplots(1, 2, figsize=(12.0, 4.4), gridspec_kw={"width_ratios": [1.15, 1.0]}) + if not rows: + for ax in axes: + apply_axis_style(ax) + fig.suptitle(title or "Fracture Precursor Timeline (no observations)") + maybe_save(fig, save_path) + return fig, axes + + by_track: dict[int, list[dict[str, Any]]] = {} + max_scan = 0 + for r in rows: + by_track.setdefault(r["track_id"], []).append(r) + if r["scan_idx"] > max_scan: + max_scan = r["scan_idx"] + + split_tids: set[int] = set() + top_tid: int | None = None + for idx_c, c in enumerate(candidates or []): + tid_c = int(float(c["track_id"])) if isinstance(c, dict) else int(c.track_id) + sub_split = ( + str(c.get("subsequent_split", "")).lower() == "true" if isinstance(c, dict) else bool(c.subsequent_split) + ) + if idx_c == 0: + top_tid = tid_c + if sub_split: + split_tids.add(tid_c) + + ranked_tids = sorted( + by_track.keys(), + key=lambda tid: (-max(x["precursor_score"] for x in by_track[tid]), tid), + )[:max_tracks] + if top_tid is None and ranked_tids: + top_tid = ranked_tids[0] + + mat = np.full((len(ranked_tids), max_scan + 1), np.nan, dtype=np.float64) + for r_i, tid in enumerate(ranked_tids): + for r in by_track[tid]: + mat[r_i, r["scan_idx"]] = r["precursor_score"] + + im = axes[0].imshow(mat, aspect="auto", cmap="YlOrRd", vmin=0.0, vmax=1.0, interpolation="nearest") + axes[0].set_yticks(list(range(len(ranked_tids)))) + axes[0].set_yticklabels( + [f"T{tid} *" if tid in split_tids else f"T{tid}" for tid in ranked_tids], + fontsize=8.5, + ) + axes[0].set_xticks(list(range(max_scan + 1))) + axes[0].set_xlabel("Scan index", fontsize=9) + axes[0].set_ylabel("Track ID (* = subsequent split)", fontsize=9) + axes[0].set_title("Precursor Score Heatmap", fontsize=10, fontweight="bold") + fig.colorbar(im, ax=axes[0], fraction=0.046, pad=0.04, label="Precursor score") + + # Right panel: component breakdown for top candidate + if top_tid is not None and top_tid in by_track: + top_obs = sorted(by_track[top_tid], key=lambda x: x["scan_idx"]) + scans = [x["scan_idx"] for x in top_obs] + axes[1].plot( + scans, + [x["anisotropy_term"] for x in top_obs], + "o--", + color=OKABE_ITO[0], + linewidth=1.4, + label="Shape anisotropy", + ) + axes[1].plot( + scans, + [x["spread_term"] for x in top_obs], + "s--", + color=OKABE_ITO[2], + linewidth=1.4, + label="Covariance broadening", + ) + axes[1].plot( + scans, + [x["roi_dispersion_term"] for x in top_obs], + "^--", + color=OKABE_ITO[3], + linewidth=1.4, + label="3D sub-peaks + DINO dispersion", + ) + axes[1].plot( + scans, + [x["precursor_score"] for x in top_obs], + "o-", + color=OKABE_ITO[1], + linewidth=2.2, + label=f"Composite score (T{top_tid})", + ) + axes[1].axhline(score_threshold, linestyle=":", color="#475569", linewidth=1.2, label="Threshold") + axes[1].set_ylim(-0.04, 1.06) + axes[1].set_xticks(list(range(max_scan + 1))) + axes[1].set_xlabel("Scan index", fontsize=9) + axes[1].set_ylabel("Normalized term [0, 1]", fontsize=9) + axes[1].set_title(f"Top Precursor Decomposition — Track T{top_tid}", fontsize=10, fontweight="bold") + axes[1].legend(fontsize=7.8, loc="best", frameon=True) + apply_axis_style(axes[1]) + + fig.suptitle(title or "Persistent 4D Fracture Precursor Analysis", fontsize=11, fontweight="bold", y=1.02) + fig.tight_layout() + maybe_save(fig, save_path) + return fig, axes diff --git a/docs/architecture.md b/docs/architecture.md index 7f9e002..c8efe27 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,17 +6,18 @@ This document describes how the repository is structured, how data flows through ```text BraggTrack/ -├── braggtrack/ # Installable Python package (v0.2.0) -│ ├── io/ # Discovery, ESRF-ID03 / NeXus HDF5 auto-discovery, AxisCalibration, validation -│ ├── segmentation/ # Classical 3D LoG+watershed, cross-scan threshold smoothing, experimental DINO segmenter, features +├── braggtrack/ # Installable Python package (v0.3.0) +│ ├── io/ # Discovery, LazyVolume streaming, AxisCalibration, DetectorGeometry, ImageD11 & NeXus export +│ ├── segmentation/ # Classical 3D LoG+watershed, chunked Otsu + threshold smoothing, experimental DINO segmenter │ ├── semantic/ # Orthogonal (mu, chi, d) MIPs + frozen ViT / patch encoders (DINOv2 default, DINOv3, mock) -│ ├── tracking/ # Costs, Hungarian assignment, kinematics, split/merge lineage DAG, fracture precursors -│ └── cli/ # Thin argparse entry points + reproducible run_pipeline orchestrator +│ ├── tracking/ # Costs, Hungarian assignment, multi-scan gap closing, kinematics, split/merge DAG, precursors +│ ├── viz/ # Publication plots (MIPs, Lineage DAG, Strain fans, Precursors) + interactive HTML 4D Explorer +│ └── cli/ # Thin argparse entry points (9 CLIs) + reproducible run_pipeline orchestrator ├── data/ │ └── sample_operando/ # Bundled ESRF-ID03 Ni-rich cathode operando scans (scan0001..scan0003) ├── notebooks/ # End-to-end demo and DINO vs. classical segmentation notebooks -├── docs/ # Architecture & coding practices -├── scripts/ # Acceptance gates, semantic ablation with bootstrap CIs, ci_report.py +├── docs/ # Architecture, quantitative benchmarks, figures, and coding practices +├── scripts/ # 9 acceptance gates, 5-scenario benchmark suite, semantic ablation, ci_report.py ├── tests/ # Unit and acceptance test suite └── artifacts/ # Generated outputs (gitignored) ``` @@ -25,11 +26,12 @@ BraggTrack/ | Layer | Responsibility | Key Types / Functions | Depends on | |---|---|---|---| -| **io** | Discover scans, auto-discover 3D HDF5 cubes, extract timestamps & `AxisCalibration`, validate sequences | `ScanVolumeMeta`, `AxisCalibration`, `load_primary_volume`, `BeamlineAdapter` | `numpy`, `h5py` | +| **io** | Discover scans, stream `LazyVolume` HDF5 slices & chunked Otsu histograms, calibrate `(mu, chi, d)` & `(2theta, eta, d, \|Q\|)`, export `ImageD11` `.flt` & NeXus `.nxs` | `LazyVolume`, `otsu_threshold_chunked`, `AxisCalibration`, `DetectorGeometry`, `export_imaged11_flt`, `export_nexus_process` | `numpy`, `h5py` | | **segmentation** | Multi-scan threshold smoothing, 3D LoG + `h`-maxima + watershed (or experimental DINO + HDBSCAN), physical/voxel feature table | `segment_classical`, `smooth_thresholds`, `segment_dino`, `extract_instance_table` | `numpy`, `scipy`, `scikit-image`, `scikit-learn` | -| **semantic** | Cropped spot cubes, orthogonal MIPs, multi-view global embeddings & dense patch-token grids | `crop_spot_cube`, `orthogonal_mips`, `make_multiview_encoder`, `make_patch_encoder` | `numpy`, optional `torch` + `transformers` | -| **tracking** | Vectorised geometry + semantic cost, Hungarian assignment, per-grain kinematics, `1 -> 2+` split & `2+ -> 1` merge lineage resolution, persistent fracture precursors | `PositionShapeCost`, `GeometrySemanticCost`, `build_tracks`, `compute_grain_kinematics`, `resolve_splits_and_merges`, `compute_fracture_precursors` | `numpy`, `scipy`, `networkx` | -| **cli** | Parse args/JSON config, orchestrate stages, write CSV/JSON/NPZ artifacts and SHA-256 `run_manifest.json` | `segment_dataset`, `embed_dataset`, `track_dataset`, `detect_events`, `run_pipeline` | All of the above | +| **semantic** | Lazy ROI spot cropping, orthogonal MIPs, multi-view global embeddings & dense patch-token grids | `crop_spot_cube`, `orthogonal_mips`, `make_multiview_encoder`, `make_patch_encoder` | `numpy`, optional `torch` + `transformers` | +| **tracking** | Vectorised geometry + semantic cost, Hungarian assignment, multi-scan gap closing (`max_gap`), per-grain kinematics, `1 -> 2+` split & `2+ -> 1` merge lineage, fracture precursors | `PositionShapeCost`, `GeometrySemanticCost`, `resolve_track_gaps`, `build_tracks`, `compute_grain_kinematics`, `resolve_splits_and_merges`, `compute_fracture_precursors` | `numpy`, `scipy`, `networkx` | +| **viz** | Publication figures and zero-CDN offline interactive HTML 4D Lineage & Strain Explorer | `plot_lineage_dag`, `plot_kinematics_dashboard`, `plot_precursor_timeline`, `plot_spot_mip_card`, `generate_html_report` | `numpy`, optional `matplotlib` (HTML explorer uses stdlib only) | +| **cli** | Parse args/JSON config, orchestrate stages, write CSV/JSON/NPZ/FLT/NXS/HTML artifacts and SHA-256 `run_manifest.json` | `segment_dataset`, `embed_dataset`, `track_dataset`, `detect_events`, `report_dataset`, `run_pipeline` | All of the above | **Rule of thumb:** keep **numeric kernels** and **typed data contracts** in subpackages; keep **CLI modules** as thin orchestration layers. @@ -38,25 +40,29 @@ BraggTrack/ ```mermaid flowchart LR subgraph inputs["Inputs"] - H5["HDF5 / NeXus 3D scans"] - CAL["AxisCalibration (auto or JSON)"] + H5["HDF5 / NeXus 3D scans (LazyVolume)"] + CAL["AxisCalibration & DetectorGeometry"] end subgraph seg["Segmentation"] - SEG["segment_dataset (smoothed Otsu + 3D watershed)"] + SEG["segment_dataset (chunked Otsu + 3D watershed)"] FEAT["features.csv + labels.npz"] end subgraph sem["Semantic"] - EMB["embed_dataset (MIPs + DINOv2 / mock)"] + EMB["embed_dataset (ROI MIPs + DINOv2 / mock)"] NPZ["embeddings.npz"] end - subgraph trk["Tracking & Kinematics"] - TRK["track_dataset (Hungarian + kinematics)"] - KIN["tracks.csv + kinematics.csv"] + subgraph trk["Tracking, Gap-Closing & Kinematics"] + TRK["track_dataset (Hungarian + max_gap + kinematics)"] + KIN["tracks.csv + kinematics.csv + .flt / .nxs"] end subgraph evt["Lineage & Precursors"] EVT["detect_events (splits, merges, ROI dispersion)"] OUT["events.csv + precursors.csv"] end + subgraph rep["Visualization"] + REP["report_dataset (braggtrack.viz)"] + HTML["report.html + figures/*.png"] + end MAN["run_manifest.json (SHA-256 + timings)"] H5 --> SEG @@ -65,19 +71,23 @@ flowchart LR H5 --> EMB FEAT --> EMB --> NPZ FEAT --> TRK + CAL --> TRK NPZ --> TRK TRK --> KIN H5 --> EVT FEAT --> EVT NPZ --> EVT EVT --> OUT + KIN --> REP + OUT --> REP --> HTML SEG --> MAN TRK --> MAN EVT --> MAN + REP --> MAN ``` - **Dataset root** is a directory containing `scan*` folders (defaults to bundled `data/sample_operando/` when present). -- **Physical coordinates vs. voxel indices:** `extract_instance_table(..., calibration=cal)` converts `centroid_mu`, `centroid_chi`, `centroid_d`, `cov_*`, and `eig_1..3` into physical reciprocal-space units while preserving integer voxel `bbox_*` indices so downstream 3D ROI cropping (`crop_spot_cube`) remains exact. +- **Physical coordinates vs. voxel indices:** `extract_instance_table(..., calibration=cal)` converts `centroid_mu`, `centroid_chi`, `centroid_d`, `cov_*`, and `eig_1..3` into physical reciprocal-space units while preserving integer voxel `bbox_*` indices so downstream 3D ROI cropping (`crop_spot_cube` over `LazyVolume`) remains exact. - **Artifacts** under `artifacts/` are reproducible build outputs and are never committed to Git. ## Extension points @@ -89,12 +99,12 @@ flowchart LR ## Performance notes -- Frame-to-frame Hungarian association builds its `(N, M)` cost matrix via vectorised `scipy.spatial.distance.cdist` and a single dense matrix multiplication (`A @ B.T`) for semantic cosine similarity. -- Cross-scan threshold stabilisation (`smooth_thresholds`) operates in $O(N \cdot W)$ on scalar per-scan Otsu thresholds without holding the entire 4D dataset in memory simultaneously during streaming workflows. +- `LazyVolume` streams `z`-slabs (`otsu_threshold_chunked`) in Pass 1 of `segment_dataset` and reads only tight 3D spot ROIs (`read_roi`) in `embed_dataset` and `detect_events`, avoiding full-volume `float64` allocations on multi-gigabyte detector stacks. +- Frame-to-frame Hungarian association and multi-scan gap closing (`resolve_track_gaps`) build cost matrices via vectorised `scipy.spatial.distance.cdist` and dense matrix multiplication (`A @ B.T`) for semantic cosine similarity. ## Configuration surfaces - **Environment variables:** - `BRAGGTRACK_DATA_ROOT`: Override default dataset root discovery. - `BRAGGTRACK_DINO_BACKEND`: Force `auto`, `mock`, or `torch` backend selection. -- **Pipeline JSON config:** Pass `--config pipeline.json` to `braggtrack-run` to declaratively configure backend, model ID (`DEFAULT_DINO_MODEL = "facebook/dinov2-small"` or DINOv3), calibration file, semantic weights (`cost_alpha`, `cost_beta`), and split/precursor thresholds. +- **Pipeline JSON config:** Pass `--config pipeline.json` to `braggtrack-run` to declaratively configure backend, model ID (`DEFAULT_DINO_MODEL = "facebook/dinov2-small"` or DINOv3), calibration/detector JSON files, `max_gap`, semantic weights (`cost_alpha`, `cost_beta`), split/precursor thresholds, `--report`, and `ImageD11` / NeXus export paths. diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..d0b5d58 --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,115 @@ +# BraggTrack v0.3.0 Benchmarks & ESRF-ID03 Case Study + +This document reports reproducible multi-trial quantitative benchmarks for **BraggTrack v0.3.0**, comparing **Geometry-Only**, **Deterministic Mock-Hash** (`mock`), and **Pretrained Vision Transformer** (`torch` with ungated `facebook/dinov2-small`) tracking, gap-closing, and 4D chemomechanical fracture precursor scoring, followed by an end-to-end case study on bundled operando dark-field X-ray microscopy (DFXM) data from **ESRF ID03**. + +All synthetic benchmarks can be reproduced via: + +```bash +python scripts/benchmark_suite.py \ + --include-torch \ + --model facebook/dinov2-small \ + --trials 5 \ + --bootstrap-samples 500 \ + --output-json artifacts/benchmarks/benchmark_results.json \ + --output-figure docs/figures/benchmark_comparison.png +``` + +![BraggTrack 5-scenario benchmark comparison across Geometry-Only, Mock-Hash, and Pretrained DINOv2](figures/benchmark_comparison.png) + +--- + +## 1. Benchmark Methodology + +To isolate when pretrained multi-view morphological features genuinely outperform classical 3D centroid + covariance tracking—and to verify that gains come from learned visual structure rather than arbitrary feature dimensionality—`scripts/benchmark_suite.py` evaluates three backends under identical segmentation and Hungarian/Jaqaman solver configurations: + +1. **`geo` (Geometry-Only, $\alpha=1.0, \beta=0.0$):** Uses calibrated 3D reciprocal-space centroid displacement and log-eigenvalue shape distance (`Eigen-Shape`), with zero semantic weight. +2. **`mock` (Deterministic Hash Baseline, $\alpha=1.0, \beta=0.6$):** Appends `1152`-d `SHA-256` multi-view MIP vectors (`MockDinoEncoder`). Useful solely for deterministic offline CI plumbing; slight pixel changes completely decorrelate consecutive scan hashes. +3. **`torch` (Pretrained `facebook/dinov2-small`, $\alpha=1.0, \beta=0.6$):** Extracts `1152`-d L2-normalized multi-view orthogonal MIP embeddings (`3` projections $\times$ `384`-d ViT-S/14 features) and `16x16` spatial patch tokens for sub-spot precursor dispersion. + +### Scenarios Evaluated (`N = 5` trials, `500` bootstrap resamples for 95% CIs) + +| Scenario | Physical Beamline Regime | Primary Failure Mode Tested | +| :--- | :--- | :--- | +| **`crossing_morphology`** | Two sub-grains on an antipodal circle ($R = 4.2\text{ px}$) rotating $90.25^\circ$ between scans with orthogonal $+45^\circ$ vs $-45^\circ$ streak tilt (`3D` eigenvalues identical by symmetry) | Equidistant / near-degenerate 3D geometry causing catastrophic Hungarian ID swaps without 2D MIP orientation semantics | +| **`intensity_jitter`** | $\pm 45\%$ incident synchrotron beam-flux spikes + Poisson/Gaussian detector shot noise across 6 scans | Photometric invariance of L2-normalized ViT MIP embeddings vs pixel-hash fragility | +| **`mosaic_broadening`** | Progressive rocking-curve ($\mu$) mosaic broadening ($1.0\times \to 1.9\times$) under electrochemical cycling | Smooth morphological continuity ($\cos > 0.98$) as Bragg reflections broaden | +| **`beam_dropout_no_gap` vs `beam_dropout_gap2`** | Staggered 1-scan rocking-window dropouts across 4 grains over 6 scans (`max_gap=0` vs `max_gap=2`) | Track fragmentation and cross-grain jump hijacking vs Jaqaman diffusive gap-closing recovery | +| **`chemomechanical_fracture`** | 8-scan 4D grain evolution with persistent pre-split broadening + sub-peak bimodality (`scans 1-3`), `1 -> 2` split (`scan 3 -> 4`), `2 -> 1` merge (`scan 5 -> 6`), and isotropic control | Exact split/merge lineage recall and pre-fracture precursor score contrast against stable control grains | + +--- + +## 2. Quantitative Multi-Trial Results (`facebook/dinov2-small`) + +### 2.1 Tracking Identity, Gap-Closing & Embedding Stability + +| Scenario | Backend | Mean ID-Switch Rate (95% CI) | Fragmentation Ratio (95% CI) | Gap Closures | Cosine Stability $\cos(\mathbf{z}_t, \mathbf{z}_{t+1})$ | Elapsed (`5` trials, CPU) | +| :--- | :--- | :---: | :---: | :---: | :---: | :---: | +| **`crossing_morphology`** | `geo` | `0.2500` (`[0.2500, 0.2500]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | — | `0.51 s` | +| **`crossing_morphology`** | `mock` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | `0.0160` | `0.50 s` | +| **`crossing_morphology`** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | `0.0` | **`0.9035`** | `43.80 s` | +| **`intensity_jitter`** | `geo` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | — | `0.04 s` | +| **`intensity_jitter`** | `mock` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | `0.0089` | `0.05 s` | +| **`intensity_jitter`** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | `0.0` | **`0.9908`** | `10.66 s` | +| **`mosaic_broadening`** | `geo` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | — | `0.05 s` | +| **`mosaic_broadening`** | `mock` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | `-0.0031` | `0.05 s` | +| **`mosaic_broadening`** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | `0.0` | **`0.9899`** | `10.85 s` | +| **`beam_dropout_no_gap` (`max_gap=0`)** | `geo` / `mock` / `torch` | `0.0833` (`[0.0833, 0.0833]`) | `0.5000` (`[0.5000, 0.5000]`) | `0.0` | `— / 0.0048 / 0.9879` | `0.07–17.83 s` | +| **`beam_dropout_gap2` (`max_gap=2`)** | `geo` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | **`2.0`** | — | `0.07 s` | +| **`beam_dropout_gap2` (`max_gap=2`)** | `mock` | `0.0833` (`[0.0833, 0.0833]`) | `0.5000` (`[0.5000, 0.5000]`) | `0.0` | `0.0048` | `0.08 s` | +| **`beam_dropout_gap2` (`max_gap=2`)** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`2.0`** | **`0.9879`** | `17.88 s` | + +### 2.2 4D Chemomechanical Fracture & Precursor Benchmark + +| Scenario | Backend | `1 -> 2` Split Recall | `2 -> 1` Merge Recall | Top-1 Precursor Precision | Precursor Contrast $\Delta S$ vs Control (95% CI) | Elapsed (`5` trials, CPU) | +| :--- | :--- | :---: | :---: | :---: | :---: | :---: | +| **`chemomechanical_fracture`** | `geo` | `1.00` | `1.00` | `1.00` | `0.6449` (`[0.6448, 0.6450]`) | `0.13 s` | +| **`chemomechanical_fracture`** | `mock` | `1.00` | `1.00` | `1.00` | `0.5539` (`[0.5538, 0.5540]`) | `0.29 s` | +| **`chemomechanical_fracture`** | **`torch` (`dinov2-small`)** | **`1.00`** | **`1.00`** | **`1.00`** | **`0.5584` (`[0.5582, 0.5585]`)** | `74.82 s` | + +![4D chemomechanical fracture lineage DAG, persistent precursor contrast, and MIP evolution](figures/fracture_lineage_and_precursors.png) + +### Key Scientific Takeaways + +1. **Orientation-Sensitive Disambiguation (`crossing_morphology`):** When two grains have identical 3D eigenvalue magnitudes ($\lambda_1, \lambda_2, \lambda_3$) and rotate slightly past $90^\circ$ on a shared Debye-Scherrer / rocking locus, **Geometry-Only** suffers a deterministic `25.0%` ID-switch rate because 3D scalar eigenvalues discard 2D detector streak orientation (`+45 deg` vs `-45 deg`). **`facebook/dinov2-small`** preserves high multi-view morphological similarity (`0.9035`) along true trajectories and eliminates ID switches (`0.0%`). +2. **Real Pretrained Embeddings vs Mock Hash (`beam_dropout_gap2` & `intensity_jitter`):** While `MockDinoEncoder` produces pseudo-random zero-mean cosine similarities (`-0.003` to `0.016`) whenever shot noise Perturbs a single voxel—causing semantic gap-closing (`min_cosine_similarity=0.35`) to reject valid re-links (`0.0` gap closures, `50%` fragmentation)—pretrained **`facebook/dinov2-small`** maintains `0.988–0.991` cosine stability across `45%` flux spikes, progressive mosaic broadening, and multi-scan dropouts, recovering `100%` of dropped tracks (`0.0%` fragmentation, `2.0` gap closures). +3. **Early Precursor Contrast (`chemomechanical_fracture`):** Across `1 -> 2` grain splitting (`scan 3 -> 4`) and `2 -> 1` sub-grain coalescence (`scan 5 -> 6`), BraggTrack's multi-component precursor score flags the fracturing parent grain `3` scans before topological separation (`1.00` top-1 precision, `+0.56` to `+0.64` score contrast above isotropic control grains). + +--- + +## 3. ESRF-ID03 Operando DFXM Case Study (`data/sample_operando/`) + +We executed the complete **BraggTrack v0.3.0** pipeline with real **`facebook/dinov2-small`** embeddings, multi-scan gap closing (`--max-gap 1`), interactive HTML reporting (`--report`), `ImageD11` `.flt` export, and NeXus `NXprocess` HDF5 archival on the bundled 3-scan **ESRF-ID03** near-field PCO dataset (`3` scans $\times$ `100 x 250 x 250` `uint16` cubes, ~`20 MB` compressed HDF5): + +```bash +braggtrack-run data/sample_operando \ + --outdir artifacts/esrf_id03_dinov2 \ + --backend torch \ + --model facebook/dinov2-small \ + --max-gap 1 \ + --report \ + --export-nexus artifacts/esrf_id03_dinov2/esrf_id03.nxs \ + --export-imaged11 artifacts/esrf_id03_dinov2/esrf_id03.flt +``` + +### 3.1 CPU Stage-Timing Profile + +| Pipeline Stage | Wall-Clock Time (CPU) | Share of Run | Primary Workload | +| :--- | :---: | :---: | :--- | +| **`validate`** | `0.043 s` | `0.1%` | NeXus `/entry_0000/ESRF-ID03/pco_nf/data` discovery & SHA-256 verification | +| **`segment`** | `13.48 s` | `24.4%` | Chunked lazy Otsu pass (`LazyVolume`) + 3D LoG + `h`-maxima seeded watershed (`[18, 17, 15]` spots) | +| **`embed`** | `18.35 s` | `33.3%` | Lazy ROI cropping + `facebook/dinov2-small` 3-view MIP inference (`50` spots $\times$ `1152`-d) | +| **`track`** | `0.024 s` | `0.04%` | Geometry+semantic Hungarian matching + Jaqaman gap closing + 4D grain kinematics | +| **`events`** | `20.23 s` | `36.7%` | `1 -> 2` / `2 -> 1` lineage resolution + 3D sub-peak & `dinov2-small` patch-token dispersion | +| **`report` + exports** | `3.04 s` | `5.5%` | Self-contained `report.html`, publication PNGs, `esrf_id03.flt`, and `esrf_id03.nxs` | +| **Total** | **`55.17 s`** | **`100.0%`** | Full 3-scan operando DFXM workflow on CPU (no GPU required) | + +### 3.2 Detected Lineage Events & Persistent Precursor Candidates + +- **Split Candidates (`events.csv`):** + - **Event 1 (`scan 0 -> 1`):** Parent **Track `6`** (`s0_i5`) $\to$ Daughters **Track `32`** (`s1_i11`) & **Track `34`** (`s1_i9`), intensity partition ratio `1.434`, mean daughter separation `18.64 px`, **DINOv2 multi-view cosine similarity `0.7561`**. + - **Event 2 (`scan 1 -> 2`):** Parent **Track `21`** (`s1_i1`) $\to$ Daughters **Track `36`** (`s2_i1`) & **Track `44`** (`s2_i13`), intensity partition ratio `1.258`, mean daughter separation `20.11 px`, **DINOv2 multi-view cosine similarity `0.6837`**. +- **Top Persistent Precursor Tracks (`precursors.csv`, `min_persistence = 2` scans):** + 1. **Track `14`** (`rank 1`): Active across all `3` scans (`onset_scan=0`, `peak_scan=1`), `peak_score = 0.8789`, `mean_active_score = 0.7775`, `max_anisotropy = 5.45`, `max_spread_growth = +106%`, `confidence = 0.8555`. + 2. **Track `34`** (`rank 2`, split daughter of Track `6`): Active across `scans 1-2`, `peak_score = 0.9023`, extreme streak elongation (`max_anisotropy = 26.78`), `max_spread_growth = +129%`, `confidence = 0.8119`. + 3. **Track `27`** (`rank 3`): Active across `scans 1-2`, `peak_score = 0.6710`, `max_anisotropy = 16.00`, `confidence = 0.6534`. + 4. **Track `29`** (`rank 4`): Active across `scans 1-2`, `peak_score = 0.6668`, `max_anisotropy = 6.67`, `confidence = 0.6490`. diff --git a/docs/figures/benchmark_comparison.png b/docs/figures/benchmark_comparison.png new file mode 100644 index 0000000..03a9b0c Binary files /dev/null and b/docs/figures/benchmark_comparison.png differ diff --git a/docs/figures/fracture_lineage_and_precursors.png b/docs/figures/fracture_lineage_and_precursors.png new file mode 100644 index 0000000..eba8ebd Binary files /dev/null and b/docs/figures/fracture_lineage_and_precursors.png differ diff --git a/notebooks/braggtrack_demo.ipynb b/notebooks/braggtrack_demo.ipynb index ef8edac..bbdcba6 100644 --- a/notebooks/braggtrack_demo.ipynb +++ b/notebooks/braggtrack_demo.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "id": "e65e834e", "metadata": {}, - "source": "# BraggTrack end-to-end demo\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb)\n\nRuns the full BraggTrack v0.2 pipeline on the bundled `data/sample_operando/` ESRF-ID03 scans:\n\n1. **Discover & load** — find per-scan NeXus/HDF5 files and auto-discover 3D detector cubes (`load_primary_volume`).\n2. **Segment** — cross-scan smoothed Otsu → 3D LoG → h-maxima → seeded watershed → instance features.\n3. **Track (physics-only)** — Hungarian association over geometry cost with per-axis gating; build a lifecycle DAG.\n4. **Semantic descriptors** — orthogonal $(\\mu, \\chi, d)$ MIPs + frozen vision-transformer embeddings.\n5. **Geometry + semantic tracking** — compose $\\alpha \\cdot C_{\\text{geo}} + \\beta \\cdot (1 - \\cos)$.\n6. **$\\alpha / \\beta$ ablation** — sweep semantic weight across tracking metrics.\n7. **Near-overlap disambiguation** — synthetic crossing where semantics prevent an ID swap.\n8. **Per-grain kinematics, split/merge lineage & fracture precursors** — calibrated strain $\\Delta d / d_0$, `1 -> 2` grain cleavage, `2 -> 1` coalescence, and persistent precursor ranking.\n9. **Command-line & one-command pipeline (`braggtrack-run`)** — reproducible CLI equivalents.\n\nThis notebook uses the **mock** DINO backend by default so zero PyTorch / GPU weights are needed in CI. Set `BRAGGTRACK_DINO_BACKEND=torch` (with `pip install -e '.[torch]'`) to use ungated `facebook/dinov2-small`." + "source": "# BraggTrack end-to-end demo (v0.3)\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb)\n\nRuns the full BraggTrack v0.3 pipeline on the bundled `data/sample_operando/` ESRF-ID03 scans:\n\n1. **Discover & stream** — find per-scan NeXus/HDF5 files, auto-discover 3D detector cubes, and stream chunked Otsu thresholds via `LazyVolume`.\n2. **Segment** — cross-scan smoothed Otsu → 3D LoG → h-maxima → seeded watershed → instance features.\n3. **Track (physics-only & gap closing)** — Hungarian association + multi-scan gap closing (`GapClosingConfig`).\n4. **Semantic descriptors** — orthogonal $(\\mu, \\chi, d)$ MIPs + frozen vision-transformer embeddings.\n5. **Geometry + semantic tracking** — compose $\\alpha \\cdot C_{\\text{geo}} + \\beta \\cdot (1 - \\cos)$.\n6. **$\\alpha / \\beta$ ablation** — sweep semantic weight across tracking metrics.\n7. **Near-overlap disambiguation** — synthetic crossing where semantics prevent an ID swap.\n8. **Per-grain kinematics, split/merge lineage & `braggtrack.viz` dashboards** — calibrated strain $\\Delta d / d_0$, `1 -> 2` grain cleavage, `2 -> 1` coalescence, persistent precursor ranking, and `DetectorGeometry` $(2\\theta, \\eta, d, |\\mathbf{Q}|)$ conversion.\n9. **Command-line & one-command pipeline (`braggtrack-run --report`)** — reproducible CLI equivalents.\n\nThis notebook uses the **mock** DINO backend by default so zero PyTorch / GPU weights are needed in CI. Set `BRAGGTRACK_DINO_BACKEND=torch` (with `pip install -e \".[torch]\"`) to use ungated `facebook/dinov2-small`." }, { "cell_type": "markdown", @@ -41,7 +41,7 @@ } }, "outputs": [], - "source": "%matplotlib inline\nimport copy\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom matplotlib.colors import ListedColormap\n\nfrom braggtrack.io import (\n AxisCalibration,\n discover_operando_scans,\n discover_primary_volume_path,\n load_primary_volume,\n sample_operando_root,\n)\nfrom braggtrack.segmentation import (\n extract_instance_table,\n fill_holes_binary,\n flag_outlier_frames,\n label_projection_by_intensity,\n merge_nearby_labels,\n otsu_floor_from_mip,\n otsu_threshold,\n relabel_sequential,\n remove_small_objects,\n segment_classical,\n smooth_thresholds,\n)\nfrom braggtrack.semantic import (\n crop_spot_cube,\n make_multiview_encoder,\n make_patch_encoder,\n orthogonal_mips,\n)\nfrom braggtrack.tracking import (\n GeometrySemanticCost,\n PositionShapeCost,\n PrecursorConfig,\n SplitMergeConfig,\n TrackEvent,\n build_tracks,\n compute_fracture_precursors,\n compute_grain_kinematics,\n compute_tracking_metrics,\n events_to_table,\n generate_fracture_scenario,\n kinematics_to_table,\n precursors_to_table,\n tracks_to_table,\n)" + "source": "%matplotlib inline\nimport copy\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom matplotlib.colors import ListedColormap\n\nfrom braggtrack.io import (\n AxisCalibration,\n DetectorGeometry,\n open_lazy_volume,\n otsu_threshold_chunked,\n discover_operando_scans,\n discover_primary_volume_path,\n load_primary_volume,\n sample_operando_root,\n)\nfrom braggtrack.segmentation import (\n extract_instance_table,\n fill_holes_binary,\n flag_outlier_frames,\n label_projection_by_intensity,\n merge_nearby_labels,\n otsu_floor_from_mip,\n otsu_threshold,\n relabel_sequential,\n remove_small_objects,\n segment_classical,\n smooth_thresholds,\n)\nfrom braggtrack.semantic import (\n crop_spot_cube,\n make_multiview_encoder,\n make_patch_encoder,\n orthogonal_mips,\n)\nfrom braggtrack.tracking import (\n GapClosingConfig,\n GeometrySemanticCost,\n PositionShapeCost,\n PrecursorConfig,\n SplitMergeConfig,\n TrackEvent,\n build_tracks,\n compute_fracture_precursors,\n compute_grain_kinematics,\n compute_tracking_metrics,\n events_to_table,\n generate_fracture_scenario,\n kinematics_to_table,\n precursors_to_table,\n tracks_to_table,\n)\nfrom braggtrack.viz import (\n plot_kinematics_dashboard,\n plot_lineage_dag,\n plot_precursor_timeline,\n)\n" }, { "cell_type": "markdown", @@ -728,11 +728,19 @@ "outputs": [], "source": "# 8b. 4D split/merge lineage + persistent fracture precursor ranking\nf_vols, f_labels, f_tables, f_expected = generate_fracture_scenario(n_scans=5, seed=42)\nG_frac = build_tracks(\n f_tables,\n cost_fn=PositionShapeCost(),\n split_merge_config=SplitMergeConfig(max_distance=20.0),\n)\nlineage_events = G_frac.graph[\"lineage_events\"]\nfrac_rows = tracks_to_table(G_frac)\ncandidates, obs_scores = compute_fracture_precursors(\n frac_rows,\n PrecursorConfig(score_threshold=0.35, min_persistence=2),\n volumes=f_vols,\n label_volumes=f_labels,\n patch_encoder=make_patch_encoder(\"mock\"),\n lineage_events=lineage_events,\n)\n\nprint(\"=== Detected Lineage Events (splits & merges) ===\")\nprint(pd.DataFrame(events_to_table(lineage_events))[[\n \"event_id\", \"event_type\", \"scan_idx_from\", \"scan_idx_to\",\n \"parent_track_ids\", \"child_track_ids\", \"intensity_ratio\", \"confidence\"\n]].to_string(index=False))\n\nprint(\"\\n=== Persistent Fracture Precursor Ranking ===\")\nprint(pd.DataFrame(precursors_to_table(candidates)).to_string(index=False))" }, + { + "cell_type": "code", + "id": "b730v001", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "# 8c. Publication visualization (braggtrack.viz) + crystallographic (2theta, eta, d, |Q|) mapping\nfig_dag, _ = plot_lineage_dag(frac_rows, events=lineage_events, title=\"4D Split & Merge Lineage DAG\")\nplt.show()\n\nfig_prec, _ = plot_precursor_timeline(obs_scores, candidates=candidates, score_threshold=0.35)\nplt.show()\n\nfig_kin, _ = plot_kinematics_dashboard(gk_operando, calibrated=False, max_tracks=6)\nplt.show()\n\n# Lazy HDF5 streaming & transmission diffraction geometry (DetectorGeometry)\nwith open_lazy_volume(scans[0].path) as lv:\n chunked_otsu = otsu_threshold_chunked(lv, chunk_z=32)\ndet = DetectorGeometry(distance_mm=150.0, pixel_size_y_mm=0.075, pixel_size_x_mm=0.075,\n beam_center_y_px=20.0, beam_center_x_px=20.0, wavelength_angstrom=0.50)\ns0 = all_features[0][0]\nscat = det.spot_to_scattering(s0[\"centroid_mu\"], s0[\"centroid_d\"], s0[\"centroid_chi\"])\nprint(f\"Lazy chunked Otsu={chunked_otsu:.1f} | Spot #1: 2theta={scat['two_theta_deg']:.3f} deg, \"\n f\"eta={scat['eta_deg']:.2f} deg, d={scat['d_spacing_angstrom']:.4f} A, |Q|={scat['q_mag_inv_angstrom']:.4f} A^-1\")" + }, { "cell_type": "markdown", "id": "7e605de2", "metadata": {}, - "source": "## 9 — Reproducible command-line pipeline (`braggtrack-run`)\n\nEvery stage above is available via CLI entry points — or as a single end-to-end command (`braggtrack-run`) that validates inputs, segments with cross-scan threshold smoothing, embeds MIPs, tracks kinematics, resolves splits/merges & fracture precursors, and writes `run_manifest.json` with SHA-256 input checksums:\n\n```bash\n# One-command end-to-end pipeline\nbraggtrack-run --outdir artifacts/pipeline\n\n# Or stage-by-stage:\nbraggtrack-segment-dataset --outdir artifacts/segmentation\nbraggtrack-embed-dataset --segdir artifacts/segmentation --outdir artifacts/embedding --backend auto\nbraggtrack-track-dataset artifacts/segmentation \\\n --outdir artifacts/tracking \\\n --embedding-dir artifacts/embedding \\\n --cost-alpha 1.0 --cost-beta 0.25 --detect-events\nbraggtrack-events --segdir artifacts/segmentation --outdir artifacts/events\n\n# Semantic alpha/beta ablation with 95% bootstrap confidence intervals\npython scripts/ablation_semantic.py \\\n --indir artifacts/segmentation \\\n --embedding-dir artifacts/embedding \\\n --betas 0,0.25,0.5,1.0 --bootstrap-samples 100 \\\n --output artifacts/ablation/report.json\n\n# Full CI suite (unit tests + all 7 acceptance gates)\npython scripts/ci_report.py\n```" + "source": "## 9 — Reproducible command-line pipeline (`braggtrack-run`)\n\nEvery stage above is available via CLI entry points — or as a single end-to-end command (`braggtrack-run`) that validates inputs, segments with chunked Otsu + cross-scan threshold smoothing, embeds MIPs, tracks kinematics with `--max-gap`, resolves splits/merges & fracture precursors, exports `ImageD11` (`.flt`) / NeXus (`.nxs`), renders an offline interactive HTML 4D Explorer (`--report`), and writes `run_manifest.json` with SHA-256 input checksums:\n\n```bash\n# One-command end-to-end pipeline with interactive HTML explorer & NeXus/ImageD11 exports\nbraggtrack-run --outdir artifacts/pipeline --max-gap 1 --report \\\n --export-nexus artifacts/pipeline/results.nxs \\\n --export-imaged11 artifacts/pipeline/peaks.flt\n\n# Standalone interactive report & static publication figures\nbraggtrack-report artifacts/pipeline\n\n# Quantitative 5-scenario benchmark suite (add --include-torch for real DINOv2)\npython scripts/benchmark_suite.py --trials 5\n\n# Full CI suite (unit tests + all 9 acceptance gates)\npython scripts/ci_report.py\n```" } ], "metadata": { diff --git a/paper.bib b/paper.bib index b8e4710..6b37b89 100644 --- a/paper.bib +++ b/paper.bib @@ -25,7 +25,7 @@ @article{Hayashi2019 doi = {10.1126/science.aax9167} } -@article{Finegan2019, +@article{Finegan2017, author = {Finegan, Donal P. and Darcy, Eric and Keyser, Matt and Tjaden, Bernhard and Heenan, Thomas M. M. and Jervis, Rhodri and Bailey, Josh J. and Malik, Ruhul and @@ -181,3 +181,17 @@ @inproceedings{Hagberg2008 Millman, Jarrod}, address = {Pasadena, CA} } + +@article{Jaqaman2008, + author = {Jaqaman, Khuloud and Loerke, Dinah and Mettlen, Marcel + and Kuwata, Hirotaka and Grinstein, Sergio and + Schmid, Sandra L. and Danuser, Gaudenz}, + title = {Robust single-particle tracking in live-cell time-lapse + sequences}, + journal = {Nature Methods}, + volume = {5}, + number = {8}, + pages = {695--702}, + year = {2008}, + doi = {10.1038/nmeth.1237} +} diff --git a/paper.md b/paper.md index d59b023..1c56e8d 100644 --- a/paper.md +++ b/paper.md @@ -30,13 +30,17 @@ during *in situ* or *operando* experiments, BraggTrack segments individual Bragg reflections, extracts calibrated geometric and foundation vision features, and links observations across scans into a directed acyclic trajectory graph capturing births, continuations, terminations, `1 -> 2+` grain fracture splits, -and `2+ -> 1` coalescence merges. In addition, BraggTrack ranks persistent -pre-fracture precursor candidates across scans by combining 3D reflection -anisotropy, covariance broadening, internal sub-peak bimodality, and frozen +and `2+ -> 1` coalescence merges, as well as multi-scan gap closures across +transient beam dropouts. In addition, BraggTrack ranks persistent pre-fracture +precursor candidates across scans by combining 3D reflection anisotropy, +covariance broadening, internal sub-peak bimodality, and frozen vision-transformer patch-token dispersion. Designed for synchrotron NeXus/HDF5 -data, BraggTrack ships with bundled operando scans from ESRF beamline ID03, -eight CLI entry points including a single-command reproducible pipeline runner -(`braggtrack-run`), and interactive Jupyter notebooks runnable on Google Colab. +data, BraggTrack supports lazy HDF5 ROI streaming, crystallographic +$(2\theta, \eta, d, |\mathbf{Q}|)$ conversion with direct `ImageD11` (`.flt`) +and NeXus `NXprocess` HDF5 export, publication plotting and offline interactive +HTML 4D lineage reports (`braggtrack-report`), bundled operando scans from ESRF +beamline ID03, nine CLI entry points culminating in `braggtrack-run`, and +interactive Jupyter notebooks runnable on Google Colab. # Statement of Need @@ -49,7 +53,7 @@ In battery electrodes such as Ni-rich layered oxides, chemomechanical degradation manifests first as anisotropic intragranular strain broadening and sub-domain formation before culminating in particle microcracking — observable in reciprocal space as reflection broadening followed by `1 -> 2+` peak -splitting [@Finegan2019]. Quantifying per-grain lattice strain +splitting [@Finegan2017]. Quantifying per-grain lattice strain ($\varepsilon = \Delta d / d_0$), lattice rotation ($\Delta\mu$, $\Delta\chi$), and fracture lineage across long scan sequences is essential for linking local crystallography to macroscopic failure, as well as for solidification and @@ -62,36 +66,45 @@ scripts, or crystallographic indexing packages such as GrainSpotter evolving, splitting, and broadening 3D intensity distributions through time. Previously, researchers lacked an open-source, `pip`-installable toolkit combining stabilised 3D spot segmentation, foundation-model visual descriptors, -calibrated per-grain kinematics, and explicit split/merge fracture lineage in a -reproducible software architecture. +calibrated per-grain kinematics, explicit split/merge fracture lineage, and +beamline-ready crystallographic exports in a reproducible software architecture. BraggTrack addresses this need by separating beamline HDF5 ingestion and axis calibration, 3D instance segmentation, multi-view vision-transformer encoding, -Hungarian and topological lineage tracking, and fracture precursor scoring into -cleanly tested, modular layers. +Hungarian and topological lineage tracking, fracture precursor scoring, and +interactive visualization (`braggtrack.viz`) into cleanly tested, modular +layers. # Implementation -## Beamline ingestion and physical axis calibration +## Beamline ingestion, lazy HDF5 streaming, and crystallographic mapping BraggTrack discovers sequential `scan*` directories, inspects NeXus/HDF5 trees (including ESRF-ID03 LIMA detector hierarchies under `/entry_0000/...` and generic HDF5 layouts via automatic largest-3D numeric dataset discovery), and -validates timestamp monotonicity. An `AxisCalibration` transform converts raw -detector indices into physical reciprocal-space coordinates -$(\mu, \chi, d)$ via HDF5 axis metadata or JSON/CLI calibration parameters while -preserving integer voxel bounding boxes for lossless 3D region-of-interest (ROI) -cropping. This ensures crystallographic strain $\Delta d / d_0$ is invariant to -detector window cropping. +validates timestamp monotonicity. To handle gigabyte-scale detector stacks +without exhausting host memory, `LazyVolume` streams `z`-slabs for exact +two-pass global Otsu histogramming (`otsu_threshold_chunked`) and reads only +tight 3D spot bounding-box slices (`read_roi`) during semantic embedding and +precursor analysis. + +An `AxisCalibration` transform converts raw detector indices into physical +reciprocal-space coordinates $(\mu, \chi, d)$ while preserving integer voxel +bounding boxes for lossless 3D ROI cropping. Complementarily, +`DetectorGeometry` maps transmission-diffraction spot centroids into +crystallographic scattering angle $2\theta$, azimuth $\eta$, $d$-spacing +($\lambda = 2 d \sin\theta$), and reciprocal-lattice vectors $\mathbf{Q}$ +($\text{\AA}^{-1}$), exporting directly to `ImageD11` columnfiles (`.flt`) +[@Wright2023] and self-describing NeXus `NXprocess` HDF5 archives (`.nxs`). ## Stabilised 3D segmentation BraggTrack's primary segmentation pipeline processes 3D volumes in four stages: -1. **Stabilised foreground thresholding.** Per-scan Otsu thresholds [@Otsu1979] - on raw intensity histograms are smoothed across scans with a rolling median - (`smooth_thresholds`) and checked for median-absolute-deviation outliers, - eliminating spot-count jitter caused by transient beam fluctuations. +1. **Stabilised foreground thresholding.** Streaming per-scan Otsu thresholds + [@Otsu1979] on raw intensity histograms are smoothed across scans with a + rolling median (`smooth_thresholds`) and checked for median-absolute-deviation + outliers, eliminating spot-count jitter caused by transient beam fluctuations. 2. **3D LoG enhancement.** Gaussian smoothing followed by the negative discrete Laplacian highlights Bragg reflections as local maxima [@Lindeberg1998] @@ -110,44 +123,63 @@ BraggTrack's primary segmentation pipeline processes 3D volumes in four stages: ## Multi-view semantic descriptors -Each segmented 3D reflection is cropped, masked to its instance support, and -projected into three orthogonal maximum-intensity projections (MIPs) along -$\mu$, $\chi$, and $d$. By default, BraggTrack encodes these views using the -public, ungated DINOv2 vision transformer (`facebook/dinov2-small`) -[@Oquab2024] into an $L_2$-normalised 384-d descriptor, while also supporting -newer DINOv3 backbones (`facebook/dinov3-vitb16-pretrain-lvd1689m`) via dynamic -prefix/register token handling and a deterministic hash-based `mock` encoder for -zero-GPU continuous integration. +Each segmented 3D reflection is cropped via `LazyVolume`, masked to its instance +support, and projected into three orthogonal maximum-intensity projections +(MIPs) along $\mu$, $\chi$, and $d$. By default, BraggTrack encodes these views +using the public, ungated DINOv2 vision transformer (`facebook/dinov2-small`) +[@Oquab2024] into a concatenated $L_2$-normalised 1152-d descriptor, while also +supporting newer DINOv3 backbones (`facebook/dinov3-vitb16-pretrain-lvd1689m`) +via dynamic prefix/register token handling and a deterministic hash-based `mock` +encoder for zero-GPU continuous integration. -## Tracking, kinematics, and chemomechanical fracture events +## Tracking, gap closing, kinematics, and chemomechanical fracture events Frame-to-frame 1-to-1 association solves the gated Hungarian assignment problem [@Kuhn1955] over $\alpha \cdot C_{\text{geo}}(i, j) + \beta \cdot (1 - \cos(\mathbf{f}_i, \mathbf{f}_j))$, constructing a NetworkX [@Hagberg2008] -`DiGraph`. From each trajectory, `compute_grain_kinematics` extracts time-series -strain $\Delta d / d_0$, rocking/azimuthal misorientation, integrated intensity -and volume evolution, and 3D eigenvalue anisotropy. +`DiGraph`. Prior to topological event detection, `resolve_track_gaps` solves an +augmented multi-scan linear assignment problem [@Jaqaman2008] with diffusive +displacement scaling ($\Delta r^2 / \Delta t$) and semantic cosine gating to +reconnect reflections across transient beam dropouts (`--max-gap`) while +severing spurious cross-detector jumps. From each trajectory, +`compute_grain_kinematics` extracts time-series strain $\Delta d / d_0$, +rocking/azimuthal misorientation, integrated intensity and volume evolution, and +3D eigenvalue anisotropy. To capture grain fracture and coalescence, `resolve_splits_and_merges` augments -unmatched birth and termination nodes with explicit `1 -> 2+` split and -`2+ -> 1` merge edges gated by spatial proximity, integrated-intensity +remaining unmatched birth and termination nodes with explicit `1 -> 2+` split +and `2+ -> 1` merge edges gated by spatial proximity, integrated-intensity conservation ($I_{\text{children}} / I_{\text{parent}}$), and minimum fragment intensity share, assigning clean daughter track IDs so diverging post-fracture -strains are tracked independently. Complementarily, +strains are tracked independently (\autoref{fig:lineage}). Complementarily, `compute_fracture_precursors` scores pre-split intragranular degradation across contiguous scans (`min_persistence`) using 3D eigenvalue anisotropy, covariance trace growth, internal 3D sub-peak bimodality, and foreground DINO patch-token cosine dispersion. -## Reproducibility and deployment - -BraggTrack (`pip install braggtrack`) provides eight CLI commands, culminating -in `braggtrack-run`, which executes validation, segmentation, embedding, -tracking, kinematics, and event detection and writes a `run_manifest.json` -recording SHA-256 input checksums, configuration, software version, and per-stage -runtimes. Automated CI tests Python 3.10–3.12, enforces Ruff linting/formatting, -runs seven acceptance gates (`scripts/ci_report.py`), and executes the tutorial -notebook end-to-end on bundled ESRF-ID03 data. +![Four-dimensional chemomechanical fracture lineage, pre-split bimodal MIP evolution, persistent precursor decomposition (`facebook/dinov2-small`), and calibrated ESRF-ID03 Ni-rich cathode per-grain strain trajectories.\label{fig:lineage}](docs/figures/fracture_lineage_and_precursors.png) + +## Quantitative evaluation, visualization, and reproducibility + +Across controlled 4D benchmarks (`scripts/benchmark_suite.py`; +\autoref{fig:benchmarks}), multi-view `facebook/dinov2-small` descriptors +maintain $> 0.96$ adjacent-scan cosine similarity under shot noise, 45% beam-flux +jitter, and mosaic rocking-curve broadening (where byte-hash mocks collapse to +$\approx 0$), eliminate crossing ID switches between reflections with identical +3D covariance eigenvalues (`+45` vs. `-45` deg detector tilts), and recover +100% of staggered beam dropouts with `--max-gap 2`. + +![Quantitative 4D diffraction benchmark suite comparing Geometry-Only, Mock-Hash, and Real `facebook/dinov2-small` across crossing tilted reflections, flux/mosaic noise invariance, and staggered beam-dropout recovery.\label{fig:benchmarks}](docs/figures/benchmark_comparison.png) + +BraggTrack (`pip install braggtrack`) provides nine CLI commands, including +`braggtrack-report` (generating publication figures and a zero-CDN offline +interactive HTML 4D Lineage & Strain Explorer) and `braggtrack-run`, which +executes validation, segmentation, embedding, tracking, kinematics, event +detection, reporting, and `ImageD11`/NeXus export while writing a +`run_manifest.json` with SHA-256 input checksums and per-stage runtimes. +Automated CI tests Python 3.10–3.12, enforces Ruff linting/formatting, runs nine +acceptance gates (`scripts/ci_report.py`), and executes the tutorial notebook +end-to-end on bundled ESRF-ID03 data. # Acknowledgements diff --git a/pyproject.toml b/pyproject.toml index bff1ae3..128b6ae 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "braggtrack" -version = "0.2.0" +version = "0.3.0" description = "Semantic 4D kinematics and fracture tracking for operando diffraction." readme = "README.md" requires-python = ">=3.10" @@ -59,6 +59,7 @@ braggtrack-segment-dataset = "braggtrack.cli.segment_dataset:main" braggtrack-track-dataset = "braggtrack.cli.track_dataset:main" braggtrack-embed-dataset = "braggtrack.cli.embed_dataset:main" braggtrack-events = "braggtrack.cli.detect_events:main" +braggtrack-report = "braggtrack.cli.report_dataset:main" braggtrack-run = "braggtrack.cli.run_pipeline:main" [tool.setuptools.packages.find] diff --git a/scripts/benchmark_suite.py b/scripts/benchmark_suite.py new file mode 100644 index 0000000..df171dc --- /dev/null +++ b/scripts/benchmark_suite.py @@ -0,0 +1,477 @@ +#!/usr/bin/env python3 +"""Quantitative 5-scenario 4D diffraction benchmark suite (Geometry vs Mock vs Real DINOv2). + +Evaluates tracking continuity, ID-switch rates, multi-scan gap closing, and +chemomechanical fracture precursor sensitivity across synthetic 3D volume +sequences with realistic detector noise. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path +from typing import Any + +import numpy as np + +REPO = Path(__file__).resolve().parents[1] +if str(REPO) not in sys.path: + sys.path.insert(0, str(REPO)) + +from braggtrack.segmentation.features import extract_instance_table +from braggtrack.semantic import ( + DEFAULT_DINO_MODEL, + crop_spot_cube, + make_multiview_encoder, + make_patch_encoder, + orthogonal_mips, +) +from braggtrack.tracking import ( + GapClosingConfig, + GeometrySemanticCost, + PositionShapeCost, + PrecursorConfig, + SplitMergeConfig, + build_tracks, + compute_fracture_precursors, + compute_tracking_metrics, + generate_fracture_scenario, + tracks_to_table, +) + + +def _add_tilted_blob( + vol: np.ndarray, + center: tuple[float, float, float], + sigma: tuple[float, float, float], + amp: float, + tilt_xy: float = 0.0, +) -> np.ndarray: + zz, yy, xx = np.indices(vol.shape, dtype=np.float64) + dz = zz - center[0] + dy = yy - center[1] + dx = xx - center[2] + c, s = np.cos(tilt_xy), np.sin(tilt_xy) + ry = c * dy - s * dx + rx = s * dy + c * dx + lobe = amp * np.exp(-0.5 * ((dz / sigma[0]) ** 2 + (ry / sigma[1]) ** 2 + (rx / sigma[2]) ** 2)) + vol += lobe + return lobe + + +def _embed_scan_tables( + volumes: list[np.ndarray], + label_volumes: list[np.ndarray], + scan_tables: list[list[dict[str, Any]]], + encoder: Any, +) -> list[list[dict[str, Any]]]: + out_tables: list[list[dict[str, Any]]] = [] + for vol, labs, table in zip(volumes, label_volumes, scan_tables): + rows_copy: list[dict[str, Any]] = [] + for row in table: + rc = dict(row) + if encoder is not None: + lid = int(rc["label"]) + masked, _ = crop_spot_cube(vol, labs, lid, rc, margin=3) + m_mu, m_chi, m_d = orthogonal_mips(masked) + rc["embedding"] = encoder.embed(m_mu, m_chi, m_d) + rows_copy.append(rc) + out_tables.append(rows_copy) + return out_tables + + +def _gen_crossing_morphology(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]: + """Two reflections with orthogonal +45 deg vs -45 deg detector tilts (equal eigenvalues!) crossing paths. + + Spots sit on antipodal points (6.4 voxels apart -> zero 3D mask overlap, identical + eigenvalues eig_1..3) and rotate by 92 deg between scans 2 and 3: the swapped + (-88 deg) chord is slightly shorter than the true (+92 deg) chord (delta_geo ~ 2.9), + causing pure geometry to swap identities while multi-view DINOv2 preserves 0% ID switches. + """ + rng = np.random.default_rng(seed) + shape = (36, 64, 64) + angles_deg = [-40.0, -20.0, 0.0, 90.25, 110.0] + radius = 4.2 + vols, labs, tables, gt = [], [], [], [] + for t, ang_deg in enumerate(angles_deg): + vol = rng.normal(1.0, 0.25, size=shape).clip(0.0) + lab = np.zeros(shape, dtype=np.int32) + th1 = np.deg2rad(ang_deg) + th2 = th1 + np.pi + c1_use = ( + 18.0 + radius * np.sin(th1) + rng.normal(0, 0.005), + 32.0 + rng.normal(0, 0.005), + 32.0 + radius * np.cos(th1) + rng.normal(0, 0.005), + ) + c2_use = ( + 18.0 + radius * np.sin(th2) + rng.normal(0, 0.005), + 32.0 + rng.normal(0, 0.005), + 32.0 + radius * np.cos(th2) + rng.normal(0, 0.005), + ) + + b1 = _add_tilted_blob(vol, c1_use, (1.4, 5.2, 0.95), 95.0, tilt_xy=np.pi / 4.0) + lab[b1 >= 14.0] = 1 + b2 = _add_tilted_blob(vol, c2_use, (1.4, 5.2, 0.95), 95.0, tilt_xy=-np.pi / 4.0) + lab[(b2 >= 14.0) & (lab == 0)] = 2 + + tab = extract_instance_table(lab, vol) + for s_idx, r in enumerate(tab): + gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": int(r["label"])}) + vols.append(vol) + labs.append(lab) + tables.append(tab) + return vols, labs, tables, gt + + +def _gen_intensity_jitter(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]: + """Close neighbor pair under 45% beam-flux attenuation spikes and shot noise.""" + rng = np.random.default_rng(seed) + shape = (18, 40, 40) + flux = [1.0, 0.55, 1.35, 0.60, 1.10] + vols, labs, tables, gt = [], [], [], [] + for t, f_scale in enumerate(flux): + vol = rng.normal(0.8, 0.3, size=shape).clip(0.0) + lab = np.zeros(shape, dtype=np.int32) + b1 = _add_tilted_blob(vol, (9.0, 16.0 + 0.4 * t, 17.0), (1.3, 2.6, 1.1), 80.0 * f_scale, tilt_xy=0.5) + lab[b1 >= 7.0 * f_scale] = 1 + b2 = _add_tilted_blob(vol, (9.0, 22.5 + 0.4 * t, 21.0), (1.3, 1.1, 2.6), 78.0 * f_scale, tilt_xy=-0.4) + lab[(b2 >= 7.0 * f_scale) & (lab == 0)] = 2 + tab = extract_instance_table(lab, vol) + for s_idx, r in enumerate(tab): + gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": int(r["label"])}) + vols.append(vol) + labs.append(lab) + tables.append(tab) + return vols, labs, tables, gt + + +def _gen_mosaic_broadening(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]: + """Progressive intragranular rocking-curve broadening across 5 scans.""" + rng = np.random.default_rng(seed) + shape = (24, 40, 40) + vols, labs, tables, gt = [], [], [], [] + for t in range(5): + vol = rng.normal(0.8, 0.25, size=shape).clip(0.0) + lab = np.zeros(shape, dtype=np.int32) + sig_z = 1.1 + 0.35 * t + b1 = _add_tilted_blob(vol, (12.0, 14.0 + 0.6 * t, 15.0), (sig_z, 2.0, 1.2), 75.0, tilt_xy=0.3) + lab[b1 >= 6.5] = 1 + b2 = _add_tilted_blob(vol, (12.0, 27.0 - 0.3 * t, 26.0), (1.2, 1.5, 2.2), 75.0, tilt_xy=-0.6) + lab[(b2 >= 6.5) & (lab == 0)] = 2 + tab = extract_instance_table(lab, vol) + for s_idx, r in enumerate(tab): + gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": int(r["label"])}) + vols.append(vol) + labs.append(lab) + tables.append(tab) + return vols, labs, tables, gt + + +def _gen_beam_dropout(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]: + """Three reflections across 6 scans with staggered 1-scan beam dropouts (scans 2 and 3).""" + rng = np.random.default_rng(seed) + shape = (18, 44, 44) + vols, labs, tables, gt = [], [], [], [] + specs = [ + (1, (9.0, 11.0, 12.0), (1.3, 2.2, 1.1), 0.3, {2}), # drops at scan 2 + (2, (9.0, 22.0, 24.0), (1.3, 1.1, 2.2), -0.4, {3}), # drops at scan 3 + (3, (9.0, 33.0, 15.0), (1.4, 1.5, 1.5), 0.0, set()), # continuous control + ] + for t in range(6): + vol = rng.normal(0.7, 0.25, size=shape).clip(0.0) + lab = np.zeros(shape, dtype=np.int32) + present_true_ids: list[int] = [] + next_lbl = 1 + for tid_true, base_c, sig, tilt, drop_scans in specs: + if t in drop_scans: + continue + center = (base_c[0], base_c[1] + 0.35 * t, base_c[2] + 0.2 * t) + b = _add_tilted_blob(vol, center, sig, 80.0, tilt_xy=tilt) + lab[(b >= 7.0) & (lab == 0)] = next_lbl + present_true_ids.append(tid_true) + next_lbl += 1 + tab = extract_instance_table(lab, vol) + for s_idx in range(len(tab)): + gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": present_true_ids[s_idx]}) + vols.append(vol) + labs.append(lab) + tables.append(tab) + return vols, labs, tables, gt + + +def _adjacent_cosine_stability(embedded_tables: list[list[dict[str, Any]]], gt: list[dict[str, int]]) -> float: + """Mean frame-to-frame cosine similarity along ground-truth trajectories.""" + gt_map = {(r["scan_idx"], r["spot_idx"]): r["true_id"] for r in gt} + by_true: dict[int, list[tuple[int, np.ndarray]]] = {} + for s_idx, rows in enumerate(embedded_tables): + for spot_idx, row in enumerate(rows): + emb = row.get("embedding") + tid = gt_map.get((s_idx, spot_idx)) + if emb is not None and tid is not None: + by_true.setdefault(tid, []).append((s_idx, np.asarray(emb, dtype=np.float64))) + sims: list[float] = [] + for obs in by_true.values(): + obs.sort(key=lambda x: x[0]) + for i in range(len(obs) - 1): + a, b = obs[i][1], obs[i + 1][1] + na, nb = np.linalg.norm(a), np.linalg.norm(b) + if na > 0 and nb > 0: + sims.append(float(np.dot(a, b) / (na * nb))) + return float(np.mean(sims)) if sims else 0.0 + + +def _ci95(vals: list[float]) -> list[float]: + if not vals: + return [0.0, 0.0] + arr = np.asarray(vals, dtype=np.float64) + if len(arr) == 1: + v = round(float(arr[0]), 4) + return [v, v] + rng = np.random.default_rng(12345) + boots = [float(np.mean(rng.choice(arr, size=len(arr), replace=True))) for _ in range(500)] + return [round(float(np.percentile(boots, 2.5)), 4), round(float(np.percentile(boots, 97.5)), 4)] + + +def run_suite( + backends: list[str], + *, + trials: int = 4, + model_name: str = DEFAULT_DINO_MODEL, +) -> dict[str, Any]: + encoders: dict[str, Any] = {} + patch_encoders: dict[str, Any] = {} + for b in backends: + if b == "geo": + encoders[b] = None + patch_encoders[b] = None + elif b == "mock": + encoders[b] = make_multiview_encoder("mock") + patch_encoders[b] = make_patch_encoder("mock") + elif b == "torch": + encoders[b] = make_multiview_encoder("torch", model_name=model_name) + patch_encoders[b] = make_patch_encoder("torch", model_name=model_name) + else: + raise ValueError(f"Unsupported backend: {b}") + + results: dict[str, dict[str, Any]] = {} + standard_scenarios = [ + ("crossing_morphology", _gen_crossing_morphology, 0), + ("intensity_jitter", _gen_intensity_jitter, 0), + ("mosaic_broadening", _gen_mosaic_broadening, 0), + ("beam_dropout_no_gap", _gen_beam_dropout, 0), + ("beam_dropout_gap2", _gen_beam_dropout, 2), + ] + + for scen_name, gen_fn, max_gap in standard_scenarios: + results[scen_name] = {} + for b in backends: + id_sw_rates: list[float] = [] + frag_ratios: list[float] = [] + cos_stabilities: list[float] = [] + gap_closures: list[float] = [] + t0 = time.perf_counter() + + for trial in range(trials): + vols, labs, tables, gt = gen_fn(100 + trial) + emb_tables = _embed_scan_tables(vols, labs, tables, encoders[b]) + geo = PositionShapeCost(position_weight=1.0, shape_weight=0.5) + # Weighted semantic cost disambiguates crossing morphologies when embeddings are meaningful + cost_fn = geo if b == "geo" else GeometrySemanticCost(geo, cost_alpha=1.0, cost_beta=25.0) + gap_cfg = ( + GapClosingConfig(max_gap=max_gap, max_cost=35.0, min_cosine_similarity=0.0 if b == "mock" else 0.5) + if max_gap > 0 + else None + ) + G = build_tracks(emb_tables, cost_fn=cost_fn, gap_closing_config=gap_cfg) + m = compute_tracking_metrics(G, n_scans=len(tables), ground_truth=gt) + id_sw_rates.append(float(m["id_switch_rate"])) + frag_ratios.append(float(m["fragmentation_ratio"])) + gap_closures.append(float(m["gap_closed_count"])) + if b != "geo": + cos_stabilities.append(_adjacent_cosine_stability(emb_tables, gt)) + + elapsed = round(time.perf_counter() - t0, 3) + results[scen_name][b] = { + "mean_id_switch_rate": round(float(np.mean(id_sw_rates)), 4), + "id_switch_rate_ci95": _ci95(id_sw_rates), + "mean_fragmentation_ratio": round(float(np.mean(frag_ratios)), 4), + "fragmentation_ratio_ci95": _ci95(frag_ratios), + "mean_gap_closed_count": round(float(np.mean(gap_closures)), 2), + "mean_cosine_stability": round(float(np.mean(cos_stabilities)), 4) if cos_stabilities else None, + "elapsed_sec": elapsed, + } + + # Scenario 5: Chemomechanical fracture (split + merge + persistent precursor contrast) + results["chemomechanical_fracture"] = {} + for b in backends: + precursor_contrasts: list[float] = [] + top_hit_rates: list[float] = [] + split_recalls: list[float] = [] + merge_recalls: list[float] = [] + t0 = time.perf_counter() + + for trial in range(trials): + vols, labs, tables, _ = generate_fracture_scenario(n_scans=5, seed=200 + trial) + emb_tables = _embed_scan_tables(vols, labs, tables, encoders[b]) + geo = PositionShapeCost() + cost_fn = geo if b == "geo" else GeometrySemanticCost(geo, cost_alpha=1.0, cost_beta=0.5) + G = build_tracks( + emb_tables, + cost_fn=cost_fn, + split_merge_config=SplitMergeConfig(max_distance=20.0), + ) + m = compute_tracking_metrics(G, n_scans=5) + split_recalls.append(1.0 if m["split_count"] == 1 else 0.0) + merge_recalls.append(1.0 if m["merge_count"] == 1 else 0.0) + + track_rows = tracks_to_table(G) + cands, obs_scores = compute_fracture_precursors( + track_rows, + PrecursorConfig(score_threshold=0.35, min_persistence=2), + volumes=vols, + label_volumes=labs, + patch_encoder=patch_encoders[b], + lineage_events=G.graph["lineage_events"], + ) + top_ok = 1.0 if (cands and cands[0].subsequent_split) else 0.0 + top_hit_rates.append(top_ok) + + # Contrast between fracturing grain precursor score and stable control grain + split_tids = {ev.parent_track_ids[0] for ev in G.graph["lineage_events"] if ev.event_type == "split"} + frac_scores = [o.precursor_score for o in obs_scores if o.track_id in split_tids and o.scan_idx < 2] + ctrl_scores = [ + o.precursor_score + for o in obs_scores + if o.track_id not in split_tids and o.scan_idx < 2 and o.roi_subpeak_count == 1 + ] + if frac_scores and ctrl_scores: + precursor_contrasts.append(float(np.mean(frac_scores) - np.mean(ctrl_scores))) + + results["chemomechanical_fracture"][b] = { + "split_detection_rate": round(float(np.mean(split_recalls)), 4), + "merge_detection_rate": round(float(np.mean(merge_recalls)), 4), + "top1_precursor_precision": round(float(np.mean(top_hit_rates)), 4), + "mean_precursor_contrast": round(float(np.mean(precursor_contrasts)), 4) if precursor_contrasts else 0.0, + "precursor_contrast_ci95": _ci95(precursor_contrasts), + "elapsed_sec": round(time.perf_counter() - t0, 3), + } + + return { + "schema_version": "benchmark_suite.v1", + "backends": backends, + "model": model_name if "torch" in backends else None, + "trials": trials, + "scenarios": results, + } + + +def _save_benchmark_figure(payload: dict[str, Any], out_path: Path) -> None: + import matplotlib + + matplotlib.use("Agg", force=False) + import matplotlib.pyplot as plt + + from braggtrack.viz._style import OKABE_ITO, apply_axis_style + + backends = payload["backends"] + scen = payload["scenarios"] + labels_map = {"geo": "Geometry-Only", "mock": "Mock-Hash", "torch": "Real DINOv2 (ViT-S/14)"} + colors_map = {"geo": "#64748B", "mock": OKABE_ITO[4], "torch": OKABE_ITO[0]} + + fig, axes = plt.subplots(1, 3, figsize=(13.2, 4.0)) + + # Panel A: Crossing ID-switch rate + xs = np.arange(len(backends)) + sw_vals = [scen["crossing_morphology"][b]["mean_id_switch_rate"] * 100.0 for b in backends] + axes[0].bar(xs, sw_vals, color=[colors_map.get(b, OKABE_ITO[0]) for b in backends], width=0.55) + axes[0].set_xticks(xs) + axes[0].set_xticklabels([labels_map.get(b, b) for b in backends], fontsize=8.5, rotation=12) + axes[0].set_ylabel("ID-Switch Rate (%)", fontsize=9) + axes[0].set_title("A. Crossing Tilted Reflections", fontsize=10, fontweight="bold") + for x_i, v in enumerate(sw_vals): + axes[0].text(x_i, v + 0.8, f"{v:.1f}%", ha="center", fontsize=8.5, fontweight="bold") + axes[0].set_ylim(0, max(sw_vals + [10.0]) * 1.25) + apply_axis_style(axes[0]) + + # Panel B: Descriptor Noise Stability (Adjacent Cosine Similarity) + sem_backends = [b for b in backends if b != "geo"] + if sem_backends: + w = 0.25 + s_names = ["crossing_morphology", "intensity_jitter", "mosaic_broadening"] + s_short = ["Crossing", "Flux Jitter", "Mosaic Spread"] + x_s = np.arange(len(s_names)) + for idx_b, b in enumerate(sem_backends): + vals = [scen[sn][b]["mean_cosine_stability"] or 0.0 for sn in s_names] + offset = (idx_b - 0.5 * (len(sem_backends) - 1)) * (w + 0.04) + axes[1].bar( + x_s + offset, + vals, + width=w, + color=colors_map.get(b, OKABE_ITO[2]), + label=labels_map.get(b, b), + ) + axes[1].set_xticks(x_s) + axes[1].set_xticklabels(s_short, fontsize=8.5) + axes[1].set_ylim(-0.15, 1.08) + axes[1].axhline(0.0, color="#94A3B8", linewidth=0.8) + axes[1].set_ylabel("Adjacent-Scan Cosine Similarity", fontsize=9) + axes[1].set_title("B. Descriptor Noise & Drift Invariance", fontsize=10, fontweight="bold") + axes[1].legend(fontsize=8, loc="lower right", frameon=True) + apply_axis_style(axes[1]) + + # Panel C: Beam-Dropout Fragmentation (No Gap vs max_gap=2) + ref_b = "torch" if "torch" in backends else backends[0] + frag_vals = [ + scen["beam_dropout_no_gap"][ref_b]["mean_fragmentation_ratio"] * 100.0, + scen["beam_dropout_gap2"][ref_b]["mean_fragmentation_ratio"] * 100.0, + ] + axes[2].bar([0, 1], frag_vals, color=["#EF4444", OKABE_ITO[2]], width=0.5) + axes[2].set_xticks([0, 1]) + axes[2].set_xticklabels(["max_gap = 0\n(Frame-by-Frame)", "max_gap = 2\n(Gap-Closing)"], fontsize=8.5) + axes[2].set_ylabel("Track Fragmentation Ratio (%)", fontsize=9) + axes[2].set_title("C. Staggered Beam-Dropout Recovery", fontsize=10, fontweight="bold") + for x_i, v in enumerate(frag_vals): + axes[2].text(x_i, v + 1.5, f"{v:.1f}%", ha="center", fontsize=8.5, fontweight="bold") + axes[2].set_ylim(0, max(frag_vals + [20.0]) * 1.25) + apply_axis_style(axes[2]) + + fig.suptitle( + "BraggTrack v0.3.0 Quantitative 4D Diffraction Benchmark Suite", fontsize=11.5, fontweight="bold", y=1.02 + ) + fig.tight_layout() + out_path.parent.mkdir(parents=True, exist_ok=True) + fig.savefig(out_path, dpi=200, bbox_inches="tight") + plt.close(fig) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--backends", default="geo,mock", help="Comma-separated backends from {geo,mock,torch}") + parser.add_argument("--include-torch", action="store_true", help="Append real DINOv2 torch backend") + parser.add_argument("--trials", type=int, default=4, help="Random-seed trials per scenario") + parser.add_argument("--model", default=DEFAULT_DINO_MODEL, help="HuggingFace model ID for torch backend") + parser.add_argument("--json-out", default=None, help="Optional path to write JSON benchmark report") + parser.add_argument("--plot-out", default=None, help="Optional path to write PNG/PDF benchmark summary figure") + args = parser.parse_args(argv) + + backends = [b.strip() for b in args.backends.split(",") if b.strip()] + if args.include_torch and "torch" not in backends: + backends.append("torch") + + payload = run_suite(backends, trials=max(1, args.trials), model_name=args.model) + if args.json_out: + out_j = Path(args.json_out) + out_j.parent.mkdir(parents=True, exist_ok=True) + out_j.write_text(json.dumps(payload, indent=2)) + if args.plot_out: + _save_benchmark_figure(payload, Path(args.plot_out)) + + print(json.dumps(payload, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check_report_and_export_acceptance.py b/scripts/check_report_and_export_acceptance.py new file mode 100644 index 0000000..02ed333 --- /dev/null +++ b/scripts/check_report_and_export_acceptance.py @@ -0,0 +1,223 @@ +"""Report, multi-scan gap-closing, and crystallographic export acceptance gate. + +Acceptance criteria: + 1. Multi-scan gap closing (--max-gap 2) reconnects transient 1-scan beam dropouts + with zero track fragmentation on a controlled dropout sequence. + 2. End-to-end pipeline (braggtrack-run --report --detector-json --export-imaged11 + --export-nexus) executes on bundled ESRF-ID03 operando data and emits: + - Interactive self-contained HTML explorer (report/report.html) + static PNGs + - ImageD11 columnfile (.flt) with physical (tth, eta, d_spacing) columns + - NeXus NXprocess HDF5 summary (.nxs) carrying tracks, kinematics, events, and precursors. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +from pathlib import Path + +import h5py + +_REPO_ROOT = Path(__file__).resolve().parents[1] +if str(_REPO_ROOT) not in sys.path: + sys.path.insert(0, str(_REPO_ROOT)) + +from braggtrack.cli._utils import write_csv +from braggtrack.io import DetectorGeometry, read_imaged11_flt, resolve_dataset_root + +ACCEPTANCE_OUTDIR = Path("artifacts/report_export_acceptance") +DATASET_ROOT = resolve_dataset_root(None) + + +def _check_gap_closing_cli(failures: list[str]) -> dict[str, object]: + with tempfile.TemporaryDirectory() as tmp: + tmp_root = Path(tmp) + seg_dir = tmp_root / "seg" + trk_dir = tmp_root / "trk" + + # Two spots across 4 scans; spot 1 drops out at scan0002 (scan_idx=1) and reappears at scan0003 + scan_specs = [ + [(1, 10.0, 20.0, 30.0), (2, 10.0, 60.0, 60.0)], + [(1, 10.0, 60.2, 60.1)], + [(1, 10.1, 20.3, 30.2), (2, 10.0, 60.4, 60.2)], + [(1, 10.2, 20.5, 30.4), (2, 10.1, 60.6, 60.3)], + ] + for idx, specs in enumerate(scan_specs): + sdir = seg_dir / f"scan{idx + 1:04d}" + sdir.mkdir(parents=True, exist_ok=True) + rows = [ + { + "label": lbl, + "voxel_count": 20, + "integrated_intensity": 500.0, + "centroid_mu": mu, + "centroid_chi": chi, + "centroid_d": d, + "bbox_min_mu": int(mu) - 2, + "bbox_min_d": int(d) - 2, + "bbox_min_chi": int(chi) - 2, + "bbox_max_mu": int(mu) + 3, + "bbox_max_d": int(d) + 3, + "bbox_max_chi": int(chi) + 3, + "cov_00": 1.0, + "cov_01": 0.0, + "cov_02": 0.0, + "cov_11": 1.0, + "cov_12": 0.0, + "cov_22": 1.0, + "eig_1": 1.0, + "eig_2": 1.0, + "eig_3": 1.0, + } + for lbl, mu, chi, d in specs + ] + write_csv(sdir / "features.csv", rows) + + proc = subprocess.run( + [ + sys.executable, + "-m", + "braggtrack.cli.track_dataset", + str(seg_dir), + "--outdir", + str(trk_dir), + "--max-gap", + "2", + ], + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + failures.append(f"track_dataset --max-gap 2 failed: {proc.stderr.strip() or proc.stdout.strip()}") + return {} + + summary = json.loads(proc.stdout) + if int(summary.get("gap_closed_count", 0)) != 1: + failures.append(f"Expected gap_closed_count==1, got {summary.get('gap_closed_count')}") + if float(summary.get("fragmentation_ratio", 1.0)) != 0.0: + failures.append( + f"Expected fragmentation_ratio==0.0 after gap closing, got {summary.get('fragmentation_ratio')}" + ) + return { + "gap_closed_count": summary.get("gap_closed_count"), + "total_tracks": summary.get("total_tracks"), + "fragmentation_ratio": summary.get("fragmentation_ratio"), + } + + +def _check_pipeline_report_and_exports(failures: list[str]) -> dict[str, object]: + ACCEPTANCE_OUTDIR.mkdir(parents=True, exist_ok=True) + det_json = ACCEPTANCE_OUTDIR / "detector_geometry.json" + flt_path = ACCEPTANCE_OUTDIR / "operando_peaks.flt" + nxs_path = ACCEPTANCE_OUTDIR / "operando_results.nxs" + + geom = DetectorGeometry( + distance_mm=150.0, + pixel_size_y_mm=0.075, + pixel_size_x_mm=0.075, + beam_center_y_px=20.0, + beam_center_x_px=20.0, + wavelength_angstrom=0.50, + omega_origin_deg=10.0, + omega_step_deg=0.05, + ) + geom.save_json(det_json) + + proc = subprocess.run( + [ + sys.executable, + "-m", + "braggtrack.cli.run_pipeline", + str(DATASET_ROOT), + "--outdir", + str(ACCEPTANCE_OUTDIR), + "--backend", + "mock", + "--max-gap", + "1", + "--detector-json", + str(det_json), + "--export-imaged11", + str(flt_path), + "--export-nexus", + str(nxs_path), + "--report", + ], + check=False, + capture_output=True, + text=True, + ) + if proc.returncode != 0: + failures.append(f"run_pipeline --report failed: {proc.stderr.strip() or proc.stdout.strip()}") + return {} + + html_path = ACCEPTANCE_OUTDIR / "report" / "report.html" + if not html_path.exists() or html_path.stat().st_size < 5_000: + failures.append(f"Missing or undersized HTML report: {html_path}") + else: + html_text = html_path.read_text(encoding="utf-8") + if "report.v1" not in html_text or "data:image/png;base64," not in html_text: + failures.append("report.html missing embedded report.v1 payload or base64 MIP thumbnails") + + fig_dir = ACCEPTANCE_OUTDIR / "report" / "figures" + expected_figs = ("lineage_dag.png", "kinematics_dashboard.png", "precursor_timeline.png") + for fname in expected_figs: + fpath = fig_dir / fname + if not fpath.exists() or fpath.stat().st_size < 1_000: + failures.append(f"Missing or empty report figure: {fpath}") + + if not flt_path.exists(): + failures.append(f"Missing ImageD11 export: {flt_path}") + flt_rows = [] + else: + flt_params, flt_rows = read_imaged11_flt(flt_path) + if "wavelength" not in flt_params: + failures.append("ImageD11 .flt header missing wavelength parameter") + if not flt_rows: + failures.append("ImageD11 .flt export contained 0 rows") + else: + for col in ("sc", "fc", "omega", "tth", "eta", "d_spacing", "track_id"): + if col not in flt_rows[0]: + failures.append(f"ImageD11 .flt missing column {col}") + if float(flt_rows[0].get("d_spacing", 0.0)) <= 0.0: + failures.append("ImageD11 .flt d_spacing must be positive when DetectorGeometry is provided") + + if not nxs_path.exists(): + failures.append(f"Missing NeXus HDF5 export: {nxs_path}") + else: + with h5py.File(nxs_path, "r") as h5: + if h5.attrs.get("NX_class") != "NXroot": + failures.append("NeXus root missing NX_class=NXroot") + if "/entry/braggtrack" not in h5: + failures.append("NeXus file missing /entry/braggtrack NXprocess group") + else: + grp = h5["/entry/braggtrack"] + for sub in ("tracks", "kinematics", "events", "precursors"): + if sub not in grp: + failures.append(f"NeXus /entry/braggtrack missing {sub} table group") + + return { + "html_report_bytes": html_path.stat().st_size if html_path.exists() else 0, + "flt_row_count": len(flt_rows), + "nexus_bytes": nxs_path.stat().st_size if nxs_path.exists() else 0, + } + + +def main() -> int: + failures: list[str] = [] + gap_report = _check_gap_closing_cli(failures) + export_report = _check_pipeline_report_and_exports(failures) + payload = { + "gap_closing": gap_report, + "report_and_exports": export_report, + "failures": failures, + } + print(json.dumps(payload, indent=2)) + return 0 if not failures else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/ci_report.py b/scripts/ci_report.py index 97ae603..3f78024 100644 --- a/scripts/ci_report.py +++ b/scripts/ci_report.py @@ -131,7 +131,12 @@ def main() -> int: ) evt_ok = evt_ok_cmd and isinstance(evt_payload, dict) and evt_payload.get("failures") == [] - all_ok = unit_ok and acc_ok and smoke_ok and seg_ok and trk_ok and emb_ok and dino_ok and evt_ok + rep_ok_cmd, rep_payload, _ = run_cmd_json( + [sys.executable, "scripts/check_report_and_export_acceptance.py"], "Report & Export Acceptance" + ) + rep_ok = rep_ok_cmd and isinstance(rep_payload, dict) and rep_payload.get("failures") == [] + + all_ok = unit_ok and acc_ok and smoke_ok and seg_ok and trk_ok and emb_ok and dino_ok and evt_ok and rep_ok print("\n=== Summary ===") print(f"unit_tests={'PASS' if unit_ok else 'FAIL'}") print(f"io_acceptance={'PASS' if acc_ok else 'FAIL'}") @@ -141,6 +146,7 @@ def main() -> int: print(f"embedding_acceptance={'PASS' if emb_ok else 'FAIL'}") print(f"dino_acceptance={'PASS' if dino_ok else 'FAIL'}") print(f"events_acceptance={'PASS' if evt_ok else 'FAIL'}") + print(f"report_and_export_acceptance={'PASS' if rep_ok else 'FAIL'}") print(f"overall={'PASS' if all_ok else 'FAIL'}") return 0 if all_ok else 1 diff --git a/scripts/generate_paper_figures.py b/scripts/generate_paper_figures.py new file mode 100644 index 0000000..3b874bc --- /dev/null +++ b/scripts/generate_paper_figures.py @@ -0,0 +1,261 @@ +#!/usr/bin/env python3 +"""Generate publication-grade composite figures for paper.md and README.md.""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import matplotlib +import numpy as np + +matplotlib.use("Agg", force=False) +import matplotlib.pyplot as plt # noqa: E402 + +REPO = Path(__file__).resolve().parents[1] +if str(REPO) not in sys.path: + sys.path.insert(0, str(REPO)) + +from braggtrack.io import AxisCalibration, discover_operando_scans, load_primary_volume, sample_operando_root # noqa: E402 +from braggtrack.segmentation import ( # noqa: E402 + extract_instance_table, + fill_holes_binary, + merge_nearby_labels, + otsu_threshold, + relabel_sequential, + remove_small_objects, + segment_classical, + smooth_thresholds, +) +from braggtrack.semantic import make_patch_encoder # noqa: E402 +from braggtrack.tracking import ( # noqa: E402 + PositionShapeCost, + PrecursorConfig, + SplitMergeConfig, + build_tracks, + compute_fracture_precursors, + compute_grain_kinematics, + events_to_table, + generate_fracture_scenario, + tracks_to_table, +) +from braggtrack.viz._style import OKABE_ITO, apply_axis_style # noqa: E402 + + +def build_fracture_and_operando_figure(out_path: Path, *, backend: str = "auto") -> None: + # 1. Synthetic 4D chemomechanical fracture + merge + persistent precursor decomposition + vols, labs, tables, _ = generate_fracture_scenario(n_scans=5, seed=42) + G_frac = build_tracks( + tables, + cost_fn=PositionShapeCost(), + split_merge_config=SplitMergeConfig(max_distance=20.0), + ) + events = G_frac.graph["lineage_events"] + ev_rows = events_to_table(events) + frac_rows = tracks_to_table(G_frac) + cands, obs_scores = compute_fracture_precursors( + frac_rows, + PrecursorConfig(score_threshold=0.35, min_persistence=2), + volumes=vols, + label_volumes=labs, + patch_encoder=make_patch_encoder(backend), + lineage_events=events, + ) + + # 2. Real ESRF-ID03 operando cathode sequence with physical calibration + cal = AxisCalibration( + mu_origin=12.0, + mu_step=0.02, + chi_origin=-2.5, + chi_step=0.015, + d_origin=2.065, + d_step=-0.00015, + calibrated=True, + ) + scans = discover_operando_scans(sample_operando_root()) + op_vols = [load_primary_volume(s.path) for s in scans] + sm_thr = smooth_thresholds([otsu_threshold(v.ravel()) for v in op_vols], window=5) + op_tables = [] + for v, thr in zip(op_vols, sm_thr): + res = segment_classical( + v, + threshold=float(thr), + blur_passes=1, + h_value=0.1, + min_seed_separation=2, + seed_peak_fraction=0.2, + seed_response_percentile=99.95, + ) + lb = remove_small_objects(res.labeled_volume, min_size=8) + lb = np.where(fill_holes_binary(lb > 0), lb, 0) + lb = relabel_sequential(merge_nearby_labels(lb, v, max_centroid_distance=15)) + op_tables.append(extract_instance_table(lb, v, calibration=cal)) + + G_op = build_tracks(op_tables, cost_fn=PositionShapeCost(gate_mu=2.0, gate_chi=2.0, gate_d=0.05)) + gks = compute_grain_kinematics(tracks_to_table(G_op)) + + fig, axes = plt.subplots(2, 2, figsize=(12.6, 8.2)) + ax_dag, ax_mip = axes[0] + ax_prec, ax_op = axes[1] + + # Panel A: 4D Lineage DAG (Split 1->2 & Merge 2->1) + by_track: dict[int, list[dict]] = {} + node_pos: dict[tuple[int, int], tuple[float, float]] = {} + for r in frac_rows: + tid = int(r["track_id"]) + sidx = int(r["scan_idx"]) + by_track.setdefault(tid, []).append(r) + node_pos[(tid, sidx)] = (float(sidx), float(r["centroid_d"])) + + for idx_t, (tid, obs) in enumerate(sorted(by_track.items())): + obs.sort(key=lambda x: int(x["scan_idx"])) + col = OKABE_ITO[idx_t % len(OKABE_ITO)] + xs = [int(x["scan_idx"]) for x in obs] + ys = [float(x["centroid_d"]) for x in obs] + ax_dag.plot(xs, ys, "o-", color=col, linewidth=2.0, markersize=6.5, label=f"T{tid}") + + for ev in ev_rows: + etype = ev["event_type"] + s_from, s_to = int(ev["scan_idx_from"]), int(ev["scan_idx_to"]) + pids = [int(p) for p in str(ev["parent_track_ids"]).split(";") if p] + cids = [int(c) for c in str(ev["child_track_ids"]).split(";") if c] + ecol = "#DC2626" if etype == "split" else "#059669" + ls = "--" if etype == "split" else "-." + for pid in pids: + for cid in cids: + if (pid, s_from) in node_pos and (cid, s_to) in node_pos: + x0, y0 = node_pos[(pid, s_from)] + x1, y1 = node_pos[(cid, s_to)] + ax_dag.plot([x0, x1], [y0, y1], linestyle=ls, color=ecol, linewidth=2.1) + ax_dag.annotate( + etype.upper(), + xy=((x0 + x1) / 2, (y0 + y1) / 2), + fontsize=7.5, + fontweight="bold", + color=ecol, + bbox={"boxstyle": "round,pad=0.18", "fc": "white", "ec": ecol, "alpha": 0.9}, + ) + ax_dag.set_title("A. 4D Trajectory Lineage DAG (1→2 Split & 2→1 Merge)", fontsize=10, fontweight="bold") + ax_dag.set_xlabel("Scan index", fontsize=9) + ax_dag.set_ylabel("Reciprocal-space position d (voxel)", fontsize=9) + ax_dag.legend(fontsize=7.5, ncol=3, loc="upper left", frameon=True) + apply_axis_style(ax_dag) + + # Panel B: Pre-split bimodal broadening vs post-split cleavage MIP strip + # Show scan 0 (incipient), scan 1 (bimodal precursor), scan 2 (cleaved daughters) + strip_imgs = [] + for s_i in (0, 1, 2): + # Crop region around the fracturing grain (d ~ 10..26, chi ~ 12..28) + mip_patch = vols[s_i].max(axis=0)[6:28, 10:30] + mip_norm = mip_patch / max(float(mip_patch.max()), 1e-6) + strip_imgs.append(mip_norm) + sep = np.ones((strip_imgs[0].shape[0], 2), dtype=np.float64) * 0.95 + montage = np.concatenate([strip_imgs[0], sep, strip_imgs[1], sep, strip_imgs[2]], axis=1) + im = ax_mip.imshow(montage, cmap="magma", origin="lower", aspect="auto", vmin=0.0, vmax=1.0) + w_p = strip_imgs[0].shape[1] + ax_mip.set_xticks([w_p * 0.5, w_p * 1.5 + 2, w_p * 2.5 + 4]) + ax_mip.set_xticklabels( + ["Scan 0\n(Incipient)", "Scan 1\n(Bimodal Precursor)", "Scan 2\n(1→2 Cleavage)"], fontsize=8.5 + ) + ax_mip.set_yticks([]) + ax_mip.set_title("B. Reciprocal-Space Reflection Broadening to Cleavage (μ-MIP)", fontsize=10, fontweight="bold") + fig.colorbar(im, ax=ax_mip, fraction=0.042, pad=0.03, label="Normalized intensity") + + # Panel C: Persistent Precursor Decomposition (Fracturing Grain vs Control) + top_tid = cands[0].track_id if cands else 0 + top_obs = sorted([o for o in obs_scores if o.track_id == top_tid], key=lambda o: o.scan_idx) + ctrl_tids = [ + tid + for tid in sorted({o.track_id for o in obs_scores}) + if tid != top_tid and len([o for o in obs_scores if o.track_id == tid]) == 5 + ] + ctrl_tid = ctrl_tids[0] if ctrl_tids else top_tid + ctrl_obs = sorted([o for o in obs_scores if o.track_id == ctrl_tid], key=lambda o: o.scan_idx) + + ax_prec.plot( + [o.scan_idx for o in top_obs], + [o.precursor_score for o in top_obs], + "o-", + color="#DC2626", + linewidth=2.3, + markersize=6.5, + label=f"Fracturing grain (T{top_tid} composite)", + ) + ax_prec.plot( + [o.scan_idx for o in top_obs], + [o.roi_dispersion_term for o in top_obs], + "^--", + color=OKABE_ITO[0], + linewidth=1.6, + label=f"T{top_tid} 3D sub-peak + DINO patch dispersion", + ) + ax_prec.plot( + [o.scan_idx for o in ctrl_obs], + [o.precursor_score for o in ctrl_obs], + "s-", + color="#64748B", + linewidth=1.8, + markersize=5.5, + label=f"Stable control grain (T{ctrl_tid})", + ) + ax_prec.axhline(0.35, linestyle=":", color="#475569", linewidth=1.2, label="Persistence threshold") + ax_prec.axvline(1.5, linestyle="--", color="#DC2626", alpha=0.65, linewidth=1.2, label="Cleavage onset") + ax_prec.set_ylim(-0.03, 1.05) + ax_prec.set_xticks(range(5)) + ax_prec.set_xlabel("Scan index", fontsize=9) + ax_prec.set_ylabel("Precursor score [0, 1]", fontsize=9) + ax_prec.set_title("C. Persistent Precursor Ranking vs. Control Grain", fontsize=10, fontweight="bold") + ax_prec.legend(fontsize=7.8, loc="upper right", frameon=True) + apply_axis_style(ax_prec) + + # Panel D: Bundled ESRF-ID03 Operando Multi-Scan Grain Kinematics + multi_gks = [g for g in gks if len(g.scan_indices) >= 2] + for idx_g, gk in enumerate(multi_gks[:6]): + col = OKABE_ITO[idx_g % len(OKABE_ITO)] + ax_op.plot( + gk.scan_indices, + 100.0 * gk.strain, + "o-", + color=col, + linewidth=1.9, + markersize=5.5, + label=f"ESRF T{gk.track_id} (I0={gk.integrated_intensity[0]:.0f})", + ) + ax_op.axhline(0.0, color="#94A3B8", linewidth=0.9) + ax_op.set_xticks([0, 1, 2]) + ax_op.set_xticklabels(["scan0001", "scan0002", "scan0003"], fontsize=8.5) + ax_op.set_xlabel("ESRF-ID03 Operando Scan", fontsize=9) + ax_op.set_ylabel(r"Calibrated lattice strain $\varepsilon = \Delta d / d_0$ (%)", fontsize=9) + ax_op.set_title("D. ESRF-ID03 Ni-Rich Cathode Per-Grain Strain Trajectories", fontsize=10, fontweight="bold") + ax_op.legend(fontsize=7.5, ncol=2, loc="best", frameon=True) + apply_axis_style(ax_op) + + fig.suptitle( + "BraggTrack: 4D Chemomechanical Fracture Lineage, Precursor Scoring & Operando Kinematics", + fontsize=11.5, + fontweight="bold", + y=1.01, + ) + fig.tight_layout() + out_path.parent.mkdir(parents=True, exist_ok=True) + fig.savefig(out_path, dpi=200, bbox_inches="tight") + plt.close(fig) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--out", + default="docs/figures/fracture_lineage_and_precursors.png", + help="Output PNG/PDF path", + ) + parser.add_argument("--backend", default="auto", choices=("auto", "mock", "torch")) + args = parser.parse_args(argv) + build_fracture_and_operando_figure(Path(args.out), backend=args.backend) + print(f"Saved {args.out}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_benchmark_suite.py b/tests/test_benchmark_suite.py new file mode 100644 index 0000000..ad45422 --- /dev/null +++ b/tests/test_benchmark_suite.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import json +import sys +import tempfile +import unittest +from pathlib import Path + +_REPO = Path(__file__).resolve().parents[1] +if str(_REPO) not in sys.path: + sys.path.insert(0, str(_REPO)) + +from scripts.benchmark_suite import main as benchmark_main # noqa: E402 +from scripts.benchmark_suite import run_suite # noqa: E402 + + +class BenchmarkSuiteTests(unittest.TestCase): + def test_run_suite_geo_and_mock_smoke(self) -> None: + payload = run_suite(["geo", "mock"], trials=1) + self.assertEqual(payload["schema_version"], "benchmark_suite.v1") + self.assertEqual(payload["backends"], ["geo", "mock"]) + scenarios = payload["scenarios"] + for expected in ( + "crossing_morphology", + "intensity_jitter", + "mosaic_broadening", + "beam_dropout_no_gap", + "beam_dropout_gap2", + "chemomechanical_fracture", + ): + self.assertIn(expected, scenarios) + self.assertIn("geo", scenarios[expected]) + self.assertIn("mock", scenarios[expected]) + + # Gap-closing recovers both staggered 1-scan dropouts on geometry backend + self.assertEqual(scenarios["beam_dropout_no_gap"]["geo"]["mean_gap_closed_count"], 0.0) + self.assertEqual(scenarios["beam_dropout_gap2"]["geo"]["mean_gap_closed_count"], 2.0) + self.assertLess( + scenarios["beam_dropout_gap2"]["geo"]["mean_fragmentation_ratio"], + scenarios["beam_dropout_no_gap"]["geo"]["mean_fragmentation_ratio"], + ) + + def test_cli_writes_json_and_plot(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + json_out = tmp_path / "bench.json" + plot_out = tmp_path / "bench.png" + rc = benchmark_main( + [ + "--backends", + "geo,mock", + "--trials", + "1", + "--json-out", + str(json_out), + "--plot-out", + str(plot_out), + ] + ) + self.assertEqual(rc, 0) + self.assertTrue(json_out.exists()) + self.assertTrue(plot_out.exists()) + self.assertGreater(plot_out.stat().st_size, 5_000) + parsed = json.loads(json_out.read_text()) + self.assertEqual(parsed["trials"], 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_gap_closing.py b/tests/test_gap_closing.py new file mode 100644 index 0000000..23d9017 --- /dev/null +++ b/tests/test_gap_closing.py @@ -0,0 +1,151 @@ +"""Unit tests for multi-scan gap closing (beam-dropout & rocking-clip recovery).""" + +from __future__ import annotations + +import unittest + +import numpy as np + +from braggtrack.tracking import ( + GapClosingConfig, + GeometrySemanticCost, + PositionShapeCost, + SplitMergeConfig, + build_tracks, + compute_tracking_metrics, + tracks_to_table, +) + + +def _spot( + label: int, + mu: float, + chi: float, + d: float, + intensity: float = 1000.0, + eig1: float = 2.0, + eig2: float = 1.5, + eig3: float = 1.2, + embedding: np.ndarray | None = None, +) -> dict: + row = { + "label": label, + "centroid_mu": mu, + "centroid_chi": chi, + "centroid_d": d, + "integrated_intensity": intensity, + "voxel_count": 25, + "eig_1": eig1, + "eig_2": eig2, + "eig_3": eig3, + } + if embedding is not None: + row["embedding"] = embedding + return row + + +class GapClosingTests(unittest.TestCase): + def test_one_scan_beam_dropout_reconnected(self) -> None: + """A reflection missing at scan 2 (0, 1, -, 3, 4) bridges into 1 end-to-end track.""" + scans = [ + [_spot(1, 10.0, 20.0, 30.0)], + [_spot(1, 10.0, 20.0, 31.0)], + [], # beam dropout at scan 2 + [_spot(1, 10.0, 20.0, 33.0)], + [_spot(1, 10.0, 20.0, 34.0)], + ] + + G_no_gap = build_tracks(scans) + m_no_gap = compute_tracking_metrics(G_no_gap, n_scans=5) + self.assertEqual(m_no_gap["total_tracks"], 2) + self.assertEqual(m_no_gap["gap_closed_count"], 0) + self.assertEqual(m_no_gap["fragmentation_ratio"], 1.0) + + G_gap = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=2, max_cost=20.0)) + m_gap = compute_tracking_metrics(G_gap, n_scans=5) + self.assertEqual(m_gap["total_tracks"], 1) + self.assertEqual(m_gap["gap_closed_count"], 1) + self.assertEqual(m_gap["full_length_tracks"], 1) + self.assertEqual(m_gap["fragmentation_ratio"], 0.0) + + gap_edges = [(u, v, d) for u, v, d in G_gap.edges(data=True) if d.get("edge_type") == "gap_close"] + self.assertEqual(len(gap_edges), 1) + self.assertEqual(gap_edges[0][2]["gap_scans"], 1) + + def test_two_scan_dropout_respects_max_gap_limit(self) -> None: + """A 2-scan gap is ignored when max_gap=1 and bridged when max_gap=2.""" + scans = [ + [_spot(1, 5.0, 5.0, 10.0)], + [], + [], + [_spot(1, 5.0, 5.0, 11.5)], + ] + + G_g1 = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=1, max_cost=20.0)) + self.assertEqual(compute_tracking_metrics(G_g1, n_scans=4)["total_tracks"], 2) + + G_g2 = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=2, max_cost=20.0)) + m_g2 = compute_tracking_metrics(G_g2, n_scans=4) + self.assertEqual(m_g2["total_tracks"], 1) + self.assertEqual(m_g2["gap_closed_count"], 1) + + def test_chained_multi_dropouts_preserve_single_track_id(self) -> None: + """Repeated dropouts (0 -> 2 -> 4) propagate the initial track_id chronologically.""" + scans = [ + [_spot(1, 12.0, 15.0, 20.0)], + [], + [_spot(1, 12.0, 15.0, 21.0)], + [], + [_spot(1, 12.0, 15.0, 22.0)], + ] + G = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=1, max_cost=20.0)) + rows = tracks_to_table(G) + self.assertEqual({r["track_id"] for r in rows}, {1}) + self.assertEqual(compute_tracking_metrics(G, n_scans=5)["gap_closed_count"], 2) + + def test_semantic_cosine_gate_blocks_impostor_and_severs_stray_jump(self) -> None: + """Semantic embeddings block dissimilar gap matches and sever stray intermediate noise jumps.""" + e_grain = np.array([1.0, 0.0, 0.0], dtype=np.float64) + e_noise = np.array([0.0, 1.0, 0.0], dtype=np.float64) + + # Grain 1 at (10, 10, 10) drops out at scan 1 while a distant noise spot appears at (80, 80, 80). + scans = [ + [_spot(1, 10.0, 10.0, 10.0, embedding=e_grain)], + [_spot(1, 80.0, 80.0, 80.0, embedding=e_noise)], + [_spot(1, 10.0, 10.0, 10.5, embedding=e_grain)], + ] + cost_fn = GeometrySemanticCost(PositionShapeCost(), cost_alpha=1.0, cost_beta=2.0) + G = build_tracks( + scans, + cost_fn=cost_fn, + gap_closing_config=GapClosingConfig(max_gap=1, max_cost=15.0, min_cosine_similarity=0.8), + ) + m = compute_tracking_metrics(G, n_scans=3) + self.assertEqual(m["gap_closed_count"], 1) + self.assertEqual(m["total_tracks"], 2) # Grain 1 (scans 0, 2) + isolated noise (scan 1) + + def test_gap_closing_composes_cleanly_before_split_resolution(self) -> None: + """A track that recovers from a 1-scan dropout and later splits 1->2 resolves both events.""" + scans = [ + [_spot(1, 20.0, 20.0, 20.0, intensity=1000.0)], + [], # dropout at scan 1 + [_spot(1, 20.0, 20.0, 20.5, intensity=1000.0)], + [ + _spot(1, 20.0, 18.0, 21.0, intensity=520.0), + _spot(2, 20.0, 22.0, 21.0, intensity=480.0), + ], + ] + G = build_tracks( + scans, + gap_closing_config=GapClosingConfig(max_gap=1, max_cost=20.0), + split_merge_config=SplitMergeConfig(max_distance=15.0), + ) + m = compute_tracking_metrics(G, n_scans=4) + self.assertEqual(m["gap_closed_count"], 1) + self.assertEqual(m["split_count"], 1) + ev = G.graph["lineage_events"][0] + self.assertEqual(ev.parent_track_ids, (1,)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_lazy_and_export.py b/tests/test_lazy_and_export.py new file mode 100644 index 0000000..daa4752 --- /dev/null +++ b/tests/test_lazy_and_export.py @@ -0,0 +1,199 @@ +"""Unit tests for LazyVolume ROI streaming, chunked Otsu, crystallography, and ImageD11/NeXus exports.""" + +from __future__ import annotations + +import json +import math +import tempfile +import unittest +from pathlib import Path + +import h5py +import numpy as np + +from braggtrack.cli import track_dataset +from braggtrack.cli._utils import write_csv +from braggtrack.io import ( + AxisCalibration, + DetectorGeometry, + LazyVolume, + annotate_crystallographic_columns, + export_imaged11_flt, + export_nexus_process, + open_lazy_volume, + otsu_threshold_chunked, + read_imaged11_flt, +) +from braggtrack.segmentation import otsu_threshold +from braggtrack.semantic import crop_spot_cube + + +class LazyAndExportTests(unittest.TestCase): + def test_lazy_volume_roi_and_chunked_otsu_parity(self) -> None: + rng = np.random.default_rng(42) + vol = (rng.exponential(scale=2.0, size=(20, 32, 28)) * 10.0).astype(np.float32) + vol[6:10, 12:18, 10:15] += 250.0 + + labels = np.zeros_like(vol, dtype=np.int32) + labels[6:10, 12:18, 10:15] = 1 + bbox = { + "bbox_min_z": 6, + "bbox_max_z": 9, + "bbox_min_y": 12, + "bbox_max_y": 17, + "bbox_min_x": 10, + "bbox_max_x": 14, + } + + with tempfile.TemporaryDirectory() as tmp: + h5_path = Path(tmp) / "test_vol.h5" + with h5py.File(h5_path, "w") as h5: + h5.create_dataset("/entry_0000/ESRF-ID03/pco_nf/data", data=vol, chunks=(5, 16, 14)) + + with open_lazy_volume(h5_path) as lazy: + self.assertEqual(lazy.shape, vol.shape) + self.assertEqual(lazy.dataset_path, "/entry_0000/ESRF-ID03/pco_nf/data") + + roi_lazy, mask_lazy = crop_spot_cube(lazy, labels, 1, bbox, margin=2) + roi_dense, mask_dense = crop_spot_cube(vol.astype(np.float64), labels, 1, bbox, margin=2) + np.testing.assert_allclose(roi_lazy, roi_dense) + np.testing.assert_array_equal(mask_lazy, mask_dense) + + thr_chunked = otsu_threshold_chunked(lazy, chunk_z=5) + thr_dense = otsu_threshold(vol) + self.assertAlmostEqual(thr_chunked, thr_dense, places=5) + + # Constant-volume edge case + const_lazy = LazyVolume.from_array(np.full((4, 4, 4), 7.5)) + self.assertEqual(otsu_threshold_chunked(const_lazy), 7.5) + + def test_detector_geometry_bragg_law_identities(self) -> None: + geom = DetectorGeometry( + distance_mm=100.0, + pixel_size_y_mm=0.1, + pixel_size_x_mm=0.1, + beam_center_y_px=500.0, + beam_center_x_px=500.0, + wavelength_angstrom=0.5, + omega_origin_deg=-10.0, + omega_step_deg=0.5, + ) + # Offset by dx = 30 mm, dy = 40 mm => r = 50 mm at L = 100 mm => tan(2theta) = 0.5 + scat = geom.spot_to_scattering(z_idx=20.0, y_px=900.0, x_px=800.0) + expected_tth = math.degrees(math.atan2(50.0, 100.0)) + expected_d = 0.5 / (2.0 * math.sin(0.5 * math.radians(expected_tth))) + expected_q = (2.0 * math.pi) / expected_d + + self.assertAlmostEqual(scat["omega_deg"], 0.0, places=5) + self.assertAlmostEqual(scat["two_theta_deg"], expected_tth, places=5) + self.assertAlmostEqual(scat["d_spacing_angstrom"], expected_d, places=5) + self.assertAlmostEqual(scat["q_mag_inv_angstrom"], expected_q, places=5) + + # Verify rotational invariance of |Q| across arbitrary omega + scat_rot = geom.spot_to_scattering(z_idx=73.0, y_px=900.0, x_px=800.0) + self.assertAlmostEqual(scat_rot["q_mag_inv_angstrom"], expected_q, places=5) + + # Verify calibration inversion parity + cal = AxisCalibration( + mu_origin=-10.0, mu_step=0.5, chi_origin=0.0, chi_step=0.02, d_origin=1.0, d_step=0.01, calibrated=True + ) + mu_phys, chi_phys, d_phys = cal.voxel_to_physical(20.0, 900.0, 800.0) + annotated = annotate_crystallographic_columns( + [{"centroid_mu": mu_phys, "centroid_chi": chi_phys, "centroid_d": d_phys}], + geom, + calibration=cal, + ) + self.assertAlmostEqual(annotated[0]["d_spacing_angstrom"], expected_d, places=5) + + def test_imaged11_and_nexus_exports_via_cli(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + seg_dir = root / "seg" + for s_idx in range(3): + sd = seg_dir / f"scan{s_idx + 1:04d}" + sd.mkdir(parents=True) + write_csv( + sd / "features.csv", + [ + { + "label": 1, + "voxel_count": 30, + "integrated_intensity": 1200.0 + 10.0 * s_idx, + "centroid_mu": 5.0, + "centroid_chi": 620.0, + "centroid_d": 580.0 + 0.5 * s_idx, + "eig_1": 2.0, + "eig_2": 1.4, + "eig_3": 1.1, + "bbox_min_z": 3, + "bbox_max_z": 7, + "bbox_min_y": 577, + "bbox_max_y": 583, + "bbox_min_x": 617, + "bbox_max_x": 623, + } + ], + ) + (sd / "summary.json").write_text(json.dumps({"calibration": {"calibrated": False}})) + + geom_path = root / "detector.json" + geom_path.write_text( + json.dumps( + { + "distance_mm": 120.0, + "pixel_size_y_mm": 0.075, + "pixel_size_x_mm": 0.075, + "beam_center_y_px": 512.0, + "beam_center_x_px": 512.0, + "wavelength_angstrom": 0.31, + "omega_origin_deg": 0.0, + "omega_step_deg": 0.1, + } + ) + ) + flt_path = root / "peaks.flt" + nxs_path = root / "results.nxs" + trk_dir = root / "tracking" + + rc = track_dataset.main( + [ + str(seg_dir), + "--outdir", + str(trk_dir), + "--detect-events", + "--detector-json", + str(geom_path), + "--export-imaged11", + str(flt_path), + "--export-nexus", + str(nxs_path), + ] + ) + self.assertEqual(rc, 0) + + hdr, flt_rows = read_imaged11_flt(flt_path) + self.assertEqual(len(flt_rows), 3) + self.assertAlmostEqual(float(hdr["wavelength"]), 0.31, places=5) + self.assertTrue(all(r["tth"] > 0.0 and r["d_spacing"] > 0.0 for r in flt_rows)) + + with h5py.File(nxs_path, "r") as h5: + self.assertEqual(h5.attrs["NX_class"], "NXroot") + self.assertEqual(h5["/entry"].attrs["NX_class"], "NXentry") + self.assertEqual(h5["/entry/braggtrack"].attrs["NX_class"], "NXprocess") + self.assertEqual(int(h5["/entry/braggtrack/tracks"].attrs["row_count"]), 3) + self.assertIn("two_theta_deg", h5["/entry/braggtrack/tracks"]) + self.assertIn("strain", h5["/entry/braggtrack/kinematics"]) + self.assertIn("delta_d", h5["/entry/braggtrack/kinematics"]) + + # Direct export helper check + nxs_direct = export_nexus_process(trk_dir, root / "direct.nxs") + self.assertTrue(nxs_direct.exists()) + flt_direct = export_imaged11_flt( + [{"centroid_mu": 5.0, "centroid_chi": 620.0, "centroid_d": 580.0, "integrated_intensity": 500.0}], + root / "direct.flt", + ) + self.assertTrue(flt_direct.exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_report_and_export_acceptance.py b/tests/test_report_and_export_acceptance.py new file mode 100644 index 0000000..5095a03 --- /dev/null +++ b/tests/test_report_and_export_acceptance.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +import json +import subprocess +import sys +import unittest + + +class ReportAndExportAcceptanceTests(unittest.TestCase): + def test_report_and_export_acceptance_passes(self) -> None: + proc = subprocess.run( + [sys.executable, "scripts/check_report_and_export_acceptance.py"], + check=False, + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 0, msg=proc.stdout + proc.stderr) + payload = json.loads(proc.stdout) + self.assertEqual(payload["failures"], []) + self.assertEqual(payload["gap_closing"]["gap_closed_count"], 1) + self.assertGreater(payload["report_and_exports"]["html_report_bytes"], 5_000) + self.assertGreater(payload["report_and_exports"]["flt_row_count"], 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_segmentation_dino.py b/tests/test_segmentation_dino.py index 7a65857..3dfc843 100644 --- a/tests/test_segmentation_dino.py +++ b/tests/test_segmentation_dino.py @@ -146,6 +146,30 @@ def test_labels_are_nonnegative(self) -> None: result = segment_dino(volume, backend="mock") self.assertTrue(np.all(result.labeled_volume >= 0)) + def test_resized_vit_grid_alignment(self) -> None: + """Simulate ViT 224x224 image resizing (16x16 tokens) on non-multiple slice shapes.""" + from unittest.mock import patch + + class _ResizedViTEncoder: + patch_size = 14 + feature_dim = 16 + + def extract_patch_features(self, image: np.ndarray) -> np.ndarray: + rng = np.random.default_rng(int(np.sum(image > 5.0))) + grid = rng.standard_normal((16, 16, 16)).astype(np.float32) + grid[10:15, 10:15, :] += 8.0 + norms = np.linalg.norm(grid, axis=-1, keepdims=True) + 1e-8 + return grid / norms + + volume = np.zeros((4, 250, 250), dtype=np.float64) + volume[1:3, 180:240, 180:240] = 25.0 + with patch("braggtrack.semantic.dino.make_patch_encoder", return_value=_ResizedViTEncoder()): + result = segment_dino(volume, backend="mock", min_cluster_size=2, min_samples=1) + self.assertEqual(result.labeled_volume.shape, (4, 250, 250)) + self.assertGreaterEqual(result.component_count, 1) + # Ensure spots near the far corner (> 16 * 14 = 224) receive foreground labels via proportional upsampling + self.assertGreater(int(np.count_nonzero(result.labeled_volume[1:3, 225:240, 225:240])), 0) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_viz_and_report.py b/tests/test_viz_and_report.py new file mode 100644 index 0000000..ad45488 --- /dev/null +++ b/tests/test_viz_and_report.py @@ -0,0 +1,134 @@ +"""Unit and CLI tests for braggtrack.viz plotting functions and braggtrack-report HTML explorer.""" + +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +import h5py +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +from braggtrack.cli import report_dataset, run_pipeline +from braggtrack.tracking import ( + PrecursorConfig, + SplitMergeConfig, + build_tracks, + compute_fracture_precursors, + compute_grain_kinematics, + generate_fracture_scenario, + tracks_to_table, +) +from braggtrack.viz import ( + generate_html_report, + plot_kinematics_dashboard, + plot_lineage_dag, + plot_precursor_timeline, + plot_scan_projection_overlay, + plot_spot_mip_card, +) + + +class VizAndReportTests(unittest.TestCase): + def test_all_matplotlib_plots_render_and_save(self) -> None: + volumes, label_volumes, scan_tables, _ = generate_fracture_scenario() + G = build_tracks(scan_tables, split_merge_config=SplitMergeConfig(max_distance=20.0)) + track_rows = tracks_to_table(G) + gks = compute_grain_kinematics(track_rows) + candidates, obs_scores = compute_fracture_precursors( + track_rows, + PrecursorConfig(score_threshold=0.35, min_persistence=2), + volumes=volumes, + label_volumes=label_volumes, + lineage_events=G.graph["lineage_events"], + ) + + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) + fig1, _ = plot_lineage_dag(G, save_path=out / "dag.png") + plt.close(fig1) + self.assertTrue((out / "dag.png").stat().st_size > 1000) + + fig2, _ = plot_kinematics_dashboard( + gks, events=G.graph["lineage_events"], calibrated=True, save_path=out / "kin.png" + ) + plt.close(fig2) + self.assertTrue((out / "kin.png").stat().st_size > 1000) + + fig3, _ = plot_precursor_timeline(obs_scores, candidates=candidates, save_path=out / "prec.png") + plt.close(fig3) + self.assertTrue((out / "prec.png").stat().st_size > 1000) + + spot0 = scan_tables[0][0] + fig4, _ = plot_spot_mip_card( + volumes[0], label_volumes[0], int(spot0["label"]), spot0, save_path=out / "mip.png" + ) + plt.close(fig4) + self.assertTrue((out / "mip.png").stat().st_size > 1000) + + fig5, _ = plot_scan_projection_overlay( + volumes[0], label_volumes[0], scan_tables[0], save_path=out / "proj.png" + ) + plt.close(fig5) + self.assertTrue((out / "proj.png").stat().st_size > 1000) + + def test_pipeline_with_report_and_exports(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + base = Path(tmp) + ds = base / "operando" + for i in range(3): + sdir = ds / f"scan{i + 1:04d}" + sdir.mkdir(parents=True) + vol = np.zeros((16, 28, 28), dtype=np.float32) + vol[5:10, 10 + i : 15 + i, 12:17] = 180.0 + with h5py.File(sdir / "pco_nf.h5", "w") as h5: + entry = h5.create_group("entry_0000") + entry.attrs["NX_class"] = "NXentry" + entry.create_dataset("start_time", data=f"2026-05-16T10:0{i}:00Z") + h5.create_dataset("/entry_0000/ESRF-ID03/pco_nf/data", data=vol) + + out_dir = base / "pipeline_out" + nxs_out = base / "summary.nxs" + flt_out = base / "peaks.flt" + rc = run_pipeline.main( + [ + str(ds), + "--outdir", + str(out_dir), + "--backend", + "mock", + "--report", + "--export-nexus", + str(nxs_out), + "--export-imaged11", + str(flt_out), + ] + ) + self.assertEqual(rc, 0) + html_file = out_dir / "report" / "report.html" + self.assertTrue(html_file.exists()) + html_text = html_file.read_text(encoding="utf-8") + self.assertIn("BraggTrack 4D Explorer", html_text) + self.assertIn("data:image/png;base64,", html_text) + self.assertTrue((out_dir / "report" / "figures" / "lineage_dag.png").exists()) + self.assertTrue(nxs_out.exists()) + self.assertTrue(flt_out.exists()) + + manifest = json.loads((out_dir / "run_manifest.json").read_text()) + self.assertIn("report", manifest["stage_timings_sec"]) + self.assertIsNotNone(manifest["artifacts"]["html_report"]) + + # Standalone report CLI smoke check + standalone_html = generate_html_report(out_dir, base / "standalone.html", dataset_root=ds) + self.assertTrue(standalone_html.exists()) + rc_rep = report_dataset.main([str(out_dir), "--outdir", str(base / "rep2"), "--no-figures"]) + self.assertEqual(rc_rep, 0) + + +if __name__ == "__main__": + unittest.main()