diff --git a/CHANGELOG.md b/CHANGELOG.md
index f6424c7..6fffb35 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,20 @@ All notable changes to BraggTrack will be documented in this file.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [0.3.0] - 2026-11-17
+
+### Added
+- **Multi-scan beam-dropout gap closing (`braggtrack.tracking.gap_closing`)**: Added Jaqaman-style augmented linear assignment (`GapClosingConfig`, `GapCloseRecord`, `resolve_track_gaps`) with diffusive displacement scaling (`pos_d2 / delta_scans`), bidirectional spurious jump-edge severing, semantic cosine gating, and chronological track ID propagation (`--max-gap` across `track_dataset`, `detect_events`, and `run_pipeline`).
+- **Chunked & lazy HDF5 streaming (`braggtrack.io.lazy`)**: Added context-managed `LazyVolume` (`open_lazy_volume`, `read_roi`, `iter_z_chunks`) and exact two-pass `otsu_threshold_chunked`, wired into `segment_dataset` Pass 1, `embed_dataset`, `detect_events`, and `crop_spot_cube` to avoid full-volume `float64` allocations on gigabyte-scale detector stacks.
+- **Crystallographic scattering geometry & downstream exports (`braggtrack.io.crystallography`, `braggtrack.io.export`)**: Added `DetectorGeometry` (`spot_to_scattering`, `annotate_crystallographic_columns`) converting spot centroids into `(omega_deg, two_theta_deg, eta_deg, d_spacing_angstrom, qx/qy/qz/q_mag_inv_angstrom)` via Bragg's law, plus direct `ImageD11` columnfile export/parsing (`export_imaged11_flt`, `read_imaged11_flt`) and self-describing NeXus `NXprocess` HDF5 archival export (`export_nexus_process`).
+- **Publication visualization package & interactive HTML 4D Explorer (`braggtrack.viz`, `braggtrack-report`)**: Added `plot_spot_mip_card`, `plot_scan_projection_overlay`, `plot_lineage_dag`, `plot_kinematics_dashboard`, `plot_precursor_timeline`, and `generate_html_report` (zero-CDN offline `report.html` with stdlib-encoded base64 3-view MIP thumbnails, pan/zoom SVG Lineage DAG, filter pills, and synchronized strain/precursor inspector), plus the 9th CLI entry point `braggtrack-report` and `--report` flag on `braggtrack-run`.
+- **Quantitative 5-scenario real DINOv2 benchmark suite (`scripts/benchmark_suite.py`, `docs/benchmarks.md`)**: Added multi-trial bootstrap-CI benchmark harness comparing **Geometry-Only**, **Mock-Hash**, and **Real `facebook/dinov2-small`** across crossing tilted reflections, beam-flux jitter, mosaic rocking broadening, staggered beam dropouts (`max_gap=2`), and 4D chemomechanical fracture/precursor contrast, alongside publication figures (`docs/figures/`) and an ESRF-ID03 `facebook/dinov2-small` case study.
+- **9th CI acceptance gate (`scripts/check_report_and_export_acceptance.py`)**: Wired automated verification of `--max-gap`, `report.html` + static figures, `ImageD11` `.flt` round-trip parsing, and NeXus `NXprocess` HDF5 schema into `scripts/ci_report.py`.
+
+### Fixed
+- Fixed patch-grid shape mismatch and far-edge clipping in `segment_dino` when HuggingFace ViT processors resize non-multiple slices (e.g. `250x250 -> 224x224`, `16x16` tokens vs `250 // 14 = 17`), and honored `BRAGGTRACK_DINO_BACKEND` when `backend="auto"` in `braggtrack.semantic.dino`.
+- Fixed `@Finegan2017` citation key/year mismatch (`2017`, DOI `10.1039/C7EE00385D`) and added `@Jaqaman2008` in `paper.bib` and `paper.md`; synced `CITATION.cff`, `CONTRIBUTING.md`, `docs/architecture.md`, and `README.md` to `v0.3.0`.
+
## [0.2.0] - 2026-11-17
### Added
diff --git a/CITATION.cff b/CITATION.cff
index 8b445ef..8d1e56b 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -2,22 +2,25 @@ cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: "BraggTrack"
type: software
-version: 0.1.0
+version: 0.3.0
date-released: "2026-05-16"
license: MIT
repository-code: "https://github.com/BASE-Laboratory/BraggTrack"
url: "https://github.com/BASE-Laboratory/BraggTrack"
abstract: >-
- Semantic 4D kinematics and fracture tracking for operando
- X-ray diffraction. BraggTrack segments Bragg reflections in
- 3D reciprocal-space volumes and tracks them across time using
- classical image processing and self-supervised vision-transformer
- embeddings.
+ Semantic 4D kinematics, multi-scan gap closing, and chemomechanical
+ fracture tracking for operando X-ray diffraction. BraggTrack segments
+ Bragg reflections in 3D reciprocal-space volumes, tracks trajectories
+ and split/merge lineage using classical crystallography and
+ vision-transformer embeddings, and exports ImageD11, NeXus HDF5, and
+ interactive HTML 4D lineage reports.
keywords:
- X-ray diffraction
- operando crystallography
- 3D segmentation
- object tracking
+ - grain fracture
+ - synchrotron
- materials science
- Python
authors:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 1833647..f9b3376 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -4,13 +4,15 @@ Thanks for helping improve BraggTrack.
## Before you open a PR
-1. Install the package in editable mode with dev/runtime dependencies you need (`pip install -e .` plus `numpy`, `scipy`, `scikit-image`, `networkx`; add `h5py` / `torch` / `transformers` as required).
-2. Run **`python scripts/pre_pr_check.py`** (conflict markers and optional sync hint against `origin/main`).
-3. Run **`python scripts/ci_report.py`** (unit tests + Week 1–4 acceptance scripts).
+1. Install the package in editable mode with developer dependencies (`pip install -e ".[dev]"` for full PyTorch, notebook, pytest, and Ruff tooling, or `pip install -e .` for the lightweight core).
+2. Run **`ruff check braggtrack/ tests/ scripts/`** and **`ruff format --check braggtrack/ tests/ scripts/`**.
+3. Run **`python scripts/pre_pr_check.py`** (conflict markers and optional sync hint against `origin/main`).
+4. Run **`python scripts/ci_report.py`** (unit tests + all 9 acceptance gates covering IO, smoke, segmentation, tracking, embedding, DINO, events/pipeline, and interactive report + ImageD11/NeXus exports).
## Where things live
-- **[Architecture](docs/architecture.md)** — package layers, data flow, extension points.
+- **[Architecture](docs/architecture.md)** — package layers (`io`, `segmentation`, `semantic`, `tracking`, `viz`, `cli`), data flow, extension points.
+- **[Benchmarks](docs/benchmarks.md)** — 5-scenario quantitative evaluation (`scripts/benchmark_suite.py`) and ESRF-ID03 `facebook/dinov2-small` case study.
- **[Coding practices](docs/coding-practices.md)** — style, testing, optional deps, CI expectations.
- **Sample scan data** — versioned under [`data/sample_operando/`](data/sample_operando/README.md) (not the repo root).
diff --git a/README.md b/README.md
index 8f43b53..d61abd5 100644
--- a/README.md
+++ b/README.md
@@ -4,15 +4,24 @@
[](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb)
-Semantic 4D kinematics, grain lineage (`1 -> 2+` fracture splits & `2+ -> 1` coalescence merges), and persistent fracture precursor tracking for operando X-ray diffraction using classical 3D crystallography and foundation vision transformers.
+Semantic 4D kinematics, multi-scan beam-dropout gap closing, grain lineage (`1 -> 2+` fracture splits & `2+ -> 1` coalescence merges), and persistent fracture precursor tracking for operando X-ray diffraction using classical 3D crystallography and foundation vision transformers.
+
+
+
+
## Key Capabilities
-- **Beamline-ready NeXus/HDF5 I/O & calibration (`braggtrack.io`)** — Auto-discovers 3D detector cubes across standard NeXus paths, ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), and arbitrary HDF5 trees (`auto_discover=True`), with optional physical `(mu, chi, d)` coordinate calibration via `AxisCalibration`.
-- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Intensity-domain Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`).
+- **Beamline-scale NeXus/HDF5 I/O, lazy ROI streaming & crystallographic mapping (`braggtrack.io`)** — Auto-discovers 3D detector cubes across standard NeXus paths, ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), and arbitrary HDF5 trees (`auto_discover=True`). Streams `z`-slabs via `LazyVolume` (`otsu_threshold_chunked`, `read_roi`), calibrates `(mu, chi, d)` via `AxisCalibration`, maps transmission geometry `(2theta, eta, d, |Q|)` via `DetectorGeometry`, and exports directly to **`ImageD11` (`.flt`)** and **NeXus `NXprocess` HDF5 (`.nxs`)**.
+- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Streaming Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`).
- **Multi-view foundation vision descriptors (`braggtrack.semantic`)** — Orthogonal $(\mu, \chi, d)$ maximum-intensity projections (MIPs) encoded with ungated **`facebook/dinov2-small`** by default (zero HuggingFace login required), drop-in support for **DINOv3** (`facebook/dinov3-vitb16-pretrain-lvd1689m`), or a deterministic zero-GPU `mock` backend for CI.
-- **4D grain kinematics, split/merge lineage & fracture precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association into a NetworkX trajectory DAG, per-grain lattice strain ($\Delta d / d_0$) and angular misorientation ($\Delta\mu$, $\Delta\chi$), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking combining 3D shape anisotropy, covariance spread growth, and spot ROI sub-peak / DINO patch-token dispersion.
-- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, segmentation, semantic embedding, tracking, kinematics, and fracture event analysis end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`.
+- **4D grain kinematics, multi-scan gap closing, split/merge lineage & precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association, Jaqaman-style multi-scan gap closing (`--max-gap`) across transient beam dropouts, per-grain lattice strain ($\Delta d / d_0$) and angular misorientation ($\Delta\mu$, $\Delta\chi$), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking combining 3D shape anisotropy, covariance spread growth, and spot ROI sub-peak / DINO patch-token dispersion.
+- **Publication figures & offline interactive HTML 4D Explorer (`braggtrack.viz`, `braggtrack-report`)** — Generates Okabe–Ito styled lineage DAGs, strain/misorientation dashboards, precursor heatmaps, 3-view MIP cards, and a single-file interactive `report.html` with embedded base64 MIP thumbnails that opens offline on air-gapped beamline networks.
+- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, segmentation, semantic embedding, tracking, kinematics, fracture event analysis, HTML reporting, and `ImageD11`/NeXus export end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`.
+
+
+
+
---
@@ -25,7 +34,7 @@ pip install -e .
# With PyTorch + HuggingFace Transformers for real DINOv2 / DINOv3 backbones
pip install -e ".[torch]"
-# With Jupyter, Matplotlib, and Pandas for interactive notebooks
+# With Jupyter, Matplotlib, and Pandas for interactive notebooks & static figures
pip install -e ".[notebook]"
# Full developer environment (torch + notebook + pytest + ruff)
@@ -38,10 +47,14 @@ pip install -e ".[dev]"
BraggTrack ships with three real ESRF ID03 operando Ni-rich cathode scans under [`data/sample_operando/`](data/sample_operando/README.md) (`100 x 250 x 250` `uint16` cubes). Omit the dataset path to run on the bundled scans automatically, or pass your own directory of `scanXXXX/` folders:
-### One-command end-to-end pipeline
+### One-command end-to-end pipeline (with interactive HTML report & NeXus/`ImageD11` exports)
```bash
-braggtrack-run --outdir artifacts/pipeline
+braggtrack-run --outdir artifacts/pipeline \
+ --max-gap 1 \
+ --report \
+ --export-nexus artifacts/pipeline/results.nxs \
+ --export-imaged11 artifacts/pipeline/peaks.flt
```
Produces:
@@ -50,26 +63,29 @@ Produces:
- `artifacts/pipeline/embedding/` (`embeddings.npz`, `embedding_manifest.json`)
- `artifacts/pipeline/tracking/` (`tracks.csv`, `kinematics.csv`, `kinematics_summary.json`, `tracking_summary.json`, QC notebook)
- `artifacts/pipeline/events/` (`events.csv`, `events.json`, `precursors.csv`, `events_summary.json`)
+- `artifacts/pipeline/report/` (`report.html` interactive 4D Explorer + `figures/*.png`)
+- `artifacts/pipeline/results.nxs` & `artifacts/pipeline/peaks.flt`
- `artifacts/pipeline/run_manifest.json` (SHA-256 input checksums, git SHA, config, stage timings)
-### Stage-by-stage CLI commands
+### Stage-by-stage CLI commands (9 Entry Points)
| CLI Entry Point | Module Equivalent | Purpose |
|---|---|---|
| `braggtrack-inspect` | `python -m braggtrack.cli.inspect_datasets` | Discover `scan*` folders, primary 3D HDF5 dataset paths, and metadata |
| `braggtrack-validate` | `python -m braggtrack.cli.validate_dataset` | Validate sequence ordering, monotonic timestamps, and axis contracts |
| `braggtrack-segment-synthetic` | `python -m braggtrack.cli.segment_synthetic` | Run fast synthetic 3D segmentation smoke test |
-| `braggtrack-segment-dataset` | `python -m braggtrack.cli.segment_dataset` | Segment 3D Bragg spots across scans with smoothed thresholds & optional calibration |
-| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Compute orthogonal MIP embeddings (`auto`, `mock`, or `torch` DINOv2/DINOv3) |
-| `braggtrack-track-dataset` | `python -m braggtrack.cli.track_dataset` | Build trajectory DAG, export `tracks.csv` + `kinematics.csv` (+ optional `--detect-events`) |
-| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and ROI-based fracture precursors |
-| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config config.json` and `run_manifest.json` |
+| `braggtrack-segment-dataset` | `python -m braggtrack.cli.segment_dataset` | Segment 3D Bragg spots via chunked Otsu + smoothed thresholds & optional calibration |
+| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Stream lazy 3D spot ROIs & compute orthogonal MIP embeddings (`auto`, `mock`, `torch`) |
+| `braggtrack-track-dataset` | `python -m braggtrack.cli.track_dataset` | Build trajectory DAG (`--max-gap`), export `tracks.csv`, `kinematics.csv`, `.flt`, `.nxs` |
+| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and lazy-ROI fracture precursors |
+| `braggtrack-report` | `python -m braggtrack.cli.report_dataset` | Render publication PNG/PDF figures and self-contained interactive `report.html` |
+| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config config.json`, `--report`, and exports |
---
-## Physical Axis Calibration (`mu`, `chi`, `d`)
+## Physical Axis Calibration (`mu`, `chi`, `d`) & Crystallographic Geometry (`2theta`, `eta`, `|Q|`)
-Cropped detector cubes store integer voxel indices `[0..N)`. For physically meaningful lattice strain $\varepsilon = (d(t) - d_0) / d_0$ and angular misorientation in degrees, supply an `AxisCalibration` JSON file (or CLI flags) when segmenting:
+Cropped detector cubes store integer voxel indices `[0..N)`. For physically meaningful lattice strain $\varepsilon = (d(t) - d_0) / d_0$ and angular misorientation in degrees, supply an `AxisCalibration` JSON file when segmenting:
```json
{
@@ -82,30 +98,36 @@ Cropped detector cubes store integer voxel indices `[0..N)`. For physically mean
}
```
-```bash
-braggtrack-segment-dataset data/sample_operando \
- --calibration-json calibration.json \
- --outdir artifacts/segmentation
-```
+For transmission 3DXRD / `ImageD11` indexing workflows, supply a `DetectorGeometry` JSON file (`--detector-json`) to annotate `omega_deg`, `two_theta_deg`, `eta_deg`, `d_spacing_angstrom`, and `(qx, qy, qz, q_mag_inv_angstrom)` and export `.flt` columnfiles:
-If HDF5 scans already include 1D `mu`, `chi`, `d` coordinate datasets or `/entry/calibration` attributes, `extract_axis_calibration()` detects them automatically.
+```json
+{
+ "distance_mm": 150.0,
+ "pixel_size_y_mm": 0.075,
+ "pixel_size_x_mm": 0.075,
+ "beam_center_y_px": 1024.0,
+ "beam_center_x_px": 1024.0,
+ "wavelength_angstrom": 0.2898,
+ "omega_origin_deg": 0.0,
+ "omega_step_deg": 0.05
+}
+```
---
-## Vision Backbone Options (DINOv2 Default & Optional DINOv3)
+## Vision Backbone Options & Quantitative Benchmarks
- **Default (`facebook/dinov2-small`)**: Public and ungated on HuggingFace — works out of the box with `pip install -e ".[torch]"` without tokens or license gates.
-- **DINOv3 (`facebook/dinov3-vitb16-pretrain-lvd1689m`)**: Supported via `--model` / `--dino-model` after accepting Meta's license on HuggingFace and setting `HF_TOKEN` (or running `huggingface-cli login`):
- ```bash
- braggtrack-embed-dataset --backend torch --model facebook/dinov3-vitb16-pretrain-lvd1689m
- ```
+- **DINOv3 (`facebook/dinov3-vitb16-pretrain-lvd1689m`)**: Supported via `--model` / `--dino-model` after accepting Meta's license on HuggingFace and setting `HF_TOKEN`.
- **Offline / CI (`mock`)**: Deterministic SHA-256 multi-view projection encoder requiring zero GPU or network access (`--backend mock` or `BRAGGTRACK_DINO_BACKEND=mock`).
+See [`docs/benchmarks.md`](docs/benchmarks.md) for the full 5-scenario benchmark evaluation (`scripts/benchmark_suite.py`) comparing **Geometry-Only**, **Mock-Hash**, and **Real `facebook/dinov2-small`** alongside the ESRF-ID03 case study.
+
---
## Notebooks
-- [`notebooks/braggtrack_demo.ipynb`](notebooks/braggtrack_demo.ipynb) — End-to-end tutorial on the bundled ESRF-ID03 scans: discovery, stabilised 3D segmentation, physics & semantic tracking, per-grain kinematics, α/β ablation, and synthetic 4D grain fracture + precursor ranking.
+- [`notebooks/braggtrack_demo.ipynb`](notebooks/braggtrack_demo.ipynb) — End-to-end tutorial on the bundled ESRF-ID03 scans: discovery, lazy/chunked segmentation, physics & semantic tracking, multi-scan gap closing, per-grain kinematics, `braggtrack.viz` lineage & precursor plots, and crystallographic geometry mapping.
- [`notebooks/dino_segmentation_comparison.ipynb`](notebooks/dino_segmentation_comparison.ipynb) — Side-by-side comparison of stabilised classical 3D watershed vs. experimental DINO patch-clustering segmentation.
---
@@ -120,9 +142,12 @@ ruff format --check braggtrack/ tests/ scripts/
# Unit test suite with coverage
pytest tests/ -v --cov=braggtrack
-# Full CI gate runner (unit tests + IO, smoke, segmentation, tracking, embedding, DINO, and events acceptance)
+# Full CI gate runner (unit tests + all 9 acceptance gates)
python scripts/ci_report.py
+# Quantitative 5-scenario benchmark suite (add --include-torch for real DINOv2)
+python scripts/benchmark_suite.py --trials 5
+
# Pre-PR hygiene check
python scripts/pre_pr_check.py
```
@@ -131,16 +156,16 @@ python scripts/pre_pr_check.py
## Troubleshooting & FAQ
-- **`MissingH5DependencyError` or custom HDF5 dataset paths:** `h5py` is installed by default with `pip install -e .`. `load_primary_volume()` automatically inspects standard NeXus paths, `/entry_0000/ESRF-ID03/pco_nf/data`, and falls back to the largest 3D numeric dataset in the file. Pass `--allow-synthetic-fallback` only when testing schema plumbing on empty/metadata-only HDF5 stubs.
+- **`MissingH5DependencyError` or custom HDF5 dataset paths:** `h5py` is installed by default with `pip install -e .`. `load_primary_volume()` and `LazyVolume` automatically inspect standard NeXus paths, `/entry_0000/ESRF-ID03/pco_nf/data`, and fall back to the largest 3D numeric dataset in the file. Pass `--allow-synthetic-fallback` only when testing schema plumbing on empty/metadata-only HDF5 stubs.
- **`401 Unauthorized` / gated HuggingFace model error:** You passed a gated model ID (such as `facebook/dinov3-*`) without authenticating. Either omit `--model` to use the ungated default (`facebook/dinov2-small`), or accept the model license on HuggingFace and export `HF_TOKEN`.
-- **Unphysical strain percentages in `kinematics.csv`:** Check `"calibrated_axes"` in `kinematics_summary.json`. When `false`, `centroid_d` is measured in cropped detector pixels (`y` index) rather than Angstroms — pass `--calibration-json` (or `--d-origin` / `--d-step`) to compute true crystallographic $\Delta d / d_0$.
+- **Unphysical strain percentages in `kinematics.csv`:** Check `"calibrated_axes"` in `kinematics_summary.json`. When `false`, `centroid_d` is measured in cropped detector pixels (`y` index) rather than Angstroms — pass `--calibration-json` (or `--detector-json`) to compute physical crystallographic quantities.
## Repository Layout & Contributing
-- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `cli`).
+- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `viz`, `cli`).
- **`data/sample_operando/`** — Bundled ESRF-ID03 operando test scans (`scan0001`–`scan0003`).
-- **`docs/`** — [Architecture](docs/architecture.md) and [Coding Practices](docs/coding-practices.md).
-- **`scripts/`** — Acceptance gates, semantic α/β ablation (`ablation_semantic.py`), and CI runner (`ci_report.py`).
+- **`docs/`** — [Architecture](docs/architecture.md), [Benchmarks](docs/benchmarks.md), [Coding Practices](docs/coding-practices.md), and publication figures (`docs/figures/`).
+- **`scripts/`** — 9 acceptance gates, 5-scenario benchmark suite (`benchmark_suite.py`), semantic α/β ablation (`ablation_semantic.py`), and CI runner (`ci_report.py`).
- **`artifacts/`** — Generated pipeline outputs (gitignored; never commit).
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines and [paper.md](paper.md) for the manuscript overview.
diff --git a/braggtrack/__init__.py b/braggtrack/__init__.py
index 20123fa..14eeacb 100644
--- a/braggtrack/__init__.py
+++ b/braggtrack/__init__.py
@@ -5,6 +5,6 @@
try:
__version__ = version("braggtrack")
except PackageNotFoundError:
- __version__ = "0.2.0.dev0"
+ __version__ = "0.3.0.dev0"
-__all__ = ["io", "segmentation", "semantic", "tracking"]
+__all__ = ["io", "segmentation", "semantic", "tracking", "viz"]
diff --git a/braggtrack/cli/detect_events.py b/braggtrack/cli/detect_events.py
index 321dfaf..9eb799d 100644
--- a/braggtrack/cli/detect_events.py
+++ b/braggtrack/cli/detect_events.py
@@ -4,6 +4,7 @@
import argparse
import json
+from contextlib import ExitStack
from pathlib import Path
from typing import Any
@@ -13,14 +14,16 @@
from braggtrack.cli.track_dataset import _load_embeddings_npz, _merge_embeddings
from braggtrack.io import (
BeamlineAdapter,
+ LazyVolume,
MissingH5DependencyError,
discover_operando_scans,
- load_primary_volume,
+ open_lazy_volume,
resolve_dataset_root,
)
from braggtrack.semantic import DEFAULT_DINO_MODEL, make_patch_encoder
from braggtrack.semantic.dino import _requested_backend, _resolve_backend
from braggtrack.tracking import (
+ GapClosingConfig,
GeometrySemanticCost,
PositionShapeCost,
PrecursorConfig,
@@ -66,6 +69,8 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--gate-chi", type=float, default=float("inf"))
parser.add_argument("--gate-d", type=float, default=float("inf"))
parser.add_argument("--max-cost", type=float, default=float("inf"))
+ parser.add_argument("--max-gap", type=int, default=0, help="Max missing intermediate scans to bridge (0 disables)")
+ parser.add_argument("--gap-max-cost", type=float, default=25.0, help="Max penalized cost for gap-closing bridges")
parser.add_argument("--cost-alpha", type=float, default=1.0)
parser.add_argument("--cost-beta", type=float, default=0.0)
parser.add_argument("--split-max-distance", type=float, default=25.0)
@@ -83,13 +88,14 @@ def build_parser() -> argparse.ArgumentParser:
def _try_load_roi_inputs(
+ stack: ExitStack,
root_arg: str | None,
scan_dirs: list[Path],
scan_names: list[str],
*,
allow_synthetic_fallback: bool,
-) -> tuple[dict[int, np.ndarray], dict[int, np.ndarray], dict[int, str | None]]:
- volumes: dict[int, np.ndarray] = {}
+) -> tuple[dict[int, Any], dict[int, np.ndarray], dict[int, str | None]]:
+ volumes: dict[int, Any] = {}
label_volumes: dict[int, np.ndarray] = {}
timestamps: dict[int, str | None] = {}
@@ -123,13 +129,13 @@ def _try_load_roi_inputs(
if sf is None or idx not in label_volumes:
continue
try:
- vol = load_primary_volume(sf.path)
+ vol: LazyVolume = stack.enter_context(open_lazy_volume(sf.path))
except MissingH5DependencyError:
- vol = synth_volume_from_file(sf.path)
+ vol = LazyVolume.from_array(synth_volume_from_file(sf.path))
except (KeyError, ValueError):
if not allow_synthetic_fallback:
continue
- vol = synth_volume_from_file(sf.path)
+ vol = LazyVolume.from_array(synth_volume_from_file(sf.path))
if vol.shape == label_volumes[idx].shape:
volumes[idx] = vol
@@ -175,63 +181,69 @@ def main(argv: list[str] | None = None) -> int:
print(json.dumps({"error": "Missing embeddings.npz", "path": str(npz)}))
return 1
- volumes, label_volumes, scan_timestamps = _try_load_roi_inputs(
- args.root,
- used_scan_dirs,
- scan_names,
- allow_synthetic_fallback=bool(args.allow_synthetic_fallback),
- )
+ with ExitStack() as stack:
+ volumes, label_volumes, scan_timestamps = _try_load_roi_inputs(
+ stack,
+ args.root,
+ used_scan_dirs,
+ scan_names,
+ allow_synthetic_fallback=bool(args.allow_synthetic_fallback),
+ )
- geo = PositionShapeCost(
- position_weight=args.position_weight,
- shape_weight=args.shape_weight,
- gate_mu=args.gate_mu,
- gate_chi=args.gate_chi,
- gate_d=args.gate_d,
- )
- cost_fn = (
- GeometrySemanticCost(geo, cost_alpha=args.cost_alpha, cost_beta=args.cost_beta)
- if args.cost_beta != 0.0
- else geo
- )
- sm_cfg = SplitMergeConfig(
- max_distance=args.split_max_distance,
- gate_mu=args.gate_mu,
- gate_chi=args.gate_chi,
- gate_d=args.gate_d,
- min_intensity_ratio=args.min_intensity_ratio,
- max_intensity_ratio=args.max_intensity_ratio,
- min_fragment_share=args.min_fragment_share,
- )
+ geo = PositionShapeCost(
+ position_weight=args.position_weight,
+ shape_weight=args.shape_weight,
+ gate_mu=args.gate_mu,
+ gate_chi=args.gate_chi,
+ gate_d=args.gate_d,
+ )
+ cost_fn = (
+ GeometrySemanticCost(geo, cost_alpha=args.cost_alpha, cost_beta=args.cost_beta)
+ if args.cost_beta != 0.0
+ else geo
+ )
+ gap_cfg = GapClosingConfig(max_gap=args.max_gap, max_cost=args.gap_max_cost) if args.max_gap > 0 else None
+ sm_cfg = SplitMergeConfig(
+ max_distance=args.split_max_distance,
+ gate_mu=args.gate_mu,
+ gate_chi=args.gate_chi,
+ gate_d=args.gate_d,
+ min_intensity_ratio=args.min_intensity_ratio,
+ max_intensity_ratio=args.max_intensity_ratio,
+ min_fragment_share=args.min_fragment_share,
+ )
- G = build_tracks(
- scan_tables,
- cost_fn=cost_fn,
- max_cost=args.max_cost,
- split_merge_config=sm_cfg,
- scan_timestamps=scan_timestamps or None,
- )
- metrics = compute_tracking_metrics(G, n_scans=len(scan_tables))
- lineage_events = list(G.graph.get("lineage_events", []))
- track_rows = tracks_to_table(G)
- for tr in track_rows:
- tr.pop("embedding", None)
+ G = build_tracks(
+ scan_tables,
+ cost_fn=cost_fn,
+ max_cost=args.max_cost,
+ gap_closing_config=gap_cfg,
+ split_merge_config=sm_cfg,
+ scan_timestamps=scan_timestamps or None,
+ )
+ metrics = compute_tracking_metrics(G, n_scans=len(scan_tables))
+ lineage_events = list(G.graph.get("lineage_events", []))
+ track_rows = tracks_to_table(G)
+ for tr in track_rows:
+ tr.pop("embedding", None)
- resolved_backend = _resolve_backend(_requested_backend(args.backend)) # type: ignore[arg-type]
- patch_encoder = make_patch_encoder(resolved_backend, model_name=args.model) if (volumes and label_volumes) else None
+ resolved_backend = _resolve_backend(_requested_backend(args.backend)) # type: ignore[arg-type]
+ patch_encoder = (
+ make_patch_encoder(resolved_backend, model_name=args.model) if (volumes and label_volumes) else None
+ )
- prec_cfg = PrecursorConfig(
- score_threshold=args.precursor_threshold,
- min_persistence=max(1, args.precursor_persistence),
- )
- candidates, obs_scores = compute_fracture_precursors(
- track_rows,
- prec_cfg,
- volumes=volumes or None,
- label_volumes=label_volumes or None,
- patch_encoder=patch_encoder,
- lineage_events=lineage_events,
- )
+ prec_cfg = PrecursorConfig(
+ score_threshold=args.precursor_threshold,
+ min_persistence=max(1, args.precursor_persistence),
+ )
+ candidates, obs_scores = compute_fracture_precursors(
+ track_rows,
+ prec_cfg,
+ volumes=volumes or None,
+ label_volumes=label_volumes or None,
+ patch_encoder=patch_encoder,
+ lineage_events=lineage_events,
+ )
event_rows = events_to_table(lineage_events)
prec_rows = precursors_to_table(candidates)
@@ -262,6 +274,7 @@ def main(argv: list[str] | None = None) -> int:
"scan_names": scan_names,
"n_scans": len(scan_tables),
"total_tracks": metrics["total_tracks"],
+ "gap_closed_count": metrics["gap_closed_count"],
"split_count": metrics["split_count"],
"merge_count": metrics["merge_count"],
"event_count": len(event_rows),
diff --git a/braggtrack/cli/embed_dataset.py b/braggtrack/cli/embed_dataset.py
index b819c01..f1baec7 100644
--- a/braggtrack/cli/embed_dataset.py
+++ b/braggtrack/cli/embed_dataset.py
@@ -10,9 +10,10 @@
from braggtrack.cli._utils import load_feature_csv, synth_volume_from_file
from braggtrack.io import (
+ LazyVolume,
MissingH5DependencyError,
discover_operando_scans,
- load_primary_volume,
+ open_lazy_volume,
resolve_dataset_root,
)
from braggtrack.semantic import DEFAULT_DINO_MODEL, crop_spot_cube, make_multiview_encoder, orthogonal_mips
@@ -83,38 +84,39 @@ def main(argv: list[str] | None = None) -> int:
return 1
try:
- volume = load_primary_volume(scan_file.path)
+ lazy_vol = open_lazy_volume(scan_file.path)
except MissingH5DependencyError:
- volume = synth_volume_from_file(scan_file.path)
+ lazy_vol = LazyVolume.from_array(synth_volume_from_file(scan_file.path))
except (KeyError, ValueError):
if not args.allow_synthetic_fallback:
raise
- volume = synth_volume_from_file(scan_file.path)
-
- if volume.shape != labels_full.shape:
- print(
- json.dumps(
- {
- "error": "Volume/labels shape mismatch",
- "scan": name,
- "volume": list(volume.shape),
- "labels": list(labels_full.shape),
- }
+ lazy_vol = LazyVolume.from_array(synth_volume_from_file(scan_file.path))
+
+ with lazy_vol:
+ if lazy_vol.shape != labels_full.shape:
+ print(
+ json.dumps(
+ {
+ "error": "Volume/labels shape mismatch",
+ "scan": name,
+ "volume": list(lazy_vol.shape),
+ "labels": list(labels_full.shape),
+ }
+ )
)
- )
- return 1
+ return 1
+
+ label_ids: list[int] = []
+ vectors: list[np.ndarray] = []
- label_ids: list[int] = []
- vectors: list[np.ndarray] = []
-
- for row in rows:
- lid = int(row["label"])
- mip_mu, mip_chi, mip_d = orthogonal_mips(
- crop_spot_cube(volume, labels_full, lid, row, margin=max(0, args.margin))[0]
- )
- vec = enc.embed(mip_mu, mip_chi, mip_d)
- label_ids.append(lid)
- vectors.append(vec)
+ for row in rows:
+ lid = int(row["label"])
+ mip_mu, mip_chi, mip_d = orthogonal_mips(
+ crop_spot_cube(lazy_vol, labels_full, lid, row, margin=max(0, args.margin))[0]
+ )
+ vec = enc.embed(mip_mu, mip_chi, mip_d)
+ label_ids.append(lid)
+ vectors.append(vec)
if not vectors:
dim = 384
diff --git a/braggtrack/cli/report_dataset.py b/braggtrack/cli/report_dataset.py
new file mode 100644
index 0000000..f6ac4fc
--- /dev/null
+++ b/braggtrack/cli/report_dataset.py
@@ -0,0 +1,140 @@
+"""Generate publication figures and an interactive HTML 4D beamtime explorer from BraggTrack outputs."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+from pathlib import Path
+from typing import Any
+
+from braggtrack.viz import (
+ generate_html_report,
+ plot_kinematics_dashboard,
+ plot_lineage_dag,
+ plot_precursor_timeline,
+)
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "artifact_dir",
+ nargs="?",
+ default="artifacts/pipeline",
+ help="Pipeline or tracking output directory containing tracks.csv / events / manifest",
+ )
+ parser.add_argument(
+ "--dataset-root",
+ default=None,
+ help="Optional raw HDF5 dataset root for embedding 3-axis MIP spot thumbnails",
+ )
+ parser.add_argument(
+ "--outdir",
+ default=None,
+ help="Optional report output directory (defaults to /report)",
+ )
+ parser.add_argument(
+ "--no-figures",
+ action="store_true",
+ help="Skip static Matplotlib PNG/PDF generation and emit only the self-contained HTML explorer",
+ )
+ parser.add_argument(
+ "--max-mip-tracks",
+ type=int,
+ default=16,
+ help="Maximum priority tracks for lazy 3-view MIP thumbnail extraction",
+ )
+ return parser
+
+
+def _read_csv_rows(path: Path) -> list[dict[str, str]]:
+ if not path.exists() or path.stat().st_size == 0:
+ return []
+ with path.open(newline="") as f:
+ return list(csv.DictReader(f))
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = build_parser().parse_args(argv)
+ in_root = Path(args.artifact_dir)
+ if not in_root.is_dir():
+ print(json.dumps({"error": "Artifact directory not found", "artifact_dir": str(in_root)}))
+ return 1
+
+ out_root = Path(args.outdir) if args.outdir else (in_root / "report")
+ out_root.mkdir(parents=True, exist_ok=True)
+
+ trk_dir = in_root / "tracking" if (in_root / "tracking").is_dir() else in_root
+ evt_dir = in_root / "events" if (in_root / "events").is_dir() else trk_dir
+ track_rows = _read_csv_rows(evt_dir / "tracks.csv") or _read_csv_rows(trk_dir / "tracks.csv")
+ if not track_rows:
+ print(json.dumps({"error": "No tracks.csv found under artifact directory", "artifact_dir": str(in_root)}))
+ return 1
+
+ html_path = generate_html_report(
+ in_root,
+ out_root / "report.html",
+ dataset_root=args.dataset_root,
+ max_mip_tracks=max(0, args.max_mip_tracks),
+ )
+
+ figures: dict[str, str] = {}
+ if not args.no_figures:
+ try:
+ import matplotlib
+
+ matplotlib.use("Agg", force=False)
+ import matplotlib.pyplot as plt
+
+ fig_dir = out_root / "figures"
+ fig_dir.mkdir(parents=True, exist_ok=True)
+ event_rows = _read_csv_rows(evt_dir / "events.csv") or _read_csv_rows(trk_dir / "events.csv")
+ prec_rows = _read_csv_rows(evt_dir / "precursors.csv") or _read_csv_rows(trk_dir / "precursors.csv")
+ prec_obs_rows = _read_csv_rows(evt_dir / "precursor_observations.csv")
+
+ calibrated = False
+ ts_path = trk_dir / "tracking_summary.json"
+ if ts_path.exists():
+ try:
+ calibrated = bool(json.loads(ts_path.read_text()).get("calibrated_axes", False))
+ except (OSError, json.JSONDecodeError):
+ calibrated = False
+
+ fig_dag, _ = plot_lineage_dag(track_rows, events=event_rows, save_path=fig_dir / "lineage_dag.png")
+ plt.close(fig_dag)
+ figures["lineage_dag"] = str((fig_dir / "lineage_dag.png").resolve())
+
+ fig_kin, _ = plot_kinematics_dashboard(
+ track_rows,
+ events=event_rows,
+ calibrated=calibrated,
+ save_path=fig_dir / "kinematics_dashboard.png",
+ )
+ plt.close(fig_kin)
+ figures["kinematics_dashboard"] = str((fig_dir / "kinematics_dashboard.png").resolve())
+
+ if prec_obs_rows:
+ fig_prec, _ = plot_precursor_timeline(
+ prec_obs_rows,
+ candidates=prec_rows,
+ save_path=fig_dir / "precursor_timeline.png",
+ )
+ plt.close(fig_prec)
+ figures["precursor_timeline"] = str((fig_dir / "precursor_timeline.png").resolve())
+ except ImportError:
+ pass
+
+ summary: dict[str, Any] = {
+ "schema_version": "report.v1",
+ "html_report": str(html_path.resolve()),
+ "figures": figures,
+ "track_row_count": len(track_rows),
+ }
+ (out_root / "report_summary.json").write_text(json.dumps(summary, indent=2))
+ print(json.dumps(summary, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/braggtrack/cli/run_pipeline.py b/braggtrack/cli/run_pipeline.py
index 12ef93e..756f5b1 100644
--- a/braggtrack/cli/run_pipeline.py
+++ b/braggtrack/cli/run_pipeline.py
@@ -14,7 +14,7 @@
from typing import Any
import braggtrack
-from braggtrack.cli import detect_events, embed_dataset, segment_dataset, track_dataset
+from braggtrack.cli import detect_events, embed_dataset, report_dataset, segment_dataset, track_dataset
from braggtrack.io import BeamlineAdapter, discover_operando_scans, resolve_dataset_root, validate_sequence
from braggtrack.semantic import DEFAULT_DINO_MODEL
@@ -48,8 +48,12 @@ def build_parser() -> argparse.ArgumentParser:
)
parser.add_argument("--smooth-window", type=int, default=5, help="Cross-scan rolling median threshold window")
parser.add_argument("--calibration-json", default=None, help="Optional AxisCalibration JSON path")
+ parser.add_argument(
+ "--detector-json", default=None, help="Optional DetectorGeometry JSON path for (2theta, eta, Q)"
+ )
parser.add_argument("--cost-alpha", type=float, default=1.0, help="Geometry weight in semantic cost")
parser.add_argument("--cost-beta", type=float, default=0.25, help="Semantic embedding weight in tracking cost")
+ parser.add_argument("--max-gap", type=int, default=0, help="Max missing intermediate scans to bridge (0 disables)")
parser.add_argument(
"--split-max-distance", type=float, default=25.0, help="Max centroid distance for splits/merges"
)
@@ -61,6 +65,13 @@ def build_parser() -> argparse.ArgumentParser:
default=True,
help="Run split/merge lineage and fracture precursor detection stage (default: enabled)",
)
+ parser.add_argument(
+ "--report",
+ action="store_true",
+ help="Generate static publication figures and interactive HTML 4D explorer under /report",
+ )
+ parser.add_argument("--export-imaged11", default=None, help="Optional ImageD11 .flt export path")
+ parser.add_argument("--export-nexus", default=None, help="Optional NeXus NXprocess .nxs/.h5 export path")
parser.add_argument(
"--allow-synthetic-fallback",
action="store_true",
@@ -102,12 +113,17 @@ def _merge_config_overrides(args: argparse.Namespace) -> dict[str, Any]:
"method": args.method,
"smooth_window": args.smooth_window,
"calibration_json": args.calibration_json,
+ "detector_json": args.detector_json,
"cost_alpha": args.cost_alpha,
"cost_beta": args.cost_beta,
+ "max_gap": args.max_gap,
"split_max_distance": args.split_max_distance,
"precursor_threshold": args.precursor_threshold,
"precursor_persistence": args.precursor_persistence,
"detect_events": bool(args.detect_events),
+ "report": bool(args.report),
+ "export_imaged11": args.export_imaged11,
+ "export_nexus": args.export_nexus,
"allow_synthetic_fallback": bool(args.allow_synthetic_fallback),
"segmentation_extra_args": [],
"tracking_extra_args": [],
@@ -132,6 +148,7 @@ def main(argv: list[str] | None = None) -> int:
emb_dir = outdir / "embedding"
trk_dir = outdir / "tracking"
evt_dir = outdir / "events"
+ rep_dir = outdir / "report"
timings: dict[str, float] = {}
t_total0 = time.perf_counter()
@@ -190,7 +207,7 @@ def main(argv: list[str] | None = None) -> int:
if rc_seg != 0:
return rc_seg
- # 3. Embed dataset (whenever cost_beta != 0 or detect_events is enabled)
+ # 3. Embed dataset (whenever cost_beta != 0)
need_embed = float(cfg["cost_beta"]) != 0.0
if need_embed:
t0 = time.perf_counter()
@@ -222,6 +239,8 @@ def main(argv: list[str] | None = None) -> int:
str(cfg["cost_alpha"]),
"--cost-beta",
str(cfg["cost_beta"]),
+ "--max-gap",
+ str(cfg["max_gap"]),
"--split-max-distance",
str(cfg["split_max_distance"]),
"--precursor-threshold",
@@ -233,6 +252,12 @@ def main(argv: list[str] | None = None) -> int:
trk_argv.extend(["--embedding-dir", str(emb_dir)])
if cfg.get("detect_events"):
trk_argv.append("--detect-events")
+ if cfg.get("detector_json"):
+ trk_argv.extend(["--detector-json", str(cfg["detector_json"])])
+ if cfg.get("export_imaged11"):
+ trk_argv.extend(["--export-imaged11", str(cfg["export_imaged11"])])
+ if cfg.get("export_nexus") and not cfg.get("detect_events"):
+ trk_argv.extend(["--export-nexus", str(cfg["export_nexus"])])
trk_argv.extend(str(x) for x in cfg.get("tracking_extra_args", []))
rc_trk = track_dataset.main(trk_argv)
timings["track"] = round(time.perf_counter() - t0, 4)
@@ -256,6 +281,8 @@ def main(argv: list[str] | None = None) -> int:
str(cfg["cost_alpha"]),
"--cost-beta",
str(cfg["cost_beta"]),
+ "--max-gap",
+ str(cfg["max_gap"]),
"--split-max-distance",
str(cfg["split_max_distance"]),
"--precursor-threshold",
@@ -272,34 +299,57 @@ def main(argv: list[str] | None = None) -> int:
if rc_evt != 0:
return rc_evt
- total_sec = round(time.perf_counter() - t_total0, 4)
- manifest = {
- "schema_version": "manifest.v1",
- "created_at": datetime.now(timezone.utc).isoformat(),
- "braggtrack_version": braggtrack.__version__,
- "git_sha": _resolve_git_sha(),
- "python_version": sys.version.split()[0],
- "platform": platform.platform(),
- "dataset_root": str(root.resolve()),
- "outdir": str(outdir.resolve()),
- "config": cfg,
- "inputs": input_files,
- "stage_timings_sec": timings,
- "total_duration_sec": total_sec,
- "artifacts": {
- "validation_summary": str((outdir / "validation_summary.json").resolve()),
- "segmentation_summary": str((seg_dir / "segmentation_summary.json").resolve()),
- "embedding_summary": str((emb_dir / "embedding_summary.json").resolve()) if need_embed else None,
- "tracking_summary": str((trk_dir / "tracking_summary.json").resolve()),
- "kinematics_csv": str((trk_dir / "kinematics.csv").resolve()),
- "kinematics_summary": str((trk_dir / "kinematics_summary.json").resolve()),
- "events_summary": str((evt_dir / "events_summary.json").resolve()) if cfg.get("detect_events") else None,
- "events_csv": str((evt_dir / "events.csv").resolve()) if cfg.get("detect_events") else None,
- "precursors_csv": str((evt_dir / "precursors.csv").resolve()) if cfg.get("detect_events") else None,
- },
- }
+ if cfg.get("export_nexus") and cfg.get("detect_events"):
+ from braggtrack.io import DetectorGeometry, export_nexus_process
+
+ dgeom = DetectorGeometry.from_json(cfg["detector_json"]) if cfg.get("detector_json") else None
+ export_nexus_process(outdir, cfg["export_nexus"], geometry=dgeom)
+
manifest_path = outdir / "run_manifest.json"
- manifest_path.write_text(json.dumps(manifest, indent=2))
+
+ def _build_manifest(cur_total_sec: float) -> dict[str, Any]:
+ return {
+ "schema_version": "manifest.v1",
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "braggtrack_version": braggtrack.__version__,
+ "git_sha": _resolve_git_sha(),
+ "python_version": sys.version.split()[0],
+ "platform": platform.platform(),
+ "dataset_root": str(root.resolve()),
+ "outdir": str(outdir.resolve()),
+ "config": cfg,
+ "inputs": input_files,
+ "stage_timings_sec": timings,
+ "total_duration_sec": cur_total_sec,
+ "artifacts": {
+ "validation_summary": str((outdir / "validation_summary.json").resolve()),
+ "segmentation_summary": str((seg_dir / "segmentation_summary.json").resolve()),
+ "embedding_summary": str((emb_dir / "embedding_summary.json").resolve()) if need_embed else None,
+ "tracking_summary": str((trk_dir / "tracking_summary.json").resolve()),
+ "kinematics_csv": str((trk_dir / "kinematics.csv").resolve()),
+ "kinematics_summary": str((trk_dir / "kinematics_summary.json").resolve()),
+ "events_summary": str((evt_dir / "events_summary.json").resolve())
+ if cfg.get("detect_events")
+ else None,
+ "events_csv": str((evt_dir / "events.csv").resolve()) if cfg.get("detect_events") else None,
+ "precursors_csv": str((evt_dir / "precursors.csv").resolve()) if cfg.get("detect_events") else None,
+ "html_report": str((rep_dir / "report.html").resolve()) if cfg.get("report") else None,
+ },
+ }
+
+ total_sec = round(time.perf_counter() - t_total0, 4)
+ manifest_path.write_text(json.dumps(_build_manifest(total_sec), indent=2))
+
+ # 6. Optional interactive HTML & static figure report
+ if cfg.get("report"):
+ t0 = time.perf_counter()
+ rc_rep = report_dataset.main([str(outdir), "--dataset-root", str(root), "--outdir", str(rep_dir)])
+ timings["report"] = round(time.perf_counter() - t0, 4)
+ if rc_rep != 0:
+ return rc_rep
+ total_sec = round(time.perf_counter() - t_total0, 4)
+ manifest_path.write_text(json.dumps(_build_manifest(total_sec), indent=2))
+
print(json.dumps({"manifest": str(manifest_path.resolve()), "total_duration_sec": total_sec}, indent=2))
return 0
diff --git a/braggtrack/cli/segment_dataset.py b/braggtrack/cli/segment_dataset.py
index aaaf79e..9f6ff29 100644
--- a/braggtrack/cli/segment_dataset.py
+++ b/braggtrack/cli/segment_dataset.py
@@ -16,6 +16,8 @@
discover_primary_volume_path,
extract_axis_calibration,
load_primary_volume,
+ open_lazy_volume,
+ otsu_threshold_chunked,
resolve_dataset_root,
)
from braggtrack.segmentation import (
@@ -191,12 +193,20 @@ def _load_one(scan_obj: object) -> tuple[np.ndarray, str, str | None, AxisCalibr
src = "synthetic_fallback"
return vol, src, dpath, scan_cal
- # First pass: compute scalar per-scan Otsu thresholds in O(1) volume memory.
+ # First pass: compute scalar per-scan Otsu thresholds via streaming z-slabs.
raw_thresholds: list[float] = []
for scan in scans:
- vol_tmp, _, _, _ = _load_one(scan)
- raw_thresholds.append(otsu_threshold(vol_tmp.ravel()) * float(args.threshold_fraction))
- del vol_tmp
+ scan_path = scan.path # type: ignore[attr-defined]
+ try:
+ with open_lazy_volume(scan_path) as lazy_tmp:
+ thr = otsu_threshold_chunked(lazy_tmp)
+ except MissingH5DependencyError:
+ thr = otsu_threshold(synth_volume_from_file(scan_path))
+ except (KeyError, ValueError):
+ if not args.allow_synthetic_fallback:
+ raise
+ thr = otsu_threshold(synth_volume_from_file(scan_path))
+ raw_thresholds.append(thr * float(args.threshold_fraction))
if args.method == "classical" and args.smooth_window > 1 and len(raw_thresholds) > 1:
effective_thresholds = [float(x) for x in smooth_thresholds(raw_thresholds, window=args.smooth_window)]
diff --git a/braggtrack/cli/track_dataset.py b/braggtrack/cli/track_dataset.py
index 2472cf2..c73c107 100644
--- a/braggtrack/cli/track_dataset.py
+++ b/braggtrack/cli/track_dataset.py
@@ -10,7 +10,15 @@
import numpy as np
from braggtrack.cli._utils import load_feature_csv, write_csv, write_qc_notebook
+from braggtrack.io import (
+ AxisCalibration,
+ DetectorGeometry,
+ annotate_crystallographic_columns,
+ export_imaged11_flt,
+ export_nexus_process,
+)
from braggtrack.tracking import (
+ GapClosingConfig,
GeometrySemanticCost,
PositionShapeCost,
PrecursorConfig,
@@ -40,6 +48,18 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--gate-chi", type=float, default=float("inf"))
parser.add_argument("--gate-d", type=float, default=float("inf"))
parser.add_argument("--max-cost", type=float, default=float("inf"))
+ parser.add_argument(
+ "--max-gap",
+ type=int,
+ default=0,
+ help="Maximum missing intermediate scans to bridge via gap closing (0 disables)",
+ )
+ parser.add_argument(
+ "--gap-max-cost",
+ type=float,
+ default=25.0,
+ help="Maximum penalized cost allowed when bridging missing scans",
+ )
parser.add_argument(
"--embedding-dir",
default=None,
@@ -80,6 +100,21 @@ def build_parser() -> argparse.ArgumentParser:
default=2,
help="Minimum contiguous scans above threshold to rank as a fracture precursor",
)
+ parser.add_argument(
+ "--detector-json",
+ default=None,
+ help="Optional DetectorGeometry JSON path to annotate (omega, 2theta, eta, d, |Q|) columns",
+ )
+ parser.add_argument(
+ "--export-imaged11",
+ default=None,
+ help="Optional path to write an ImageD11 .flt columnfile of tracked reflections",
+ )
+ parser.add_argument(
+ "--export-nexus",
+ default=None,
+ help="Optional path to write a NeXus NXprocess HDF5 summary file",
+ )
return parser
@@ -100,7 +135,7 @@ def _merge_embeddings(rows: list[dict[str, Any]], emb: dict[int, np.ndarray]) ->
row["embedding"] = emb[lid]
-def _detect_calibrated_axes(scan_dirs: list[Path]) -> bool:
+def _detect_calibration(scan_dirs: list[Path]) -> AxisCalibration | None:
for sd in scan_dirs:
summary_path = sd / "summary.json"
if not summary_path.exists():
@@ -111,8 +146,8 @@ def _detect_calibrated_axes(scan_dirs: list[Path]) -> bool:
continue
cal = meta.get("calibration")
if isinstance(cal, dict) and bool(cal.get("calibrated", False)):
- return True
- return False
+ return AxisCalibration.from_dict(cal)
+ return None
_TRACKING_QC_CODE = [
@@ -204,6 +239,7 @@ def main(argv: list[str] | None = None) -> int:
else:
cost_fn = geo
+ gap_cfg = GapClosingConfig(max_gap=args.max_gap, max_cost=args.gap_max_cost) if args.max_gap > 0 else None
sm_cfg = (
SplitMergeConfig(
max_distance=args.split_max_distance,
@@ -214,17 +250,28 @@ def main(argv: list[str] | None = None) -> int:
if args.detect_events
else None
)
- G = build_tracks(scan_tables, cost_fn=cost_fn, max_cost=args.max_cost, split_merge_config=sm_cfg)
+ G = build_tracks(
+ scan_tables,
+ cost_fn=cost_fn,
+ max_cost=args.max_cost,
+ gap_closing_config=gap_cfg,
+ split_merge_config=sm_cfg,
+ )
metrics = compute_tracking_metrics(G, n_scans=len(scan_tables))
+ active_cal = _detect_calibration(used_scan_dirs)
+ calibrated_axes = active_cal is not None
+ detector_geom = DetectorGeometry.from_json(args.detector_json) if args.detector_json else None
+
track_rows = tracks_to_table(G)
for tr in track_rows:
tr.pop("embedding", None)
+ if detector_geom is not None:
+ track_rows = annotate_crystallographic_columns(track_rows, detector_geom, calibration=active_cal)
write_csv(outdir / "tracks.csv", track_rows)
(outdir / "tracking_metrics.json").write_text(json.dumps(metrics, indent=2))
# Per-grain kinematics export.
- calibrated_axes = _detect_calibrated_axes(used_scan_dirs)
gk = compute_grain_kinematics(track_rows)
kin_summary = summarize_kinematics(gk, n_scans=len(scan_tables), calibrated_axes=calibrated_axes)
write_csv(outdir / "kinematics.csv", kinematics_to_table(gk, include_delta_d=True))
@@ -257,12 +304,19 @@ def main(argv: list[str] | None = None) -> int:
"schema_version": schema_version,
"cost_alpha": args.cost_alpha,
"cost_beta": args.cost_beta,
+ "max_gap": args.max_gap,
"embedding_dir": str(emb_root) if emb_root else None,
"detect_events": bool(args.detect_events),
+ "detector_geometry": detector_geom.to_dict() if detector_geom is not None else None,
}
(outdir / "tracking_summary.json").write_text(json.dumps(summary, indent=2))
write_qc_notebook(outdir / "qc" / "tracking_qc.ipynb", title="Tracking QC", code_source=_TRACKING_QC_CODE)
+ if args.export_imaged11:
+ export_imaged11_flt(track_rows, args.export_imaged11, geometry=detector_geom, calibration=active_cal)
+ if args.export_nexus:
+ export_nexus_process(outdir, args.export_nexus, geometry=detector_geom, calibration=active_cal)
+
print(json.dumps(summary, indent=2))
return 0
diff --git a/braggtrack/io/__init__.py b/braggtrack/io/__init__.py
index c2b0577..9a37a54 100644
--- a/braggtrack/io/__init__.py
+++ b/braggtrack/io/__init__.py
@@ -1,7 +1,10 @@
"""Data I/O helpers for BraggTrack."""
from .beamline import BeamlineAdapter
+from .crystallography import DetectorGeometry, annotate_crystallographic_columns
from .discovery import discover_operando_scans
+from .export import IMAGED11_COLUMNS, export_imaged11_flt, export_nexus_process, read_imaged11_flt
+from .lazy import LazyVolume, open_lazy_volume, otsu_threshold_chunked
from .models import AxisCalibration, AxisSpec, ExperimentSequence, ScanVolumeMeta
from .nexus import (
MissingH5DependencyError,
@@ -15,21 +18,30 @@
from .validation import ValidationIssue, validate_sequence
__all__ = [
+ "AxisCalibration",
+ "AxisSpec",
+ "BeamlineAdapter",
+ "DetectorGeometry",
+ "ExperimentSequence",
+ "IMAGED11_COLUMNS",
+ "LazyVolume",
+ "MissingH5DependencyError",
+ "ScanVolumeMeta",
+ "ValidationIssue",
+ "annotate_crystallographic_columns",
"default_dataset_root",
- "resolve_dataset_root",
- "sample_operando_root",
"discover_operando_scans",
"discover_primary_volume_path",
+ "export_imaged11_flt",
+ "export_nexus_process",
"extract_axis_calibration",
"extract_scan_metadata",
- "summarize_hdf5_tree",
"load_primary_volume",
- "MissingH5DependencyError",
- "AxisCalibration",
- "AxisSpec",
- "ScanVolumeMeta",
- "ExperimentSequence",
- "BeamlineAdapter",
- "ValidationIssue",
+ "open_lazy_volume",
+ "otsu_threshold_chunked",
+ "read_imaged11_flt",
+ "resolve_dataset_root",
+ "sample_operando_root",
+ "summarize_hdf5_tree",
"validate_sequence",
]
diff --git a/braggtrack/io/crystallography.py b/braggtrack/io/crystallography.py
new file mode 100644
index 0000000..d7169f8
--- /dev/null
+++ b/braggtrack/io/crystallography.py
@@ -0,0 +1,146 @@
+"""Transmission diffraction detector geometry and reciprocal-space (2theta, eta, |Q|) mapping."""
+
+from __future__ import annotations
+
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from .models import AxisCalibration
+
+
+@dataclass(frozen=True)
+class DetectorGeometry:
+ """Area-detector diffraction geometry for crystallographic angle & Q conversion.
+
+ Follows standard transmission 3DXRD conventions (e.g. ESRF / ImageD11):
+ incident beam along ``+X_lab``, detector plane at ``X_lab = distance_mm``,
+ horizontal detector coordinate ``x_px`` (BraggTrack ``chi`` axis),
+ vertical detector coordinate ``y_px`` (BraggTrack ``d`` axis), and
+ rotation stack ``z_idx`` (BraggTrack ``mu`` / ``omega`` axis).
+ """
+
+ distance_mm: float = 150.0
+ pixel_size_y_mm: float = 0.075
+ pixel_size_x_mm: float = 0.075
+ beam_center_y_px: float = 1024.0
+ beam_center_x_px: float = 1024.0
+ wavelength_angstrom: float = 0.2898
+ omega_origin_deg: float = 0.0
+ omega_step_deg: float = 0.05
+
+ def __post_init__(self) -> None:
+ if self.distance_mm <= 0.0:
+ raise ValueError("distance_mm must be positive.")
+ if self.pixel_size_y_mm <= 0.0 or self.pixel_size_x_mm <= 0.0:
+ raise ValueError("pixel_size_y_mm and pixel_size_x_mm must be positive.")
+ if self.wavelength_angstrom <= 0.0:
+ raise ValueError("wavelength_angstrom must be positive.")
+
+ def to_dict(self) -> dict[str, float]:
+ return {
+ "distance_mm": float(self.distance_mm),
+ "pixel_size_y_mm": float(self.pixel_size_y_mm),
+ "pixel_size_x_mm": float(self.pixel_size_x_mm),
+ "beam_center_y_px": float(self.beam_center_y_px),
+ "beam_center_x_px": float(self.beam_center_x_px),
+ "wavelength_angstrom": float(self.wavelength_angstrom),
+ "omega_origin_deg": float(self.omega_origin_deg),
+ "omega_step_deg": float(self.omega_step_deg),
+ }
+
+ @classmethod
+ def from_dict(cls, data: dict[str, Any]) -> DetectorGeometry:
+ return cls(
+ distance_mm=float(data.get("distance_mm", 150.0)),
+ pixel_size_y_mm=float(data.get("pixel_size_y_mm", 0.075)),
+ pixel_size_x_mm=float(data.get("pixel_size_x_mm", 0.075)),
+ beam_center_y_px=float(data.get("beam_center_y_px", 1024.0)),
+ beam_center_x_px=float(data.get("beam_center_x_px", 1024.0)),
+ wavelength_angstrom=float(data.get("wavelength_angstrom", 0.2898)),
+ omega_origin_deg=float(data.get("omega_origin_deg", 0.0)),
+ omega_step_deg=float(data.get("omega_step_deg", 0.05)),
+ )
+
+ @classmethod
+ def from_json(cls, path: str | Path) -> DetectorGeometry:
+ raw = json.loads(Path(path).read_text())
+ if not isinstance(raw, dict):
+ raise ValueError("DetectorGeometry JSON must contain an object.")
+ return cls.from_dict(raw)
+
+ def save_json(self, path: str | Path) -> None:
+ out = Path(path)
+ out.parent.mkdir(parents=True, exist_ok=True)
+ out.write_text(json.dumps(self.to_dict(), indent=2), encoding="utf-8")
+
+ def spot_to_scattering(self, z_idx: float, y_px: float, x_px: float) -> dict[str, float]:
+ """Convert detector voxel centroid ``(z_idx, y_px, x_px)`` to ``(omega, 2theta, eta, d, Q)``."""
+ dx_mm = (float(x_px) - self.beam_center_x_px) * self.pixel_size_x_mm
+ dy_mm = (float(y_px) - self.beam_center_y_px) * self.pixel_size_y_mm
+ r_mm = math.hypot(dx_mm, dy_mm)
+ path_len = math.hypot(self.distance_mm, r_mm)
+
+ two_theta_rad = math.atan2(r_mm, self.distance_mm)
+ two_theta_deg = math.degrees(two_theta_rad)
+ eta_deg = math.degrees(math.atan2(dy_mm, dx_mm))
+ omega_deg = self.omega_origin_deg + float(z_idx) * self.omega_step_deg
+ omega_rad = math.radians(omega_deg)
+
+ sin_theta = math.sin(0.5 * two_theta_rad)
+ if sin_theta > 1e-12:
+ d_angstrom = self.wavelength_angstrom / (2.0 * sin_theta)
+ else:
+ d_angstrom = float("inf")
+
+ k_mag = (2.0 * math.pi) / self.wavelength_angstrom
+ qlab_x = k_mag * (self.distance_mm / path_len - 1.0)
+ qlab_y = k_mag * (dx_mm / path_len)
+ qlab_z = k_mag * (dy_mm / path_len)
+
+ cos_w = math.cos(omega_rad)
+ sin_w = math.sin(omega_rad)
+ qx = qlab_x * cos_w + qlab_y * sin_w
+ qy = -qlab_x * sin_w + qlab_y * cos_w
+ qz = qlab_z
+ q_mag = math.sqrt(qx * qx + qy * qy + qz * qz)
+
+ return {
+ "omega_deg": round(omega_deg, 6),
+ "two_theta_deg": round(two_theta_deg, 6),
+ "eta_deg": round(eta_deg, 6),
+ "d_spacing_angstrom": round(d_angstrom, 6) if math.isfinite(d_angstrom) else float("inf"),
+ "qx_inv_angstrom": round(qx, 6),
+ "qy_inv_angstrom": round(qy, 6),
+ "qz_inv_angstrom": round(qz, 6),
+ "q_mag_inv_angstrom": round(q_mag, 6),
+ }
+
+
+def _recover_voxel_centroid(row: dict[str, Any], calibration: AxisCalibration | None) -> tuple[float, float, float]:
+ mu = float(row["centroid_mu"])
+ chi = float(row["centroid_chi"])
+ d_val = float(row["centroid_d"])
+ if calibration is not None and calibration.calibrated:
+ z_idx = (mu - calibration.mu_origin) / calibration.mu_step
+ x_px = (chi - calibration.chi_origin) / calibration.chi_step
+ y_px = (d_val - calibration.d_origin) / calibration.d_step
+ return z_idx, y_px, x_px
+ return mu, d_val, chi
+
+
+def annotate_crystallographic_columns(
+ rows: list[dict[str, Any]],
+ geometry: DetectorGeometry,
+ *,
+ calibration: AxisCalibration | None = None,
+) -> list[dict[str, Any]]:
+ """Return shallow-copied rows augmented with ``(omega_deg, two_theta_deg, eta_deg, d_spacing_angstrom, q_*)``."""
+ enriched: list[dict[str, Any]] = []
+ for row in rows:
+ z_idx, y_px, x_px = _recover_voxel_centroid(row, calibration)
+ scat = geometry.spot_to_scattering(z_idx, y_px, x_px)
+ enriched.append({**row, **scat})
+ return enriched
diff --git a/braggtrack/io/export.py b/braggtrack/io/export.py
new file mode 100644
index 0000000..5b67a2d
--- /dev/null
+++ b/braggtrack/io/export.py
@@ -0,0 +1,256 @@
+"""Interoperability exporters for ImageD11 columnfiles (.flt) and NeXus NXprocess HDF5."""
+
+from __future__ import annotations
+
+import csv
+import json
+import math
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+import braggtrack
+
+from .crystallography import DetectorGeometry, _recover_voxel_centroid
+from .models import AxisCalibration
+from .nexus import _require_h5py
+
+IMAGED11_COLUMNS = (
+ "sc",
+ "fc",
+ "omega",
+ "Number_of_pixels",
+ "avg_intensity",
+ "sum_intensity",
+ "tth",
+ "eta",
+ "d_spacing",
+ "gx",
+ "gy",
+ "gz",
+ "scan_idx",
+ "spot_id",
+ "track_id",
+)
+
+
+def export_imaged11_flt(
+ rows: list[dict[str, Any]],
+ out_path: str | Path,
+ *,
+ geometry: DetectorGeometry | None = None,
+ calibration: AxisCalibration | None = None,
+) -> Path:
+ """Write segmented or tracked Bragg reflections as an ImageD11 ``.flt`` columnfile.
+
+ In ImageD11 convention, ``sc`` is the slow (vertical, ``y``) pixel coordinate,
+ ``fc`` is the fast (horizontal, ``x``) pixel coordinate, and ``(gx, gy, gz)``
+ are reciprocal-space scattering vectors in ``1/d`` (``Q / 2*pi``) units.
+ """
+ dest = Path(out_path)
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ geom = geometry or DetectorGeometry()
+ two_pi = 2.0 * math.pi
+
+ lines: list[str] = [
+ f"# ImageD11 columnfile exported by BraggTrack {braggtrack.__version__}",
+ f"# wavelength = {geom.wavelength_angstrom:.6f}",
+ f"# distance = {geom.distance_mm * 1000.0:.3f}",
+ f"# y_center = {geom.beam_center_y_px:.3f}",
+ f"# z_center = {geom.beam_center_x_px:.3f}",
+ f"# y_size = {geom.pixel_size_y_mm * 1000.0:.3f}",
+ f"# z_size = {geom.pixel_size_x_mm * 1000.0:.3f}",
+ "# " + " ".join(IMAGED11_COLUMNS),
+ ]
+
+ for idx, row in enumerate(rows):
+ z_idx, y_px, x_px = _recover_voxel_centroid(row, calibration)
+ scat = geom.spot_to_scattering(z_idx, y_px, x_px)
+ vox = max(1, int(float(row.get("voxel_count", 1))))
+ sum_int = float(row.get("integrated_intensity", 0.0))
+ avg_int = sum_int / float(vox)
+ scan_idx = int(float(row.get("scan_idx", 0)))
+ spot_id = int(float(row.get("label", row.get("spot_idx", idx + 1))))
+ track_id = int(float(row.get("track_id", -1)))
+ d_val = scat["d_spacing_angstrom"] if math.isfinite(scat["d_spacing_angstrom"]) else 0.0
+ gx = scat["qx_inv_angstrom"] / two_pi
+ gy = scat["qy_inv_angstrom"] / two_pi
+ gz = scat["qz_inv_angstrom"] / two_pi
+
+ lines.append(
+ f"{y_px:10.4f} {x_px:10.4f} {scat['omega_deg']:10.4f} "
+ f"{vox:8d} {avg_int:12.4f} {sum_int:14.4f} "
+ f"{scat['two_theta_deg']:10.5f} {scat['eta_deg']:10.4f} {d_val:10.5f} "
+ f"{gx:11.6f} {gy:11.6f} {gz:11.6f} "
+ f"{scan_idx:6d} {spot_id:6d} {track_id:6d}"
+ )
+
+ dest.write_text("\n".join(lines) + "\n")
+ return dest
+
+
+def read_imaged11_flt(path: str | Path) -> tuple[dict[str, str], list[dict[str, float]]]:
+ """Parse an ImageD11 ``.flt`` columnfile back into header parameters and numeric row dicts."""
+ params: dict[str, str] = {}
+ columns: list[str] = []
+ rows: list[dict[str, float]] = []
+
+ for raw_line in Path(path).read_text().splitlines():
+ line = raw_line.strip()
+ if not line:
+ continue
+ if line.startswith("#"):
+ body = line[1:].strip()
+ if "=" in body:
+ k, v = body.split("=", 1)
+ params[k.strip()] = v.strip()
+ elif body.startswith("sc ") or body.startswith("sc\t"):
+ columns = body.split()
+ continue
+ if not columns:
+ raise ValueError("Missing '# sc fc ...' column header in ImageD11 file.")
+ parts = line.split()
+ if len(parts) != len(columns):
+ raise ValueError(f"Expected {len(columns)} columns, got {len(parts)} in line: {line}")
+ rows.append({col: float(val) for col, val in zip(columns, parts)})
+
+ return params, rows
+
+
+def _read_csv_rows(path: Path) -> list[dict[str, str]]:
+ if not path.exists() or path.stat().st_size == 0:
+ return []
+ with path.open(newline="") as f:
+ return list(csv.DictReader(f))
+
+
+def _write_table_group(grp: Any, h5py: Any, rows: list[dict[str, Any]]) -> None:
+ if not rows:
+ grp.attrs["row_count"] = 0
+ return
+ grp.attrs["row_count"] = len(rows)
+ str_dt = h5py.string_dtype(encoding="utf-8")
+ keys = list(rows[0].keys())
+ for key in keys:
+ vals = [r.get(key, "") for r in rows]
+ # Attempt numeric conversion first (float64 or int64), else UTF-8 strings.
+ is_int = True
+ is_float = True
+ int_vals: list[int] = []
+ float_vals: list[float] = []
+ for v in vals:
+ s = str(v).strip()
+ if s == "" or s.lower() in ("true", "false") or ";" in s:
+ is_int = False
+ is_float = False
+ break
+ try:
+ iv = int(s)
+ int_vals.append(iv)
+ except ValueError:
+ is_int = False
+ try:
+ fv = float(s)
+ float_vals.append(fv)
+ except ValueError:
+ is_float = False
+ break
+ if is_int:
+ grp.create_dataset(key, data=np.asarray(int_vals, dtype=np.int64), compression="gzip")
+ elif is_float:
+ grp.create_dataset(key, data=np.asarray(float_vals, dtype=np.float64), compression="gzip")
+ else:
+ arr = np.asarray([str(v) for v in vals], dtype=object)
+ grp.create_dataset(key, data=arr, dtype=str_dt)
+
+
+def export_nexus_process(
+ artifact_dir: str | Path,
+ out_h5: str | Path,
+ *,
+ geometry: DetectorGeometry | None = None,
+ calibration: AxisCalibration | None = None,
+) -> Path:
+ """Package BraggTrack pipeline/tracking outputs into a NeXus ``NXentry`` / ``NXprocess`` HDF5 file."""
+ h5py = _require_h5py()
+ root = Path(artifact_dir)
+ dest = Path(out_h5)
+ dest.parent.mkdir(parents=True, exist_ok=True)
+
+ # Support either a pipeline root (containing tracking/ & events/) or a direct tracking output dir.
+ trk_dir = root / "tracking" if (root / "tracking").is_dir() else root
+ evt_dir = root / "events" if (root / "events").is_dir() else trk_dir
+
+ track_rows: list[dict[str, Any]] = [dict(r) for r in _read_csv_rows(trk_dir / "tracks.csv")]
+ if geometry is not None and track_rows:
+ from .crystallography import annotate_crystallographic_columns
+
+ track_rows = annotate_crystallographic_columns(track_rows, geometry, calibration=calibration)
+
+ kin_rows = _read_csv_rows(trk_dir / "kinematics.csv")
+ event_rows = _read_csv_rows(evt_dir / "events.csv")
+ prec_rows = _read_csv_rows(evt_dir / "precursors.csv")
+
+ provenance_json = "{}"
+ for cand in (root / "run_manifest.json", trk_dir / "tracking_summary.json"):
+ if cand.exists():
+ provenance_json = cand.read_text()
+ break
+
+ str_dt = h5py.string_dtype(encoding="utf-8")
+ with h5py.File(dest, "w") as h5:
+ h5.attrs["NX_class"] = "NXroot"
+ h5.attrs["default"] = "entry"
+
+ entry = h5.create_group("entry")
+ entry.attrs["NX_class"] = "NXentry"
+ entry.attrs["default"] = "braggtrack"
+ entry.create_dataset("title", data="BraggTrack 4D Diffraction Tracking & Fracture Analysis", dtype=str_dt)
+
+ proc = entry.create_group("braggtrack")
+ proc.attrs["NX_class"] = "NXprocess"
+ proc.create_dataset("program", data="braggtrack", dtype=str_dt)
+ proc.create_dataset("version", data=str(braggtrack.__version__), dtype=str_dt)
+ proc.create_dataset("date", data=datetime.now(timezone.utc).isoformat(), dtype=str_dt)
+ proc.create_dataset("provenance_json", data=provenance_json, dtype=str_dt)
+
+ if geometry is not None:
+ geom_grp = proc.create_group("detector_geometry")
+ geom_grp.attrs["NX_class"] = "NXparameters"
+ for k, v in geometry.to_dict().items():
+ geom_grp.create_dataset(k, data=float(v))
+
+ tracks_grp = proc.create_group("tracks")
+ tracks_grp.attrs["NX_class"] = "NXdata"
+ _write_table_group(tracks_grp, h5py, track_rows)
+
+ kin_grp = proc.create_group("kinematics")
+ kin_grp.attrs["NX_class"] = "NXdata"
+ _write_table_group(kin_grp, h5py, kin_rows)
+
+ evt_grp = proc.create_group("events")
+ evt_grp.attrs["NX_class"] = "NXcollection"
+ _write_table_group(evt_grp, h5py, event_rows)
+
+ prec_grp = proc.create_group("precursors")
+ prec_grp.attrs["NX_class"] = "NXcollection"
+ _write_table_group(prec_grp, h5py, prec_rows)
+
+ # Attach summary metrics if available
+ trk_summary_path = trk_dir / "tracking_summary.json"
+ if trk_summary_path.exists():
+ try:
+ ts = json.loads(trk_summary_path.read_text())
+ metrics_grp = proc.create_group("metrics")
+ metrics_grp.attrs["NX_class"] = "NXparameters"
+ for k, v in ts.items():
+ if isinstance(v, bool):
+ metrics_grp.create_dataset(k, data=bool(v))
+ elif isinstance(v, (int, float)):
+ metrics_grp.create_dataset(k, data=v)
+ except (OSError, json.JSONDecodeError):
+ pass
+
+ return dest
diff --git a/braggtrack/io/lazy.py b/braggtrack/io/lazy.py
new file mode 100644
index 0000000..968a26e
--- /dev/null
+++ b/braggtrack/io/lazy.py
@@ -0,0 +1,144 @@
+"""Chunked and ROI-lazy 3D volume access for multi-gigabyte HDF5 detector stacks."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+from skimage.filters import threshold_otsu
+
+from .nexus import _require_h5py, _resolve_volume_dataset
+
+
+class LazyVolume:
+ """Lazy 3D volume wrapper supporting bounded ROI reads and z-slab streaming.
+
+ Works seamlessly over either an open ``h5py.Dataset`` or an in-memory
+ ``numpy.ndarray`` fallback so downstream stages (:func:`crop_spot_cube`,
+ chunked Otsu thresholding) never materialize full ``float64`` detector
+ cubes unless explicitly requested via ``vol[()]``.
+ """
+
+ def __init__(
+ self,
+ dataset: Any,
+ *,
+ handle: Any | None = None,
+ dataset_path: str | None = None,
+ ) -> None:
+ shape = tuple(int(x) for x in dataset.shape)
+ if len(shape) != 3:
+ if handle is not None:
+ handle.close()
+ raise ValueError(f"LazyVolume requires a 3D dataset, got shape={shape}")
+ self._dataset = dataset
+ self._handle = handle
+ self._shape: tuple[int, int, int] = (shape[0], shape[1], shape[2])
+ self._dtype = np.dtype(dataset.dtype)
+ self._dataset_path = dataset_path
+
+ @classmethod
+ def from_array(cls, array: np.ndarray, *, dataset_path: str | None = None) -> LazyVolume:
+ arr = np.asarray(array)
+ return cls(arr, handle=None, dataset_path=dataset_path)
+
+ @property
+ def shape(self) -> tuple[int, int, int]:
+ return self._shape
+
+ @property
+ def ndim(self) -> int:
+ return 3
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._dtype
+
+ @property
+ def dataset_path(self) -> str | None:
+ return self._dataset_path
+
+ def read_roi(self, z_slice: slice, y_slice: slice, x_slice: slice) -> np.ndarray:
+ """Read a bounded 3D sub-block into memory as ``float64``."""
+ return np.asarray(self._dataset[z_slice, y_slice, x_slice], dtype=np.float64)
+
+ def __getitem__(self, key: Any) -> np.ndarray:
+ return np.asarray(self._dataset[key], dtype=np.float64)
+
+ def iter_z_chunks(self, chunk_z: int = 32) -> Iterator[np.ndarray]:
+ """Yield contiguous ``(dz, ny, nx)`` ``float64`` slabs along axis 0."""
+ step = max(1, int(chunk_z))
+ nz = self._shape[0]
+ for z0 in range(0, nz, step):
+ z1 = min(nz, z0 + step)
+ yield np.asarray(self._dataset[z0:z1, :, :], dtype=np.float64)
+
+ def to_dense(self) -> np.ndarray:
+ """Materialize the full 3D volume as ``float64``."""
+ return np.asarray(self._dataset[()], dtype=np.float64)
+
+ def close(self) -> None:
+ if self._handle is not None:
+ self._handle.close()
+ self._handle = None
+
+ def __enter__(self) -> LazyVolume:
+ return self
+
+ def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None:
+ self.close()
+
+
+def open_lazy_volume(
+ path: str | Path,
+ candidates: list[str] | None = None,
+ *,
+ auto_discover: bool = True,
+) -> LazyVolume:
+ """Open an HDF5 file and return a :class:`LazyVolume` bound to its primary 3D dataset."""
+ h5py = _require_h5py()
+ handle = h5py.File(path, "r")
+ try:
+ ds_path = _resolve_volume_dataset(handle, h5py, candidates=candidates, auto_discover=auto_discover)
+ ds = handle[ds_path]
+ except Exception:
+ handle.close()
+ raise
+ return LazyVolume(ds, handle=handle, dataset_path=ds_path)
+
+
+def otsu_threshold_chunked(
+ volume: LazyVolume | np.ndarray,
+ *,
+ nbins: int = 256,
+ chunk_z: int = 32,
+) -> float:
+ """Compute global Otsu threshold via streaming z-slabs in ``O(chunk)`` memory."""
+ lazy = volume if isinstance(volume, LazyVolume) else LazyVolume.from_array(np.asarray(volume))
+ nz, ny, nx = lazy.shape
+ if nz * ny * nx == 0:
+ raise ValueError("Otsu threshold requires at least one value.")
+
+ vmin = float("inf")
+ vmax = float("-inf")
+ for slab in lazy.iter_z_chunks(chunk_z=chunk_z):
+ s_min = float(np.min(slab))
+ s_max = float(np.max(slab))
+ if s_min < vmin:
+ vmin = s_min
+ if s_max > vmax:
+ vmax = s_max
+
+ if vmin == vmax:
+ return float(vmin)
+
+ edges = np.linspace(vmin, vmax, max(2, int(nbins)) + 1, dtype=np.float64)
+ counts = np.zeros(len(edges) - 1, dtype=np.int64)
+ for slab in lazy.iter_z_chunks(chunk_z=chunk_z):
+ c, _ = np.histogram(slab, bins=edges)
+ counts += c
+
+ bin_centers = 0.5 * (edges[:-1] + edges[1:])
+ return float(threshold_otsu(hist=(counts, bin_centers)))
diff --git a/braggtrack/segmentation/dino_segment.py b/braggtrack/segmentation/dino_segment.py
index d5b48ef..1250b23 100644
--- a/braggtrack/segmentation/dino_segment.py
+++ b/braggtrack/segmentation/dino_segment.py
@@ -61,10 +61,13 @@ def _patch_foreground_masks(
threshold: float,
patch_size: int,
axis: int = 0,
+ grid_hw: tuple[int, int] | None = None,
) -> list[np.ndarray]:
"""Boolean foreground mask at patch resolution for each slice.
A patch is foreground if any of its pixels exceed *threshold*.
+ Handles both fixed-stride grids (``mock``) and resized ViT patch grids
+ (e.g. ``16x16`` ``facebook/dinov2-small`` tokens on ``250x250`` slices).
"""
n_slices = volume.shape[axis]
masks: list[np.ndarray] = []
@@ -72,11 +75,22 @@ def _patch_foreground_masks(
slc = np.take(volume, i, axis=axis)
fg = (slc >= threshold).astype(np.float64)
h, w = slc.shape
- hp = max(1, h // patch_size)
- wp = max(1, w // patch_size)
- cropped = fg[: hp * patch_size, : wp * patch_size]
- blocks = cropped.reshape(hp, patch_size, wp, patch_size)
- masks.append(blocks.max(axis=(1, 3)) > 0)
+ default_hw = (max(1, h // patch_size), max(1, w // patch_size))
+ hp, wp = grid_hw if grid_hw is not None else default_hw
+ if (hp, wp) == default_hw and hp * patch_size <= h and wp * patch_size <= w:
+ cropped = fg[: hp * patch_size, : wp * patch_size]
+ blocks = cropped.reshape(hp, patch_size, wp, patch_size)
+ masks.append(blocks.max(axis=(1, 3)) > 0)
+ else:
+ r_edges = np.linspace(0, h, hp + 1, dtype=int)
+ c_edges = np.linspace(0, w, wp + 1, dtype=int)
+ pooled = np.zeros((hp, wp), dtype=bool)
+ for r_i in range(hp):
+ r0, r1 = r_edges[r_i], max(r_edges[r_i] + 1, r_edges[r_i + 1])
+ for c_i in range(wp):
+ c0, c1 = c_edges[c_i], max(c_edges[c_i] + 1, c_edges[c_i + 1])
+ pooled[r_i, c_i] = bool(np.any(fg[r0:r1, c0:c1] > 0))
+ masks.append(pooled)
return masks
@@ -159,6 +173,13 @@ def _upsample_labels(
patch_size: int,
) -> np.ndarray:
"""Nearest-neighbor upsample patch-resolution labels to pixel resolution."""
+ th, tw = target_shape
+ hp, wp = patch_labels.shape
+ default_hw = (max(1, th // patch_size), max(1, tw // patch_size))
+ if (hp, wp) != default_hw and th > 0 and tw > 0 and hp > 0 and wp > 0:
+ r_idx = (np.arange(th) * hp) // th
+ c_idx = (np.arange(tw) * wp) // tw
+ return patch_labels[np.ix_(r_idx, c_idx)].astype(np.int32)
expanded = np.repeat(np.repeat(patch_labels, patch_size, axis=0), patch_size, axis=1)
out = np.zeros(target_shape, dtype=np.int32)
h = min(expanded.shape[0], target_shape[0])
@@ -302,7 +323,13 @@ def segment_dino(
features, slice_hw = _extract_slice_features(volume, encoder, axis=axis)
# Foreground mask at patch resolution — exclude background from clustering.
- fg_masks = _patch_foreground_masks(volume, threshold, encoder.patch_size, axis=axis)
+ fg_masks = _patch_foreground_masks(
+ volume,
+ threshold,
+ encoder.patch_size,
+ axis=axis,
+ grid_hw=(features.shape[1], features.shape[2]),
+ )
# Global PCA across all foreground patches for a consistent feature space
# (per-slice PCA causes wildly different cluster counts).
diff --git a/braggtrack/semantic/dino.py b/braggtrack/semantic/dino.py
index 4430380..c636052 100644
--- a/braggtrack/semantic/dino.py
+++ b/braggtrack/semantic/dino.py
@@ -25,6 +25,9 @@ def _resolve_backend(requested: BackendName) -> Literal["mock", "torch"]:
return "mock"
if requested == "torch":
return "torch"
+ env_override = os.environ.get("BRAGGTRACK_DINO_BACKEND", "").strip().lower()
+ if env_override in ("mock", "torch"):
+ return "mock" if env_override == "mock" else "torch"
try:
import torch # noqa: F401
except ImportError:
diff --git a/braggtrack/semantic/mips.py b/braggtrack/semantic/mips.py
index e3240c7..11e58af 100644
--- a/braggtrack/semantic/mips.py
+++ b/braggtrack/semantic/mips.py
@@ -5,14 +5,16 @@
from __future__ import annotations
+from typing import Any
+
import numpy as np
def crop_spot_cube(
- volume: np.ndarray,
+ volume: np.ndarray | Any,
labels: np.ndarray,
label_id: int,
- bbox: dict[str, int],
+ bbox: dict[str, Any],
margin: int = 2,
) -> tuple[np.ndarray, np.ndarray]:
"""Crop a padded sub-volume and binary mask for one instance.
diff --git a/braggtrack/tracking/__init__.py b/braggtrack/tracking/__init__.py
index bc5d52d..68a1f62 100644
--- a/braggtrack/tracking/__init__.py
+++ b/braggtrack/tracking/__init__.py
@@ -3,6 +3,7 @@
from .assignment import associate_frames
from .cost import CostFunction, GeometrySemanticCost, PositionShapeCost
from .events import LineageEvent, SplitMergeConfig, events_to_table, resolve_splits_and_merges
+from .gap_closing import GapCloseRecord, GapClosingConfig, resolve_track_gaps
from .kinematics import (
GrainKinematics,
KinematicsSummary,
@@ -24,6 +25,8 @@
__all__ = [
"CostFunction",
+ "GapCloseRecord",
+ "GapClosingConfig",
"GeometrySemanticCost",
"GrainKinematics",
"KinematicsSummary",
@@ -45,6 +48,7 @@
"kinematics_to_table",
"precursors_to_table",
"resolve_splits_and_merges",
+ "resolve_track_gaps",
"summarize_kinematics",
"summary_to_dict",
"tracks_to_table",
diff --git a/braggtrack/tracking/gap_closing.py b/braggtrack/tracking/gap_closing.py
new file mode 100644
index 0000000..c40ce8f
--- /dev/null
+++ b/braggtrack/tracking/gap_closing.py
@@ -0,0 +1,275 @@
+"""Multi-scan gap closing for transient beam dropouts and rocking-clip recovery."""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+from typing import Any
+
+import networkx as nx
+import numpy as np
+from scipy.optimize import linear_sum_assignment
+
+from .cost import CostFunction, GeometrySemanticCost, PositionShapeCost
+from .lifecycle import TrackEvent
+
+
+@dataclass(frozen=True)
+class GapClosingConfig:
+ """Configuration for reconnecting tracks across missing intermediate scans.
+
+ Parameters
+ ----------
+ max_gap : int
+ Maximum number of missing intermediate scans (``1 <= gap_scans <= max_gap``).
+ ``0`` disables gap closing.
+ gap_penalty_per_scan : float
+ Multiplicative penalty per missing scan: ``cost * (1 + gap_penalty_per_scan * gap_scans)``.
+ max_cost : float
+ Upper bound on the penalized association cost for accepting a gap bridge.
+ min_cosine_similarity : float
+ Minimum embedding cosine similarity when both endpoints carry ``embedding``.
+ """
+
+ max_gap: int = 2
+ gap_penalty_per_scan: float = 0.15
+ max_cost: float = 25.0
+ min_cosine_similarity: float = 0.50
+
+
+@dataclass(frozen=True)
+class GapCloseRecord:
+ """Metadata record for an accepted multi-scan gap closure."""
+
+ track_id: int
+ from_scan: int
+ to_scan: int
+ gap_scans: int
+ from_node: str
+ to_node: str
+ cost: float
+
+
+def _cosine_sim(a_raw: Any, b_raw: Any) -> float | None:
+ if a_raw is None or b_raw is None:
+ return None
+ a = np.asarray(a_raw, dtype=np.float64).ravel()
+ b = np.asarray(b_raw, dtype=np.float64).ravel()
+ if a.size == 0 or b.size == 0 or a.shape != b.shape:
+ return None
+ na = float(np.linalg.norm(a))
+ nb = float(np.linalg.norm(b))
+ if na <= 0.0 or nb <= 0.0:
+ return None
+ return float(np.clip(np.dot(a, b) / (na * nb), -1.0, 1.0))
+
+
+def _eval_gap_cost(
+ cost_fn: CostFunction,
+ u_data: dict[str, Any],
+ v_data: dict[str, Any],
+ delta_scans: int,
+ min_cos_sim: float,
+) -> float:
+ """Evaluate gap-normalized cost across ``delta_scans = to_scan - from_scan >= 2``."""
+ sim = _cosine_sim(u_data.get("embedding"), v_data.get("embedding"))
+ if sim is not None and sim < min_cos_sim:
+ return math.inf
+
+ geo: PositionShapeCost | None = None
+ alpha = 1.0
+ beta = 0.0
+ if isinstance(cost_fn, PositionShapeCost):
+ geo = cost_fn
+ elif isinstance(cost_fn, GeometrySemanticCost):
+ geo = cost_fn.geometry
+ alpha = cost_fn.cost_alpha
+ beta = cost_fn.cost_beta
+
+ if geo is not None:
+ dmu = abs(float(u_data["centroid_mu"]) - float(v_data["centroid_mu"]))
+ dchi = abs(float(u_data["centroid_chi"]) - float(v_data["centroid_chi"]))
+ dd = abs(float(u_data["centroid_d"]) - float(v_data["centroid_d"]))
+ step_scale = float(max(1, delta_scans))
+ if (dmu / step_scale) > geo.gate_mu or (dchi / step_scale) > geo.gate_chi or (dd / step_scale) > geo.gate_d:
+ return math.inf
+
+ # Diffusive scaling (dx^2 / delta_t) keeps gap costs comparable to 1-step costs.
+ pos_dist2 = (dmu**2 + dchi**2 + dd**2) / step_scale
+ deig1 = float(u_data.get("eig_1", 0.0)) - float(v_data.get("eig_1", 0.0))
+ deig2 = float(u_data.get("eig_2", 0.0)) - float(v_data.get("eig_2", 0.0))
+ deig3 = float(u_data.get("eig_3", 0.0)) - float(v_data.get("eig_3", 0.0))
+ shape_dist2 = deig1**2 + deig2**2 + deig3**2
+ g_cost = geo.position_weight * pos_dist2 + geo.shape_weight * shape_dist2
+ if beta == 0.0:
+ return alpha * g_cost
+ if sim is None:
+ return math.inf
+ return alpha * g_cost + beta * (1.0 - sim)
+
+ return float(cost_fn(u_data, v_data))
+
+
+def _propagate_track_id_forward(G: nx.DiGraph, start_nid: str, old_tid: int, new_tid: int) -> None:
+ queue = [start_nid]
+ visited: set[str] = set()
+ while queue:
+ curr = queue.pop(0)
+ if curr in visited:
+ continue
+ visited.add(curr)
+ if G.nodes[curr].get("track_id") == old_tid:
+ G.nodes[curr]["track_id"] = new_tid
+ for _, succ in G.out_edges(curr):
+ queue.append(succ)
+
+
+def resolve_track_gaps(
+ G: nx.DiGraph,
+ *,
+ n_scans: int,
+ cost_fn: CostFunction | None = None,
+ config: GapClosingConfig | None = None,
+) -> list[GapCloseRecord]:
+ """Reconnect prematurely terminated tracks to matching births across ``1..max_gap`` missing scans."""
+ cfg = config or GapClosingConfig()
+ if cfg.max_gap <= 0 or G.number_of_nodes() == 0 or n_scans < 3:
+ G.graph.setdefault("gap_closures", [])
+ return []
+
+ active_cost: CostFunction = cost_fn if cost_fn is not None else PositionShapeCost()
+ next_tid = max((int(G.nodes[n]["track_id"]) for n in G.nodes), default=0) + 1
+
+ # Pass 0: Sever forced cross-detector 1-to-1 jump edges created when max_cost=inf
+ # paired a dropping-out track with a distant stray spot on an intermediate frame.
+ nodes_by_scan: dict[int, list[str]] = {}
+ for nid in G.nodes:
+ nodes_by_scan.setdefault(int(G.nodes[nid]["scan_idx"]), []).append(nid)
+
+ sever_threshold = max(cfg.max_cost * 4.0, 100.0)
+
+ def _edge_is_jump(a: str, b: str) -> bool:
+ c_ab = float(active_cost(G.nodes[a], G.nodes[b]))
+ return not math.isfinite(c_ab) or c_ab > sever_threshold
+
+ edges_to_sever: list[tuple[str, str]] = []
+ for u, v, edata in list(G.edges(data=True)):
+ if edata.get("edge_type", "continuation") != "continuation":
+ continue
+ if not _edge_is_jump(u, v):
+ continue
+ su = int(G.nodes[u]["scan_idx"])
+ sv = int(G.nodes[v]["scan_idx"])
+ has_gap_partner = False
+ # Forward check: u can bridge over the jump to w at su + gap + 1
+ for gap in range(1, cfg.max_gap + 1):
+ for w in nodes_by_scan.get(su + gap + 1, []):
+ if G.in_degree(w) == 0 or all(_edge_is_jump(p, w) for p, _ in G.in_edges(w)):
+ gc = _eval_gap_cost(active_cost, G.nodes[u], G.nodes[w], gap + 1, cfg.min_cosine_similarity)
+ if gc * (1.0 + cfg.gap_penalty_per_scan * gap) <= cfg.max_cost * 0.6:
+ has_gap_partner = True
+ break
+ if has_gap_partner:
+ break
+ # Backward check: v can receive a bridge from p at sv - gap - 1
+ if not has_gap_partner:
+ for gap in range(1, cfg.max_gap + 1):
+ for p in nodes_by_scan.get(sv - gap - 1, []):
+ if G.out_degree(p) == 0 or all(_edge_is_jump(p, q) for _, q in G.out_edges(p)):
+ gc = _eval_gap_cost(active_cost, G.nodes[p], G.nodes[v], gap + 1, cfg.min_cosine_similarity)
+ if gc * (1.0 + cfg.gap_penalty_per_scan * gap) <= cfg.max_cost * 0.6:
+ has_gap_partner = True
+ break
+ if has_gap_partner:
+ break
+ if has_gap_partner:
+ edges_to_sever.append((u, v))
+
+ for u, v in edges_to_sever:
+ old_tid = int(G.nodes[u]["track_id"])
+ G.remove_edge(u, v)
+ G.nodes[u]["event"] = TrackEvent.TERMINATED
+ G.nodes[v]["event"] = TrackEvent.BORN
+ _propagate_track_id_forward(G, v, old_tid, next_tid)
+ next_tid += 1
+
+ term_nodes = sorted(
+ [n for n in G.nodes if G.out_degree(n) == 0 and int(G.nodes[n]["scan_idx"]) < n_scans - 1],
+ key=lambda n: (int(G.nodes[n]["scan_idx"]), int(G.nodes[n]["spot_idx"])),
+ )
+ born_nodes = sorted(
+ [n for n in G.nodes if G.in_degree(n) == 0 and int(G.nodes[n]["scan_idx"]) > 0],
+ key=lambda n: (int(G.nodes[n]["scan_idx"]), int(G.nodes[n]["spot_idx"])),
+ )
+
+ n_term = len(term_nodes)
+ n_born = len(born_nodes)
+ if n_term == 0 or n_born == 0:
+ G.graph["gap_closures"] = []
+ return []
+
+ inadmissible = 1e12
+ unmatched_cost = cfg.max_cost + 1e-6
+ aug_matrix = np.full((n_term, n_born + n_term), inadmissible, dtype=np.float64)
+ for i in range(n_term):
+ aug_matrix[i, n_born + i] = unmatched_cost
+
+ for i, u in enumerate(term_nodes):
+ u_data = G.nodes[u]
+ su = int(u_data["scan_idx"])
+ for j, v in enumerate(born_nodes):
+ v_data = G.nodes[v]
+ sv = int(v_data["scan_idx"])
+ gap_scans = sv - su - 1
+ if gap_scans < 1 or gap_scans > cfg.max_gap:
+ continue
+ base = _eval_gap_cost(active_cost, u_data, v_data, gap_scans + 1, cfg.min_cosine_similarity)
+ if not math.isfinite(base):
+ continue
+ penalized = base * (1.0 + cfg.gap_penalty_per_scan * gap_scans)
+ if penalized <= cfg.max_cost:
+ aug_matrix[i, j] = penalized
+
+ row_ind, col_ind = linear_sum_assignment(aug_matrix)
+
+ accepted_pairs: list[tuple[int, int, str, str, float]] = []
+ for r, c in zip(row_ind, col_ind):
+ if c < n_born and aug_matrix[r, c] <= cfg.max_cost:
+ u = term_nodes[int(r)]
+ v = born_nodes[int(c)]
+ su = int(G.nodes[u]["scan_idx"])
+ sv = int(G.nodes[v]["scan_idx"])
+ accepted_pairs.append((su, sv, u, v, float(aug_matrix[r, c])))
+
+ # Process chronologically so multi-gap chains (e.g., 0 -> 2 -> 4) propagate a single track_id.
+ accepted_pairs.sort(key=lambda item: (item[0], item[1], item[2]))
+
+ records: list[GapCloseRecord] = []
+ for su, sv, u, v, cost_val in accepted_pairs:
+ tid = int(G.nodes[u]["track_id"])
+ old_v_tid = int(G.nodes[v]["track_id"])
+ gap_scans = sv - su - 1
+
+ G.add_edge(u, v, edge_type="gap_close", gap_scans=gap_scans, cost=round(cost_val, 4))
+ _propagate_track_id_forward(G, v, old_v_tid, tid)
+
+ G.nodes[u]["event"] = TrackEvent.BORN if G.in_degree(u) == 0 else TrackEvent.CONTINUED
+ if G.out_degree(v) > 0 or sv == n_scans - 1:
+ G.nodes[v]["event"] = TrackEvent.CONTINUED
+ else:
+ G.nodes[v]["event"] = TrackEvent.TERMINATED
+
+ records.append(
+ GapCloseRecord(
+ track_id=tid,
+ from_scan=su,
+ to_scan=sv,
+ gap_scans=gap_scans,
+ from_node=u,
+ to_node=v,
+ cost=round(cost_val, 4),
+ )
+ )
+
+ G.graph["gap_closures"] = records
+ return records
diff --git a/braggtrack/tracking/lifecycle.py b/braggtrack/tracking/lifecycle.py
index d72089c..54b8c8c 100644
--- a/braggtrack/tracking/lifecycle.py
+++ b/braggtrack/tracking/lifecycle.py
@@ -12,6 +12,7 @@
if TYPE_CHECKING:
from .events import SplitMergeConfig
+ from .gap_closing import GapClosingConfig
class TrackEvent(str, enum.Enum):
@@ -30,6 +31,7 @@ def build_tracks(
cost_fn: CostFunction | None = None,
max_cost: float = float("inf"),
*,
+ gap_closing_config: GapClosingConfig | None = None,
split_merge_config: SplitMergeConfig | None = None,
scan_timestamps: dict[int, str | None] | None = None,
) -> nx.DiGraph:
@@ -37,7 +39,8 @@ def build_tracks(
Each node represents a spot observation ``(scan_index, spot_index)``
and carries the full feature dict plus a ``track_id`` and ``event``
- attribute. Edges connect matched spots across consecutive frames, and
+ attribute. Edges connect matched spots across consecutive frames,
+ optional *gap_closing_config* reconnects missing intermediate scans, and
optional *split_merge_config* resolves ``1 -> 2+`` splits and ``2+ -> 1``
merges.
"""
@@ -106,6 +109,16 @@ def build_tracks(
# Still alive at end — leave as CONTINUED (not terminated)
pass
+ if gap_closing_config is not None and gap_closing_config.max_gap > 0:
+ from .gap_closing import resolve_track_gaps
+
+ resolve_track_gaps(
+ G,
+ n_scans=n_scans,
+ cost_fn=active_cost,
+ config=gap_closing_config,
+ )
+
if split_merge_config is not None:
from .events import resolve_splits_and_merges
diff --git a/braggtrack/tracking/metrics.py b/braggtrack/tracking/metrics.py
index 6901b51..3b6ae49 100644
--- a/braggtrack/tracking/metrics.py
+++ b/braggtrack/tracking/metrics.py
@@ -53,7 +53,17 @@ def compute_tracking_metrics(
terminated_count += 1
total_tracks = len(tracks)
- full_length_tracks = sum(1 for obs in tracks.values() if len(obs) >= n_scans)
+ full_length_tracks = sum(
+ 1
+ for obs in tracks.values()
+ if len(obs) >= n_scans
+ or (
+ n_scans > 0
+ and bool(G.graph.get("gap_closures"))
+ and min(int(r["scan_idx"]) for r in obs) == 0
+ and max(int(r["scan_idx"]) for r in obs) >= n_scans - 1
+ )
+ )
fragmentation_ratio = 1.0 - full_length_tracks / total_tracks if total_tracks > 0 else 0.0
# ID-switch rate (requires ground truth).
@@ -89,6 +99,7 @@ def compute_tracking_metrics(
lineage_events = G.graph.get("lineage_events", [])
split_count = sum(1 for ev in lineage_events if getattr(ev, "event_type", None) == "split")
merge_count = sum(1 for ev in lineage_events if getattr(ev, "event_type", None) == "merge")
+ gap_closed_count = len(G.graph.get("gap_closures", []))
return {
"total_tracks": total_tracks,
@@ -97,6 +108,7 @@ def compute_tracking_metrics(
"born_count": born_count,
"continued_count": continued_count,
"terminated_count": terminated_count,
+ "gap_closed_count": gap_closed_count,
"split_count": split_count,
"merge_count": merge_count,
"id_switch_count": id_switch_count,
diff --git a/braggtrack/viz/__init__.py b/braggtrack/viz/__init__.py
new file mode 100644
index 0000000..52705e6
--- /dev/null
+++ b/braggtrack/viz/__init__.py
@@ -0,0 +1,17 @@
+"""Publication plotting and interactive HTML 4D beamtime explorer for BraggTrack."""
+
+from .html_report import build_explorer_payload, generate_html_report
+from .kinematics import plot_kinematics_dashboard
+from .lineage import plot_lineage_dag
+from .mips import plot_scan_projection_overlay, plot_spot_mip_card
+from .precursors import plot_precursor_timeline
+
+__all__ = [
+ "build_explorer_payload",
+ "generate_html_report",
+ "plot_kinematics_dashboard",
+ "plot_lineage_dag",
+ "plot_precursor_timeline",
+ "plot_scan_projection_overlay",
+ "plot_spot_mip_card",
+]
diff --git a/braggtrack/viz/_style.py b/braggtrack/viz/_style.py
new file mode 100644
index 0000000..7306263
--- /dev/null
+++ b/braggtrack/viz/_style.py
@@ -0,0 +1,45 @@
+"""Shared styling and optional-dependency guards for braggtrack.viz."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+# Colorblind-safe Okabe-Ito inspired scientific palette
+OKABE_ITO = (
+ "#0072B2", # blue
+ "#D55E00", # vermilion / split highlight
+ "#009E73", # bluish green / merge highlight
+ "#CC79A7", # reddish purple
+ "#E69F00", # amber / gap-close highlight
+ "#56B4E9", # sky blue
+ "#F0E442", # yellow
+ "#333333", # charcoal
+)
+
+
+def _require_matplotlib() -> tuple[Any, Any]:
+ try:
+ import matplotlib
+ import matplotlib.pyplot as plt
+ except ImportError as exc:
+ raise ImportError(
+ "braggtrack.viz requires matplotlib. Install with `pip install braggtrack[notebook]`."
+ ) from exc
+ return matplotlib, plt
+
+
+def apply_axis_style(ax: Any) -> None:
+ """Apply crisp publication-ready spine and grid styling to a Matplotlib Axes."""
+ ax.spines["top"].set_visible(False)
+ ax.spines["right"].set_visible(False)
+ ax.grid(True, linestyle="--", linewidth=0.5, alpha=0.35)
+
+
+def maybe_save(fig: Any, save_path: str | Path | None, *, dpi: int = 200) -> Path | None:
+ if save_path is None:
+ return None
+ out = Path(save_path)
+ out.parent.mkdir(parents=True, exist_ok=True)
+ fig.savefig(out, dpi=dpi, bbox_inches="tight")
+ return out
diff --git a/braggtrack/viz/html_report.py b/braggtrack/viz/html_report.py
new file mode 100644
index 0000000..49518ef
--- /dev/null
+++ b/braggtrack/viz/html_report.py
@@ -0,0 +1,696 @@
+"""Self-contained offline interactive HTML 4D Lineage, Kinematics & Fracture Explorer."""
+
+from __future__ import annotations
+
+import base64
+import csv
+import html
+import json
+import struct
+import zlib
+from contextlib import ExitStack
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from braggtrack.io import (
+ LazyVolume,
+ MissingH5DependencyError,
+ discover_operando_scans,
+ open_lazy_volume,
+ resolve_dataset_root,
+)
+from braggtrack.semantic.mips import crop_spot_cube, orthogonal_mips
+from braggtrack.tracking.kinematics import compute_grain_kinematics
+
+
+def _encode_rgb_png_base64(rgb: np.ndarray) -> str:
+ """Encode a ``(H, W, 3)`` ``uint8`` RGB array to a base64 PNG data URI using stdlib ``zlib``."""
+ h, w, _ = rgb.shape
+ raw_rows = b"".join(b"\x00" + rgb[y].tobytes() for y in range(h))
+
+ def _chunk(tag: bytes, payload: bytes) -> bytes:
+ crc = zlib.crc32(tag + payload) & 0xFFFFFFFF
+ return struct.pack(">I", len(payload)) + tag + payload + struct.pack(">I", crc)
+
+ ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)
+ png = (
+ b"\x89PNG\r\n\x1a\n"
+ + _chunk(b"IHDR", ihdr)
+ + _chunk(b"IDAT", zlib.compress(raw_rows, 6))
+ + _chunk(b"IEND", b"")
+ )
+ return "data:image/png;base64," + base64.b64encode(png).decode("ascii")
+
+
+def _mip_to_magma_like_rgb(mip: np.ndarray, size: int = 44) -> np.ndarray:
+ """Nearest-neighbor resize a 2D MIP to ``(size, size)`` and apply a high-contrast magma-like LUT."""
+ arr = np.asarray(mip, dtype=np.float64)
+ if arr.size == 0:
+ return np.zeros((size, size, 3), dtype=np.uint8)
+ yi = np.linspace(0, arr.shape[0] - 1, size).astype(np.intp)
+ xi = np.linspace(0, arr.shape[1] - 1, size).astype(np.intp)
+ resized = arr[np.ix_(yi, xi)]
+ lo = float(np.min(resized))
+ hi = float(np.percentile(resized, 99.5)) if resized.size > 1 else float(np.max(resized))
+ if hi <= lo:
+ hi = lo + 1.0
+ t = np.clip((resized - lo) / (hi - lo), 0.0, 1.0)
+ # Smooth 4-stop magma-like ramp: #0b0924 -> #6b1d5c -> #dc493a -> #f9e784
+ r = np.clip(11.0 + 235.0 * (t**0.75), 0.0, 255.0).astype(np.uint8)
+ g = np.clip(9.0 + 222.0 * (t**1.65), 0.0, 255.0).astype(np.uint8)
+ b = np.clip(36.0 + 140.0 * np.sin(np.pi * np.clip(t * 0.85, 0.0, 1.0)) + 80.0 * (t**3), 0.0, 255.0).astype(np.uint8)
+ return np.stack([r, g, b], axis=-1)
+
+
+def _stitch_three_mips_b64(mip_mu: np.ndarray, mip_chi: np.ndarray, mip_d: np.ndarray, tile: int = 40) -> str:
+ """Stitch ``(d, chi) | (mu, d) | (mu, chi)`` MIPs horizontally with 2-px dividers into one base64 PNG."""
+ t1 = _mip_to_magma_like_rgb(mip_mu, size=tile)
+ t2 = _mip_to_magma_like_rgb(mip_chi, size=tile)
+ t3 = _mip_to_magma_like_rgb(mip_d, size=tile)
+ sep = np.full((tile, 2, 3), 30, dtype=np.uint8)
+ strip = np.concatenate([t1, sep, t2, sep, t3], axis=1)
+ return _encode_rgb_png_base64(strip)
+
+
+def _read_csv(path: Path) -> list[dict[str, str]]:
+ if not path.exists() or path.stat().st_size == 0:
+ return []
+ with path.open(newline="") as f:
+ return list(csv.DictReader(f))
+
+
+def _read_json(path: Path) -> Any:
+ if not path.exists():
+ return None
+ try:
+ return json.loads(path.read_text())
+ except (OSError, json.JSONDecodeError):
+ return None
+
+
+def _collect_mip_thumbnails(
+ seg_dir: Path,
+ dataset_root: Path | None,
+ track_rows: list[dict[str, Any]],
+ priority_tids: list[int],
+ *,
+ max_tracks: int,
+) -> dict[str, str]:
+ """Extract compact 3-view MIP strips keyed by ``'{track_id}:{scan_idx}'`` via lazy HDF5 ROI reads."""
+ if not seg_dir.is_dir() or dataset_root is None or not dataset_root.is_dir():
+ return {}
+ try:
+ scans_fs = discover_operando_scans(dataset_root)
+ except Exception:
+ return {}
+ if not scans_fs:
+ return {}
+
+ scan_dirs = sorted(d for d in seg_dir.iterdir() if d.is_dir() and d.name.startswith("scan"))
+ scan_by_name = {s.scan_name: s for s in scans_fs}
+ selected = set(priority_tids[:max_tracks])
+ wanted_by_scan: dict[int, list[dict[str, Any]]] = {}
+ for r in track_rows:
+ tid = int(float(r["track_id"]))
+ if tid in selected and "bbox_min_z" in r and "label" in r:
+ s_idx = int(float(r["scan_idx"]))
+ wanted_by_scan.setdefault(s_idx, []).append(r)
+
+ thumbs: dict[str, str] = {}
+ with ExitStack() as stack:
+ for s_idx, obs_list in wanted_by_scan.items():
+ if s_idx >= len(scan_dirs):
+ continue
+ sd = scan_dirs[s_idx]
+ lab_path = sd / "labels.npz"
+ sf = scan_by_name.get(sd.name)
+ if not lab_path.exists() or sf is None:
+ continue
+ labels = np.load(lab_path)["labels"]
+ try:
+ lazy_vol: LazyVolume = stack.enter_context(open_lazy_volume(sf.path))
+ except (MissingH5DependencyError, KeyError, ValueError, OSError):
+ continue
+ if lazy_vol.shape != labels.shape:
+ continue
+ for obs in obs_list:
+ lid = int(float(obs["label"]))
+ tid = int(float(obs["track_id"]))
+ masked_cube, _ = crop_spot_cube(lazy_vol, labels, lid, obs, margin=2)
+ m_mu, m_chi, m_d = orthogonal_mips(masked_cube)
+ thumbs[f"{tid}:{s_idx}"] = _stitch_three_mips_b64(m_mu, m_chi, m_d, tile=38)
+ return thumbs
+
+
+def build_explorer_payload(
+ artifact_dir: str | Path,
+ *,
+ dataset_root: str | Path | None = None,
+ max_mip_tracks: int = 16,
+) -> dict[str, Any]:
+ """Assemble structured JSON payload for the interactive 4D explorer from pipeline/tracking outputs."""
+ root = Path(artifact_dir)
+ trk_dir = root / "tracking" if (root / "tracking").is_dir() else root
+ evt_dir = root / "events" if (root / "events").is_dir() else trk_dir
+ seg_dir = root / "segmentation" if (root / "segmentation").is_dir() else root.parent / "segmentation"
+
+ manifest = _read_json(root / "run_manifest.json") or {}
+ trk_summary = _read_json(trk_dir / "tracking_summary.json") or {}
+ evt_summary = _read_json(evt_dir / "events_summary.json") or {}
+
+ ds_root: Path | None = None
+ if dataset_root is not None:
+ ds_root = resolve_dataset_root(str(dataset_root))
+ elif manifest.get("dataset_root"):
+ cand = Path(str(manifest["dataset_root"]))
+ if cand.is_dir():
+ ds_root = cand
+
+ raw_tracks = _read_csv(evt_dir / "tracks.csv") or _read_csv(trk_dir / "tracks.csv")
+ raw_events = _read_csv(evt_dir / "events.csv") or _read_csv(trk_dir / "events.csv")
+ raw_precursors = _read_csv(evt_dir / "precursors.csv") or _read_csv(trk_dir / "precursors.csv")
+ raw_prec_obs = _read_csv(evt_dir / "precursor_observations.csv")
+
+ track_rows: list[dict[str, Any]] = []
+ for r in raw_tracks:
+ parsed: dict[str, Any] = {}
+ for k, v in r.items():
+ if k in ("event", "lineage_role", "parent_track_ids"):
+ parsed[k] = v
+ elif k in ("track_id", "scan_idx", "spot_idx", "label", "voxel_count"):
+ parsed[k] = int(float(v)) if v != "" else 0
+ else:
+ try:
+ parsed[k] = float(v)
+ except (TypeError, ValueError):
+ parsed[k] = v
+ track_rows.append(parsed)
+
+ gks = compute_grain_kinematics(track_rows) if track_rows else []
+ gk_by_tid = {g.track_id: g for g in gks}
+
+ prec_obs_by_key: dict[tuple[int, int], dict[str, float]] = {}
+ for r in raw_prec_obs:
+ key = (int(float(r["track_id"])), int(float(r["scan_idx"])))
+ prec_obs_by_key[key] = {
+ "score": float(r.get("precursor_score", 0.0)),
+ "anisotropy_term": float(r.get("anisotropy_term", 0.0)),
+ "spread_term": float(r.get("spread_term", 0.0)),
+ "roi_dispersion_term": float(r.get("roi_dispersion_term", 0.0)),
+ "subpeaks": int(float(r.get("roi_subpeak_count", 1))),
+ }
+
+ events_list: list[dict[str, Any]] = []
+ split_tids: set[int] = set()
+ merge_tids: set[int] = set()
+ for r in raw_events:
+ ptids = [int(x) for x in str(r.get("parent_track_ids", "")).split(";") if x.strip()]
+ ctids = [int(x) for x in str(r.get("child_track_ids", "")).split(";") if x.strip()]
+ etype = str(r.get("event_type", ""))
+ if etype == "split":
+ split_tids.update(ptids)
+ split_tids.update(ctids)
+ elif etype == "merge":
+ merge_tids.update(ptids)
+ merge_tids.update(ctids)
+ events_list.append(
+ {
+ "event_type": etype,
+ "parent_scan": int(float(r.get("scan_idx_from", r.get("parent_scan", 0)))),
+ "child_scan": int(float(r.get("scan_idx_to", r.get("child_scan", 1)))),
+ "parent_track_ids": ptids,
+ "child_track_ids": ctids,
+ "intensity_ratio": round(float(r.get("intensity_ratio", 1.0)), 3),
+ "centroid_distance": round(float(r.get("mean_distance", r.get("centroid_distance", 0.0))), 3),
+ "confidence": round(float(r.get("confidence", 0.0)), 3),
+ }
+ )
+
+ precursors_list: list[dict[str, Any]] = []
+ prec_tids: set[int] = set()
+ for r in raw_precursors:
+ tid = int(float(r["track_id"]))
+ prec_tids.add(tid)
+ precursors_list.append(
+ {
+ "rank": int(float(r.get("rank", 0))),
+ "track_id": tid,
+ "onset_scan": int(float(r.get("onset_scan", 0))),
+ "peak_scan": int(float(r.get("peak_scan", 0))),
+ "persistence_scans": int(float(r.get("persistence_scans", 0))),
+ "peak_score": round(float(r.get("peak_score", 0.0)), 4),
+ "max_anisotropy": round(float(r.get("max_anisotropy", 1.0)), 3),
+ "max_subpeak_count": int(float(r.get("max_subpeak_count", 1))),
+ "confidence": round(float(r.get("confidence", 0.0)), 4),
+ "subsequent_split": str(r.get("subsequent_split", "")).lower() == "true",
+ }
+ )
+
+ by_tid: dict[int, list[dict[str, Any]]] = {}
+ for r in track_rows:
+ by_tid.setdefault(int(r["track_id"]), []).append(r)
+
+ n_scans = int(trk_summary.get("n_scans") or evt_summary.get("n_scans") or 0)
+ if n_scans == 0 and track_rows:
+ n_scans = max(int(r["scan_idx"]) for r in track_rows) + 1
+
+ priority_tids = sorted(
+ by_tid.keys(),
+ key=lambda tid: (
+ 0 if (tid in split_tids or tid in merge_tids or tid in prec_tids) else 1,
+ -len(by_tid[tid]),
+ -max(float(x.get("integrated_intensity", 0.0)) for x in by_tid[tid]),
+ tid,
+ ),
+ )
+ thumbs = _collect_mip_thumbnails(seg_dir, ds_root, track_rows, priority_tids, max_tracks=max_mip_tracks)
+
+ tracks_payload: list[dict[str, Any]] = []
+ for tid in sorted(by_tid.keys(), key=lambda t: (float(by_tid[t][0]["centroid_d"]), t)):
+ obs_sorted = sorted(by_tid[tid], key=lambda x: int(x["scan_idx"]))
+ gk = gk_by_tid.get(tid)
+ scans = [int(x["scan_idx"]) for x in obs_sorted]
+ has_gap = any(scans[i + 1] - scans[i] > 1 for i in range(len(scans) - 1))
+ is_full = n_scans > 0 and min(scans) == 0 and max(scans) >= n_scans - 1
+
+ obs_items: list[dict[str, Any]] = []
+ for idx_o, o in enumerate(obs_sorted):
+ s_i = int(o["scan_idx"])
+ strain_pct = float(gk.strain[idx_o] * 100.0) if gk is not None and idx_o < len(gk.strain) else 0.0
+ miso = (
+ float(gk.misorientation_total[idx_o])
+ if gk is not None and idx_o < len(gk.misorientation_total)
+ else 0.0
+ )
+ aniso = float(gk.anisotropy[idx_o]) if gk is not None and idx_o < len(gk.anisotropy) else 1.0
+ po = prec_obs_by_key.get((tid, s_i), {})
+ obs_items.append(
+ {
+ "scan_idx": s_i,
+ "label": int(o.get("label", 0)),
+ "mu": round(float(o["centroid_mu"]), 4),
+ "chi": round(float(o["centroid_chi"]), 4),
+ "d": round(float(o["centroid_d"]), 4),
+ "intensity": round(float(o.get("integrated_intensity", 0.0)), 2),
+ "voxels": int(o.get("voxel_count", 0)),
+ "strain_pct": round(strain_pct, 4),
+ "misorientation": round(miso, 4),
+ "anisotropy": round(aniso, 3),
+ "precursor_score": round(float(po.get("score", 0.0)), 4),
+ "roi_dispersion_term": round(float(po.get("roi_dispersion_term", 0.0)), 4),
+ "subpeaks": int(po.get("subpeaks", 1)),
+ "mip_b64": thumbs.get(f"{tid}:{s_i}"),
+ }
+ )
+
+ tracks_payload.append(
+ {
+ "track_id": tid,
+ "is_full": is_full,
+ "has_gap": has_gap,
+ "is_split": tid in split_tids,
+ "is_merge": tid in merge_tids,
+ "is_precursor": tid in prec_tids,
+ "max_abs_strain_pct": round(max((abs(x["strain_pct"]) for x in obs_items), default=0.0), 4),
+ "max_precursor": round(max((x["precursor_score"] for x in obs_items), default=0.0), 4),
+ "observations": obs_items,
+ }
+ )
+
+ scan_names = (
+ trk_summary.get("scan_names") or evt_summary.get("scan_names") or [f"scan{i + 1:04d}" for i in range(n_scans)]
+ )
+ return {
+ "title": f"BraggTrack 4D Explorer — {Path(manifest.get('dataset_root') or root).name}",
+ "schema_version": "report.v1",
+ "n_scans": n_scans,
+ "scan_names": scan_names,
+ "calibrated_axes": bool(trk_summary.get("calibrated_axes", False)),
+ "backend": evt_summary.get("backend") or manifest.get("config", {}).get("backend") or "n/a",
+ "model": evt_summary.get("model") or manifest.get("config", {}).get("model") or "n/a",
+ "braggtrack_version": manifest.get("braggtrack_version", "0.3.0"),
+ "total_duration_sec": manifest.get("total_duration_sec"),
+ "kpis": {
+ "total_tracks": len(tracks_payload),
+ "full_length_tracks": sum(1 for t in tracks_payload if t["is_full"]),
+ "gap_closed_count": int(trk_summary.get("gap_closed_count", evt_summary.get("gap_closed_count", 0))),
+ "split_count": sum(1 for e in events_list if e["event_type"] == "split"),
+ "merge_count": sum(1 for e in events_list if e["event_type"] == "merge"),
+ "precursor_count": len(precursors_list),
+ },
+ "tracks": tracks_payload,
+ "events": events_list,
+ "precursors": precursors_list,
+ }
+
+
+_HTML_TEMPLATE = """
+
+
+
+
+__TITLE__
+
+
+
+
+
+
BraggTrack 4D Explorer
+
+
+
+
+
+
+
+
+ 4D Trajectory & Lineage DAG
+ Click any track node to inspect
+
+
+
+
+
+
+ Track Inspector
+
+
+
+
+
+
+
+
+
Split & Merge Lineage Events
+
+
+
+
Persistent Fracture Precursor Candidates
+
+
+
+
+
+
+
+"""
+
+
+def generate_html_report(
+ artifact_dir: str | Path,
+ out_html: str | Path | None = None,
+ *,
+ dataset_root: str | Path | None = None,
+ max_mip_tracks: int = 16,
+) -> Path:
+ """Render a self-contained interactive HTML 4D explorer from a BraggTrack output directory."""
+ root = Path(artifact_dir)
+ dest = Path(out_html) if out_html is not None else (root / "report.html")
+ dest.parent.mkdir(parents=True, exist_ok=True)
+
+ payload = build_explorer_payload(root, dataset_root=dataset_root, max_mip_tracks=max_mip_tracks)
+ safe_json = json.dumps(payload).replace("", "<\\/")
+ html_doc = _HTML_TEMPLATE.replace("__TITLE__", html.escape(str(payload["title"]))).replace(
+ "__PAYLOAD_JSON__", safe_json
+ )
+ dest.write_text(html_doc, encoding="utf-8")
+ return dest
diff --git a/braggtrack/viz/kinematics.py b/braggtrack/viz/kinematics.py
new file mode 100644
index 0000000..2496469
--- /dev/null
+++ b/braggtrack/viz/kinematics.py
@@ -0,0 +1,103 @@
+"""Multi-panel per-grain kinematics dashboard (strain, misorientation, and intensity/anisotropy)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from braggtrack.tracking.kinematics import GrainKinematics, compute_grain_kinematics
+
+from ._style import OKABE_ITO, _require_matplotlib, apply_axis_style, maybe_save
+
+
+def plot_kinematics_dashboard(
+ kinematics_or_tracks: list[GrainKinematics] | list[dict[str, Any]],
+ *,
+ events: list[Any] | None = None,
+ max_tracks: int = 12,
+ calibrated: bool = False,
+ title: str | None = None,
+ save_path: str | Path | None = None,
+) -> tuple[Any, Any]:
+ """Plot 3-panel kinematic evolution: lattice strain, angular misorientation, and intensity/anisotropy."""
+ _, plt = _require_matplotlib()
+
+ if kinematics_or_tracks and isinstance(kinematics_or_tracks[0], GrainKinematics):
+ gks: list[GrainKinematics] = list(kinematics_or_tracks) # type: ignore[arg-type]
+ else:
+ rows = [dict(r) for r in kinematics_or_tracks] # type: ignore[arg-type]
+ if rows and "strain" in rows[0] and "misorientation_total" in rows[0] and "relative_intensity" in rows[0]:
+ # Reconstruct GrainKinematics from kinematics.csv rows or compute from tracks.csv rows
+ gks = compute_grain_kinematics(rows)
+ else:
+ gks = compute_grain_kinematics(rows)
+
+ fig, axes = plt.subplots(1, 3, figsize=(13.5, 4.2))
+ if not gks:
+ for ax in axes:
+ apply_axis_style(ax)
+ fig.suptitle(title or "Per-Grain Kinematics Dashboard (empty)")
+ maybe_save(fig, save_path)
+ return fig, axes
+
+ ranked = sorted(gks, key=lambda g: (-len(g.scan_indices), -float(g.integrated_intensity[0]), g.track_id))[
+ :max_tracks
+ ]
+
+ for idx, gk in enumerate(ranked):
+ color = OKABE_ITO[idx % len(OKABE_ITO)]
+ lbl = f"T{gk.track_id}"
+ scans = gk.scan_indices
+ axes[0].plot(scans, 100.0 * gk.strain, "o-", color=color, linewidth=1.7, markersize=4.5, label=lbl)
+ axes[1].plot(scans, gk.misorientation_total, "o-", color=color, linewidth=1.7, markersize=4.5, label=lbl)
+ axes[2].plot(scans, gk.anisotropy, "s-", color=color, linewidth=1.6, markersize=4.0, label=lbl)
+
+ for ev in events or []:
+ etype = ev.get("event_type") if isinstance(ev, dict) else getattr(ev, "event_type", "")
+ raw_scan = (
+ ev.get("scan_idx_to", ev.get("child_scan", 0))
+ if isinstance(ev, dict)
+ else getattr(ev, "scan_idx_to", getattr(ev, "child_scan", 0))
+ )
+ cscan = int(float(raw_scan))
+ ev_col = OKABE_ITO[1] if etype == "split" else OKABE_ITO[2]
+ for ax in axes:
+ ax.axvline(cscan - 0.5, linestyle=":", linewidth=1.2, color=ev_col, alpha=0.8)
+
+ strain_label = (
+ r"Lattice strain $\varepsilon = \Delta d / d_0$ (%)"
+ if calibrated
+ else r"Relative $d$-shift $\Delta d / d_0$ (%)"
+ )
+ miso_label = (
+ r"Misorientation $\sqrt{\Delta\mu^2 + \Delta\chi^2}$ (deg)"
+ if calibrated
+ else r"Angular shift $\sqrt{\Delta\mu^2 + \Delta\chi^2}$ (px)"
+ )
+
+ axes[0].axhline(0.0, color="#94A3B8", linewidth=0.8)
+ axes[0].set_title("Strain / d-Spacing Evolution", fontsize=10, fontweight="bold")
+ axes[0].set_ylabel(strain_label, fontsize=8.8)
+
+ axes[1].set_title("Lattice Rotation / Misorientation", fontsize=10, fontweight="bold")
+ axes[1].set_ylabel(miso_label, fontsize=8.8)
+
+ axes[2].set_title(r"3D Shape Anisotropy ($\lambda_1 / \lambda_3$)", fontsize=10, fontweight="bold")
+ axes[2].set_ylabel(r"Eigenvalue ratio $\lambda_1 / \lambda_3$", fontsize=8.8)
+
+ for ax in axes:
+ ax.set_xlabel("Scan index", fontsize=9)
+ apply_axis_style(ax)
+
+ if len(ranked) <= 10:
+ axes[0].legend(fontsize=7.5, ncol=2, frameon=True)
+
+ fig.suptitle(
+ title or f"Operando Grain Kinematics ({'Calibrated Physical Units' if calibrated else 'Detector Units'})",
+ fontsize=11,
+ fontweight="bold",
+ y=1.02,
+ )
+ fig.tight_layout()
+ maybe_save(fig, save_path)
+ return fig, axes
diff --git a/braggtrack/viz/lineage.py b/braggtrack/viz/lineage.py
new file mode 100644
index 0000000..4dff72a
--- /dev/null
+++ b/braggtrack/viz/lineage.py
@@ -0,0 +1,216 @@
+"""Publication-grade 4D trajectory and split/merge lineage DAG visualization."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import networkx as nx
+import numpy as np
+
+from braggtrack.tracking.lifecycle import tracks_to_table
+
+from ._style import OKABE_ITO, _require_matplotlib, apply_axis_style, maybe_save
+
+
+def _parse_int_list(val: Any) -> list[int]:
+ if val is None or val == "":
+ return []
+ if isinstance(val, (list, tuple)):
+ return [int(x) for x in val]
+ return [int(tok) for tok in str(val).split(";") if tok.strip()]
+
+
+def plot_lineage_dag(
+ tracks_or_graph: nx.DiGraph | list[dict[str, Any]],
+ *,
+ events: list[Any] | None = None,
+ max_tracks: int = 40,
+ title: str | None = None,
+ save_path: str | Path | None = None,
+) -> tuple[Any, Any]:
+ """Plot a scan-layered 4D trajectory & lineage DAG highlighting splits, merges, and gap closures."""
+ _, plt = _require_matplotlib()
+
+ if isinstance(tracks_or_graph, nx.DiGraph):
+ rows = tracks_to_table(tracks_or_graph)
+ if events is None:
+ events = list(tracks_or_graph.graph.get("lineage_events", []))
+ else:
+ rows = [dict(r) for r in tracks_or_graph]
+
+ fig, ax = plt.subplots(figsize=(9.5, 5.2))
+ if not rows:
+ ax.set_title(title or "4D Trajectory Lineage DAG (empty)")
+ apply_axis_style(ax)
+ maybe_save(fig, save_path)
+ return fig, ax
+
+ by_track: dict[int, list[dict[str, Any]]] = {}
+ for r in rows:
+ by_track.setdefault(int(float(r["track_id"])), []).append(r)
+ for tid in by_track:
+ by_track[tid].sort(key=lambda r: int(float(r["scan_idx"])))
+
+ # Prioritize tracks involved in split/merge events, then longest & brightest tracks.
+ event_records: list[dict[str, Any]] = []
+ involved_tids: set[int] = set()
+ for ev in events or []:
+ if isinstance(ev, dict):
+ etype = str(ev.get("event_type", ""))
+ pscan = int(float(ev.get("scan_idx_from", ev.get("parent_scan", 0))))
+ cscan = int(float(ev.get("scan_idx_to", ev.get("child_scan", pscan + 1))))
+ ptids = _parse_int_list(ev.get("parent_track_ids"))
+ ctids = _parse_int_list(ev.get("child_track_ids"))
+ else:
+ etype = str(getattr(ev, "event_type", ""))
+ pscan = int(getattr(ev, "scan_idx_from", getattr(ev, "parent_scan", 0)))
+ cscan = int(getattr(ev, "scan_idx_to", getattr(ev, "child_scan", pscan + 1)))
+ ptids = [int(x) for x in getattr(ev, "parent_track_ids", ())]
+ ctids = [int(x) for x in getattr(ev, "child_track_ids", ())]
+ event_records.append(
+ {
+ "event_type": etype,
+ "parent_scan": pscan,
+ "child_scan": cscan,
+ "parent_track_ids": ptids,
+ "child_track_ids": ctids,
+ }
+ )
+ involved_tids.update(ptids)
+ involved_tids.update(ctids)
+
+ ranked_tids = sorted(
+ by_track.keys(),
+ key=lambda tid: (
+ 0 if tid in involved_tids else 1,
+ -len(by_track[tid]),
+ -max(float(r.get("integrated_intensity", 0.0)) for r in by_track[tid]),
+ tid,
+ ),
+ )[:max_tracks]
+ selected_set = set(ranked_tids)
+
+ # Order vertical lanes by initial centroid_d so spatially nearby fragments sit adjacent.
+ ordered_tids = sorted(ranked_tids, key=lambda tid: (float(by_track[tid][0]["centroid_d"]), tid))
+ lane_y = {tid: float(idx) for idx, tid in enumerate(ordered_tids)}
+
+ node_xy: dict[tuple[int, int], tuple[float, float]] = {}
+ all_x: list[float] = []
+ all_y: list[float] = []
+ all_strain: list[float] = []
+
+ gap_labeled = False
+ cont_labeled = False
+ for tid in ordered_tids:
+ obs = by_track[tid]
+ y_val = lane_y[tid]
+ d0 = float(obs[0]["centroid_d"])
+ scans = [int(float(r["scan_idx"])) for r in obs]
+ for r in obs:
+ s_idx = int(float(r["scan_idx"]))
+ d_cur = float(r["centroid_d"])
+ strain_pct = 100.0 * (d_cur - d0) / d0 if abs(d0) > 1e-9 else 0.0
+ node_xy[(tid, s_idx)] = (float(s_idx), y_val)
+ all_x.append(float(s_idx))
+ all_y.append(y_val)
+ all_strain.append(strain_pct)
+
+ for i in range(len(scans) - 1):
+ s0, s1 = scans[i], scans[i + 1]
+ if s1 - s0 > 1:
+ ax.plot(
+ [s0, s1],
+ [y_val, y_val],
+ linestyle="--",
+ linewidth=1.8,
+ color=OKABE_ITO[4],
+ label="Gap-closed bridge" if not gap_labeled else None,
+ zorder=2,
+ )
+ gap_labeled = True
+ else:
+ ax.plot(
+ [s0, s1],
+ [y_val, y_val],
+ linestyle="-",
+ linewidth=1.8,
+ color="#64748B",
+ alpha=0.75,
+ label="Continuation" if not cont_labeled else None,
+ zorder=2,
+ )
+ cont_labeled = True
+
+ split_labeled = False
+ merge_labeled = False
+ for ev in event_records:
+ etype = ev["event_type"]
+ pscan = ev["parent_scan"]
+ cscan = ev["child_scan"]
+ for ptid in ev["parent_track_ids"]:
+ for ctid in ev["child_track_ids"]:
+ if ptid not in selected_set or ctid not in selected_set:
+ continue
+ p_xy = node_xy.get((ptid, pscan), (float(pscan), lane_y[ptid]))
+ c_xy = node_xy.get((ctid, cscan), (float(cscan), lane_y[ctid]))
+ color = OKABE_ITO[1] if etype == "split" else OKABE_ITO[2]
+ lbl = None
+ if etype == "split" and not split_labeled:
+ lbl = "1 -> 2+ Split"
+ split_labeled = True
+ elif etype == "merge" and not merge_labeled:
+ lbl = "2+ -> 1 Merge"
+ merge_labeled = True
+ ax.plot(
+ [p_xy[0], c_xy[0]],
+ [p_xy[1], c_xy[1]],
+ color=color,
+ linewidth=2.4,
+ alpha=0.92,
+ label=lbl,
+ zorder=3,
+ )
+ ax.scatter(
+ [c_xy[0] if etype == "split" else p_xy[0]],
+ [c_xy[1] if etype == "split" else p_xy[1]],
+ marker="D" if etype == "split" else "s",
+ s=58,
+ color=color,
+ edgecolor="#FFFFFF",
+ linewidth=0.8,
+ zorder=5,
+ )
+
+ vmax = max(0.05, float(np.max(np.abs(all_strain)))) if all_strain else 0.1
+ sc = ax.scatter(
+ all_x,
+ all_y,
+ c=all_strain,
+ cmap="coolwarm",
+ vmin=-vmax,
+ vmax=vmax,
+ s=42,
+ edgecolors="#1E293B",
+ linewidths=0.6,
+ zorder=4,
+ )
+ cbar = fig.colorbar(sc, ax=ax, pad=0.02)
+ cbar.set_label(r"Relative $d$-shift $\Delta d / d_0$ (%)", fontsize=9)
+
+ ax.set_yticks([lane_y[tid] for tid in ordered_tids])
+ ax.set_yticklabels([f"T{tid}" for tid in ordered_tids], fontsize=8)
+ max_scan = int(max(all_x)) if all_x else 0
+ ax.set_xticks(list(range(max_scan + 1)))
+ ax.set_xlabel("Scan index", fontsize=9.5)
+ ax.set_ylabel("Grain trajectory lane", fontsize=9.5)
+ ax.set_title(
+ title or f"4D Bragg Reflection Lineage DAG ({len(ordered_tids)} tracks)", fontsize=11, fontweight="bold"
+ )
+ apply_axis_style(ax)
+ handles, labels_legend = ax.get_legend_handles_labels()
+ if handles:
+ ax.legend(loc="upper left", fontsize=8, frameon=True)
+ fig.tight_layout()
+ maybe_save(fig, save_path)
+ return fig, ax
diff --git a/braggtrack/viz/mips.py b/braggtrack/viz/mips.py
new file mode 100644
index 0000000..8328f37
--- /dev/null
+++ b/braggtrack/viz/mips.py
@@ -0,0 +1,104 @@
+"""Orthogonal 3-view MIP spot cards and full-scan projection overlays."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from braggtrack.semantic.mips import crop_spot_cube, orthogonal_mips
+
+from ._style import _require_matplotlib, apply_axis_style, maybe_save
+
+
+def plot_spot_mip_card(
+ volume: np.ndarray | Any,
+ labels: np.ndarray,
+ label_id: int,
+ bbox: dict[str, Any],
+ *,
+ margin: int = 3,
+ title: str | None = None,
+ save_path: str | Path | None = None,
+) -> tuple[Any, Any]:
+ """Plot a 3-panel orthogonal MIP card ``(d, chi)``, ``(mu, d)``, ``(mu, chi)`` for one 3D reflection."""
+ _, plt = _require_matplotlib()
+ masked_cube, mask = crop_spot_cube(volume, labels, int(label_id), bbox, margin=margin)
+ mip_mu, mip_chi, mip_d = orthogonal_mips(masked_cube)
+ mask_mu, mask_chi, mask_d = orthogonal_mips(mask)
+
+ fig, axes = plt.subplots(1, 3, figsize=(10.5, 3.4))
+ panels = [
+ (mip_mu, mask_mu, r"$\mu$-collapsed $(d, \chi)$", r"$\chi$ (px)", r"$d$ (px)"),
+ (mip_chi, mask_chi, r"$\chi$-collapsed $(\mu, d)$", r"$d$ (px)", r"$\mu$ (frame)"),
+ (mip_d, mask_d, r"$d$-collapsed $(\mu, \chi)$", r"$\chi$ (px)", r"$\mu$ (frame)"),
+ ]
+
+ for ax, (mip, msk, subtitle, xlabel, ylabel) in zip(axes, panels):
+ im = ax.imshow(mip, cmap="magma", origin="lower", aspect="auto")
+ if msk.shape[0] >= 2 and msk.shape[1] >= 2 and np.any(msk > 0) and np.any(msk == 0):
+ ax.contour(msk, levels=[0.5], colors=["#56B4E9"], linewidths=1.1)
+ ax.set_title(subtitle, fontsize=9.5, fontweight="bold")
+ ax.set_xlabel(xlabel, fontsize=8.5)
+ ax.set_ylabel(ylabel, fontsize=8.5)
+ fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
+
+ vox = int(np.sum(mask > 0))
+ tot_int = float(np.sum(masked_cube))
+ header = title or f"Spot #{label_id} | {vox} voxels | I = {tot_int:.1f}"
+ fig.suptitle(header, fontsize=10.5, fontweight="bold", y=1.02)
+ fig.tight_layout()
+ maybe_save(fig, save_path)
+ return fig, axes
+
+
+def plot_scan_projection_overlay(
+ volume: np.ndarray | Any,
+ labels: np.ndarray,
+ feature_rows: list[dict[str, Any]],
+ *,
+ max_labels: int = 25,
+ title: str | None = None,
+ save_path: str | Path | None = None,
+) -> tuple[Any, Any]:
+ """Plot a full-scan ``(d, chi)`` MIP with instance contours and ranked spot IDs."""
+ _, plt = _require_matplotlib()
+ vol_arr = volume[()] if not isinstance(volume, np.ndarray) else volume
+ mip_dc = np.max(np.asarray(vol_arr, dtype=np.float64), axis=0)
+ disp = np.log1p(np.maximum(0.0, mip_dc))
+ fg_proj = np.max(labels > 0, axis=0).astype(np.float64)
+
+ fig, ax = plt.subplots(figsize=(7.5, 6.0))
+ im = ax.imshow(disp, cmap="inferno", origin="lower", aspect="auto")
+ if fg_proj.shape[0] >= 2 and fg_proj.shape[1] >= 2 and np.any(fg_proj > 0) and np.any(fg_proj == 0):
+ ax.contour(fg_proj, levels=[0.5], colors=["#56B4E9"], linewidths=0.8, alpha=0.8)
+
+ ranked = sorted(feature_rows, key=lambda r: float(r.get("integrated_intensity", 0.0)), reverse=True)[:max_labels]
+ for row in ranked:
+ if "bbox_min_x" in row and "bbox_max_x" in row:
+ cx = 0.5 * (float(row["bbox_min_x"]) + float(row["bbox_max_x"]))
+ cy = 0.5 * (float(row["bbox_min_y"]) + float(row["bbox_max_y"]))
+ else:
+ cx = float(row["centroid_chi"])
+ cy = float(row["centroid_d"])
+ lid = int(float(row["label"]))
+ ax.plot(cx, cy, marker="+", color="#00E5FF", markersize=6, markeredgewidth=1.2)
+ ax.annotate(
+ f"#{lid}",
+ (cx, cy),
+ xytext=(4, 4),
+ textcoords="offset points",
+ fontsize=7.5,
+ color="#FFFFFF",
+ bbox={"boxstyle": "round,pad=0.15", "fc": "#111827", "ec": "none", "alpha": 0.65},
+ )
+
+ ax.set_title(title or f"Detector Projection (log1p MIP) — {len(feature_rows)} segmented reflections", fontsize=10.5)
+ ax.set_xlabel(r"Detector $\chi$ / fast pixel", fontsize=9)
+ ax.set_ylabel(r"Detector $d$ / slow pixel", fontsize=9)
+ apply_axis_style(ax)
+ fig.colorbar(im, ax=ax, label="log(1 + max intensity)")
+ fig.tight_layout()
+ maybe_save(fig, save_path)
+ return fig, ax
diff --git a/braggtrack/viz/precursors.py b/braggtrack/viz/precursors.py
new file mode 100644
index 0000000..771fcad
--- /dev/null
+++ b/braggtrack/viz/precursors.py
@@ -0,0 +1,149 @@
+"""Persistent 4D fracture precursor heatmap and component decomposition plots."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from braggtrack.tracking.precursors import ObservationPrecursorScore, PrecursorCandidate
+
+from ._style import OKABE_ITO, _require_matplotlib, apply_axis_style, maybe_save
+
+
+def _obs_to_dict(obs: ObservationPrecursorScore | dict[str, Any]) -> dict[str, Any]:
+ if isinstance(obs, dict):
+ return {
+ "track_id": int(float(obs["track_id"])),
+ "scan_idx": int(float(obs["scan_idx"])),
+ "anisotropy_term": float(obs.get("anisotropy_term", 0.0)),
+ "spread_term": float(obs.get("spread_term", 0.0)),
+ "roi_dispersion_term": float(obs.get("roi_dispersion_term", 0.0)),
+ "roi_subpeak_count": int(float(obs.get("roi_subpeak_count", 1))),
+ "precursor_score": float(obs.get("precursor_score", 0.0)),
+ }
+ return {
+ "track_id": obs.track_id,
+ "scan_idx": obs.scan_idx,
+ "anisotropy_term": obs.anisotropy_term,
+ "spread_term": obs.spread_term,
+ "roi_dispersion_term": obs.roi_dispersion_term,
+ "roi_subpeak_count": obs.roi_subpeak_count,
+ "precursor_score": obs.precursor_score,
+ }
+
+
+def plot_precursor_timeline(
+ obs_scores: list[ObservationPrecursorScore] | list[dict[str, Any]],
+ *,
+ candidates: list[PrecursorCandidate] | list[dict[str, Any]] | None = None,
+ max_tracks: int = 15,
+ score_threshold: float = 0.40,
+ title: str | None = None,
+ save_path: str | Path | None = None,
+) -> tuple[Any, Any]:
+ """Plot a track-by-scan fracture precursor heatmap and top-candidate component decomposition."""
+ _, plt = _require_matplotlib()
+ rows = [_obs_to_dict(x) for x in obs_scores]
+
+ fig, axes = plt.subplots(1, 2, figsize=(12.0, 4.4), gridspec_kw={"width_ratios": [1.15, 1.0]})
+ if not rows:
+ for ax in axes:
+ apply_axis_style(ax)
+ fig.suptitle(title or "Fracture Precursor Timeline (no observations)")
+ maybe_save(fig, save_path)
+ return fig, axes
+
+ by_track: dict[int, list[dict[str, Any]]] = {}
+ max_scan = 0
+ for r in rows:
+ by_track.setdefault(r["track_id"], []).append(r)
+ if r["scan_idx"] > max_scan:
+ max_scan = r["scan_idx"]
+
+ split_tids: set[int] = set()
+ top_tid: int | None = None
+ for idx_c, c in enumerate(candidates or []):
+ tid_c = int(float(c["track_id"])) if isinstance(c, dict) else int(c.track_id)
+ sub_split = (
+ str(c.get("subsequent_split", "")).lower() == "true" if isinstance(c, dict) else bool(c.subsequent_split)
+ )
+ if idx_c == 0:
+ top_tid = tid_c
+ if sub_split:
+ split_tids.add(tid_c)
+
+ ranked_tids = sorted(
+ by_track.keys(),
+ key=lambda tid: (-max(x["precursor_score"] for x in by_track[tid]), tid),
+ )[:max_tracks]
+ if top_tid is None and ranked_tids:
+ top_tid = ranked_tids[0]
+
+ mat = np.full((len(ranked_tids), max_scan + 1), np.nan, dtype=np.float64)
+ for r_i, tid in enumerate(ranked_tids):
+ for r in by_track[tid]:
+ mat[r_i, r["scan_idx"]] = r["precursor_score"]
+
+ im = axes[0].imshow(mat, aspect="auto", cmap="YlOrRd", vmin=0.0, vmax=1.0, interpolation="nearest")
+ axes[0].set_yticks(list(range(len(ranked_tids))))
+ axes[0].set_yticklabels(
+ [f"T{tid} *" if tid in split_tids else f"T{tid}" for tid in ranked_tids],
+ fontsize=8.5,
+ )
+ axes[0].set_xticks(list(range(max_scan + 1)))
+ axes[0].set_xlabel("Scan index", fontsize=9)
+ axes[0].set_ylabel("Track ID (* = subsequent split)", fontsize=9)
+ axes[0].set_title("Precursor Score Heatmap", fontsize=10, fontweight="bold")
+ fig.colorbar(im, ax=axes[0], fraction=0.046, pad=0.04, label="Precursor score")
+
+ # Right panel: component breakdown for top candidate
+ if top_tid is not None and top_tid in by_track:
+ top_obs = sorted(by_track[top_tid], key=lambda x: x["scan_idx"])
+ scans = [x["scan_idx"] for x in top_obs]
+ axes[1].plot(
+ scans,
+ [x["anisotropy_term"] for x in top_obs],
+ "o--",
+ color=OKABE_ITO[0],
+ linewidth=1.4,
+ label="Shape anisotropy",
+ )
+ axes[1].plot(
+ scans,
+ [x["spread_term"] for x in top_obs],
+ "s--",
+ color=OKABE_ITO[2],
+ linewidth=1.4,
+ label="Covariance broadening",
+ )
+ axes[1].plot(
+ scans,
+ [x["roi_dispersion_term"] for x in top_obs],
+ "^--",
+ color=OKABE_ITO[3],
+ linewidth=1.4,
+ label="3D sub-peaks + DINO dispersion",
+ )
+ axes[1].plot(
+ scans,
+ [x["precursor_score"] for x in top_obs],
+ "o-",
+ color=OKABE_ITO[1],
+ linewidth=2.2,
+ label=f"Composite score (T{top_tid})",
+ )
+ axes[1].axhline(score_threshold, linestyle=":", color="#475569", linewidth=1.2, label="Threshold")
+ axes[1].set_ylim(-0.04, 1.06)
+ axes[1].set_xticks(list(range(max_scan + 1)))
+ axes[1].set_xlabel("Scan index", fontsize=9)
+ axes[1].set_ylabel("Normalized term [0, 1]", fontsize=9)
+ axes[1].set_title(f"Top Precursor Decomposition — Track T{top_tid}", fontsize=10, fontweight="bold")
+ axes[1].legend(fontsize=7.8, loc="best", frameon=True)
+ apply_axis_style(axes[1])
+
+ fig.suptitle(title or "Persistent 4D Fracture Precursor Analysis", fontsize=11, fontweight="bold", y=1.02)
+ fig.tight_layout()
+ maybe_save(fig, save_path)
+ return fig, axes
diff --git a/docs/architecture.md b/docs/architecture.md
index 7f9e002..c8efe27 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -6,17 +6,18 @@ This document describes how the repository is structured, how data flows through
```text
BraggTrack/
-├── braggtrack/ # Installable Python package (v0.2.0)
-│ ├── io/ # Discovery, ESRF-ID03 / NeXus HDF5 auto-discovery, AxisCalibration, validation
-│ ├── segmentation/ # Classical 3D LoG+watershed, cross-scan threshold smoothing, experimental DINO segmenter, features
+├── braggtrack/ # Installable Python package (v0.3.0)
+│ ├── io/ # Discovery, LazyVolume streaming, AxisCalibration, DetectorGeometry, ImageD11 & NeXus export
+│ ├── segmentation/ # Classical 3D LoG+watershed, chunked Otsu + threshold smoothing, experimental DINO segmenter
│ ├── semantic/ # Orthogonal (mu, chi, d) MIPs + frozen ViT / patch encoders (DINOv2 default, DINOv3, mock)
-│ ├── tracking/ # Costs, Hungarian assignment, kinematics, split/merge lineage DAG, fracture precursors
-│ └── cli/ # Thin argparse entry points + reproducible run_pipeline orchestrator
+│ ├── tracking/ # Costs, Hungarian assignment, multi-scan gap closing, kinematics, split/merge DAG, precursors
+│ ├── viz/ # Publication plots (MIPs, Lineage DAG, Strain fans, Precursors) + interactive HTML 4D Explorer
+│ └── cli/ # Thin argparse entry points (9 CLIs) + reproducible run_pipeline orchestrator
├── data/
│ └── sample_operando/ # Bundled ESRF-ID03 Ni-rich cathode operando scans (scan0001..scan0003)
├── notebooks/ # End-to-end demo and DINO vs. classical segmentation notebooks
-├── docs/ # Architecture & coding practices
-├── scripts/ # Acceptance gates, semantic ablation with bootstrap CIs, ci_report.py
+├── docs/ # Architecture, quantitative benchmarks, figures, and coding practices
+├── scripts/ # 9 acceptance gates, 5-scenario benchmark suite, semantic ablation, ci_report.py
├── tests/ # Unit and acceptance test suite
└── artifacts/ # Generated outputs (gitignored)
```
@@ -25,11 +26,12 @@ BraggTrack/
| Layer | Responsibility | Key Types / Functions | Depends on |
|---|---|---|---|
-| **io** | Discover scans, auto-discover 3D HDF5 cubes, extract timestamps & `AxisCalibration`, validate sequences | `ScanVolumeMeta`, `AxisCalibration`, `load_primary_volume`, `BeamlineAdapter` | `numpy`, `h5py` |
+| **io** | Discover scans, stream `LazyVolume` HDF5 slices & chunked Otsu histograms, calibrate `(mu, chi, d)` & `(2theta, eta, d, \|Q\|)`, export `ImageD11` `.flt` & NeXus `.nxs` | `LazyVolume`, `otsu_threshold_chunked`, `AxisCalibration`, `DetectorGeometry`, `export_imaged11_flt`, `export_nexus_process` | `numpy`, `h5py` |
| **segmentation** | Multi-scan threshold smoothing, 3D LoG + `h`-maxima + watershed (or experimental DINO + HDBSCAN), physical/voxel feature table | `segment_classical`, `smooth_thresholds`, `segment_dino`, `extract_instance_table` | `numpy`, `scipy`, `scikit-image`, `scikit-learn` |
-| **semantic** | Cropped spot cubes, orthogonal MIPs, multi-view global embeddings & dense patch-token grids | `crop_spot_cube`, `orthogonal_mips`, `make_multiview_encoder`, `make_patch_encoder` | `numpy`, optional `torch` + `transformers` |
-| **tracking** | Vectorised geometry + semantic cost, Hungarian assignment, per-grain kinematics, `1 -> 2+` split & `2+ -> 1` merge lineage resolution, persistent fracture precursors | `PositionShapeCost`, `GeometrySemanticCost`, `build_tracks`, `compute_grain_kinematics`, `resolve_splits_and_merges`, `compute_fracture_precursors` | `numpy`, `scipy`, `networkx` |
-| **cli** | Parse args/JSON config, orchestrate stages, write CSV/JSON/NPZ artifacts and SHA-256 `run_manifest.json` | `segment_dataset`, `embed_dataset`, `track_dataset`, `detect_events`, `run_pipeline` | All of the above |
+| **semantic** | Lazy ROI spot cropping, orthogonal MIPs, multi-view global embeddings & dense patch-token grids | `crop_spot_cube`, `orthogonal_mips`, `make_multiview_encoder`, `make_patch_encoder` | `numpy`, optional `torch` + `transformers` |
+| **tracking** | Vectorised geometry + semantic cost, Hungarian assignment, multi-scan gap closing (`max_gap`), per-grain kinematics, `1 -> 2+` split & `2+ -> 1` merge lineage, fracture precursors | `PositionShapeCost`, `GeometrySemanticCost`, `resolve_track_gaps`, `build_tracks`, `compute_grain_kinematics`, `resolve_splits_and_merges`, `compute_fracture_precursors` | `numpy`, `scipy`, `networkx` |
+| **viz** | Publication figures and zero-CDN offline interactive HTML 4D Lineage & Strain Explorer | `plot_lineage_dag`, `plot_kinematics_dashboard`, `plot_precursor_timeline`, `plot_spot_mip_card`, `generate_html_report` | `numpy`, optional `matplotlib` (HTML explorer uses stdlib only) |
+| **cli** | Parse args/JSON config, orchestrate stages, write CSV/JSON/NPZ/FLT/NXS/HTML artifacts and SHA-256 `run_manifest.json` | `segment_dataset`, `embed_dataset`, `track_dataset`, `detect_events`, `report_dataset`, `run_pipeline` | All of the above |
**Rule of thumb:** keep **numeric kernels** and **typed data contracts** in subpackages; keep **CLI modules** as thin orchestration layers.
@@ -38,25 +40,29 @@ BraggTrack/
```mermaid
flowchart LR
subgraph inputs["Inputs"]
- H5["HDF5 / NeXus 3D scans"]
- CAL["AxisCalibration (auto or JSON)"]
+ H5["HDF5 / NeXus 3D scans (LazyVolume)"]
+ CAL["AxisCalibration & DetectorGeometry"]
end
subgraph seg["Segmentation"]
- SEG["segment_dataset (smoothed Otsu + 3D watershed)"]
+ SEG["segment_dataset (chunked Otsu + 3D watershed)"]
FEAT["features.csv + labels.npz"]
end
subgraph sem["Semantic"]
- EMB["embed_dataset (MIPs + DINOv2 / mock)"]
+ EMB["embed_dataset (ROI MIPs + DINOv2 / mock)"]
NPZ["embeddings.npz"]
end
- subgraph trk["Tracking & Kinematics"]
- TRK["track_dataset (Hungarian + kinematics)"]
- KIN["tracks.csv + kinematics.csv"]
+ subgraph trk["Tracking, Gap-Closing & Kinematics"]
+ TRK["track_dataset (Hungarian + max_gap + kinematics)"]
+ KIN["tracks.csv + kinematics.csv + .flt / .nxs"]
end
subgraph evt["Lineage & Precursors"]
EVT["detect_events (splits, merges, ROI dispersion)"]
OUT["events.csv + precursors.csv"]
end
+ subgraph rep["Visualization"]
+ REP["report_dataset (braggtrack.viz)"]
+ HTML["report.html + figures/*.png"]
+ end
MAN["run_manifest.json (SHA-256 + timings)"]
H5 --> SEG
@@ -65,19 +71,23 @@ flowchart LR
H5 --> EMB
FEAT --> EMB --> NPZ
FEAT --> TRK
+ CAL --> TRK
NPZ --> TRK
TRK --> KIN
H5 --> EVT
FEAT --> EVT
NPZ --> EVT
EVT --> OUT
+ KIN --> REP
+ OUT --> REP --> HTML
SEG --> MAN
TRK --> MAN
EVT --> MAN
+ REP --> MAN
```
- **Dataset root** is a directory containing `scan*` folders (defaults to bundled `data/sample_operando/` when present).
-- **Physical coordinates vs. voxel indices:** `extract_instance_table(..., calibration=cal)` converts `centroid_mu`, `centroid_chi`, `centroid_d`, `cov_*`, and `eig_1..3` into physical reciprocal-space units while preserving integer voxel `bbox_*` indices so downstream 3D ROI cropping (`crop_spot_cube`) remains exact.
+- **Physical coordinates vs. voxel indices:** `extract_instance_table(..., calibration=cal)` converts `centroid_mu`, `centroid_chi`, `centroid_d`, `cov_*`, and `eig_1..3` into physical reciprocal-space units while preserving integer voxel `bbox_*` indices so downstream 3D ROI cropping (`crop_spot_cube` over `LazyVolume`) remains exact.
- **Artifacts** under `artifacts/` are reproducible build outputs and are never committed to Git.
## Extension points
@@ -89,12 +99,12 @@ flowchart LR
## Performance notes
-- Frame-to-frame Hungarian association builds its `(N, M)` cost matrix via vectorised `scipy.spatial.distance.cdist` and a single dense matrix multiplication (`A @ B.T`) for semantic cosine similarity.
-- Cross-scan threshold stabilisation (`smooth_thresholds`) operates in $O(N \cdot W)$ on scalar per-scan Otsu thresholds without holding the entire 4D dataset in memory simultaneously during streaming workflows.
+- `LazyVolume` streams `z`-slabs (`otsu_threshold_chunked`) in Pass 1 of `segment_dataset` and reads only tight 3D spot ROIs (`read_roi`) in `embed_dataset` and `detect_events`, avoiding full-volume `float64` allocations on multi-gigabyte detector stacks.
+- Frame-to-frame Hungarian association and multi-scan gap closing (`resolve_track_gaps`) build cost matrices via vectorised `scipy.spatial.distance.cdist` and dense matrix multiplication (`A @ B.T`) for semantic cosine similarity.
## Configuration surfaces
- **Environment variables:**
- `BRAGGTRACK_DATA_ROOT`: Override default dataset root discovery.
- `BRAGGTRACK_DINO_BACKEND`: Force `auto`, `mock`, or `torch` backend selection.
-- **Pipeline JSON config:** Pass `--config pipeline.json` to `braggtrack-run` to declaratively configure backend, model ID (`DEFAULT_DINO_MODEL = "facebook/dinov2-small"` or DINOv3), calibration file, semantic weights (`cost_alpha`, `cost_beta`), and split/precursor thresholds.
+- **Pipeline JSON config:** Pass `--config pipeline.json` to `braggtrack-run` to declaratively configure backend, model ID (`DEFAULT_DINO_MODEL = "facebook/dinov2-small"` or DINOv3), calibration/detector JSON files, `max_gap`, semantic weights (`cost_alpha`, `cost_beta`), split/precursor thresholds, `--report`, and `ImageD11` / NeXus export paths.
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
new file mode 100644
index 0000000..d0b5d58
--- /dev/null
+++ b/docs/benchmarks.md
@@ -0,0 +1,115 @@
+# BraggTrack v0.3.0 Benchmarks & ESRF-ID03 Case Study
+
+This document reports reproducible multi-trial quantitative benchmarks for **BraggTrack v0.3.0**, comparing **Geometry-Only**, **Deterministic Mock-Hash** (`mock`), and **Pretrained Vision Transformer** (`torch` with ungated `facebook/dinov2-small`) tracking, gap-closing, and 4D chemomechanical fracture precursor scoring, followed by an end-to-end case study on bundled operando dark-field X-ray microscopy (DFXM) data from **ESRF ID03**.
+
+All synthetic benchmarks can be reproduced via:
+
+```bash
+python scripts/benchmark_suite.py \
+ --include-torch \
+ --model facebook/dinov2-small \
+ --trials 5 \
+ --bootstrap-samples 500 \
+ --output-json artifacts/benchmarks/benchmark_results.json \
+ --output-figure docs/figures/benchmark_comparison.png
+```
+
+
+
+---
+
+## 1. Benchmark Methodology
+
+To isolate when pretrained multi-view morphological features genuinely outperform classical 3D centroid + covariance tracking—and to verify that gains come from learned visual structure rather than arbitrary feature dimensionality—`scripts/benchmark_suite.py` evaluates three backends under identical segmentation and Hungarian/Jaqaman solver configurations:
+
+1. **`geo` (Geometry-Only, $\alpha=1.0, \beta=0.0$):** Uses calibrated 3D reciprocal-space centroid displacement and log-eigenvalue shape distance (`Eigen-Shape`), with zero semantic weight.
+2. **`mock` (Deterministic Hash Baseline, $\alpha=1.0, \beta=0.6$):** Appends `1152`-d `SHA-256` multi-view MIP vectors (`MockDinoEncoder`). Useful solely for deterministic offline CI plumbing; slight pixel changes completely decorrelate consecutive scan hashes.
+3. **`torch` (Pretrained `facebook/dinov2-small`, $\alpha=1.0, \beta=0.6$):** Extracts `1152`-d L2-normalized multi-view orthogonal MIP embeddings (`3` projections $\times$ `384`-d ViT-S/14 features) and `16x16` spatial patch tokens for sub-spot precursor dispersion.
+
+### Scenarios Evaluated (`N = 5` trials, `500` bootstrap resamples for 95% CIs)
+
+| Scenario | Physical Beamline Regime | Primary Failure Mode Tested |
+| :--- | :--- | :--- |
+| **`crossing_morphology`** | Two sub-grains on an antipodal circle ($R = 4.2\text{ px}$) rotating $90.25^\circ$ between scans with orthogonal $+45^\circ$ vs $-45^\circ$ streak tilt (`3D` eigenvalues identical by symmetry) | Equidistant / near-degenerate 3D geometry causing catastrophic Hungarian ID swaps without 2D MIP orientation semantics |
+| **`intensity_jitter`** | $\pm 45\%$ incident synchrotron beam-flux spikes + Poisson/Gaussian detector shot noise across 6 scans | Photometric invariance of L2-normalized ViT MIP embeddings vs pixel-hash fragility |
+| **`mosaic_broadening`** | Progressive rocking-curve ($\mu$) mosaic broadening ($1.0\times \to 1.9\times$) under electrochemical cycling | Smooth morphological continuity ($\cos > 0.98$) as Bragg reflections broaden |
+| **`beam_dropout_no_gap` vs `beam_dropout_gap2`** | Staggered 1-scan rocking-window dropouts across 4 grains over 6 scans (`max_gap=0` vs `max_gap=2`) | Track fragmentation and cross-grain jump hijacking vs Jaqaman diffusive gap-closing recovery |
+| **`chemomechanical_fracture`** | 8-scan 4D grain evolution with persistent pre-split broadening + sub-peak bimodality (`scans 1-3`), `1 -> 2` split (`scan 3 -> 4`), `2 -> 1` merge (`scan 5 -> 6`), and isotropic control | Exact split/merge lineage recall and pre-fracture precursor score contrast against stable control grains |
+
+---
+
+## 2. Quantitative Multi-Trial Results (`facebook/dinov2-small`)
+
+### 2.1 Tracking Identity, Gap-Closing & Embedding Stability
+
+| Scenario | Backend | Mean ID-Switch Rate (95% CI) | Fragmentation Ratio (95% CI) | Gap Closures | Cosine Stability $\cos(\mathbf{z}_t, \mathbf{z}_{t+1})$ | Elapsed (`5` trials, CPU) |
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
+| **`crossing_morphology`** | `geo` | `0.2500` (`[0.2500, 0.2500]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | — | `0.51 s` |
+| **`crossing_morphology`** | `mock` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | `0.0160` | `0.50 s` |
+| **`crossing_morphology`** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | `0.0` | **`0.9035`** | `43.80 s` |
+| **`intensity_jitter`** | `geo` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | — | `0.04 s` |
+| **`intensity_jitter`** | `mock` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | `0.0089` | `0.05 s` |
+| **`intensity_jitter`** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | `0.0` | **`0.9908`** | `10.66 s` |
+| **`mosaic_broadening`** | `geo` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | — | `0.05 s` |
+| **`mosaic_broadening`** | `mock` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | `0.0` | `-0.0031` | `0.05 s` |
+| **`mosaic_broadening`** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | `0.0` | **`0.9899`** | `10.85 s` |
+| **`beam_dropout_no_gap` (`max_gap=0`)** | `geo` / `mock` / `torch` | `0.0833` (`[0.0833, 0.0833]`) | `0.5000` (`[0.5000, 0.5000]`) | `0.0` | `— / 0.0048 / 0.9879` | `0.07–17.83 s` |
+| **`beam_dropout_gap2` (`max_gap=2`)** | `geo` | `0.0000` (`[0.0000, 0.0000]`) | `0.0000` (`[0.0000, 0.0000]`) | **`2.0`** | — | `0.07 s` |
+| **`beam_dropout_gap2` (`max_gap=2`)** | `mock` | `0.0833` (`[0.0833, 0.0833]`) | `0.5000` (`[0.5000, 0.5000]`) | `0.0` | `0.0048` | `0.08 s` |
+| **`beam_dropout_gap2` (`max_gap=2`)** | **`torch` (`dinov2-small`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`0.0000` (`[0.0000, 0.0000]`)** | **`2.0`** | **`0.9879`** | `17.88 s` |
+
+### 2.2 4D Chemomechanical Fracture & Precursor Benchmark
+
+| Scenario | Backend | `1 -> 2` Split Recall | `2 -> 1` Merge Recall | Top-1 Precursor Precision | Precursor Contrast $\Delta S$ vs Control (95% CI) | Elapsed (`5` trials, CPU) |
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
+| **`chemomechanical_fracture`** | `geo` | `1.00` | `1.00` | `1.00` | `0.6449` (`[0.6448, 0.6450]`) | `0.13 s` |
+| **`chemomechanical_fracture`** | `mock` | `1.00` | `1.00` | `1.00` | `0.5539` (`[0.5538, 0.5540]`) | `0.29 s` |
+| **`chemomechanical_fracture`** | **`torch` (`dinov2-small`)** | **`1.00`** | **`1.00`** | **`1.00`** | **`0.5584` (`[0.5582, 0.5585]`)** | `74.82 s` |
+
+
+
+### Key Scientific Takeaways
+
+1. **Orientation-Sensitive Disambiguation (`crossing_morphology`):** When two grains have identical 3D eigenvalue magnitudes ($\lambda_1, \lambda_2, \lambda_3$) and rotate slightly past $90^\circ$ on a shared Debye-Scherrer / rocking locus, **Geometry-Only** suffers a deterministic `25.0%` ID-switch rate because 3D scalar eigenvalues discard 2D detector streak orientation (`+45 deg` vs `-45 deg`). **`facebook/dinov2-small`** preserves high multi-view morphological similarity (`0.9035`) along true trajectories and eliminates ID switches (`0.0%`).
+2. **Real Pretrained Embeddings vs Mock Hash (`beam_dropout_gap2` & `intensity_jitter`):** While `MockDinoEncoder` produces pseudo-random zero-mean cosine similarities (`-0.003` to `0.016`) whenever shot noise Perturbs a single voxel—causing semantic gap-closing (`min_cosine_similarity=0.35`) to reject valid re-links (`0.0` gap closures, `50%` fragmentation)—pretrained **`facebook/dinov2-small`** maintains `0.988–0.991` cosine stability across `45%` flux spikes, progressive mosaic broadening, and multi-scan dropouts, recovering `100%` of dropped tracks (`0.0%` fragmentation, `2.0` gap closures).
+3. **Early Precursor Contrast (`chemomechanical_fracture`):** Across `1 -> 2` grain splitting (`scan 3 -> 4`) and `2 -> 1` sub-grain coalescence (`scan 5 -> 6`), BraggTrack's multi-component precursor score flags the fracturing parent grain `3` scans before topological separation (`1.00` top-1 precision, `+0.56` to `+0.64` score contrast above isotropic control grains).
+
+---
+
+## 3. ESRF-ID03 Operando DFXM Case Study (`data/sample_operando/`)
+
+We executed the complete **BraggTrack v0.3.0** pipeline with real **`facebook/dinov2-small`** embeddings, multi-scan gap closing (`--max-gap 1`), interactive HTML reporting (`--report`), `ImageD11` `.flt` export, and NeXus `NXprocess` HDF5 archival on the bundled 3-scan **ESRF-ID03** near-field PCO dataset (`3` scans $\times$ `100 x 250 x 250` `uint16` cubes, ~`20 MB` compressed HDF5):
+
+```bash
+braggtrack-run data/sample_operando \
+ --outdir artifacts/esrf_id03_dinov2 \
+ --backend torch \
+ --model facebook/dinov2-small \
+ --max-gap 1 \
+ --report \
+ --export-nexus artifacts/esrf_id03_dinov2/esrf_id03.nxs \
+ --export-imaged11 artifacts/esrf_id03_dinov2/esrf_id03.flt
+```
+
+### 3.1 CPU Stage-Timing Profile
+
+| Pipeline Stage | Wall-Clock Time (CPU) | Share of Run | Primary Workload |
+| :--- | :---: | :---: | :--- |
+| **`validate`** | `0.043 s` | `0.1%` | NeXus `/entry_0000/ESRF-ID03/pco_nf/data` discovery & SHA-256 verification |
+| **`segment`** | `13.48 s` | `24.4%` | Chunked lazy Otsu pass (`LazyVolume`) + 3D LoG + `h`-maxima seeded watershed (`[18, 17, 15]` spots) |
+| **`embed`** | `18.35 s` | `33.3%` | Lazy ROI cropping + `facebook/dinov2-small` 3-view MIP inference (`50` spots $\times$ `1152`-d) |
+| **`track`** | `0.024 s` | `0.04%` | Geometry+semantic Hungarian matching + Jaqaman gap closing + 4D grain kinematics |
+| **`events`** | `20.23 s` | `36.7%` | `1 -> 2` / `2 -> 1` lineage resolution + 3D sub-peak & `dinov2-small` patch-token dispersion |
+| **`report` + exports** | `3.04 s` | `5.5%` | Self-contained `report.html`, publication PNGs, `esrf_id03.flt`, and `esrf_id03.nxs` |
+| **Total** | **`55.17 s`** | **`100.0%`** | Full 3-scan operando DFXM workflow on CPU (no GPU required) |
+
+### 3.2 Detected Lineage Events & Persistent Precursor Candidates
+
+- **Split Candidates (`events.csv`):**
+ - **Event 1 (`scan 0 -> 1`):** Parent **Track `6`** (`s0_i5`) $\to$ Daughters **Track `32`** (`s1_i11`) & **Track `34`** (`s1_i9`), intensity partition ratio `1.434`, mean daughter separation `18.64 px`, **DINOv2 multi-view cosine similarity `0.7561`**.
+ - **Event 2 (`scan 1 -> 2`):** Parent **Track `21`** (`s1_i1`) $\to$ Daughters **Track `36`** (`s2_i1`) & **Track `44`** (`s2_i13`), intensity partition ratio `1.258`, mean daughter separation `20.11 px`, **DINOv2 multi-view cosine similarity `0.6837`**.
+- **Top Persistent Precursor Tracks (`precursors.csv`, `min_persistence = 2` scans):**
+ 1. **Track `14`** (`rank 1`): Active across all `3` scans (`onset_scan=0`, `peak_scan=1`), `peak_score = 0.8789`, `mean_active_score = 0.7775`, `max_anisotropy = 5.45`, `max_spread_growth = +106%`, `confidence = 0.8555`.
+ 2. **Track `34`** (`rank 2`, split daughter of Track `6`): Active across `scans 1-2`, `peak_score = 0.9023`, extreme streak elongation (`max_anisotropy = 26.78`), `max_spread_growth = +129%`, `confidence = 0.8119`.
+ 3. **Track `27`** (`rank 3`): Active across `scans 1-2`, `peak_score = 0.6710`, `max_anisotropy = 16.00`, `confidence = 0.6534`.
+ 4. **Track `29`** (`rank 4`): Active across `scans 1-2`, `peak_score = 0.6668`, `max_anisotropy = 6.67`, `confidence = 0.6490`.
diff --git a/docs/figures/benchmark_comparison.png b/docs/figures/benchmark_comparison.png
new file mode 100644
index 0000000..03a9b0c
Binary files /dev/null and b/docs/figures/benchmark_comparison.png differ
diff --git a/docs/figures/fracture_lineage_and_precursors.png b/docs/figures/fracture_lineage_and_precursors.png
new file mode 100644
index 0000000..eba8ebd
Binary files /dev/null and b/docs/figures/fracture_lineage_and_precursors.png differ
diff --git a/notebooks/braggtrack_demo.ipynb b/notebooks/braggtrack_demo.ipynb
index ef8edac..bbdcba6 100644
--- a/notebooks/braggtrack_demo.ipynb
+++ b/notebooks/braggtrack_demo.ipynb
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"id": "e65e834e",
"metadata": {},
- "source": "# BraggTrack end-to-end demo\n\n[](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb)\n\nRuns the full BraggTrack v0.2 pipeline on the bundled `data/sample_operando/` ESRF-ID03 scans:\n\n1. **Discover & load** — find per-scan NeXus/HDF5 files and auto-discover 3D detector cubes (`load_primary_volume`).\n2. **Segment** — cross-scan smoothed Otsu → 3D LoG → h-maxima → seeded watershed → instance features.\n3. **Track (physics-only)** — Hungarian association over geometry cost with per-axis gating; build a lifecycle DAG.\n4. **Semantic descriptors** — orthogonal $(\\mu, \\chi, d)$ MIPs + frozen vision-transformer embeddings.\n5. **Geometry + semantic tracking** — compose $\\alpha \\cdot C_{\\text{geo}} + \\beta \\cdot (1 - \\cos)$.\n6. **$\\alpha / \\beta$ ablation** — sweep semantic weight across tracking metrics.\n7. **Near-overlap disambiguation** — synthetic crossing where semantics prevent an ID swap.\n8. **Per-grain kinematics, split/merge lineage & fracture precursors** — calibrated strain $\\Delta d / d_0$, `1 -> 2` grain cleavage, `2 -> 1` coalescence, and persistent precursor ranking.\n9. **Command-line & one-command pipeline (`braggtrack-run`)** — reproducible CLI equivalents.\n\nThis notebook uses the **mock** DINO backend by default so zero PyTorch / GPU weights are needed in CI. Set `BRAGGTRACK_DINO_BACKEND=torch` (with `pip install -e '.[torch]'`) to use ungated `facebook/dinov2-small`."
+ "source": "# BraggTrack end-to-end demo (v0.3)\n\n[](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb)\n\nRuns the full BraggTrack v0.3 pipeline on the bundled `data/sample_operando/` ESRF-ID03 scans:\n\n1. **Discover & stream** — find per-scan NeXus/HDF5 files, auto-discover 3D detector cubes, and stream chunked Otsu thresholds via `LazyVolume`.\n2. **Segment** — cross-scan smoothed Otsu → 3D LoG → h-maxima → seeded watershed → instance features.\n3. **Track (physics-only & gap closing)** — Hungarian association + multi-scan gap closing (`GapClosingConfig`).\n4. **Semantic descriptors** — orthogonal $(\\mu, \\chi, d)$ MIPs + frozen vision-transformer embeddings.\n5. **Geometry + semantic tracking** — compose $\\alpha \\cdot C_{\\text{geo}} + \\beta \\cdot (1 - \\cos)$.\n6. **$\\alpha / \\beta$ ablation** — sweep semantic weight across tracking metrics.\n7. **Near-overlap disambiguation** — synthetic crossing where semantics prevent an ID swap.\n8. **Per-grain kinematics, split/merge lineage & `braggtrack.viz` dashboards** — calibrated strain $\\Delta d / d_0$, `1 -> 2` grain cleavage, `2 -> 1` coalescence, persistent precursor ranking, and `DetectorGeometry` $(2\\theta, \\eta, d, |\\mathbf{Q}|)$ conversion.\n9. **Command-line & one-command pipeline (`braggtrack-run --report`)** — reproducible CLI equivalents.\n\nThis notebook uses the **mock** DINO backend by default so zero PyTorch / GPU weights are needed in CI. Set `BRAGGTRACK_DINO_BACKEND=torch` (with `pip install -e \".[torch]\"`) to use ungated `facebook/dinov2-small`."
},
{
"cell_type": "markdown",
@@ -41,7 +41,7 @@
}
},
"outputs": [],
- "source": "%matplotlib inline\nimport copy\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom matplotlib.colors import ListedColormap\n\nfrom braggtrack.io import (\n AxisCalibration,\n discover_operando_scans,\n discover_primary_volume_path,\n load_primary_volume,\n sample_operando_root,\n)\nfrom braggtrack.segmentation import (\n extract_instance_table,\n fill_holes_binary,\n flag_outlier_frames,\n label_projection_by_intensity,\n merge_nearby_labels,\n otsu_floor_from_mip,\n otsu_threshold,\n relabel_sequential,\n remove_small_objects,\n segment_classical,\n smooth_thresholds,\n)\nfrom braggtrack.semantic import (\n crop_spot_cube,\n make_multiview_encoder,\n make_patch_encoder,\n orthogonal_mips,\n)\nfrom braggtrack.tracking import (\n GeometrySemanticCost,\n PositionShapeCost,\n PrecursorConfig,\n SplitMergeConfig,\n TrackEvent,\n build_tracks,\n compute_fracture_precursors,\n compute_grain_kinematics,\n compute_tracking_metrics,\n events_to_table,\n generate_fracture_scenario,\n kinematics_to_table,\n precursors_to_table,\n tracks_to_table,\n)"
+ "source": "%matplotlib inline\nimport copy\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom matplotlib.colors import ListedColormap\n\nfrom braggtrack.io import (\n AxisCalibration,\n DetectorGeometry,\n open_lazy_volume,\n otsu_threshold_chunked,\n discover_operando_scans,\n discover_primary_volume_path,\n load_primary_volume,\n sample_operando_root,\n)\nfrom braggtrack.segmentation import (\n extract_instance_table,\n fill_holes_binary,\n flag_outlier_frames,\n label_projection_by_intensity,\n merge_nearby_labels,\n otsu_floor_from_mip,\n otsu_threshold,\n relabel_sequential,\n remove_small_objects,\n segment_classical,\n smooth_thresholds,\n)\nfrom braggtrack.semantic import (\n crop_spot_cube,\n make_multiview_encoder,\n make_patch_encoder,\n orthogonal_mips,\n)\nfrom braggtrack.tracking import (\n GapClosingConfig,\n GeometrySemanticCost,\n PositionShapeCost,\n PrecursorConfig,\n SplitMergeConfig,\n TrackEvent,\n build_tracks,\n compute_fracture_precursors,\n compute_grain_kinematics,\n compute_tracking_metrics,\n events_to_table,\n generate_fracture_scenario,\n kinematics_to_table,\n precursors_to_table,\n tracks_to_table,\n)\nfrom braggtrack.viz import (\n plot_kinematics_dashboard,\n plot_lineage_dag,\n plot_precursor_timeline,\n)\n"
},
{
"cell_type": "markdown",
@@ -728,11 +728,19 @@
"outputs": [],
"source": "# 8b. 4D split/merge lineage + persistent fracture precursor ranking\nf_vols, f_labels, f_tables, f_expected = generate_fracture_scenario(n_scans=5, seed=42)\nG_frac = build_tracks(\n f_tables,\n cost_fn=PositionShapeCost(),\n split_merge_config=SplitMergeConfig(max_distance=20.0),\n)\nlineage_events = G_frac.graph[\"lineage_events\"]\nfrac_rows = tracks_to_table(G_frac)\ncandidates, obs_scores = compute_fracture_precursors(\n frac_rows,\n PrecursorConfig(score_threshold=0.35, min_persistence=2),\n volumes=f_vols,\n label_volumes=f_labels,\n patch_encoder=make_patch_encoder(\"mock\"),\n lineage_events=lineage_events,\n)\n\nprint(\"=== Detected Lineage Events (splits & merges) ===\")\nprint(pd.DataFrame(events_to_table(lineage_events))[[\n \"event_id\", \"event_type\", \"scan_idx_from\", \"scan_idx_to\",\n \"parent_track_ids\", \"child_track_ids\", \"intensity_ratio\", \"confidence\"\n]].to_string(index=False))\n\nprint(\"\\n=== Persistent Fracture Precursor Ranking ===\")\nprint(pd.DataFrame(precursors_to_table(candidates)).to_string(index=False))"
},
+ {
+ "cell_type": "code",
+ "id": "b730v001",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": "# 8c. Publication visualization (braggtrack.viz) + crystallographic (2theta, eta, d, |Q|) mapping\nfig_dag, _ = plot_lineage_dag(frac_rows, events=lineage_events, title=\"4D Split & Merge Lineage DAG\")\nplt.show()\n\nfig_prec, _ = plot_precursor_timeline(obs_scores, candidates=candidates, score_threshold=0.35)\nplt.show()\n\nfig_kin, _ = plot_kinematics_dashboard(gk_operando, calibrated=False, max_tracks=6)\nplt.show()\n\n# Lazy HDF5 streaming & transmission diffraction geometry (DetectorGeometry)\nwith open_lazy_volume(scans[0].path) as lv:\n chunked_otsu = otsu_threshold_chunked(lv, chunk_z=32)\ndet = DetectorGeometry(distance_mm=150.0, pixel_size_y_mm=0.075, pixel_size_x_mm=0.075,\n beam_center_y_px=20.0, beam_center_x_px=20.0, wavelength_angstrom=0.50)\ns0 = all_features[0][0]\nscat = det.spot_to_scattering(s0[\"centroid_mu\"], s0[\"centroid_d\"], s0[\"centroid_chi\"])\nprint(f\"Lazy chunked Otsu={chunked_otsu:.1f} | Spot #1: 2theta={scat['two_theta_deg']:.3f} deg, \"\n f\"eta={scat['eta_deg']:.2f} deg, d={scat['d_spacing_angstrom']:.4f} A, |Q|={scat['q_mag_inv_angstrom']:.4f} A^-1\")"
+ },
{
"cell_type": "markdown",
"id": "7e605de2",
"metadata": {},
- "source": "## 9 — Reproducible command-line pipeline (`braggtrack-run`)\n\nEvery stage above is available via CLI entry points — or as a single end-to-end command (`braggtrack-run`) that validates inputs, segments with cross-scan threshold smoothing, embeds MIPs, tracks kinematics, resolves splits/merges & fracture precursors, and writes `run_manifest.json` with SHA-256 input checksums:\n\n```bash\n# One-command end-to-end pipeline\nbraggtrack-run --outdir artifacts/pipeline\n\n# Or stage-by-stage:\nbraggtrack-segment-dataset --outdir artifacts/segmentation\nbraggtrack-embed-dataset --segdir artifacts/segmentation --outdir artifacts/embedding --backend auto\nbraggtrack-track-dataset artifacts/segmentation \\\n --outdir artifacts/tracking \\\n --embedding-dir artifacts/embedding \\\n --cost-alpha 1.0 --cost-beta 0.25 --detect-events\nbraggtrack-events --segdir artifacts/segmentation --outdir artifacts/events\n\n# Semantic alpha/beta ablation with 95% bootstrap confidence intervals\npython scripts/ablation_semantic.py \\\n --indir artifacts/segmentation \\\n --embedding-dir artifacts/embedding \\\n --betas 0,0.25,0.5,1.0 --bootstrap-samples 100 \\\n --output artifacts/ablation/report.json\n\n# Full CI suite (unit tests + all 7 acceptance gates)\npython scripts/ci_report.py\n```"
+ "source": "## 9 — Reproducible command-line pipeline (`braggtrack-run`)\n\nEvery stage above is available via CLI entry points — or as a single end-to-end command (`braggtrack-run`) that validates inputs, segments with chunked Otsu + cross-scan threshold smoothing, embeds MIPs, tracks kinematics with `--max-gap`, resolves splits/merges & fracture precursors, exports `ImageD11` (`.flt`) / NeXus (`.nxs`), renders an offline interactive HTML 4D Explorer (`--report`), and writes `run_manifest.json` with SHA-256 input checksums:\n\n```bash\n# One-command end-to-end pipeline with interactive HTML explorer & NeXus/ImageD11 exports\nbraggtrack-run --outdir artifacts/pipeline --max-gap 1 --report \\\n --export-nexus artifacts/pipeline/results.nxs \\\n --export-imaged11 artifacts/pipeline/peaks.flt\n\n# Standalone interactive report & static publication figures\nbraggtrack-report artifacts/pipeline\n\n# Quantitative 5-scenario benchmark suite (add --include-torch for real DINOv2)\npython scripts/benchmark_suite.py --trials 5\n\n# Full CI suite (unit tests + all 9 acceptance gates)\npython scripts/ci_report.py\n```"
}
],
"metadata": {
diff --git a/paper.bib b/paper.bib
index b8e4710..6b37b89 100644
--- a/paper.bib
+++ b/paper.bib
@@ -25,7 +25,7 @@ @article{Hayashi2019
doi = {10.1126/science.aax9167}
}
-@article{Finegan2019,
+@article{Finegan2017,
author = {Finegan, Donal P. and Darcy, Eric and Keyser, Matt and
Tjaden, Bernhard and Heenan, Thomas M. M. and
Jervis, Rhodri and Bailey, Josh J. and Malik, Ruhul and
@@ -181,3 +181,17 @@ @inproceedings{Hagberg2008
Millman, Jarrod},
address = {Pasadena, CA}
}
+
+@article{Jaqaman2008,
+ author = {Jaqaman, Khuloud and Loerke, Dinah and Mettlen, Marcel
+ and Kuwata, Hirotaka and Grinstein, Sergio and
+ Schmid, Sandra L. and Danuser, Gaudenz},
+ title = {Robust single-particle tracking in live-cell time-lapse
+ sequences},
+ journal = {Nature Methods},
+ volume = {5},
+ number = {8},
+ pages = {695--702},
+ year = {2008},
+ doi = {10.1038/nmeth.1237}
+}
diff --git a/paper.md b/paper.md
index d59b023..1c56e8d 100644
--- a/paper.md
+++ b/paper.md
@@ -30,13 +30,17 @@ during *in situ* or *operando* experiments, BraggTrack segments individual
Bragg reflections, extracts calibrated geometric and foundation vision features,
and links observations across scans into a directed acyclic trajectory graph
capturing births, continuations, terminations, `1 -> 2+` grain fracture splits,
-and `2+ -> 1` coalescence merges. In addition, BraggTrack ranks persistent
-pre-fracture precursor candidates across scans by combining 3D reflection
-anisotropy, covariance broadening, internal sub-peak bimodality, and frozen
+and `2+ -> 1` coalescence merges, as well as multi-scan gap closures across
+transient beam dropouts. In addition, BraggTrack ranks persistent pre-fracture
+precursor candidates across scans by combining 3D reflection anisotropy,
+covariance broadening, internal sub-peak bimodality, and frozen
vision-transformer patch-token dispersion. Designed for synchrotron NeXus/HDF5
-data, BraggTrack ships with bundled operando scans from ESRF beamline ID03,
-eight CLI entry points including a single-command reproducible pipeline runner
-(`braggtrack-run`), and interactive Jupyter notebooks runnable on Google Colab.
+data, BraggTrack supports lazy HDF5 ROI streaming, crystallographic
+$(2\theta, \eta, d, |\mathbf{Q}|)$ conversion with direct `ImageD11` (`.flt`)
+and NeXus `NXprocess` HDF5 export, publication plotting and offline interactive
+HTML 4D lineage reports (`braggtrack-report`), bundled operando scans from ESRF
+beamline ID03, nine CLI entry points culminating in `braggtrack-run`, and
+interactive Jupyter notebooks runnable on Google Colab.
# Statement of Need
@@ -49,7 +53,7 @@ In battery electrodes such as Ni-rich layered oxides, chemomechanical
degradation manifests first as anisotropic intragranular strain broadening and
sub-domain formation before culminating in particle microcracking — observable
in reciprocal space as reflection broadening followed by `1 -> 2+` peak
-splitting [@Finegan2019]. Quantifying per-grain lattice strain
+splitting [@Finegan2017]. Quantifying per-grain lattice strain
($\varepsilon = \Delta d / d_0$), lattice rotation ($\Delta\mu$, $\Delta\chi$),
and fracture lineage across long scan sequences is essential for linking local
crystallography to macroscopic failure, as well as for solidification and
@@ -62,36 +66,45 @@ scripts, or crystallographic indexing packages such as GrainSpotter
evolving, splitting, and broadening 3D intensity distributions through time.
Previously, researchers lacked an open-source, `pip`-installable toolkit
combining stabilised 3D spot segmentation, foundation-model visual descriptors,
-calibrated per-grain kinematics, and explicit split/merge fracture lineage in a
-reproducible software architecture.
+calibrated per-grain kinematics, explicit split/merge fracture lineage, and
+beamline-ready crystallographic exports in a reproducible software architecture.
BraggTrack addresses this need by separating beamline HDF5 ingestion and axis
calibration, 3D instance segmentation, multi-view vision-transformer encoding,
-Hungarian and topological lineage tracking, and fracture precursor scoring into
-cleanly tested, modular layers.
+Hungarian and topological lineage tracking, fracture precursor scoring, and
+interactive visualization (`braggtrack.viz`) into cleanly tested, modular
+layers.
# Implementation
-## Beamline ingestion and physical axis calibration
+## Beamline ingestion, lazy HDF5 streaming, and crystallographic mapping
BraggTrack discovers sequential `scan*` directories, inspects NeXus/HDF5 trees
(including ESRF-ID03 LIMA detector hierarchies under `/entry_0000/...` and
generic HDF5 layouts via automatic largest-3D numeric dataset discovery), and
-validates timestamp monotonicity. An `AxisCalibration` transform converts raw
-detector indices into physical reciprocal-space coordinates
-$(\mu, \chi, d)$ via HDF5 axis metadata or JSON/CLI calibration parameters while
-preserving integer voxel bounding boxes for lossless 3D region-of-interest (ROI)
-cropping. This ensures crystallographic strain $\Delta d / d_0$ is invariant to
-detector window cropping.
+validates timestamp monotonicity. To handle gigabyte-scale detector stacks
+without exhausting host memory, `LazyVolume` streams `z`-slabs for exact
+two-pass global Otsu histogramming (`otsu_threshold_chunked`) and reads only
+tight 3D spot bounding-box slices (`read_roi`) during semantic embedding and
+precursor analysis.
+
+An `AxisCalibration` transform converts raw detector indices into physical
+reciprocal-space coordinates $(\mu, \chi, d)$ while preserving integer voxel
+bounding boxes for lossless 3D ROI cropping. Complementarily,
+`DetectorGeometry` maps transmission-diffraction spot centroids into
+crystallographic scattering angle $2\theta$, azimuth $\eta$, $d$-spacing
+($\lambda = 2 d \sin\theta$), and reciprocal-lattice vectors $\mathbf{Q}$
+($\text{\AA}^{-1}$), exporting directly to `ImageD11` columnfiles (`.flt`)
+[@Wright2023] and self-describing NeXus `NXprocess` HDF5 archives (`.nxs`).
## Stabilised 3D segmentation
BraggTrack's primary segmentation pipeline processes 3D volumes in four stages:
-1. **Stabilised foreground thresholding.** Per-scan Otsu thresholds [@Otsu1979]
- on raw intensity histograms are smoothed across scans with a rolling median
- (`smooth_thresholds`) and checked for median-absolute-deviation outliers,
- eliminating spot-count jitter caused by transient beam fluctuations.
+1. **Stabilised foreground thresholding.** Streaming per-scan Otsu thresholds
+ [@Otsu1979] on raw intensity histograms are smoothed across scans with a
+ rolling median (`smooth_thresholds`) and checked for median-absolute-deviation
+ outliers, eliminating spot-count jitter caused by transient beam fluctuations.
2. **3D LoG enhancement.** Gaussian smoothing followed by the negative discrete
Laplacian highlights Bragg reflections as local maxima [@Lindeberg1998]
@@ -110,44 +123,63 @@ BraggTrack's primary segmentation pipeline processes 3D volumes in four stages:
## Multi-view semantic descriptors
-Each segmented 3D reflection is cropped, masked to its instance support, and
-projected into three orthogonal maximum-intensity projections (MIPs) along
-$\mu$, $\chi$, and $d$. By default, BraggTrack encodes these views using the
-public, ungated DINOv2 vision transformer (`facebook/dinov2-small`)
-[@Oquab2024] into an $L_2$-normalised 384-d descriptor, while also supporting
-newer DINOv3 backbones (`facebook/dinov3-vitb16-pretrain-lvd1689m`) via dynamic
-prefix/register token handling and a deterministic hash-based `mock` encoder for
-zero-GPU continuous integration.
+Each segmented 3D reflection is cropped via `LazyVolume`, masked to its instance
+support, and projected into three orthogonal maximum-intensity projections
+(MIPs) along $\mu$, $\chi$, and $d$. By default, BraggTrack encodes these views
+using the public, ungated DINOv2 vision transformer (`facebook/dinov2-small`)
+[@Oquab2024] into a concatenated $L_2$-normalised 1152-d descriptor, while also
+supporting newer DINOv3 backbones (`facebook/dinov3-vitb16-pretrain-lvd1689m`)
+via dynamic prefix/register token handling and a deterministic hash-based `mock`
+encoder for zero-GPU continuous integration.
-## Tracking, kinematics, and chemomechanical fracture events
+## Tracking, gap closing, kinematics, and chemomechanical fracture events
Frame-to-frame 1-to-1 association solves the gated Hungarian assignment problem
[@Kuhn1955] over $\alpha \cdot C_{\text{geo}}(i, j) + \beta \cdot (1 -
\cos(\mathbf{f}_i, \mathbf{f}_j))$, constructing a NetworkX [@Hagberg2008]
-`DiGraph`. From each trajectory, `compute_grain_kinematics` extracts time-series
-strain $\Delta d / d_0$, rocking/azimuthal misorientation, integrated intensity
-and volume evolution, and 3D eigenvalue anisotropy.
+`DiGraph`. Prior to topological event detection, `resolve_track_gaps` solves an
+augmented multi-scan linear assignment problem [@Jaqaman2008] with diffusive
+displacement scaling ($\Delta r^2 / \Delta t$) and semantic cosine gating to
+reconnect reflections across transient beam dropouts (`--max-gap`) while
+severing spurious cross-detector jumps. From each trajectory,
+`compute_grain_kinematics` extracts time-series strain $\Delta d / d_0$,
+rocking/azimuthal misorientation, integrated intensity and volume evolution, and
+3D eigenvalue anisotropy.
To capture grain fracture and coalescence, `resolve_splits_and_merges` augments
-unmatched birth and termination nodes with explicit `1 -> 2+` split and
-`2+ -> 1` merge edges gated by spatial proximity, integrated-intensity
+remaining unmatched birth and termination nodes with explicit `1 -> 2+` split
+and `2+ -> 1` merge edges gated by spatial proximity, integrated-intensity
conservation ($I_{\text{children}} / I_{\text{parent}}$), and minimum fragment
intensity share, assigning clean daughter track IDs so diverging post-fracture
-strains are tracked independently. Complementarily,
+strains are tracked independently (\autoref{fig:lineage}). Complementarily,
`compute_fracture_precursors` scores pre-split intragranular degradation across
contiguous scans (`min_persistence`) using 3D eigenvalue anisotropy, covariance
trace growth, internal 3D sub-peak bimodality, and foreground DINO patch-token
cosine dispersion.
-## Reproducibility and deployment
-
-BraggTrack (`pip install braggtrack`) provides eight CLI commands, culminating
-in `braggtrack-run`, which executes validation, segmentation, embedding,
-tracking, kinematics, and event detection and writes a `run_manifest.json`
-recording SHA-256 input checksums, configuration, software version, and per-stage
-runtimes. Automated CI tests Python 3.10–3.12, enforces Ruff linting/formatting,
-runs seven acceptance gates (`scripts/ci_report.py`), and executes the tutorial
-notebook end-to-end on bundled ESRF-ID03 data.
+
+
+## Quantitative evaluation, visualization, and reproducibility
+
+Across controlled 4D benchmarks (`scripts/benchmark_suite.py`;
+\autoref{fig:benchmarks}), multi-view `facebook/dinov2-small` descriptors
+maintain $> 0.96$ adjacent-scan cosine similarity under shot noise, 45% beam-flux
+jitter, and mosaic rocking-curve broadening (where byte-hash mocks collapse to
+$\approx 0$), eliminate crossing ID switches between reflections with identical
+3D covariance eigenvalues (`+45` vs. `-45` deg detector tilts), and recover
+100% of staggered beam dropouts with `--max-gap 2`.
+
+
+
+BraggTrack (`pip install braggtrack`) provides nine CLI commands, including
+`braggtrack-report` (generating publication figures and a zero-CDN offline
+interactive HTML 4D Lineage & Strain Explorer) and `braggtrack-run`, which
+executes validation, segmentation, embedding, tracking, kinematics, event
+detection, reporting, and `ImageD11`/NeXus export while writing a
+`run_manifest.json` with SHA-256 input checksums and per-stage runtimes.
+Automated CI tests Python 3.10–3.12, enforces Ruff linting/formatting, runs nine
+acceptance gates (`scripts/ci_report.py`), and executes the tutorial notebook
+end-to-end on bundled ESRF-ID03 data.
# Acknowledgements
diff --git a/pyproject.toml b/pyproject.toml
index bff1ae3..128b6ae 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "braggtrack"
-version = "0.2.0"
+version = "0.3.0"
description = "Semantic 4D kinematics and fracture tracking for operando diffraction."
readme = "README.md"
requires-python = ">=3.10"
@@ -59,6 +59,7 @@ braggtrack-segment-dataset = "braggtrack.cli.segment_dataset:main"
braggtrack-track-dataset = "braggtrack.cli.track_dataset:main"
braggtrack-embed-dataset = "braggtrack.cli.embed_dataset:main"
braggtrack-events = "braggtrack.cli.detect_events:main"
+braggtrack-report = "braggtrack.cli.report_dataset:main"
braggtrack-run = "braggtrack.cli.run_pipeline:main"
[tool.setuptools.packages.find]
diff --git a/scripts/benchmark_suite.py b/scripts/benchmark_suite.py
new file mode 100644
index 0000000..df171dc
--- /dev/null
+++ b/scripts/benchmark_suite.py
@@ -0,0 +1,477 @@
+#!/usr/bin/env python3
+"""Quantitative 5-scenario 4D diffraction benchmark suite (Geometry vs Mock vs Real DINOv2).
+
+Evaluates tracking continuity, ID-switch rates, multi-scan gap closing, and
+chemomechanical fracture precursor sensitivity across synthetic 3D volume
+sequences with realistic detector noise.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+REPO = Path(__file__).resolve().parents[1]
+if str(REPO) not in sys.path:
+ sys.path.insert(0, str(REPO))
+
+from braggtrack.segmentation.features import extract_instance_table
+from braggtrack.semantic import (
+ DEFAULT_DINO_MODEL,
+ crop_spot_cube,
+ make_multiview_encoder,
+ make_patch_encoder,
+ orthogonal_mips,
+)
+from braggtrack.tracking import (
+ GapClosingConfig,
+ GeometrySemanticCost,
+ PositionShapeCost,
+ PrecursorConfig,
+ SplitMergeConfig,
+ build_tracks,
+ compute_fracture_precursors,
+ compute_tracking_metrics,
+ generate_fracture_scenario,
+ tracks_to_table,
+)
+
+
+def _add_tilted_blob(
+ vol: np.ndarray,
+ center: tuple[float, float, float],
+ sigma: tuple[float, float, float],
+ amp: float,
+ tilt_xy: float = 0.0,
+) -> np.ndarray:
+ zz, yy, xx = np.indices(vol.shape, dtype=np.float64)
+ dz = zz - center[0]
+ dy = yy - center[1]
+ dx = xx - center[2]
+ c, s = np.cos(tilt_xy), np.sin(tilt_xy)
+ ry = c * dy - s * dx
+ rx = s * dy + c * dx
+ lobe = amp * np.exp(-0.5 * ((dz / sigma[0]) ** 2 + (ry / sigma[1]) ** 2 + (rx / sigma[2]) ** 2))
+ vol += lobe
+ return lobe
+
+
+def _embed_scan_tables(
+ volumes: list[np.ndarray],
+ label_volumes: list[np.ndarray],
+ scan_tables: list[list[dict[str, Any]]],
+ encoder: Any,
+) -> list[list[dict[str, Any]]]:
+ out_tables: list[list[dict[str, Any]]] = []
+ for vol, labs, table in zip(volumes, label_volumes, scan_tables):
+ rows_copy: list[dict[str, Any]] = []
+ for row in table:
+ rc = dict(row)
+ if encoder is not None:
+ lid = int(rc["label"])
+ masked, _ = crop_spot_cube(vol, labs, lid, rc, margin=3)
+ m_mu, m_chi, m_d = orthogonal_mips(masked)
+ rc["embedding"] = encoder.embed(m_mu, m_chi, m_d)
+ rows_copy.append(rc)
+ out_tables.append(rows_copy)
+ return out_tables
+
+
+def _gen_crossing_morphology(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]:
+ """Two reflections with orthogonal +45 deg vs -45 deg detector tilts (equal eigenvalues!) crossing paths.
+
+ Spots sit on antipodal points (6.4 voxels apart -> zero 3D mask overlap, identical
+ eigenvalues eig_1..3) and rotate by 92 deg between scans 2 and 3: the swapped
+ (-88 deg) chord is slightly shorter than the true (+92 deg) chord (delta_geo ~ 2.9),
+ causing pure geometry to swap identities while multi-view DINOv2 preserves 0% ID switches.
+ """
+ rng = np.random.default_rng(seed)
+ shape = (36, 64, 64)
+ angles_deg = [-40.0, -20.0, 0.0, 90.25, 110.0]
+ radius = 4.2
+ vols, labs, tables, gt = [], [], [], []
+ for t, ang_deg in enumerate(angles_deg):
+ vol = rng.normal(1.0, 0.25, size=shape).clip(0.0)
+ lab = np.zeros(shape, dtype=np.int32)
+ th1 = np.deg2rad(ang_deg)
+ th2 = th1 + np.pi
+ c1_use = (
+ 18.0 + radius * np.sin(th1) + rng.normal(0, 0.005),
+ 32.0 + rng.normal(0, 0.005),
+ 32.0 + radius * np.cos(th1) + rng.normal(0, 0.005),
+ )
+ c2_use = (
+ 18.0 + radius * np.sin(th2) + rng.normal(0, 0.005),
+ 32.0 + rng.normal(0, 0.005),
+ 32.0 + radius * np.cos(th2) + rng.normal(0, 0.005),
+ )
+
+ b1 = _add_tilted_blob(vol, c1_use, (1.4, 5.2, 0.95), 95.0, tilt_xy=np.pi / 4.0)
+ lab[b1 >= 14.0] = 1
+ b2 = _add_tilted_blob(vol, c2_use, (1.4, 5.2, 0.95), 95.0, tilt_xy=-np.pi / 4.0)
+ lab[(b2 >= 14.0) & (lab == 0)] = 2
+
+ tab = extract_instance_table(lab, vol)
+ for s_idx, r in enumerate(tab):
+ gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": int(r["label"])})
+ vols.append(vol)
+ labs.append(lab)
+ tables.append(tab)
+ return vols, labs, tables, gt
+
+
+def _gen_intensity_jitter(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]:
+ """Close neighbor pair under 45% beam-flux attenuation spikes and shot noise."""
+ rng = np.random.default_rng(seed)
+ shape = (18, 40, 40)
+ flux = [1.0, 0.55, 1.35, 0.60, 1.10]
+ vols, labs, tables, gt = [], [], [], []
+ for t, f_scale in enumerate(flux):
+ vol = rng.normal(0.8, 0.3, size=shape).clip(0.0)
+ lab = np.zeros(shape, dtype=np.int32)
+ b1 = _add_tilted_blob(vol, (9.0, 16.0 + 0.4 * t, 17.0), (1.3, 2.6, 1.1), 80.0 * f_scale, tilt_xy=0.5)
+ lab[b1 >= 7.0 * f_scale] = 1
+ b2 = _add_tilted_blob(vol, (9.0, 22.5 + 0.4 * t, 21.0), (1.3, 1.1, 2.6), 78.0 * f_scale, tilt_xy=-0.4)
+ lab[(b2 >= 7.0 * f_scale) & (lab == 0)] = 2
+ tab = extract_instance_table(lab, vol)
+ for s_idx, r in enumerate(tab):
+ gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": int(r["label"])})
+ vols.append(vol)
+ labs.append(lab)
+ tables.append(tab)
+ return vols, labs, tables, gt
+
+
+def _gen_mosaic_broadening(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]:
+ """Progressive intragranular rocking-curve broadening across 5 scans."""
+ rng = np.random.default_rng(seed)
+ shape = (24, 40, 40)
+ vols, labs, tables, gt = [], [], [], []
+ for t in range(5):
+ vol = rng.normal(0.8, 0.25, size=shape).clip(0.0)
+ lab = np.zeros(shape, dtype=np.int32)
+ sig_z = 1.1 + 0.35 * t
+ b1 = _add_tilted_blob(vol, (12.0, 14.0 + 0.6 * t, 15.0), (sig_z, 2.0, 1.2), 75.0, tilt_xy=0.3)
+ lab[b1 >= 6.5] = 1
+ b2 = _add_tilted_blob(vol, (12.0, 27.0 - 0.3 * t, 26.0), (1.2, 1.5, 2.2), 75.0, tilt_xy=-0.6)
+ lab[(b2 >= 6.5) & (lab == 0)] = 2
+ tab = extract_instance_table(lab, vol)
+ for s_idx, r in enumerate(tab):
+ gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": int(r["label"])})
+ vols.append(vol)
+ labs.append(lab)
+ tables.append(tab)
+ return vols, labs, tables, gt
+
+
+def _gen_beam_dropout(seed: int) -> tuple[list[np.ndarray], list[np.ndarray], list[list[dict]], list[dict]]:
+ """Three reflections across 6 scans with staggered 1-scan beam dropouts (scans 2 and 3)."""
+ rng = np.random.default_rng(seed)
+ shape = (18, 44, 44)
+ vols, labs, tables, gt = [], [], [], []
+ specs = [
+ (1, (9.0, 11.0, 12.0), (1.3, 2.2, 1.1), 0.3, {2}), # drops at scan 2
+ (2, (9.0, 22.0, 24.0), (1.3, 1.1, 2.2), -0.4, {3}), # drops at scan 3
+ (3, (9.0, 33.0, 15.0), (1.4, 1.5, 1.5), 0.0, set()), # continuous control
+ ]
+ for t in range(6):
+ vol = rng.normal(0.7, 0.25, size=shape).clip(0.0)
+ lab = np.zeros(shape, dtype=np.int32)
+ present_true_ids: list[int] = []
+ next_lbl = 1
+ for tid_true, base_c, sig, tilt, drop_scans in specs:
+ if t in drop_scans:
+ continue
+ center = (base_c[0], base_c[1] + 0.35 * t, base_c[2] + 0.2 * t)
+ b = _add_tilted_blob(vol, center, sig, 80.0, tilt_xy=tilt)
+ lab[(b >= 7.0) & (lab == 0)] = next_lbl
+ present_true_ids.append(tid_true)
+ next_lbl += 1
+ tab = extract_instance_table(lab, vol)
+ for s_idx in range(len(tab)):
+ gt.append({"scan_idx": t, "spot_idx": s_idx, "true_id": present_true_ids[s_idx]})
+ vols.append(vol)
+ labs.append(lab)
+ tables.append(tab)
+ return vols, labs, tables, gt
+
+
+def _adjacent_cosine_stability(embedded_tables: list[list[dict[str, Any]]], gt: list[dict[str, int]]) -> float:
+ """Mean frame-to-frame cosine similarity along ground-truth trajectories."""
+ gt_map = {(r["scan_idx"], r["spot_idx"]): r["true_id"] for r in gt}
+ by_true: dict[int, list[tuple[int, np.ndarray]]] = {}
+ for s_idx, rows in enumerate(embedded_tables):
+ for spot_idx, row in enumerate(rows):
+ emb = row.get("embedding")
+ tid = gt_map.get((s_idx, spot_idx))
+ if emb is not None and tid is not None:
+ by_true.setdefault(tid, []).append((s_idx, np.asarray(emb, dtype=np.float64)))
+ sims: list[float] = []
+ for obs in by_true.values():
+ obs.sort(key=lambda x: x[0])
+ for i in range(len(obs) - 1):
+ a, b = obs[i][1], obs[i + 1][1]
+ na, nb = np.linalg.norm(a), np.linalg.norm(b)
+ if na > 0 and nb > 0:
+ sims.append(float(np.dot(a, b) / (na * nb)))
+ return float(np.mean(sims)) if sims else 0.0
+
+
+def _ci95(vals: list[float]) -> list[float]:
+ if not vals:
+ return [0.0, 0.0]
+ arr = np.asarray(vals, dtype=np.float64)
+ if len(arr) == 1:
+ v = round(float(arr[0]), 4)
+ return [v, v]
+ rng = np.random.default_rng(12345)
+ boots = [float(np.mean(rng.choice(arr, size=len(arr), replace=True))) for _ in range(500)]
+ return [round(float(np.percentile(boots, 2.5)), 4), round(float(np.percentile(boots, 97.5)), 4)]
+
+
+def run_suite(
+ backends: list[str],
+ *,
+ trials: int = 4,
+ model_name: str = DEFAULT_DINO_MODEL,
+) -> dict[str, Any]:
+ encoders: dict[str, Any] = {}
+ patch_encoders: dict[str, Any] = {}
+ for b in backends:
+ if b == "geo":
+ encoders[b] = None
+ patch_encoders[b] = None
+ elif b == "mock":
+ encoders[b] = make_multiview_encoder("mock")
+ patch_encoders[b] = make_patch_encoder("mock")
+ elif b == "torch":
+ encoders[b] = make_multiview_encoder("torch", model_name=model_name)
+ patch_encoders[b] = make_patch_encoder("torch", model_name=model_name)
+ else:
+ raise ValueError(f"Unsupported backend: {b}")
+
+ results: dict[str, dict[str, Any]] = {}
+ standard_scenarios = [
+ ("crossing_morphology", _gen_crossing_morphology, 0),
+ ("intensity_jitter", _gen_intensity_jitter, 0),
+ ("mosaic_broadening", _gen_mosaic_broadening, 0),
+ ("beam_dropout_no_gap", _gen_beam_dropout, 0),
+ ("beam_dropout_gap2", _gen_beam_dropout, 2),
+ ]
+
+ for scen_name, gen_fn, max_gap in standard_scenarios:
+ results[scen_name] = {}
+ for b in backends:
+ id_sw_rates: list[float] = []
+ frag_ratios: list[float] = []
+ cos_stabilities: list[float] = []
+ gap_closures: list[float] = []
+ t0 = time.perf_counter()
+
+ for trial in range(trials):
+ vols, labs, tables, gt = gen_fn(100 + trial)
+ emb_tables = _embed_scan_tables(vols, labs, tables, encoders[b])
+ geo = PositionShapeCost(position_weight=1.0, shape_weight=0.5)
+ # Weighted semantic cost disambiguates crossing morphologies when embeddings are meaningful
+ cost_fn = geo if b == "geo" else GeometrySemanticCost(geo, cost_alpha=1.0, cost_beta=25.0)
+ gap_cfg = (
+ GapClosingConfig(max_gap=max_gap, max_cost=35.0, min_cosine_similarity=0.0 if b == "mock" else 0.5)
+ if max_gap > 0
+ else None
+ )
+ G = build_tracks(emb_tables, cost_fn=cost_fn, gap_closing_config=gap_cfg)
+ m = compute_tracking_metrics(G, n_scans=len(tables), ground_truth=gt)
+ id_sw_rates.append(float(m["id_switch_rate"]))
+ frag_ratios.append(float(m["fragmentation_ratio"]))
+ gap_closures.append(float(m["gap_closed_count"]))
+ if b != "geo":
+ cos_stabilities.append(_adjacent_cosine_stability(emb_tables, gt))
+
+ elapsed = round(time.perf_counter() - t0, 3)
+ results[scen_name][b] = {
+ "mean_id_switch_rate": round(float(np.mean(id_sw_rates)), 4),
+ "id_switch_rate_ci95": _ci95(id_sw_rates),
+ "mean_fragmentation_ratio": round(float(np.mean(frag_ratios)), 4),
+ "fragmentation_ratio_ci95": _ci95(frag_ratios),
+ "mean_gap_closed_count": round(float(np.mean(gap_closures)), 2),
+ "mean_cosine_stability": round(float(np.mean(cos_stabilities)), 4) if cos_stabilities else None,
+ "elapsed_sec": elapsed,
+ }
+
+ # Scenario 5: Chemomechanical fracture (split + merge + persistent precursor contrast)
+ results["chemomechanical_fracture"] = {}
+ for b in backends:
+ precursor_contrasts: list[float] = []
+ top_hit_rates: list[float] = []
+ split_recalls: list[float] = []
+ merge_recalls: list[float] = []
+ t0 = time.perf_counter()
+
+ for trial in range(trials):
+ vols, labs, tables, _ = generate_fracture_scenario(n_scans=5, seed=200 + trial)
+ emb_tables = _embed_scan_tables(vols, labs, tables, encoders[b])
+ geo = PositionShapeCost()
+ cost_fn = geo if b == "geo" else GeometrySemanticCost(geo, cost_alpha=1.0, cost_beta=0.5)
+ G = build_tracks(
+ emb_tables,
+ cost_fn=cost_fn,
+ split_merge_config=SplitMergeConfig(max_distance=20.0),
+ )
+ m = compute_tracking_metrics(G, n_scans=5)
+ split_recalls.append(1.0 if m["split_count"] == 1 else 0.0)
+ merge_recalls.append(1.0 if m["merge_count"] == 1 else 0.0)
+
+ track_rows = tracks_to_table(G)
+ cands, obs_scores = compute_fracture_precursors(
+ track_rows,
+ PrecursorConfig(score_threshold=0.35, min_persistence=2),
+ volumes=vols,
+ label_volumes=labs,
+ patch_encoder=patch_encoders[b],
+ lineage_events=G.graph["lineage_events"],
+ )
+ top_ok = 1.0 if (cands and cands[0].subsequent_split) else 0.0
+ top_hit_rates.append(top_ok)
+
+ # Contrast between fracturing grain precursor score and stable control grain
+ split_tids = {ev.parent_track_ids[0] for ev in G.graph["lineage_events"] if ev.event_type == "split"}
+ frac_scores = [o.precursor_score for o in obs_scores if o.track_id in split_tids and o.scan_idx < 2]
+ ctrl_scores = [
+ o.precursor_score
+ for o in obs_scores
+ if o.track_id not in split_tids and o.scan_idx < 2 and o.roi_subpeak_count == 1
+ ]
+ if frac_scores and ctrl_scores:
+ precursor_contrasts.append(float(np.mean(frac_scores) - np.mean(ctrl_scores)))
+
+ results["chemomechanical_fracture"][b] = {
+ "split_detection_rate": round(float(np.mean(split_recalls)), 4),
+ "merge_detection_rate": round(float(np.mean(merge_recalls)), 4),
+ "top1_precursor_precision": round(float(np.mean(top_hit_rates)), 4),
+ "mean_precursor_contrast": round(float(np.mean(precursor_contrasts)), 4) if precursor_contrasts else 0.0,
+ "precursor_contrast_ci95": _ci95(precursor_contrasts),
+ "elapsed_sec": round(time.perf_counter() - t0, 3),
+ }
+
+ return {
+ "schema_version": "benchmark_suite.v1",
+ "backends": backends,
+ "model": model_name if "torch" in backends else None,
+ "trials": trials,
+ "scenarios": results,
+ }
+
+
+def _save_benchmark_figure(payload: dict[str, Any], out_path: Path) -> None:
+ import matplotlib
+
+ matplotlib.use("Agg", force=False)
+ import matplotlib.pyplot as plt
+
+ from braggtrack.viz._style import OKABE_ITO, apply_axis_style
+
+ backends = payload["backends"]
+ scen = payload["scenarios"]
+ labels_map = {"geo": "Geometry-Only", "mock": "Mock-Hash", "torch": "Real DINOv2 (ViT-S/14)"}
+ colors_map = {"geo": "#64748B", "mock": OKABE_ITO[4], "torch": OKABE_ITO[0]}
+
+ fig, axes = plt.subplots(1, 3, figsize=(13.2, 4.0))
+
+ # Panel A: Crossing ID-switch rate
+ xs = np.arange(len(backends))
+ sw_vals = [scen["crossing_morphology"][b]["mean_id_switch_rate"] * 100.0 for b in backends]
+ axes[0].bar(xs, sw_vals, color=[colors_map.get(b, OKABE_ITO[0]) for b in backends], width=0.55)
+ axes[0].set_xticks(xs)
+ axes[0].set_xticklabels([labels_map.get(b, b) for b in backends], fontsize=8.5, rotation=12)
+ axes[0].set_ylabel("ID-Switch Rate (%)", fontsize=9)
+ axes[0].set_title("A. Crossing Tilted Reflections", fontsize=10, fontweight="bold")
+ for x_i, v in enumerate(sw_vals):
+ axes[0].text(x_i, v + 0.8, f"{v:.1f}%", ha="center", fontsize=8.5, fontweight="bold")
+ axes[0].set_ylim(0, max(sw_vals + [10.0]) * 1.25)
+ apply_axis_style(axes[0])
+
+ # Panel B: Descriptor Noise Stability (Adjacent Cosine Similarity)
+ sem_backends = [b for b in backends if b != "geo"]
+ if sem_backends:
+ w = 0.25
+ s_names = ["crossing_morphology", "intensity_jitter", "mosaic_broadening"]
+ s_short = ["Crossing", "Flux Jitter", "Mosaic Spread"]
+ x_s = np.arange(len(s_names))
+ for idx_b, b in enumerate(sem_backends):
+ vals = [scen[sn][b]["mean_cosine_stability"] or 0.0 for sn in s_names]
+ offset = (idx_b - 0.5 * (len(sem_backends) - 1)) * (w + 0.04)
+ axes[1].bar(
+ x_s + offset,
+ vals,
+ width=w,
+ color=colors_map.get(b, OKABE_ITO[2]),
+ label=labels_map.get(b, b),
+ )
+ axes[1].set_xticks(x_s)
+ axes[1].set_xticklabels(s_short, fontsize=8.5)
+ axes[1].set_ylim(-0.15, 1.08)
+ axes[1].axhline(0.0, color="#94A3B8", linewidth=0.8)
+ axes[1].set_ylabel("Adjacent-Scan Cosine Similarity", fontsize=9)
+ axes[1].set_title("B. Descriptor Noise & Drift Invariance", fontsize=10, fontweight="bold")
+ axes[1].legend(fontsize=8, loc="lower right", frameon=True)
+ apply_axis_style(axes[1])
+
+ # Panel C: Beam-Dropout Fragmentation (No Gap vs max_gap=2)
+ ref_b = "torch" if "torch" in backends else backends[0]
+ frag_vals = [
+ scen["beam_dropout_no_gap"][ref_b]["mean_fragmentation_ratio"] * 100.0,
+ scen["beam_dropout_gap2"][ref_b]["mean_fragmentation_ratio"] * 100.0,
+ ]
+ axes[2].bar([0, 1], frag_vals, color=["#EF4444", OKABE_ITO[2]], width=0.5)
+ axes[2].set_xticks([0, 1])
+ axes[2].set_xticklabels(["max_gap = 0\n(Frame-by-Frame)", "max_gap = 2\n(Gap-Closing)"], fontsize=8.5)
+ axes[2].set_ylabel("Track Fragmentation Ratio (%)", fontsize=9)
+ axes[2].set_title("C. Staggered Beam-Dropout Recovery", fontsize=10, fontweight="bold")
+ for x_i, v in enumerate(frag_vals):
+ axes[2].text(x_i, v + 1.5, f"{v:.1f}%", ha="center", fontsize=8.5, fontweight="bold")
+ axes[2].set_ylim(0, max(frag_vals + [20.0]) * 1.25)
+ apply_axis_style(axes[2])
+
+ fig.suptitle(
+ "BraggTrack v0.3.0 Quantitative 4D Diffraction Benchmark Suite", fontsize=11.5, fontweight="bold", y=1.02
+ )
+ fig.tight_layout()
+ out_path.parent.mkdir(parents=True, exist_ok=True)
+ fig.savefig(out_path, dpi=200, bbox_inches="tight")
+ plt.close(fig)
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--backends", default="geo,mock", help="Comma-separated backends from {geo,mock,torch}")
+ parser.add_argument("--include-torch", action="store_true", help="Append real DINOv2 torch backend")
+ parser.add_argument("--trials", type=int, default=4, help="Random-seed trials per scenario")
+ parser.add_argument("--model", default=DEFAULT_DINO_MODEL, help="HuggingFace model ID for torch backend")
+ parser.add_argument("--json-out", default=None, help="Optional path to write JSON benchmark report")
+ parser.add_argument("--plot-out", default=None, help="Optional path to write PNG/PDF benchmark summary figure")
+ args = parser.parse_args(argv)
+
+ backends = [b.strip() for b in args.backends.split(",") if b.strip()]
+ if args.include_torch and "torch" not in backends:
+ backends.append("torch")
+
+ payload = run_suite(backends, trials=max(1, args.trials), model_name=args.model)
+ if args.json_out:
+ out_j = Path(args.json_out)
+ out_j.parent.mkdir(parents=True, exist_ok=True)
+ out_j.write_text(json.dumps(payload, indent=2))
+ if args.plot_out:
+ _save_benchmark_figure(payload, Path(args.plot_out))
+
+ print(json.dumps(payload, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/check_report_and_export_acceptance.py b/scripts/check_report_and_export_acceptance.py
new file mode 100644
index 0000000..02ed333
--- /dev/null
+++ b/scripts/check_report_and_export_acceptance.py
@@ -0,0 +1,223 @@
+"""Report, multi-scan gap-closing, and crystallographic export acceptance gate.
+
+Acceptance criteria:
+ 1. Multi-scan gap closing (--max-gap 2) reconnects transient 1-scan beam dropouts
+ with zero track fragmentation on a controlled dropout sequence.
+ 2. End-to-end pipeline (braggtrack-run --report --detector-json --export-imaged11
+ --export-nexus) executes on bundled ESRF-ID03 operando data and emits:
+ - Interactive self-contained HTML explorer (report/report.html) + static PNGs
+ - ImageD11 columnfile (.flt) with physical (tth, eta, d_spacing) columns
+ - NeXus NXprocess HDF5 summary (.nxs) carrying tracks, kinematics, events, and precursors.
+"""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import h5py
+
+_REPO_ROOT = Path(__file__).resolve().parents[1]
+if str(_REPO_ROOT) not in sys.path:
+ sys.path.insert(0, str(_REPO_ROOT))
+
+from braggtrack.cli._utils import write_csv
+from braggtrack.io import DetectorGeometry, read_imaged11_flt, resolve_dataset_root
+
+ACCEPTANCE_OUTDIR = Path("artifacts/report_export_acceptance")
+DATASET_ROOT = resolve_dataset_root(None)
+
+
+def _check_gap_closing_cli(failures: list[str]) -> dict[str, object]:
+ with tempfile.TemporaryDirectory() as tmp:
+ tmp_root = Path(tmp)
+ seg_dir = tmp_root / "seg"
+ trk_dir = tmp_root / "trk"
+
+ # Two spots across 4 scans; spot 1 drops out at scan0002 (scan_idx=1) and reappears at scan0003
+ scan_specs = [
+ [(1, 10.0, 20.0, 30.0), (2, 10.0, 60.0, 60.0)],
+ [(1, 10.0, 60.2, 60.1)],
+ [(1, 10.1, 20.3, 30.2), (2, 10.0, 60.4, 60.2)],
+ [(1, 10.2, 20.5, 30.4), (2, 10.1, 60.6, 60.3)],
+ ]
+ for idx, specs in enumerate(scan_specs):
+ sdir = seg_dir / f"scan{idx + 1:04d}"
+ sdir.mkdir(parents=True, exist_ok=True)
+ rows = [
+ {
+ "label": lbl,
+ "voxel_count": 20,
+ "integrated_intensity": 500.0,
+ "centroid_mu": mu,
+ "centroid_chi": chi,
+ "centroid_d": d,
+ "bbox_min_mu": int(mu) - 2,
+ "bbox_min_d": int(d) - 2,
+ "bbox_min_chi": int(chi) - 2,
+ "bbox_max_mu": int(mu) + 3,
+ "bbox_max_d": int(d) + 3,
+ "bbox_max_chi": int(chi) + 3,
+ "cov_00": 1.0,
+ "cov_01": 0.0,
+ "cov_02": 0.0,
+ "cov_11": 1.0,
+ "cov_12": 0.0,
+ "cov_22": 1.0,
+ "eig_1": 1.0,
+ "eig_2": 1.0,
+ "eig_3": 1.0,
+ }
+ for lbl, mu, chi, d in specs
+ ]
+ write_csv(sdir / "features.csv", rows)
+
+ proc = subprocess.run(
+ [
+ sys.executable,
+ "-m",
+ "braggtrack.cli.track_dataset",
+ str(seg_dir),
+ "--outdir",
+ str(trk_dir),
+ "--max-gap",
+ "2",
+ ],
+ check=False,
+ capture_output=True,
+ text=True,
+ )
+ if proc.returncode != 0:
+ failures.append(f"track_dataset --max-gap 2 failed: {proc.stderr.strip() or proc.stdout.strip()}")
+ return {}
+
+ summary = json.loads(proc.stdout)
+ if int(summary.get("gap_closed_count", 0)) != 1:
+ failures.append(f"Expected gap_closed_count==1, got {summary.get('gap_closed_count')}")
+ if float(summary.get("fragmentation_ratio", 1.0)) != 0.0:
+ failures.append(
+ f"Expected fragmentation_ratio==0.0 after gap closing, got {summary.get('fragmentation_ratio')}"
+ )
+ return {
+ "gap_closed_count": summary.get("gap_closed_count"),
+ "total_tracks": summary.get("total_tracks"),
+ "fragmentation_ratio": summary.get("fragmentation_ratio"),
+ }
+
+
+def _check_pipeline_report_and_exports(failures: list[str]) -> dict[str, object]:
+ ACCEPTANCE_OUTDIR.mkdir(parents=True, exist_ok=True)
+ det_json = ACCEPTANCE_OUTDIR / "detector_geometry.json"
+ flt_path = ACCEPTANCE_OUTDIR / "operando_peaks.flt"
+ nxs_path = ACCEPTANCE_OUTDIR / "operando_results.nxs"
+
+ geom = DetectorGeometry(
+ distance_mm=150.0,
+ pixel_size_y_mm=0.075,
+ pixel_size_x_mm=0.075,
+ beam_center_y_px=20.0,
+ beam_center_x_px=20.0,
+ wavelength_angstrom=0.50,
+ omega_origin_deg=10.0,
+ omega_step_deg=0.05,
+ )
+ geom.save_json(det_json)
+
+ proc = subprocess.run(
+ [
+ sys.executable,
+ "-m",
+ "braggtrack.cli.run_pipeline",
+ str(DATASET_ROOT),
+ "--outdir",
+ str(ACCEPTANCE_OUTDIR),
+ "--backend",
+ "mock",
+ "--max-gap",
+ "1",
+ "--detector-json",
+ str(det_json),
+ "--export-imaged11",
+ str(flt_path),
+ "--export-nexus",
+ str(nxs_path),
+ "--report",
+ ],
+ check=False,
+ capture_output=True,
+ text=True,
+ )
+ if proc.returncode != 0:
+ failures.append(f"run_pipeline --report failed: {proc.stderr.strip() or proc.stdout.strip()}")
+ return {}
+
+ html_path = ACCEPTANCE_OUTDIR / "report" / "report.html"
+ if not html_path.exists() or html_path.stat().st_size < 5_000:
+ failures.append(f"Missing or undersized HTML report: {html_path}")
+ else:
+ html_text = html_path.read_text(encoding="utf-8")
+ if "report.v1" not in html_text or "data:image/png;base64," not in html_text:
+ failures.append("report.html missing embedded report.v1 payload or base64 MIP thumbnails")
+
+ fig_dir = ACCEPTANCE_OUTDIR / "report" / "figures"
+ expected_figs = ("lineage_dag.png", "kinematics_dashboard.png", "precursor_timeline.png")
+ for fname in expected_figs:
+ fpath = fig_dir / fname
+ if not fpath.exists() or fpath.stat().st_size < 1_000:
+ failures.append(f"Missing or empty report figure: {fpath}")
+
+ if not flt_path.exists():
+ failures.append(f"Missing ImageD11 export: {flt_path}")
+ flt_rows = []
+ else:
+ flt_params, flt_rows = read_imaged11_flt(flt_path)
+ if "wavelength" not in flt_params:
+ failures.append("ImageD11 .flt header missing wavelength parameter")
+ if not flt_rows:
+ failures.append("ImageD11 .flt export contained 0 rows")
+ else:
+ for col in ("sc", "fc", "omega", "tth", "eta", "d_spacing", "track_id"):
+ if col not in flt_rows[0]:
+ failures.append(f"ImageD11 .flt missing column {col}")
+ if float(flt_rows[0].get("d_spacing", 0.0)) <= 0.0:
+ failures.append("ImageD11 .flt d_spacing must be positive when DetectorGeometry is provided")
+
+ if not nxs_path.exists():
+ failures.append(f"Missing NeXus HDF5 export: {nxs_path}")
+ else:
+ with h5py.File(nxs_path, "r") as h5:
+ if h5.attrs.get("NX_class") != "NXroot":
+ failures.append("NeXus root missing NX_class=NXroot")
+ if "/entry/braggtrack" not in h5:
+ failures.append("NeXus file missing /entry/braggtrack NXprocess group")
+ else:
+ grp = h5["/entry/braggtrack"]
+ for sub in ("tracks", "kinematics", "events", "precursors"):
+ if sub not in grp:
+ failures.append(f"NeXus /entry/braggtrack missing {sub} table group")
+
+ return {
+ "html_report_bytes": html_path.stat().st_size if html_path.exists() else 0,
+ "flt_row_count": len(flt_rows),
+ "nexus_bytes": nxs_path.stat().st_size if nxs_path.exists() else 0,
+ }
+
+
+def main() -> int:
+ failures: list[str] = []
+ gap_report = _check_gap_closing_cli(failures)
+ export_report = _check_pipeline_report_and_exports(failures)
+ payload = {
+ "gap_closing": gap_report,
+ "report_and_exports": export_report,
+ "failures": failures,
+ }
+ print(json.dumps(payload, indent=2))
+ return 0 if not failures else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/ci_report.py b/scripts/ci_report.py
index 97ae603..3f78024 100644
--- a/scripts/ci_report.py
+++ b/scripts/ci_report.py
@@ -131,7 +131,12 @@ def main() -> int:
)
evt_ok = evt_ok_cmd and isinstance(evt_payload, dict) and evt_payload.get("failures") == []
- all_ok = unit_ok and acc_ok and smoke_ok and seg_ok and trk_ok and emb_ok and dino_ok and evt_ok
+ rep_ok_cmd, rep_payload, _ = run_cmd_json(
+ [sys.executable, "scripts/check_report_and_export_acceptance.py"], "Report & Export Acceptance"
+ )
+ rep_ok = rep_ok_cmd and isinstance(rep_payload, dict) and rep_payload.get("failures") == []
+
+ all_ok = unit_ok and acc_ok and smoke_ok and seg_ok and trk_ok and emb_ok and dino_ok and evt_ok and rep_ok
print("\n=== Summary ===")
print(f"unit_tests={'PASS' if unit_ok else 'FAIL'}")
print(f"io_acceptance={'PASS' if acc_ok else 'FAIL'}")
@@ -141,6 +146,7 @@ def main() -> int:
print(f"embedding_acceptance={'PASS' if emb_ok else 'FAIL'}")
print(f"dino_acceptance={'PASS' if dino_ok else 'FAIL'}")
print(f"events_acceptance={'PASS' if evt_ok else 'FAIL'}")
+ print(f"report_and_export_acceptance={'PASS' if rep_ok else 'FAIL'}")
print(f"overall={'PASS' if all_ok else 'FAIL'}")
return 0 if all_ok else 1
diff --git a/scripts/generate_paper_figures.py b/scripts/generate_paper_figures.py
new file mode 100644
index 0000000..3b874bc
--- /dev/null
+++ b/scripts/generate_paper_figures.py
@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""Generate publication-grade composite figures for paper.md and README.md."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+import matplotlib
+import numpy as np
+
+matplotlib.use("Agg", force=False)
+import matplotlib.pyplot as plt # noqa: E402
+
+REPO = Path(__file__).resolve().parents[1]
+if str(REPO) not in sys.path:
+ sys.path.insert(0, str(REPO))
+
+from braggtrack.io import AxisCalibration, discover_operando_scans, load_primary_volume, sample_operando_root # noqa: E402
+from braggtrack.segmentation import ( # noqa: E402
+ extract_instance_table,
+ fill_holes_binary,
+ merge_nearby_labels,
+ otsu_threshold,
+ relabel_sequential,
+ remove_small_objects,
+ segment_classical,
+ smooth_thresholds,
+)
+from braggtrack.semantic import make_patch_encoder # noqa: E402
+from braggtrack.tracking import ( # noqa: E402
+ PositionShapeCost,
+ PrecursorConfig,
+ SplitMergeConfig,
+ build_tracks,
+ compute_fracture_precursors,
+ compute_grain_kinematics,
+ events_to_table,
+ generate_fracture_scenario,
+ tracks_to_table,
+)
+from braggtrack.viz._style import OKABE_ITO, apply_axis_style # noqa: E402
+
+
+def build_fracture_and_operando_figure(out_path: Path, *, backend: str = "auto") -> None:
+ # 1. Synthetic 4D chemomechanical fracture + merge + persistent precursor decomposition
+ vols, labs, tables, _ = generate_fracture_scenario(n_scans=5, seed=42)
+ G_frac = build_tracks(
+ tables,
+ cost_fn=PositionShapeCost(),
+ split_merge_config=SplitMergeConfig(max_distance=20.0),
+ )
+ events = G_frac.graph["lineage_events"]
+ ev_rows = events_to_table(events)
+ frac_rows = tracks_to_table(G_frac)
+ cands, obs_scores = compute_fracture_precursors(
+ frac_rows,
+ PrecursorConfig(score_threshold=0.35, min_persistence=2),
+ volumes=vols,
+ label_volumes=labs,
+ patch_encoder=make_patch_encoder(backend),
+ lineage_events=events,
+ )
+
+ # 2. Real ESRF-ID03 operando cathode sequence with physical calibration
+ cal = AxisCalibration(
+ mu_origin=12.0,
+ mu_step=0.02,
+ chi_origin=-2.5,
+ chi_step=0.015,
+ d_origin=2.065,
+ d_step=-0.00015,
+ calibrated=True,
+ )
+ scans = discover_operando_scans(sample_operando_root())
+ op_vols = [load_primary_volume(s.path) for s in scans]
+ sm_thr = smooth_thresholds([otsu_threshold(v.ravel()) for v in op_vols], window=5)
+ op_tables = []
+ for v, thr in zip(op_vols, sm_thr):
+ res = segment_classical(
+ v,
+ threshold=float(thr),
+ blur_passes=1,
+ h_value=0.1,
+ min_seed_separation=2,
+ seed_peak_fraction=0.2,
+ seed_response_percentile=99.95,
+ )
+ lb = remove_small_objects(res.labeled_volume, min_size=8)
+ lb = np.where(fill_holes_binary(lb > 0), lb, 0)
+ lb = relabel_sequential(merge_nearby_labels(lb, v, max_centroid_distance=15))
+ op_tables.append(extract_instance_table(lb, v, calibration=cal))
+
+ G_op = build_tracks(op_tables, cost_fn=PositionShapeCost(gate_mu=2.0, gate_chi=2.0, gate_d=0.05))
+ gks = compute_grain_kinematics(tracks_to_table(G_op))
+
+ fig, axes = plt.subplots(2, 2, figsize=(12.6, 8.2))
+ ax_dag, ax_mip = axes[0]
+ ax_prec, ax_op = axes[1]
+
+ # Panel A: 4D Lineage DAG (Split 1->2 & Merge 2->1)
+ by_track: dict[int, list[dict]] = {}
+ node_pos: dict[tuple[int, int], tuple[float, float]] = {}
+ for r in frac_rows:
+ tid = int(r["track_id"])
+ sidx = int(r["scan_idx"])
+ by_track.setdefault(tid, []).append(r)
+ node_pos[(tid, sidx)] = (float(sidx), float(r["centroid_d"]))
+
+ for idx_t, (tid, obs) in enumerate(sorted(by_track.items())):
+ obs.sort(key=lambda x: int(x["scan_idx"]))
+ col = OKABE_ITO[idx_t % len(OKABE_ITO)]
+ xs = [int(x["scan_idx"]) for x in obs]
+ ys = [float(x["centroid_d"]) for x in obs]
+ ax_dag.plot(xs, ys, "o-", color=col, linewidth=2.0, markersize=6.5, label=f"T{tid}")
+
+ for ev in ev_rows:
+ etype = ev["event_type"]
+ s_from, s_to = int(ev["scan_idx_from"]), int(ev["scan_idx_to"])
+ pids = [int(p) for p in str(ev["parent_track_ids"]).split(";") if p]
+ cids = [int(c) for c in str(ev["child_track_ids"]).split(";") if c]
+ ecol = "#DC2626" if etype == "split" else "#059669"
+ ls = "--" if etype == "split" else "-."
+ for pid in pids:
+ for cid in cids:
+ if (pid, s_from) in node_pos and (cid, s_to) in node_pos:
+ x0, y0 = node_pos[(pid, s_from)]
+ x1, y1 = node_pos[(cid, s_to)]
+ ax_dag.plot([x0, x1], [y0, y1], linestyle=ls, color=ecol, linewidth=2.1)
+ ax_dag.annotate(
+ etype.upper(),
+ xy=((x0 + x1) / 2, (y0 + y1) / 2),
+ fontsize=7.5,
+ fontweight="bold",
+ color=ecol,
+ bbox={"boxstyle": "round,pad=0.18", "fc": "white", "ec": ecol, "alpha": 0.9},
+ )
+ ax_dag.set_title("A. 4D Trajectory Lineage DAG (1→2 Split & 2→1 Merge)", fontsize=10, fontweight="bold")
+ ax_dag.set_xlabel("Scan index", fontsize=9)
+ ax_dag.set_ylabel("Reciprocal-space position d (voxel)", fontsize=9)
+ ax_dag.legend(fontsize=7.5, ncol=3, loc="upper left", frameon=True)
+ apply_axis_style(ax_dag)
+
+ # Panel B: Pre-split bimodal broadening vs post-split cleavage MIP strip
+ # Show scan 0 (incipient), scan 1 (bimodal precursor), scan 2 (cleaved daughters)
+ strip_imgs = []
+ for s_i in (0, 1, 2):
+ # Crop region around the fracturing grain (d ~ 10..26, chi ~ 12..28)
+ mip_patch = vols[s_i].max(axis=0)[6:28, 10:30]
+ mip_norm = mip_patch / max(float(mip_patch.max()), 1e-6)
+ strip_imgs.append(mip_norm)
+ sep = np.ones((strip_imgs[0].shape[0], 2), dtype=np.float64) * 0.95
+ montage = np.concatenate([strip_imgs[0], sep, strip_imgs[1], sep, strip_imgs[2]], axis=1)
+ im = ax_mip.imshow(montage, cmap="magma", origin="lower", aspect="auto", vmin=0.0, vmax=1.0)
+ w_p = strip_imgs[0].shape[1]
+ ax_mip.set_xticks([w_p * 0.5, w_p * 1.5 + 2, w_p * 2.5 + 4])
+ ax_mip.set_xticklabels(
+ ["Scan 0\n(Incipient)", "Scan 1\n(Bimodal Precursor)", "Scan 2\n(1→2 Cleavage)"], fontsize=8.5
+ )
+ ax_mip.set_yticks([])
+ ax_mip.set_title("B. Reciprocal-Space Reflection Broadening to Cleavage (μ-MIP)", fontsize=10, fontweight="bold")
+ fig.colorbar(im, ax=ax_mip, fraction=0.042, pad=0.03, label="Normalized intensity")
+
+ # Panel C: Persistent Precursor Decomposition (Fracturing Grain vs Control)
+ top_tid = cands[0].track_id if cands else 0
+ top_obs = sorted([o for o in obs_scores if o.track_id == top_tid], key=lambda o: o.scan_idx)
+ ctrl_tids = [
+ tid
+ for tid in sorted({o.track_id for o in obs_scores})
+ if tid != top_tid and len([o for o in obs_scores if o.track_id == tid]) == 5
+ ]
+ ctrl_tid = ctrl_tids[0] if ctrl_tids else top_tid
+ ctrl_obs = sorted([o for o in obs_scores if o.track_id == ctrl_tid], key=lambda o: o.scan_idx)
+
+ ax_prec.plot(
+ [o.scan_idx for o in top_obs],
+ [o.precursor_score for o in top_obs],
+ "o-",
+ color="#DC2626",
+ linewidth=2.3,
+ markersize=6.5,
+ label=f"Fracturing grain (T{top_tid} composite)",
+ )
+ ax_prec.plot(
+ [o.scan_idx for o in top_obs],
+ [o.roi_dispersion_term for o in top_obs],
+ "^--",
+ color=OKABE_ITO[0],
+ linewidth=1.6,
+ label=f"T{top_tid} 3D sub-peak + DINO patch dispersion",
+ )
+ ax_prec.plot(
+ [o.scan_idx for o in ctrl_obs],
+ [o.precursor_score for o in ctrl_obs],
+ "s-",
+ color="#64748B",
+ linewidth=1.8,
+ markersize=5.5,
+ label=f"Stable control grain (T{ctrl_tid})",
+ )
+ ax_prec.axhline(0.35, linestyle=":", color="#475569", linewidth=1.2, label="Persistence threshold")
+ ax_prec.axvline(1.5, linestyle="--", color="#DC2626", alpha=0.65, linewidth=1.2, label="Cleavage onset")
+ ax_prec.set_ylim(-0.03, 1.05)
+ ax_prec.set_xticks(range(5))
+ ax_prec.set_xlabel("Scan index", fontsize=9)
+ ax_prec.set_ylabel("Precursor score [0, 1]", fontsize=9)
+ ax_prec.set_title("C. Persistent Precursor Ranking vs. Control Grain", fontsize=10, fontweight="bold")
+ ax_prec.legend(fontsize=7.8, loc="upper right", frameon=True)
+ apply_axis_style(ax_prec)
+
+ # Panel D: Bundled ESRF-ID03 Operando Multi-Scan Grain Kinematics
+ multi_gks = [g for g in gks if len(g.scan_indices) >= 2]
+ for idx_g, gk in enumerate(multi_gks[:6]):
+ col = OKABE_ITO[idx_g % len(OKABE_ITO)]
+ ax_op.plot(
+ gk.scan_indices,
+ 100.0 * gk.strain,
+ "o-",
+ color=col,
+ linewidth=1.9,
+ markersize=5.5,
+ label=f"ESRF T{gk.track_id} (I0={gk.integrated_intensity[0]:.0f})",
+ )
+ ax_op.axhline(0.0, color="#94A3B8", linewidth=0.9)
+ ax_op.set_xticks([0, 1, 2])
+ ax_op.set_xticklabels(["scan0001", "scan0002", "scan0003"], fontsize=8.5)
+ ax_op.set_xlabel("ESRF-ID03 Operando Scan", fontsize=9)
+ ax_op.set_ylabel(r"Calibrated lattice strain $\varepsilon = \Delta d / d_0$ (%)", fontsize=9)
+ ax_op.set_title("D. ESRF-ID03 Ni-Rich Cathode Per-Grain Strain Trajectories", fontsize=10, fontweight="bold")
+ ax_op.legend(fontsize=7.5, ncol=2, loc="best", frameon=True)
+ apply_axis_style(ax_op)
+
+ fig.suptitle(
+ "BraggTrack: 4D Chemomechanical Fracture Lineage, Precursor Scoring & Operando Kinematics",
+ fontsize=11.5,
+ fontweight="bold",
+ y=1.01,
+ )
+ fig.tight_layout()
+ out_path.parent.mkdir(parents=True, exist_ok=True)
+ fig.savefig(out_path, dpi=200, bbox_inches="tight")
+ plt.close(fig)
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--out",
+ default="docs/figures/fracture_lineage_and_precursors.png",
+ help="Output PNG/PDF path",
+ )
+ parser.add_argument("--backend", default="auto", choices=("auto", "mock", "torch"))
+ args = parser.parse_args(argv)
+ build_fracture_and_operando_figure(Path(args.out), backend=args.backend)
+ print(f"Saved {args.out}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tests/test_benchmark_suite.py b/tests/test_benchmark_suite.py
new file mode 100644
index 0000000..ad45422
--- /dev/null
+++ b/tests/test_benchmark_suite.py
@@ -0,0 +1,69 @@
+from __future__ import annotations
+
+import json
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+_REPO = Path(__file__).resolve().parents[1]
+if str(_REPO) not in sys.path:
+ sys.path.insert(0, str(_REPO))
+
+from scripts.benchmark_suite import main as benchmark_main # noqa: E402
+from scripts.benchmark_suite import run_suite # noqa: E402
+
+
+class BenchmarkSuiteTests(unittest.TestCase):
+ def test_run_suite_geo_and_mock_smoke(self) -> None:
+ payload = run_suite(["geo", "mock"], trials=1)
+ self.assertEqual(payload["schema_version"], "benchmark_suite.v1")
+ self.assertEqual(payload["backends"], ["geo", "mock"])
+ scenarios = payload["scenarios"]
+ for expected in (
+ "crossing_morphology",
+ "intensity_jitter",
+ "mosaic_broadening",
+ "beam_dropout_no_gap",
+ "beam_dropout_gap2",
+ "chemomechanical_fracture",
+ ):
+ self.assertIn(expected, scenarios)
+ self.assertIn("geo", scenarios[expected])
+ self.assertIn("mock", scenarios[expected])
+
+ # Gap-closing recovers both staggered 1-scan dropouts on geometry backend
+ self.assertEqual(scenarios["beam_dropout_no_gap"]["geo"]["mean_gap_closed_count"], 0.0)
+ self.assertEqual(scenarios["beam_dropout_gap2"]["geo"]["mean_gap_closed_count"], 2.0)
+ self.assertLess(
+ scenarios["beam_dropout_gap2"]["geo"]["mean_fragmentation_ratio"],
+ scenarios["beam_dropout_no_gap"]["geo"]["mean_fragmentation_ratio"],
+ )
+
+ def test_cli_writes_json_and_plot(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ tmp_path = Path(tmp)
+ json_out = tmp_path / "bench.json"
+ plot_out = tmp_path / "bench.png"
+ rc = benchmark_main(
+ [
+ "--backends",
+ "geo,mock",
+ "--trials",
+ "1",
+ "--json-out",
+ str(json_out),
+ "--plot-out",
+ str(plot_out),
+ ]
+ )
+ self.assertEqual(rc, 0)
+ self.assertTrue(json_out.exists())
+ self.assertTrue(plot_out.exists())
+ self.assertGreater(plot_out.stat().st_size, 5_000)
+ parsed = json.loads(json_out.read_text())
+ self.assertEqual(parsed["trials"], 1)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_gap_closing.py b/tests/test_gap_closing.py
new file mode 100644
index 0000000..23d9017
--- /dev/null
+++ b/tests/test_gap_closing.py
@@ -0,0 +1,151 @@
+"""Unit tests for multi-scan gap closing (beam-dropout & rocking-clip recovery)."""
+
+from __future__ import annotations
+
+import unittest
+
+import numpy as np
+
+from braggtrack.tracking import (
+ GapClosingConfig,
+ GeometrySemanticCost,
+ PositionShapeCost,
+ SplitMergeConfig,
+ build_tracks,
+ compute_tracking_metrics,
+ tracks_to_table,
+)
+
+
+def _spot(
+ label: int,
+ mu: float,
+ chi: float,
+ d: float,
+ intensity: float = 1000.0,
+ eig1: float = 2.0,
+ eig2: float = 1.5,
+ eig3: float = 1.2,
+ embedding: np.ndarray | None = None,
+) -> dict:
+ row = {
+ "label": label,
+ "centroid_mu": mu,
+ "centroid_chi": chi,
+ "centroid_d": d,
+ "integrated_intensity": intensity,
+ "voxel_count": 25,
+ "eig_1": eig1,
+ "eig_2": eig2,
+ "eig_3": eig3,
+ }
+ if embedding is not None:
+ row["embedding"] = embedding
+ return row
+
+
+class GapClosingTests(unittest.TestCase):
+ def test_one_scan_beam_dropout_reconnected(self) -> None:
+ """A reflection missing at scan 2 (0, 1, -, 3, 4) bridges into 1 end-to-end track."""
+ scans = [
+ [_spot(1, 10.0, 20.0, 30.0)],
+ [_spot(1, 10.0, 20.0, 31.0)],
+ [], # beam dropout at scan 2
+ [_spot(1, 10.0, 20.0, 33.0)],
+ [_spot(1, 10.0, 20.0, 34.0)],
+ ]
+
+ G_no_gap = build_tracks(scans)
+ m_no_gap = compute_tracking_metrics(G_no_gap, n_scans=5)
+ self.assertEqual(m_no_gap["total_tracks"], 2)
+ self.assertEqual(m_no_gap["gap_closed_count"], 0)
+ self.assertEqual(m_no_gap["fragmentation_ratio"], 1.0)
+
+ G_gap = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=2, max_cost=20.0))
+ m_gap = compute_tracking_metrics(G_gap, n_scans=5)
+ self.assertEqual(m_gap["total_tracks"], 1)
+ self.assertEqual(m_gap["gap_closed_count"], 1)
+ self.assertEqual(m_gap["full_length_tracks"], 1)
+ self.assertEqual(m_gap["fragmentation_ratio"], 0.0)
+
+ gap_edges = [(u, v, d) for u, v, d in G_gap.edges(data=True) if d.get("edge_type") == "gap_close"]
+ self.assertEqual(len(gap_edges), 1)
+ self.assertEqual(gap_edges[0][2]["gap_scans"], 1)
+
+ def test_two_scan_dropout_respects_max_gap_limit(self) -> None:
+ """A 2-scan gap is ignored when max_gap=1 and bridged when max_gap=2."""
+ scans = [
+ [_spot(1, 5.0, 5.0, 10.0)],
+ [],
+ [],
+ [_spot(1, 5.0, 5.0, 11.5)],
+ ]
+
+ G_g1 = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=1, max_cost=20.0))
+ self.assertEqual(compute_tracking_metrics(G_g1, n_scans=4)["total_tracks"], 2)
+
+ G_g2 = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=2, max_cost=20.0))
+ m_g2 = compute_tracking_metrics(G_g2, n_scans=4)
+ self.assertEqual(m_g2["total_tracks"], 1)
+ self.assertEqual(m_g2["gap_closed_count"], 1)
+
+ def test_chained_multi_dropouts_preserve_single_track_id(self) -> None:
+ """Repeated dropouts (0 -> 2 -> 4) propagate the initial track_id chronologically."""
+ scans = [
+ [_spot(1, 12.0, 15.0, 20.0)],
+ [],
+ [_spot(1, 12.0, 15.0, 21.0)],
+ [],
+ [_spot(1, 12.0, 15.0, 22.0)],
+ ]
+ G = build_tracks(scans, gap_closing_config=GapClosingConfig(max_gap=1, max_cost=20.0))
+ rows = tracks_to_table(G)
+ self.assertEqual({r["track_id"] for r in rows}, {1})
+ self.assertEqual(compute_tracking_metrics(G, n_scans=5)["gap_closed_count"], 2)
+
+ def test_semantic_cosine_gate_blocks_impostor_and_severs_stray_jump(self) -> None:
+ """Semantic embeddings block dissimilar gap matches and sever stray intermediate noise jumps."""
+ e_grain = np.array([1.0, 0.0, 0.0], dtype=np.float64)
+ e_noise = np.array([0.0, 1.0, 0.0], dtype=np.float64)
+
+ # Grain 1 at (10, 10, 10) drops out at scan 1 while a distant noise spot appears at (80, 80, 80).
+ scans = [
+ [_spot(1, 10.0, 10.0, 10.0, embedding=e_grain)],
+ [_spot(1, 80.0, 80.0, 80.0, embedding=e_noise)],
+ [_spot(1, 10.0, 10.0, 10.5, embedding=e_grain)],
+ ]
+ cost_fn = GeometrySemanticCost(PositionShapeCost(), cost_alpha=1.0, cost_beta=2.0)
+ G = build_tracks(
+ scans,
+ cost_fn=cost_fn,
+ gap_closing_config=GapClosingConfig(max_gap=1, max_cost=15.0, min_cosine_similarity=0.8),
+ )
+ m = compute_tracking_metrics(G, n_scans=3)
+ self.assertEqual(m["gap_closed_count"], 1)
+ self.assertEqual(m["total_tracks"], 2) # Grain 1 (scans 0, 2) + isolated noise (scan 1)
+
+ def test_gap_closing_composes_cleanly_before_split_resolution(self) -> None:
+ """A track that recovers from a 1-scan dropout and later splits 1->2 resolves both events."""
+ scans = [
+ [_spot(1, 20.0, 20.0, 20.0, intensity=1000.0)],
+ [], # dropout at scan 1
+ [_spot(1, 20.0, 20.0, 20.5, intensity=1000.0)],
+ [
+ _spot(1, 20.0, 18.0, 21.0, intensity=520.0),
+ _spot(2, 20.0, 22.0, 21.0, intensity=480.0),
+ ],
+ ]
+ G = build_tracks(
+ scans,
+ gap_closing_config=GapClosingConfig(max_gap=1, max_cost=20.0),
+ split_merge_config=SplitMergeConfig(max_distance=15.0),
+ )
+ m = compute_tracking_metrics(G, n_scans=4)
+ self.assertEqual(m["gap_closed_count"], 1)
+ self.assertEqual(m["split_count"], 1)
+ ev = G.graph["lineage_events"][0]
+ self.assertEqual(ev.parent_track_ids, (1,))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_lazy_and_export.py b/tests/test_lazy_and_export.py
new file mode 100644
index 0000000..daa4752
--- /dev/null
+++ b/tests/test_lazy_and_export.py
@@ -0,0 +1,199 @@
+"""Unit tests for LazyVolume ROI streaming, chunked Otsu, crystallography, and ImageD11/NeXus exports."""
+
+from __future__ import annotations
+
+import json
+import math
+import tempfile
+import unittest
+from pathlib import Path
+
+import h5py
+import numpy as np
+
+from braggtrack.cli import track_dataset
+from braggtrack.cli._utils import write_csv
+from braggtrack.io import (
+ AxisCalibration,
+ DetectorGeometry,
+ LazyVolume,
+ annotate_crystallographic_columns,
+ export_imaged11_flt,
+ export_nexus_process,
+ open_lazy_volume,
+ otsu_threshold_chunked,
+ read_imaged11_flt,
+)
+from braggtrack.segmentation import otsu_threshold
+from braggtrack.semantic import crop_spot_cube
+
+
+class LazyAndExportTests(unittest.TestCase):
+ def test_lazy_volume_roi_and_chunked_otsu_parity(self) -> None:
+ rng = np.random.default_rng(42)
+ vol = (rng.exponential(scale=2.0, size=(20, 32, 28)) * 10.0).astype(np.float32)
+ vol[6:10, 12:18, 10:15] += 250.0
+
+ labels = np.zeros_like(vol, dtype=np.int32)
+ labels[6:10, 12:18, 10:15] = 1
+ bbox = {
+ "bbox_min_z": 6,
+ "bbox_max_z": 9,
+ "bbox_min_y": 12,
+ "bbox_max_y": 17,
+ "bbox_min_x": 10,
+ "bbox_max_x": 14,
+ }
+
+ with tempfile.TemporaryDirectory() as tmp:
+ h5_path = Path(tmp) / "test_vol.h5"
+ with h5py.File(h5_path, "w") as h5:
+ h5.create_dataset("/entry_0000/ESRF-ID03/pco_nf/data", data=vol, chunks=(5, 16, 14))
+
+ with open_lazy_volume(h5_path) as lazy:
+ self.assertEqual(lazy.shape, vol.shape)
+ self.assertEqual(lazy.dataset_path, "/entry_0000/ESRF-ID03/pco_nf/data")
+
+ roi_lazy, mask_lazy = crop_spot_cube(lazy, labels, 1, bbox, margin=2)
+ roi_dense, mask_dense = crop_spot_cube(vol.astype(np.float64), labels, 1, bbox, margin=2)
+ np.testing.assert_allclose(roi_lazy, roi_dense)
+ np.testing.assert_array_equal(mask_lazy, mask_dense)
+
+ thr_chunked = otsu_threshold_chunked(lazy, chunk_z=5)
+ thr_dense = otsu_threshold(vol)
+ self.assertAlmostEqual(thr_chunked, thr_dense, places=5)
+
+ # Constant-volume edge case
+ const_lazy = LazyVolume.from_array(np.full((4, 4, 4), 7.5))
+ self.assertEqual(otsu_threshold_chunked(const_lazy), 7.5)
+
+ def test_detector_geometry_bragg_law_identities(self) -> None:
+ geom = DetectorGeometry(
+ distance_mm=100.0,
+ pixel_size_y_mm=0.1,
+ pixel_size_x_mm=0.1,
+ beam_center_y_px=500.0,
+ beam_center_x_px=500.0,
+ wavelength_angstrom=0.5,
+ omega_origin_deg=-10.0,
+ omega_step_deg=0.5,
+ )
+ # Offset by dx = 30 mm, dy = 40 mm => r = 50 mm at L = 100 mm => tan(2theta) = 0.5
+ scat = geom.spot_to_scattering(z_idx=20.0, y_px=900.0, x_px=800.0)
+ expected_tth = math.degrees(math.atan2(50.0, 100.0))
+ expected_d = 0.5 / (2.0 * math.sin(0.5 * math.radians(expected_tth)))
+ expected_q = (2.0 * math.pi) / expected_d
+
+ self.assertAlmostEqual(scat["omega_deg"], 0.0, places=5)
+ self.assertAlmostEqual(scat["two_theta_deg"], expected_tth, places=5)
+ self.assertAlmostEqual(scat["d_spacing_angstrom"], expected_d, places=5)
+ self.assertAlmostEqual(scat["q_mag_inv_angstrom"], expected_q, places=5)
+
+ # Verify rotational invariance of |Q| across arbitrary omega
+ scat_rot = geom.spot_to_scattering(z_idx=73.0, y_px=900.0, x_px=800.0)
+ self.assertAlmostEqual(scat_rot["q_mag_inv_angstrom"], expected_q, places=5)
+
+ # Verify calibration inversion parity
+ cal = AxisCalibration(
+ mu_origin=-10.0, mu_step=0.5, chi_origin=0.0, chi_step=0.02, d_origin=1.0, d_step=0.01, calibrated=True
+ )
+ mu_phys, chi_phys, d_phys = cal.voxel_to_physical(20.0, 900.0, 800.0)
+ annotated = annotate_crystallographic_columns(
+ [{"centroid_mu": mu_phys, "centroid_chi": chi_phys, "centroid_d": d_phys}],
+ geom,
+ calibration=cal,
+ )
+ self.assertAlmostEqual(annotated[0]["d_spacing_angstrom"], expected_d, places=5)
+
+ def test_imaged11_and_nexus_exports_via_cli(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ seg_dir = root / "seg"
+ for s_idx in range(3):
+ sd = seg_dir / f"scan{s_idx + 1:04d}"
+ sd.mkdir(parents=True)
+ write_csv(
+ sd / "features.csv",
+ [
+ {
+ "label": 1,
+ "voxel_count": 30,
+ "integrated_intensity": 1200.0 + 10.0 * s_idx,
+ "centroid_mu": 5.0,
+ "centroid_chi": 620.0,
+ "centroid_d": 580.0 + 0.5 * s_idx,
+ "eig_1": 2.0,
+ "eig_2": 1.4,
+ "eig_3": 1.1,
+ "bbox_min_z": 3,
+ "bbox_max_z": 7,
+ "bbox_min_y": 577,
+ "bbox_max_y": 583,
+ "bbox_min_x": 617,
+ "bbox_max_x": 623,
+ }
+ ],
+ )
+ (sd / "summary.json").write_text(json.dumps({"calibration": {"calibrated": False}}))
+
+ geom_path = root / "detector.json"
+ geom_path.write_text(
+ json.dumps(
+ {
+ "distance_mm": 120.0,
+ "pixel_size_y_mm": 0.075,
+ "pixel_size_x_mm": 0.075,
+ "beam_center_y_px": 512.0,
+ "beam_center_x_px": 512.0,
+ "wavelength_angstrom": 0.31,
+ "omega_origin_deg": 0.0,
+ "omega_step_deg": 0.1,
+ }
+ )
+ )
+ flt_path = root / "peaks.flt"
+ nxs_path = root / "results.nxs"
+ trk_dir = root / "tracking"
+
+ rc = track_dataset.main(
+ [
+ str(seg_dir),
+ "--outdir",
+ str(trk_dir),
+ "--detect-events",
+ "--detector-json",
+ str(geom_path),
+ "--export-imaged11",
+ str(flt_path),
+ "--export-nexus",
+ str(nxs_path),
+ ]
+ )
+ self.assertEqual(rc, 0)
+
+ hdr, flt_rows = read_imaged11_flt(flt_path)
+ self.assertEqual(len(flt_rows), 3)
+ self.assertAlmostEqual(float(hdr["wavelength"]), 0.31, places=5)
+ self.assertTrue(all(r["tth"] > 0.0 and r["d_spacing"] > 0.0 for r in flt_rows))
+
+ with h5py.File(nxs_path, "r") as h5:
+ self.assertEqual(h5.attrs["NX_class"], "NXroot")
+ self.assertEqual(h5["/entry"].attrs["NX_class"], "NXentry")
+ self.assertEqual(h5["/entry/braggtrack"].attrs["NX_class"], "NXprocess")
+ self.assertEqual(int(h5["/entry/braggtrack/tracks"].attrs["row_count"]), 3)
+ self.assertIn("two_theta_deg", h5["/entry/braggtrack/tracks"])
+ self.assertIn("strain", h5["/entry/braggtrack/kinematics"])
+ self.assertIn("delta_d", h5["/entry/braggtrack/kinematics"])
+
+ # Direct export helper check
+ nxs_direct = export_nexus_process(trk_dir, root / "direct.nxs")
+ self.assertTrue(nxs_direct.exists())
+ flt_direct = export_imaged11_flt(
+ [{"centroid_mu": 5.0, "centroid_chi": 620.0, "centroid_d": 580.0, "integrated_intensity": 500.0}],
+ root / "direct.flt",
+ )
+ self.assertTrue(flt_direct.exists())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_report_and_export_acceptance.py b/tests/test_report_and_export_acceptance.py
new file mode 100644
index 0000000..5095a03
--- /dev/null
+++ b/tests/test_report_and_export_acceptance.py
@@ -0,0 +1,26 @@
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import unittest
+
+
+class ReportAndExportAcceptanceTests(unittest.TestCase):
+ def test_report_and_export_acceptance_passes(self) -> None:
+ proc = subprocess.run(
+ [sys.executable, "scripts/check_report_and_export_acceptance.py"],
+ check=False,
+ capture_output=True,
+ text=True,
+ )
+ self.assertEqual(proc.returncode, 0, msg=proc.stdout + proc.stderr)
+ payload = json.loads(proc.stdout)
+ self.assertEqual(payload["failures"], [])
+ self.assertEqual(payload["gap_closing"]["gap_closed_count"], 1)
+ self.assertGreater(payload["report_and_exports"]["html_report_bytes"], 5_000)
+ self.assertGreater(payload["report_and_exports"]["flt_row_count"], 0)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_segmentation_dino.py b/tests/test_segmentation_dino.py
index 7a65857..3dfc843 100644
--- a/tests/test_segmentation_dino.py
+++ b/tests/test_segmentation_dino.py
@@ -146,6 +146,30 @@ def test_labels_are_nonnegative(self) -> None:
result = segment_dino(volume, backend="mock")
self.assertTrue(np.all(result.labeled_volume >= 0))
+ def test_resized_vit_grid_alignment(self) -> None:
+ """Simulate ViT 224x224 image resizing (16x16 tokens) on non-multiple slice shapes."""
+ from unittest.mock import patch
+
+ class _ResizedViTEncoder:
+ patch_size = 14
+ feature_dim = 16
+
+ def extract_patch_features(self, image: np.ndarray) -> np.ndarray:
+ rng = np.random.default_rng(int(np.sum(image > 5.0)))
+ grid = rng.standard_normal((16, 16, 16)).astype(np.float32)
+ grid[10:15, 10:15, :] += 8.0
+ norms = np.linalg.norm(grid, axis=-1, keepdims=True) + 1e-8
+ return grid / norms
+
+ volume = np.zeros((4, 250, 250), dtype=np.float64)
+ volume[1:3, 180:240, 180:240] = 25.0
+ with patch("braggtrack.semantic.dino.make_patch_encoder", return_value=_ResizedViTEncoder()):
+ result = segment_dino(volume, backend="mock", min_cluster_size=2, min_samples=1)
+ self.assertEqual(result.labeled_volume.shape, (4, 250, 250))
+ self.assertGreaterEqual(result.component_count, 1)
+ # Ensure spots near the far corner (> 16 * 14 = 224) receive foreground labels via proportional upsampling
+ self.assertGreater(int(np.count_nonzero(result.labeled_volume[1:3, 225:240, 225:240])), 0)
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_viz_and_report.py b/tests/test_viz_and_report.py
new file mode 100644
index 0000000..ad45488
--- /dev/null
+++ b/tests/test_viz_and_report.py
@@ -0,0 +1,134 @@
+"""Unit and CLI tests for braggtrack.viz plotting functions and braggtrack-report HTML explorer."""
+
+from __future__ import annotations
+
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+import h5py
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+
+from braggtrack.cli import report_dataset, run_pipeline
+from braggtrack.tracking import (
+ PrecursorConfig,
+ SplitMergeConfig,
+ build_tracks,
+ compute_fracture_precursors,
+ compute_grain_kinematics,
+ generate_fracture_scenario,
+ tracks_to_table,
+)
+from braggtrack.viz import (
+ generate_html_report,
+ plot_kinematics_dashboard,
+ plot_lineage_dag,
+ plot_precursor_timeline,
+ plot_scan_projection_overlay,
+ plot_spot_mip_card,
+)
+
+
+class VizAndReportTests(unittest.TestCase):
+ def test_all_matplotlib_plots_render_and_save(self) -> None:
+ volumes, label_volumes, scan_tables, _ = generate_fracture_scenario()
+ G = build_tracks(scan_tables, split_merge_config=SplitMergeConfig(max_distance=20.0))
+ track_rows = tracks_to_table(G)
+ gks = compute_grain_kinematics(track_rows)
+ candidates, obs_scores = compute_fracture_precursors(
+ track_rows,
+ PrecursorConfig(score_threshold=0.35, min_persistence=2),
+ volumes=volumes,
+ label_volumes=label_volumes,
+ lineage_events=G.graph["lineage_events"],
+ )
+
+ with tempfile.TemporaryDirectory() as tmp:
+ out = Path(tmp)
+ fig1, _ = plot_lineage_dag(G, save_path=out / "dag.png")
+ plt.close(fig1)
+ self.assertTrue((out / "dag.png").stat().st_size > 1000)
+
+ fig2, _ = plot_kinematics_dashboard(
+ gks, events=G.graph["lineage_events"], calibrated=True, save_path=out / "kin.png"
+ )
+ plt.close(fig2)
+ self.assertTrue((out / "kin.png").stat().st_size > 1000)
+
+ fig3, _ = plot_precursor_timeline(obs_scores, candidates=candidates, save_path=out / "prec.png")
+ plt.close(fig3)
+ self.assertTrue((out / "prec.png").stat().st_size > 1000)
+
+ spot0 = scan_tables[0][0]
+ fig4, _ = plot_spot_mip_card(
+ volumes[0], label_volumes[0], int(spot0["label"]), spot0, save_path=out / "mip.png"
+ )
+ plt.close(fig4)
+ self.assertTrue((out / "mip.png").stat().st_size > 1000)
+
+ fig5, _ = plot_scan_projection_overlay(
+ volumes[0], label_volumes[0], scan_tables[0], save_path=out / "proj.png"
+ )
+ plt.close(fig5)
+ self.assertTrue((out / "proj.png").stat().st_size > 1000)
+
+ def test_pipeline_with_report_and_exports(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ base = Path(tmp)
+ ds = base / "operando"
+ for i in range(3):
+ sdir = ds / f"scan{i + 1:04d}"
+ sdir.mkdir(parents=True)
+ vol = np.zeros((16, 28, 28), dtype=np.float32)
+ vol[5:10, 10 + i : 15 + i, 12:17] = 180.0
+ with h5py.File(sdir / "pco_nf.h5", "w") as h5:
+ entry = h5.create_group("entry_0000")
+ entry.attrs["NX_class"] = "NXentry"
+ entry.create_dataset("start_time", data=f"2026-05-16T10:0{i}:00Z")
+ h5.create_dataset("/entry_0000/ESRF-ID03/pco_nf/data", data=vol)
+
+ out_dir = base / "pipeline_out"
+ nxs_out = base / "summary.nxs"
+ flt_out = base / "peaks.flt"
+ rc = run_pipeline.main(
+ [
+ str(ds),
+ "--outdir",
+ str(out_dir),
+ "--backend",
+ "mock",
+ "--report",
+ "--export-nexus",
+ str(nxs_out),
+ "--export-imaged11",
+ str(flt_out),
+ ]
+ )
+ self.assertEqual(rc, 0)
+ html_file = out_dir / "report" / "report.html"
+ self.assertTrue(html_file.exists())
+ html_text = html_file.read_text(encoding="utf-8")
+ self.assertIn("BraggTrack 4D Explorer", html_text)
+ self.assertIn("data:image/png;base64,", html_text)
+ self.assertTrue((out_dir / "report" / "figures" / "lineage_dag.png").exists())
+ self.assertTrue(nxs_out.exists())
+ self.assertTrue(flt_out.exists())
+
+ manifest = json.loads((out_dir / "run_manifest.json").read_text())
+ self.assertIn("report", manifest["stage_timings_sec"])
+ self.assertIsNotNone(manifest["artifacts"]["html_report"])
+
+ # Standalone report CLI smoke check
+ standalone_html = generate_html_report(out_dir, base / "standalone.html", dataset_root=ds)
+ self.assertTrue(standalone_html.exists())
+ rc_rep = report_dataset.main([str(out_dir), "--outdir", str(base / "rep2"), "--no-figures"])
+ self.assertEqual(rc_rep, 0)
+
+
+if __name__ == "__main__":
+ unittest.main()