Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,6 @@ build/
artifacts/
.ipynb_checkpoints/
_braggtrack_repo/
.coverage
.pytest_cache/
.ruff_cache/
40 changes: 21 additions & 19 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,31 +5,33 @@ All notable changes to BraggTrack will be documented in this file.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [0.2.0] - 2026-11-17

### Added
- Rolling-median threshold smoother (`smooth_thresholds`) for stable multi-frame segmentation
- Outlier frame detection (`flag_outlier_frames`) via MAD-based statistics
- Label projection by intensity (`label_projection_by_intensity`) — replaces broken `labels.max(axis=k)`
- MIP-floor masking via 2-D Otsu (`otsu_floor_from_mip`)
- Tri-axis segmented mask visualisation in demo notebook
- Semantic MIP gallery and PCA embedding space plots
- Google Colab support with auto-install cell and "Open in Colab" badge
- CI matrix (Python 3.10/3.11/3.12) with pip caching and notebook execution job
- Optional dependency groups: `[torch]`, `[notebook]`, `[dev]`
- `BRAGGTRACK_DATA_ROOT` env var for custom data locations
- Ruff linting and formatting configuration
- PEP 561 `py.typed` marker
- **ESRF-ID03 & generic NeXus HDF5 auto-discovery (`braggtrack.io.nexus`)**: Added `/entry_0000/ESRF-ID03/pco_nf/data` to primary volume candidates, largest-3D numeric dataset fallback (`discover_primary_volume_path`, `load_primary_volume(..., auto_discover=True)`), and `/entry*` / `NXentry` metadata discovery in `extract_scan_metadata`.
- **Physical reciprocal-space calibration (`AxisCalibration`)**: Added `AxisCalibration` contract (`voxel_to_physical`, JSON serialisation), HDF5 axis auto-detection (`extract_axis_calibration`), CLI calibration flags (`--calibration-json`, `--mu-origin/--mu-step`, `--chi-origin/--chi-step`, `--d-origin/--d-step`), and calibrated physical centroids/covariances/eigenvalues in `extract_instance_table` while preserving integer voxel bounding boxes.
- **Per-grain 4D kinematics CLI export**: Wired `compute_grain_kinematics` and `summarize_kinematics` into `braggtrack-track-dataset`, automatically emitting `kinematics.csv` (including `delta_d`) and `kinematics_summary.json` alongside `tracks.csv`.
- **Split (`1 -> 2+`) and merge (`2+ -> 1`) lineage DAG resolution (`braggtrack.tracking.events`)**: Added intensity-partitioning-gated split and merge detection (`resolve_splits_and_merges`, `SplitMergeConfig`, `LineageEvent`, `events_to_table`) with anti-noise minimum fragment share guards and distinct daughter/merged track ID propagation.
- **Persistent 4D fracture precursor scoring (`braggtrack.tracking.precursors`)**: Added `compute_fracture_precursors` combining 3D eigenvalue anisotropy, covariance trace growth, 3D spot ROI sub-peak bimodality, and upsampled MIP DINO patch-token cosine dispersion with multi-scan contiguous persistence filtering (`min_persistence=2`).
- **Synthetic 4D fracture ground-truth generator**: Added `generate_fracture_scenario` producing 4D intensity cubes, label masks, and ground-truth split, merge, persistent precursor, and isotropic control trajectories.
- **New CLI entry points**:
- `braggtrack-events` (`braggtrack.cli.detect_events`) for standalone split/merge lineage and ROI fracture precursor ranking (`events.csv`, `events.json`, `precursors.csv`, `events_summary.json`).
- `braggtrack-run` (`braggtrack.cli.run_pipeline`) for one-command reproducible end-to-end execution with JSON config support, stage timing profiling, SHA-256 input checksums, and `run_manifest.json`.
- **Semantic ablation bootstrap confidence intervals**: Added `--bootstrap-samples` to `scripts/ablation_semantic.py` reporting 95% bootstrap CIs (`fragmentation_ratio_ci95`, `id_switch_rate_ci95`).
- **Events & pipeline acceptance gate**: Added `scripts/check_events_acceptance.py` and wired it into `scripts/ci_report.py`.
- Rolling-median threshold smoother (`smooth_thresholds`) wired into `braggtrack-segment-dataset` (`--smooth-window 5`) and outlier frame detection (`flag_outlier_frames`) via MAD-based statistics.
- Label projection by intensity (`label_projection_by_intensity`) and MIP-floor masking via 2-D Otsu (`otsu_floor_from_mip`).
- Optional dependency groups (`[torch]`, `[notebook]`, `[dev]`), `BRAGGTRACK_DATA_ROOT` env var, Ruff configuration, and PEP 561 `py.typed` marker.

### Changed
- Seed floor now uses robust peak reference (p99.99) instead of absolute max
- Default `seed_response_percentile` raised from 99.5 to 99.95
- `torch` and `transformers` moved to optional `[torch]` extra (bare install is lightweight)
- **Ungated vision backbone default**: Standardised `DEFAULT_DINO_MODEL` to `"facebook/dinov2-small"` across `semantic` and `segmentation` so local installs, CI, and Google Colab run zero-auth out of the box, while keeping gated DINOv3 (`facebook/dinov3-vitb16-pretrain-lvd1689m`) fully supported via `--model` / `--dino-model` with universal dynamic prefix-token skipping (`cls` + register tokens).
- `segment_dataset`, `embed_dataset`, and `detect_events` now load real bundled ESRF-ID03 `(100, 250, 250)` volumes directly via `load_primary_volume` and require explicit `--allow-synthetic-fallback` before substituting synthetic blobs on unreadable HDF5 files.
- Seed floor uses robust peak reference (`p99.99`) and `seed_response_percentile=99.95`.

### Fixed
- Critical threshold domain mismatch: intensity Otsu was applied to LoG response domain
- Spot count instability across scans (11/22/36 → 18/20/16 on bundled data)
- Label projection picking highest label ID instead of brightest voxel's label
- Fixed silent fallback to fake `(24, 24, 24)` synthetic volumes when running CLI tools on bundled ESRF-ID03 NeXus/LIMA HDF5 files (`/entry_0000/ESRF-ID03/pco_nf/data`).
- Fixed uncalibrated detector-crop pixel offset distortion in crystallographic strain ($\Delta d / d_0$) when `AxisCalibration` is supplied.
- Fixed threshold domain mismatch (intensity Otsu vs. LoG response domain) and label projection highest-ID bias.

## [0.1.0] - 2025-12-01

Expand Down
174 changes: 105 additions & 69 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,109 +2,145 @@

# BraggTrack

Semantic 4D kinematics and fracture tracking for operando diffraction using foundation vision models.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/BASE-Laboratory/BraggTrack/blob/main/notebooks/braggtrack_demo.ipynb)

## Repository layout
Semantic 4D kinematics, grain lineage (`1 -> 2+` fracture splits & `2+ -> 1` coalescence merges), and persistent fracture precursor tracking for operando X-ray diffraction using classical 3D crystallography and foundation vision transformers.

- **`braggtrack/`** — Python package (I/O, segmentation, semantic embeddings, tracking, CLIs).
- **`data/sample_operando/`** — bundled example scans (`scan0001` … `scan0003`, each with `pco_nf_0000_cropped.h5`). See [`data/sample_operando/README.md`](data/sample_operando/README.md).
- **`docs/`** — [architecture](docs/architecture.md) and [coding practices](docs/coding-practices.md).
- **`artifacts/`** — generated outputs (gitignored).
## Key Capabilities

Contributing: see [CONTRIBUTING.md](CONTRIBUTING.md).
- **Beamline-ready NeXus/HDF5 I/O & calibration (`braggtrack.io`)** — Auto-discovers 3D detector cubes across standard NeXus paths, ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), and arbitrary HDF5 trees (`auto_discover=True`), with optional physical `(mu, chi, d)` coordinate calibration via `AxisCalibration`.
- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Intensity-domain Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`).
- **Multi-view foundation vision descriptors (`braggtrack.semantic`)** — Orthogonal $(\mu, \chi, d)$ maximum-intensity projections (MIPs) encoded with ungated **`facebook/dinov2-small`** by default (zero HuggingFace login required), drop-in support for **DINOv3** (`facebook/dinov3-vitb16-pretrain-lvd1689m`), or a deterministic zero-GPU `mock` backend for CI.
- **4D grain kinematics, split/merge lineage & fracture precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association into a NetworkX trajectory DAG, per-grain lattice strain ($\Delta d / d_0$) and angular misorientation ($\Delta\mu$, $\Delta\chi$), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking combining 3D shape anisotropy, covariance spread growth, and spot ROI sub-peak / DINO patch-token dispersion.
- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, segmentation, semantic embedding, tracking, kinematics, and fracture event analysis end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`.

## Included example scans
---

Bundled under **`data/sample_operando/`**:
## Installation

- `data/sample_operando/scan0001/pco_nf_0000_cropped.h5`
- `data/sample_operando/scan0002/pco_nf_0000_cropped.h5`
- `data/sample_operando/scan0003/pco_nf_0000_cropped.h5`

## Week 2 classical baseline status

Implemented segmentation stack:
```bash
# Core package (numpy, scipy, scikit-image, scikit-learn, networkx, h5py)
pip install -e .

- 3D LoG enhancement (`log_enhance_3d`)
- h-maxima seed detection (`h_maxima_seeds`)
- 3D seeded watershed-style region growing (`watershed_from_seeds`)
- post-processing (small-object cleanup + 3D hole fill)
- feature extraction table (centroid, bbox, integrated intensity, covariance, eigenvalues)
# With PyTorch + HuggingFace Transformers for real DINOv2 / DINOv3 backbones
pip install -e ".[torch]"

Run end-to-end segmentation artifacts on the three scans (omit the path to use the bundled sample automatically, or pass it explicitly):
# With Jupyter, Matplotlib, and Pandas for interactive notebooks
pip install -e ".[notebook]"

```bash
python -m braggtrack.cli.segment_dataset --outdir artifacts/week2
python -m braggtrack.cli.segment_dataset data/sample_operando --outdir artifacts/week2
# Full developer environment (torch + notebook + pytest + ruff)
pip install -e ".[dev]"
```

Outputs:
---

- `artifacts/week2/segmentation_summary.json`
- `artifacts/week2/segmentation_summary.csv`
- `artifacts/week2/scanXXXX/summary.json`
- `artifacts/week2/scanXXXX/features.csv`
- `artifacts/week2/qc/week2_visual_qc.ipynb`
## Quick Start

Note: if `h5py` is not available, the CLI uses a deterministic synthetic fallback volume per scan file so CI and schema checks remain reproducible.
BraggTrack ships with three real ESRF ID03 operando Ni-rich cathode scans under [`data/sample_operando/`](data/sample_operando/README.md) (`100 x 250 x 250` `uint16` cubes). Omit the dataset path to run on the bundled scans automatically, or pass your own directory of `scanXXXX/` folders:

## CI/CD
### One-command end-to-end pipeline

GitHub Actions CI runs:

- `python scripts/ci_report.py`

`ci_report.py` executes unit tests, Week 1 acceptance, Week 2 smoke, and Week 2 acceptance checks, then prints explicit PASS/FAIL criteria and overall status.
```bash
braggtrack-run --outdir artifacts/pipeline
```

Before opening a PR, run:
Produces:
- `artifacts/pipeline/validation_summary.json`
- `artifacts/pipeline/segmentation/` (`features.csv`, `labels.npz`, `summary.json`, QC notebook)
- `artifacts/pipeline/embedding/` (`embeddings.npz`, `embedding_manifest.json`)
- `artifacts/pipeline/tracking/` (`tracks.csv`, `kinematics.csv`, `kinematics_summary.json`, `tracking_summary.json`, QC notebook)
- `artifacts/pipeline/events/` (`events.csv`, `events.json`, `precursors.csv`, `events_summary.json`)
- `artifacts/pipeline/run_manifest.json` (SHA-256 input checksums, git SHA, config, stage timings)

### Stage-by-stage CLI commands

| CLI Entry Point | Module Equivalent | Purpose |
|---|---|---|
| `braggtrack-inspect` | `python -m braggtrack.cli.inspect_datasets` | Discover `scan*` folders, primary 3D HDF5 dataset paths, and metadata |
| `braggtrack-validate` | `python -m braggtrack.cli.validate_dataset` | Validate sequence ordering, monotonic timestamps, and axis contracts |
| `braggtrack-segment-synthetic` | `python -m braggtrack.cli.segment_synthetic` | Run fast synthetic 3D segmentation smoke test |
| `braggtrack-segment-dataset` | `python -m braggtrack.cli.segment_dataset` | Segment 3D Bragg spots across scans with smoothed thresholds & optional calibration |
| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Compute orthogonal MIP embeddings (`auto`, `mock`, or `torch` DINOv2/DINOv3) |
| `braggtrack-track-dataset` | `python -m braggtrack.cli.track_dataset` | Build trajectory DAG, export `tracks.csv` + `kinematics.csv` (+ optional `--detect-events`) |
| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and ROI-based fracture precursors |
| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config config.json` and `run_manifest.json` |

---

## Physical Axis Calibration (`mu`, `chi`, `d`)

Cropped detector cubes store integer voxel indices `[0..N)`. For physically meaningful lattice strain $\varepsilon = (d(t) - d_0) / d_0$ and angular misorientation in degrees, supply an `AxisCalibration` JSON file (or CLI flags) when segmenting:

```json
{
"mu_origin": 12.50,
"mu_step": 0.02,
"chi_origin": -3.20,
"chi_step": 0.015,
"d_origin": 2.042,
"d_step": -0.00012
}
```

```bash
python scripts/pre_pr_check.py
braggtrack-segment-dataset data/sample_operando \
--calibration-json calibration.json \
--outdir artifacts/segmentation
```

This checks for unresolved conflict markers and (when `origin/main` exists) reports ahead/behind divergence with a rebase hint.
## Week 1 implementation status
If HDF5 scans already include 1D `mu`, `chi`, `d` coordinate datasets or `/entry/calibration` attributes, `extract_axis_calibration()` detects them automatically.

Week 1 focuses on data contracts, beamline adaptation, and dataset validation.
---

Implemented modules:
## Vision Backbone Options (DINOv2 Default & Optional DINOv3)

- `braggtrack.io.models`: canonical contracts (`ScanVolumeMeta`, `ExperimentSequence`, axis definitions)
- `braggtrack.io.beamline`: beamline adapter to map discovered scan files into contract objects
- `braggtrack.io.validation`: sequence and metadata validation checks
- `braggtrack.cli.inspect_datasets`: scan discovery + metadata extraction report
- `braggtrack.cli.validate_dataset`: sequence validation report
- **Default (`facebook/dinov2-small`)**: Public and ungated on HuggingFace — works out of the box with `pip install -e ".[torch]"` without tokens or license gates.
- **DINOv3 (`facebook/dinov3-vitb16-pretrain-lvd1689m`)**: Supported via `--model` / `--dino-model` after accepting Meta's license on HuggingFace and setting `HF_TOKEN` (or running `huggingface-cli login`):
```bash
braggtrack-embed-dataset --backend torch --model facebook/dinov3-vitb16-pretrain-lvd1689m
```
- **Offline / CI (`mock`)**: Deterministic SHA-256 multi-view projection encoder requiring zero GPU or network access (`--backend mock` or `BRAGGTRACK_DINO_BACKEND=mock`).

## Week 2 baseline (segmentation)
---

Week 2 starts with a robust classical baseline, including **Otsu thresholding** as the standard first-pass separator.
## Notebooks

Implemented modules:
- [`notebooks/braggtrack_demo.ipynb`](notebooks/braggtrack_demo.ipynb) — End-to-end tutorial on the bundled ESRF-ID03 scans: discovery, stabilised 3D segmentation, physics & semantic tracking, per-grain kinematics, α/β ablation, and synthetic 4D grain fracture + precursor ranking.
- [`notebooks/dino_segmentation_comparison.ipynb`](notebooks/dino_segmentation_comparison.ipynb) — Side-by-side comparison of stabilised classical 3D watershed vs. experimental DINO patch-clustering segmentation.

- `braggtrack.segmentation.otsu`: pure-Python Otsu threshold estimation
- `braggtrack.segmentation.pipeline`: threshold segmentation + 3D connected component counting
- `braggtrack.segmentation.classical`: LoG enhancement + local-max seed extraction + seeded region growing
- `braggtrack.cli.segment_synthetic`: synthetic smoke test for segmentation harness
- `braggtrack.cli.segment_dataset`: dataset scan segmentation CLI
---

## Quick start
## Testing, Acceptance Gates & Pre-PR Checks

```bash
python -m braggtrack.cli.inspect_datasets
python -m braggtrack.cli.validate_dataset
python -m braggtrack.cli.segment_synthetic
python -m braggtrack.cli.segment_dataset --outdir artifacts/week2
# Lint & format check
ruff check braggtrack/ tests/ scripts/
ruff format --check braggtrack/ tests/ scripts/

# Unit test suite with coverage
pytest tests/ -v --cov=braggtrack

# Full CI gate runner (unit tests + IO, smoke, segmentation, tracking, embedding, DINO, and events acceptance)
python scripts/ci_report.py

# Pre-PR hygiene check
python scripts/pre_pr_check.py
```

Each command resolves the dataset root via `braggtrack.io.paths`: **bundled `data/sample_operando` when present**, otherwise the current working directory. Pass an explicit path to use another dataset root.
---

If `h5py` is unavailable in your environment, scan discovery and validation still run and include a clear warning in the output payload.
## Troubleshooting & FAQ

## CI/CD
- **`MissingH5DependencyError` or custom HDF5 dataset paths:** `h5py` is installed by default with `pip install -e .`. `load_primary_volume()` automatically inspects standard NeXus paths, `/entry_0000/ESRF-ID03/pco_nf/data`, and falls back to the largest 3D numeric dataset in the file. Pass `--allow-synthetic-fallback` only when testing schema plumbing on empty/metadata-only HDF5 stubs.
- **`401 Unauthorized` / gated HuggingFace model error:** You passed a gated model ID (such as `facebook/dinov3-*`) without authenticating. Either omit `--model` to use the ungated default (`facebook/dinov2-small`), or accept the model license on HuggingFace and export `HF_TOKEN`.
- **Unphysical strain percentages in `kinematics.csv`:** Check `"calibrated_axes"` in `kinematics_summary.json`. When `false`, `centroid_d` is measured in cropped detector pixels (`y` index) rather than Angstroms — pass `--calibration-json` (or `--d-origin` / `--d-step`) to compute true crystallographic $\Delta d / d_0$.

GitHub Actions CI now runs:
## Repository Layout & Contributing

- unit tests (`python -m unittest discover -s tests -v`)
- Week 1 acceptance checks (`python scripts/check_acceptance.py`)
- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `cli`).
- **`data/sample_operando/`** — Bundled ESRF-ID03 operando test scans (`scan0001`–`scan0003`).
- **`docs/`** — [Architecture](docs/architecture.md) and [Coding Practices](docs/coding-practices.md).
- **`scripts/`** — Acceptance gates, semantic α/β ablation (`ablation_semantic.py`), and CI runner (`ci_report.py`).
- **`artifacts/`** — Generated pipeline outputs (gitignored; never commit).

The acceptance script verifies scan discovery/order/monotonicity and fails only on unmet acceptance criteria (warnings are reported but do not fail by themselves).
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines and [paper.md](paper.md) for the manuscript overview.
2 changes: 1 addition & 1 deletion braggtrack/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
try:
__version__ = version("braggtrack")
except PackageNotFoundError:
__version__ = "0.1.0.dev0"
__version__ = "0.2.0.dev0"

__all__ = ["io", "segmentation", "semantic", "tracking"]
Loading
Loading