Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
name: Release

on:
push:
tags:
- "v*.*.*"
workflow_dispatch:

permissions:
contents: write

jobs:
build-and-verify:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install build & verification dependencies
run: |
python -m pip install --upgrade pip build twine ruff pytest
python -m pip install -e ".[notebook]"

- name: Verify lint and unit test suite
run: |
ruff check braggtrack/ tests/ scripts/
pytest tests/ -q

- name: Build sdist and wheel
run: python -m build

- name: Validate distribution metadata with Twine
run: twine check dist/*

- name: Verify clean wheel installation in isolated venv
run: |
python -m venv /tmp/release-smoke-venv
/tmp/release-smoke-venv/bin/pip install dist/*.whl
/tmp/release-smoke-venv/bin/braggtrack-inspect --help
/tmp/release-smoke-venv/bin/braggtrack-watch --help
/tmp/release-smoke-venv/bin/braggtrack-run --help

- name: Upload distribution artifacts
uses: actions/upload-artifact@v4
with:
name: python-package-distributions
path: dist/

github-release:
needs: build-and-verify
if: startsWith(github.ref, 'refs/tags/v')
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Download distribution artifacts
uses: actions/download-artifact@v4
with:
name: python-package-distributions
path: dist/

- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
files: dist/*
generate_release_notes: true

publish-pypi:
needs: build-and-verify
if: startsWith(github.ref, 'refs/tags/v') && vars.ENABLE_PYPI_PUBLISH == 'true'
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/braggtrack
permissions:
id-token: write
steps:
- name: Download distribution artifacts
uses: actions/download-artifact@v4
with:
name: python-package-distributions
path: dist/

- name: Publish distribution to PyPI via Trusted Publisher
uses: pypa/gh-action-pypi-publish@release/v1
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,29 @@ All notable changes to BraggTrack will be documented in this file.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.4.0] - 2026-11-17

### Added
- **Analytical Poisson + quantization centroid/strain uncertainties & ancestor-referenced kinematics (`braggtrack.segmentation.features`, `braggtrack.tracking.kinematics`, `braggtrack.tracking.assignment`)**:
- Emits per-spot `centroid_mu_err`, `centroid_chi_err`, and `centroid_d_err` combining Poisson photon-counting variance and `1/12` voxel quantization variance.
- Propagates `strain_err` ($\sigma_\varepsilon$), ancestor-referenced `root_track_id`, `lineage_strain`, and `lineage_strain_err` across `1 -> 2+` fracture splits, plus causal velocities (`rate_mu`, `rate_chi`, `rate_d`, `strain_rate`) and Hungarian runner-up `assignment_margin`.
- **Operando electrochemistry & external stimulus synchronization (`braggtrack.io.operando_log`, `braggtrack.viz.operando`)**:
- Parses `.csv`/`.tsv` and Biologic EC-Lab `.mpt` potentiostat logs (`load_operando_log`, `synchronize_stimulus_to_scans`), computing Savitzky–Golay regularized differential capacity $\mathrm{d}Q/\mathrm{d}V$ and per-grain electro-strain susceptibility `d_strain_d_v` ($\partial\varepsilon/\partial V$).
- Added 3-panel publication plot `plot_operando_strain_and_events` and wired `--operando-log` / `--operando-tz-offset` into `braggtrack-report`, `braggtrack-run`, and `report.html`.
- **Batched `DINOv2` ViT inference & zero-config `darfix` / ESRF BLISS NeXus auto-calibration (`braggtrack.semantic.dino`, `braggtrack.io.nexus`)**:
- Added `embed_batch` and `extract_patch_features_batch` across `MultiviewEncoder` and `PatchFeatureEncoder` (with CPU L3-friendly micro-batching), accelerating `braggtrack-embed-dataset`, `compute_fracture_precursors`, and `segment_dino` while guaranteeing bit-identical cosine parity.
- Added `NXdata` `@default` / `@signal` / `@axes` (`darfix` processed HDF5) and ESRF BLISS `instrument/positioners` auto-calibration in `discover_primary_volume_path` and `extract_axis_calibration`.
- **Online causal streaming, Gaussian Process trajectory forecasting & Bayesian "Scout-and-Zoom" DFXM engine (`braggtrack.autonomous`, `braggtrack-watch`)**:
- Added `forecast_1d_gp`, `forecast_grain_trajectory_gp`, `evaluate_acquisition`, and `score_candidates_gp` (`gp.py`), combining inverse-variance-weighted linear trend estimation with a heteroscedastic `Matérn-3/2` Gaussian Process (`alpha = sigma_pos**2`) for smooth `(mu, chi, d)` setpoint forecasting and `UCB` / `EI` / `max_variance` Bayesian target ranking.
- Added `Bluesky-Adaptive` style `tell()` / `ask()` / `tell_zoom_outcome()` / `report()` methods and $O(1)$ bounded sliding-window incremental tracking (`sliding_window_scans=24`) in `CausalStreamingTracker` (`streaming.py`), maintaining constant per-scan latency across `1,000+` scan operando runs while preserving global track IDs and root strain anchors.
- Added staged execution modes (`--mode {replay, shadow, watch, active}`), `--acquisition {ucb, ei, max_variance}`, diffractometer motor soft-limit clamping, electrochemical voltage bounds, per-lineage X-ray dose caps, synchrotron beam-dump suppression, and live HDF5 file-quiescence + `SWMR` polling in `braggtrack-watch`, emitting `bliss_dfxm_zoom.v1` commands (`zoom_triggers.jsonl`) and baseline replay metrics including `mean_setpoint_mae`.
- **Crystallographic physics & Red-Team hardening (`segmentation`, `semantic`, `tracking`, `viz`, `cli`)**:
- Added outer-perimeter background pedestal subtraction (`background_level`, `net_intensity`), gain-aware Poisson counting variance (`detector_gain`), standardized Pearson rocking-curve excess kurtosis (`kurtosis_mu`, `kurtosis_chi`, `kurtosis_d`), dimensionless voxel eigenvalues (`vox_eig_1..3`), and boundary truncation flags (`touches_border`, `touches_rocking_border`) in `extract_instance_table`, plus `float32` classical segmentation memory reduction (`segment_classical`).
- Added step-normalized physical-unit invariance across `PositionShapeCost`, `resolve_track_gaps`, and `resolve_splits_and_merges` so calibrated `AxisCalibration` runs (`deg`, `Å`) produce identical trajectories to uncalibrated voxel space, plus `mechanical_split` vs. `phase_split` (`H2 -> H3`) split classification and wall-clock `strain_rate_per_s`.
- Added aspect-ratio-preserving square letterbox padding for global `DINOv2` MIP embeddings, strict `--config` key validation in `braggtrack-run`, and Content-Security-Policy (CSP) + HTML entity escaping against DOM XSS in `generate_html_report`.
- **Ecosystem positioning, autonomous facility bibliography & release workflow (`docs/positioning.md`, `.github/workflows/release.yml`)**:
- Added `docs/positioning.md` positioning `BraggTrack` alongside `darfix`, `ImageD11`, `HEXRD`, `BraggNN`, `TrackMate 7`, `LapTrack`, `gpCAM`, `Bluesky-Adaptive`, `ANDiE`, and the Facility Autonomy Levels (FAL) taxonomy, updated `paper.md` and `paper.bib` with verified DOIs (`@GarrigaFerrer2023`, `@Isern2025`, `@Bernier2011`, `@Liu2022`, `@Fukai2023`, `@Ershov2022`, `@Drnec2025`, `@LeHoux2026`, `@Lu2025`, `@LeHouxFAL2026`, `@Szymanski2023`, `@McDannald2022`, `@Noack2021`, `@Maffettone2023`, `@Kalinin2021`), bumped metadata to `0.4.0`, and added tag-triggered sdist/wheel + Twine + GitHub Release / PyPI Trusted Publisher workflow.

## [0.3.0] - 2026-11-17

### Added
Expand Down
16 changes: 10 additions & 6 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,25 @@ cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: "BraggTrack"
type: software
version: 0.3.0
version: 0.4.0
date-released: "2026-05-16"
license: MIT
repository-code: "https://github.com/BASE-Laboratory/BraggTrack"
url: "https://github.com/BASE-Laboratory/BraggTrack"
abstract: >-
Semantic 4D kinematics, multi-scan gap closing, and chemomechanical
fracture tracking for operando X-ray diffraction. BraggTrack segments
Bragg reflections in 3D reciprocal-space volumes, tracks trajectories
and split/merge lineage using classical crystallography and
vision-transformer embeddings, and exports ImageD11, NeXus HDF5, and
Semantic 4D kinematics, multi-scan gap closing, chemomechanical fracture
tracking, operando electrochemistry synchronization, and online autonomous
Dark-Field X-ray Microscopy (DFXM) target selection for operando X-ray
diffraction. BraggTrack segments Bragg reflections in 3D reciprocal-space
volumes, tracks trajectories and split/merge lineage with analytical
uncertainties using classical crystallography and batched vision-transformer
embeddings, and exports ImageD11, NeXus HDF5, BLISS zoom triggers, and
interactive HTML 4D lineage reports.
keywords:
- X-ray diffraction
- dark-field X-ray microscopy
- operando crystallography
- autonomous experiments
- 3D segmentation
- object tracking
- grain fracture
Expand Down
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,16 @@ Semantic 4D kinematics, multi-scan beam-dropout gap closing, grain lineage (`1 -

## Key Capabilities

- **Beamline-scale NeXus/HDF5 I/O, lazy ROI streaming & crystallographic mapping (`braggtrack.io`)** — Auto-discovers 3D detector cubes across standard NeXus paths, ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), and arbitrary HDF5 trees (`auto_discover=True`). Streams `z`-slabs via `LazyVolume` (`otsu_threshold_chunked`, `read_roi`), calibrates `(mu, chi, d)` via `AxisCalibration`, maps transmission geometry `(2theta, eta, d, |Q|)` via `DetectorGeometry`, and exports directly to **`ImageD11` (`.flt`)** and **NeXus `NXprocess` HDF5 (`.nxs`)**.
- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Streaming Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`).
- **Multi-view foundation vision descriptors (`braggtrack.semantic`)** — Orthogonal $(\mu, \chi, d)$ maximum-intensity projections (MIPs) encoded with ungated **`facebook/dinov2-small`** by default (zero HuggingFace login required), drop-in support for **DINOv3** (`facebook/dinov3-vitb16-pretrain-lvd1689m`), or a deterministic zero-GPU `mock` backend for CI.
- **4D grain kinematics, multi-scan gap closing, split/merge lineage & precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association, Jaqaman-style multi-scan gap closing (`--max-gap`) across transient beam dropouts, per-grain lattice strain ($\Delta d / d_0$) and angular misorientation ($\Delta\mu$, $\Delta\chi$), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking combining 3D shape anisotropy, covariance spread growth, and spot ROI sub-peak / DINO patch-token dispersion.
- **Publication figures & offline interactive HTML 4D Explorer (`braggtrack.viz`, `braggtrack-report`)** — Generates Okabe–Ito styled lineage DAGs, strain/misorientation dashboards, precursor heatmaps, 3-view MIP cards, and a single-file interactive `report.html` with embedded base64 MIP thumbnails that opens offline on air-gapped beamline networks.
- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, segmentation, semantic embedding, tracking, kinematics, fracture event analysis, HTML reporting, and `ImageD11`/NeXus export end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`.
- **Beamline-scale NeXus/HDF5 I/O, `darfix`/BLISS auto-calibration & operando sync (`braggtrack.io`)** — Auto-discovers 3D detector cubes across `darfix` `NXdata` (`@default` / `@signal` / `@axes`), ESRF-ID03 LIMA files (`/entry_0000/ESRF-ID03/pco_nf/data`), ESRF BLISS `instrument/positioners`, and arbitrary HDF5 trees (`auto_discover=True`). Streams `z`-slabs via `LazyVolume`, synchronizes external potentiostat / load-frame logs (`CSV`/`TSV` and Biologic EC-Lab `.mpt`, regularized $\mathrm{d}Q/\mathrm{d}V$, $\partial\varepsilon/\partial V$), maps transmission geometry `(2theta, eta, d, |Q|)` via `DetectorGeometry`, and exports directly to **`ImageD11` (`.flt`)** and **NeXus `NXprocess` HDF5 (`.nxs`)**.
- **Stabilised 3D Bragg reflection segmentation (`braggtrack.segmentation`)** — Streaming Otsu foreground masking with cross-scan rolling-median smoothing (`smooth_thresholds`), 3D LoG peak enhancement, robust `h`-maxima seeding, 3D watershed, nearby-basin merging, analytical Poisson + quantization centroid uncertainties (`centroid_*_err`), and physical or voxel-space covariance/eigenvalue extraction. Includes an experimental batched dense patch-feature + HDBSCAN slice-stitching segmenter (`--method dino`).
- **Batched multi-view foundation vision descriptors (`braggtrack.semantic`)** — Orthogonal $(\mu, \chi, d)$ maximum-intensity projections (MIPs) encoded in batched forward passes (`embed_batch`, `extract_patch_features_batch`) with ungated **`facebook/dinov2-small`** by default (zero HuggingFace login required), drop-in support for **DINOv3** (`facebook/dinov3-vitb16-pretrain-lvd1689m`), or a deterministic zero-GPU `mock` backend for CI.
- **4D grain kinematics, uncertainty propagation, split/merge lineage & precursors (`braggtrack.tracking`)** — Vectorised geometry + semantic Hungarian association with runner-up `assignment_margin`, Jaqaman-style multi-scan gap closing (`--max-gap`), local strain ($\Delta d / d_0$), ancestor-referenced `lineage_strain` across splits, propagated $\sigma_\varepsilon$, causal angular/lattice velocities (`rate_mu`, `rate_chi`, `rate_d`, `strain_rate`), intensity-partitioned `1 -> 2+` split and `2+ -> 1` merge DAG edges, and persistent multi-scan fracture precursor ranking.
- **Online causal streaming & autonomous DFXM "Scout-and-Zoom" (`braggtrack.autonomous`, `braggtrack-watch`)** — Strictly causal trailing-window Otsu streaming tracker (`CausalStreamingTracker`) with $O(1)$-volume-RAM cached ROI precursors, velocity-extrapolated diffractometer setpoint prediction during rapid phase transitions (e.g., Ni92 `H2 -> H3` collapse), `bliss_dfxm_zoom.v1` JSONL trigger emission (`zoom_triggers.jsonl`), and offline causal replay benchmarking against naive baselines.
- **Publication figures, operando $\mathrm{d}Q/\mathrm{d}V$ dashboards & offline HTML 4D Explorer (`braggtrack.viz`, `braggtrack-report`)** — Generates Okabe–Ito styled lineage DAGs, strain/misorientation dashboards, 3-panel operando electrochemistry + fracture synchronization plots, precursor heatmaps, 3-view MIP cards, and a single-file interactive `report.html` that opens offline on air-gapped beamline networks.
- **One-command reproducible pipeline (`braggtrack-run`)** — Runs validation, operando stimulus sync, segmentation, semantic embedding, tracking, kinematics, fracture event analysis, HTML reporting, and `ImageD11`/NeXus export end-to-end, emitting SHA-256 input checksums, stage timings, and `run_manifest.json`.

<p align="center">
<img src="docs/figures/benchmark_comparison.png" alt="BraggTrack v0.3.0 Quantitative 4D Diffraction Benchmark Suite" width="94%" />
<img src="docs/figures/benchmark_comparison.png" alt="BraggTrack Quantitative 4D Diffraction Benchmark Suite" width="94%" />
</p>

---
Expand Down Expand Up @@ -67,19 +68,20 @@ Produces:
- `artifacts/pipeline/results.nxs` & `artifacts/pipeline/peaks.flt`
- `artifacts/pipeline/run_manifest.json` (SHA-256 input checksums, git SHA, config, stage timings)

### Stage-by-stage CLI commands (9 Entry Points)
### Stage-by-stage CLI commands (10 Entry Points)

| CLI Entry Point | Module Equivalent | Purpose |
|---|---|---|
| `braggtrack-inspect` | `python -m braggtrack.cli.inspect_datasets` | Discover `scan*` folders, primary 3D HDF5 dataset paths, and metadata |
| `braggtrack-validate` | `python -m braggtrack.cli.validate_dataset` | Validate sequence ordering, monotonic timestamps, and axis contracts |
| `braggtrack-segment-synthetic` | `python -m braggtrack.cli.segment_synthetic` | Run fast synthetic 3D segmentation smoke test |
| `braggtrack-segment-dataset` | `python -m braggtrack.cli.segment_dataset` | Segment 3D Bragg spots via chunked Otsu + smoothed thresholds & optional calibration |
| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Stream lazy 3D spot ROIs & compute orthogonal MIP embeddings (`auto`, `mock`, `torch`) |
| `braggtrack-embed-dataset` | `python -m braggtrack.cli.embed_dataset` | Stream lazy 3D spot ROIs & compute batched MIP embeddings (`auto`, `mock`, `torch`) |
| `braggtrack-track-dataset` | `python -m braggtrack.cli.track_dataset` | Build trajectory DAG (`--max-gap`), export `tracks.csv`, `kinematics.csv`, `.flt`, `.nxs` |
| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and lazy-ROI fracture precursors |
| `braggtrack-report` | `python -m braggtrack.cli.report_dataset` | Render publication PNG/PDF figures and self-contained interactive `report.html` |
| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config config.json`, `--report`, and exports |
| `braggtrack-events` | `python -m braggtrack.cli.detect_events` | Resolve `1 -> 2+` splits, `2+ -> 1` merges, and batched lazy-ROI fracture precursors |
| `braggtrack-report` | `python -m braggtrack.cli.report_dataset` | Render publication PNG/PDF figures (`--operando-log`) and self-contained `report.html` |
| `braggtrack-watch` | `python -m braggtrack.cli.watch_autonomous` | Live causal streaming watcher (`--mode watch`) & autonomous DFXM replay (`--mode replay`) |
| `braggtrack-run` | `python -m braggtrack.cli.run_pipeline` | Execute the full pipeline with optional `--config`, `--operando-log`, `--report`, and exports |

---

Expand Down Expand Up @@ -162,9 +164,9 @@ python scripts/pre_pr_check.py

## Repository Layout & Contributing

- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `viz`, `cli`).
- **`braggtrack/`** — Installable Python library (`io`, `segmentation`, `semantic`, `tracking`, `autonomous`, `viz`, `cli`).
- **`data/sample_operando/`** — Bundled ESRF-ID03 operando test scans (`scan0001`–`scan0003`).
- **`docs/`** — [Architecture](docs/architecture.md), [Benchmarks](docs/benchmarks.md), [Coding Practices](docs/coding-practices.md), and publication figures (`docs/figures/`).
- **`docs/`** — [Ecosystem Positioning & Autonomous DFXM Guide](docs/positioning.md), [Architecture](docs/architecture.md), [Benchmarks](docs/benchmarks.md), [Coding Practices](docs/coding-practices.md), and publication figures (`docs/figures/`).
- **`scripts/`** — 9 acceptance gates, 5-scenario benchmark suite (`benchmark_suite.py`), semantic α/β ablation (`ablation_semantic.py`), and CI runner (`ci_report.py`).
- **`artifacts/`** — Generated pipeline outputs (gitignored; never commit).

Expand Down
4 changes: 2 additions & 2 deletions braggtrack/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
try:
__version__ = version("braggtrack")
except PackageNotFoundError:
__version__ = "0.3.0.dev0"
__version__ = "0.4.0.dev0"

__all__ = ["io", "segmentation", "semantic", "tracking", "viz"]
__all__ = ["autonomous", "io", "segmentation", "semantic", "tracking", "viz"]
Loading
Loading