2DTM_FIB_Damage_analysis

Use 2DTM (leopard EM or cisTEM) outputs to measure FIB-Damage.

Contents

• damage_comparrison.ipynb (with sample Nilas and 30kV data) / 30kV_ion_comparison.ipynb — the driver notebooks. Each instantiates one Pipeline per condition (2/8/30 kV gallium, or gallium/argon/xenon at 30 kV respectively) and calls the visualization functions below. See "Notebook structure".
• pipeline/ — data loading/cleaning/transform package:
  • coordinate_transform.py — upstream, one-time per-lamella preprocessing (see "Upstream preprocessing" below); not run by the notebooks themselves.
  • extract_data.py — walks a condition's folder under fib_data/ (optionally per ion/kV subfolder), reads every *_final_dataframe.csv, drops NaN rows, concatenates into one DataFrame, prints unique image count.
  • clean_data.py — renames columns from either legacy cisTEM or newer leopard-em export schemas onto one canonical set (Psi/Theta/Phi/Peak/ Thickness/...), coalesces resulting duplicate columns, fills missing required columns with NaN, drops rows with Peak < 7.5.
  • transform_data.py — per image: computes a robust midpoint (trimmed- slice estimate of lamella center along Z, resistant to outlier particles), derives Distance/Distance_abs/From_edge from Thickness, computes each image's mean Peak, tags each row with its image's total particle Count, drops images below min_peaks.
  • utils.py — shared stats/plotting helpers: standardized_colors / pretty_condition_name (fixed ion→color map and label formatting), fit_gaussian / get_gaussian_parameters (per-bin Gaussian fit + t-test vs. undamaged reference), exp_decay / plot_fitted_exponential (across-bin exponential decay fit), compare_conditions_by_bin (between-condition t-test per shared bin), compute_norm_factor / compute_ribosome_count / normalized_ribosome_count / mean_and_sd (ribosome-count normalization and stats). See "Undamaged reference depth bin" and "Fitting" below for exact cutoffs and fit structure.
  • run_pipeline.py — the Pipeline class orchestrating the above: extract_data → clean_data → add_metadata → transform_data, plus filtering methods (filter_by_thickness, filter_by_defocus_1/2, filter_by_image_count) and snr_bin (bins particles into 10 nm depth bins, computes SNRU and SNR/SNRU).
• visualizations/ — plotting modules the notebooks call:
  • snr_log_visual.py — general-case SNR-vs-depth plot for an arbitrary list of conditions: scatter + error bars + fitted exponential decay per group, legend ordered gallium/argon/xenon-first. Used by the 30 kV ion comparison.
  • snr_log_visual_lowkv.py — specialized variant for the low-kV (2/8 kV) vs. 30 kV comparison: same per-condition plotting, but relabels "Gallium 2 kV" → "Nilas" for display, orders the legend Nilas → Gallium 8 kV → Gallium 30 kV, and adds an optional between_condition_dfs arg that runs compare_conditions_by_bin between two raw binned dataframes and marks any depth bin with a significant (p < 0.05) between-condition difference with a black star. Prints both the within-condition and between-condition t-test tables to stdout, and saves a CSV of plotted points (merged with fit parameters) alongside the PNG/PDF.
  • ribosome_count_comparison_visual.py — plots normalized ribosome count (not SNR) vs. depth-from-edge per condition, filtered to images with ≥100 particles and groups with ≥5 images. Integrates the fitted mean curve over 0-80 nm to print an estimated "expected ribosome count loss %" per condition for a 160 nm lamella.
• fib_data/ — per-image *_final_dataframe.csv inputs (see "Data").
• damage_plots/ — output figures/CSVs, created on first run.

Setup

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
jupyter notebook damage_comparrison.ipynb

Requires Python >=3.10.

Data

The CSVs needed to reproduce this notebook's plots are bundled in-repo under fib_data/Nilas_30kV_comparrison/ (just the per-image *_final_dataframe.csv files each Pipeline reads — not the full upstream match-template output):

FIB_Damage_analysis/
└── fib_data/
    └── Nilas_30kV_comparrison/
        ├── gallium2kv/{lamella10,lamella23,lamella27}/...
        ├── gallium30kv/{lamella26,lamella32}/...
        └── gallium8kv/2025Nov12/...

gallium8kv is loaded and thickness-filtered (cells 2 and 4) but not currently plotted by any remaining cell — kept in case it's reintroduced.

Cell 3's Pipeline(...) calls point data_root at this folder.

Upstream preprocessing

Each per-image CSV that pipeline.extract_data reads must already contain Lamella_x/Lamella_y/Lamella_z and Rad_Tilt_axis/Rad_Tilt_angle columns — the particle's position rotated from raw image-pixel coordinates into the lamella's own coordinate frame, using that image's tilt axis and tilt angle. This transform is a one-time preprocessing step per lamella/session, run before this notebook, and lives in pipeline/coordinate_transform.py (build_final_dataframes).

It merges two per-lamella inputs:

• a directory of per-image particle-coordinate CSVs (from template matching / refinement),
• a CTF/tilt metadata CSV (Estimated Tilt Axis Angle, Estimated Tilt Angle, pixel_size, one row per image),

and writes one {image}_final_dataframe.csv (the file that becomes an input to Pipeline.run()) plus a Lamella_z vs. SNR scatter plot per image.

"Undamaged" reference depth bin

Depth is measured as From_edge (distance from the lamella surface), binned in 10 nm bins. Two different, unrelated cutoffs both get called "the undamaged reference" — don't conflate them:

• SNRU normalization (Pipeline.snr_bin, pipeline/run_pipeline.py): SNRU = mean Peak over particles with From_edge >= snru_from_edge. Every particle's Peak is divided by its own image's SNRU to get SNR/SNRU. This cutoff is set per condition in the notebook (see step 4 below: 40 nm for 2 kV, 70 nm for 30 kV and 8 kV) — tuned to where each condition's own damage profile plateaus.
• Statistical-test reference (get_gaussian_parameters, mean_and_sd, compute_norm_factor in pipeline/utils.py): a fixed 50-80 nm combined bin (falling back to the deepest available bin on thinner lamellae) used only as the comparison group for each bin's t-test / Cohen's d — this is unrelated to the SNRU cutoff above and does not vary per condition.

Fitting: Gaussian per bin, then exponential across bins

Two independent fits are used, one nested inside the other:

1. Per depth bin (get_gaussian_parameters): that bin's SNR/SNRU values are histogrammed (0.075-wide sub-bins via generate_xy_data), then fit to A * exp(-(x-mu)^2 / (2*sigma^2)) (fit_gaussian). The fitted mu/sigma become that bin's plotted point and error bar; R² compares the fitted curve to the histogram counts.
2. Across bins (plot_fitted_exponential): a second, separate fit to 1 - y0 * exp(-x/k) is run through the per-bin mu values themselves — this is the dashed decay curve drawn on the plot. Bins are excluded from this second fit if particle count n < 150 (both visualization scripts) or, in snr_log_visual only (not _lowkv), if the per-bin Gaussian R² <= 0.95.

Notebook structure

1. Imports and plot style setup.
2. Load pipeline/visualizations from this folder.
3. Run the FIB-damage pipeline for gallium 2 kV, 8 kV, and 30 kV.
4. Apply each condition's thickness cutoff and SNRU reference window:
   • 2 kV: thickness <=160 nm, SNRU = mean Peak for ribosomes >=40 nm from edge
   • 30 kV: thickness >=140 nm and <=180 nm, SNRU = mean Peak for ribosomes >=70 nm from edge
   • 8 kV: thickness >=160 nm and <=200 nm, SNRU = mean Peak for ribosomes >=70 nm from edge (computed but not currently plotted by any cell below)
5. Gaussian fit parameters (get_gaussian_parameters) for 2 kV and 30 kV, then the main SNR-vs-depth comparison plot (snr_log_visual_lowkv, 2 kV vs 30 kV). Bins are excluded from the fitted exponential curve if their particle count n <150 (kept as scatter points either way; see "Fitting" above for the R² <=0.95 exclusion used by snr_log_visual but not _lowkv). Also prints each bin's T-test/Cohen's d vs. the combined 50-80 nm (undamaged) reference bins.
6. Ribosome count comparison: 30 kV vs 2 kV (ribosome_count_comparison_visual), plus the expected ribosome-count loss (integral over 0-80 nm) printed per condition.

All figures are written to a damage_plots/ subfolder created next to the notebook on first run.