Repository navigation
Conversation
Implements a parameter-free polar decoder separating amplitude (R, dark-base
detection) from direction (angle theta, base identity).
Two strategies, chosen from encoding matrix E (no chemistry-specific if/else
on channel count):
Orthogonal E (4-ch identity, 3-ch dark-base): argmax(d . E_bright.T)
Non-orthogonal E (2-col Illumina, A fires both channels):
atan2-based angular classification with xtalk-inferred thresholds
Angle thresholds derived parameter-free from the xtalk correction matrix
via _polar_thresholds_from_wcor (midpoints between E_soft centroids).
Results matching polar_basecalling.ipynb:
4-ch Exp C: mapping_rate ~79.4% cells ~74.7% (2,966 watershed cells)
3-ch NIS-seq: mapping_rate ~43.5% nuclei ~85% (1,001 CellPose nuclei)
2-col NIS-seq: mapping_rate ~29.4% nuclei ~84% (superior cell coverage vs SE)
Test data added to scallops/tests/data/:
experimentC/cells.npy watershed cell mask (1024x1024)
nis_seq/ pre-processed NIS-seq HeLa tile40
spots_3ch_corrected.npy (17520, 14, 3) xtalk-corrected 3-ch spots
spots_2col_corrected.npy (17520, 14, 2) synthesised 2-col channels
w_cor_3ch.npy (3, 3) xtalk correction matrix
peaks/labels/nuclei.npy spot metadata and segmentation
barcodes_whitelist.npy 154,646 RC Brunello barcodes
secondary_thr.npy per-chemistry secondary peak thresholds
…for SE (polar was already independent)
…ar-basecalling
…ar-basecalling
…ar-basecalling
…ar-basecalling
…ar-basecalling # Conflicts: # scallops/tests/conftest.py
…ar-basecalling # Conflicts: # scallops/reads.py
Quality scores (Q) - decode_max thresholded bases: Q used the smaller of two softmax probabilities, which capped it at about 3. It now uses the probability of the base that was called. - _decode_polar_chunk: Q was a softmax over raw intensities. It is now the phred score of P(dark/bright call) × P(base | bright), each computed from how close that decision came to flipping. - Non-orthogonal encodings with one bright base no longer crash. - The intermediate np.where is simplified. r_frac fallback - When no whitelist barcode has length T, _rfrac_from_sweep returns None. - decode_polar then logs a warning and uses 0.11·√(channels). Before, it scaled by the number of cycles instead of channels. Non-orthogonal angle classification - New helpers: - _polar_angle_bins sorts bright bases by the angle of their encoding row and puts N−1 edges at midpoints between adjacent bases. - _polar_angle_calls replaces both copies of the hard-coded bright_rows[0..2] logic. - Errors: - Non-orthogonal encodings with more than 2 channels now raise a clear ValueError. - So do bright bases with duplicate directions. - Fallback: if the angles inferred from w_cor are out of encoding order, it warns and uses the encoding angles. _polar_thresholds_from_wcor - It returns per-base angles instead of t_lo/t_hi. - New w_cor_bases argument on decode_polar. It names the base in each w_cor channel, so channel order no longer matters. - It is required when w_cor has more channels than the data. - It is optional for a w_cor over the data's own channels, where it is used to reorder w_cor. - The 3-channel→2-color projection is derived from the encoding. - The broken 2×2 fallback is fixed, and the unreachable spherical branch is removed. Other fixes - decode_polar rejects more than one dark base, since dark bases can't be told apart. Docs - decode_polar now has a "Quality scores" section. - Its docstring describes the angle bins as they work now, and documents w_cor_bases and the Q return column. scallops/tests/test_reads.py There are 10 new tests (20 in total, all passing): ┌─────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐ │ Test │ What it covers │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_quality │ Q is lower for ambiguous calls than clear ones │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_non_orthogonal_row_order │ Calls don't depend on encoding row order (4 cases) │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_w_cor_channel_order │ A 3-channel w_cor in any order, plus the missing-names error │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_w_cor_2_channels │ A 2×2 w_cor still calls A │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_w_cor_2_channels_order │ A swapped 2×2 w_cor is reordered by w_cor_bases │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_single_bright_base │ One bright base doesn't crash │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_w_cor_out_of_order_falls_back │ Warning plus fallback to encoding angles │ ├─────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤ │ test_decode_polar_multiple_dark_bases │ More than one dark base raises │ I also passed w_cor_bases to the 2-color NIS-seq test, corrected the fixture's channel comment (ch03=A, ch04=T, ch06=C) Notebooks docs/notebooks/polar_basecalling.ipynb - Code: - Updated for the new API. - The 2-color r_frac now uses the same angle-bin calls as decode_polar, so it prints 0.161 instead of 0.179. - The cell 16 plots use the baseline-subtracted geometry. - Cell 18 has corrected channel labels. - Explanations: - Channel identities. - How polar dark detection and the boundaries work. - What decode_se actually does: per-channel thresholds, not "range weighting". - How α is computed and the A/C near-tie margin. - Contradictory or unsupported claims are removed, such as cell 9's "no sweep needed" and the Bernoulli justification for f = α/3. - The method counts in cells 19 and 22, and the comments. - Outputs: the results tables didn't change.
decode_em / fit_base_mixture: built-in crosstalk correction - New parameter xtalk_tol: float | None = 0.15. - For orthogonal encodings (4-color, or 3-color with a dark base), the fit estimates the median crosstalk matrix on its fit reads, the same method channel_crosstalk_matrix uses. - It applies the matrix only if the largest off-diagonal term, relative to the diagonal, exceeds xtalk_tol, so spots you already corrected are left alone. - Non-orthogonal encodings such as 2-color are never corrected. - The matrix is stored as model["xtalk"] (None if not applied) and applied to every chunk when calling, so calls still don't depend on Dask chunking. - Effect on the test data: uncorrected 4-color goes from 84.3 % to 85.5 %, uncorrected 3-color from 53.7 % to 54.1 %, and tied=True on uncorrected 3-color from 2.1 % to 43.2 %. Results on corrected spots are unchanged. decode_se - Now raises a ValueError for more than one dark base, matching decode_polar and decode_em. Before, it silently never called the extra ones. decode_polar - Return annotation widened to pd.DataFrame | dd.DataFrame. Tests in scallops/tests/test_reads.py - New test_decode_em_crosstalk, using simulated bleed. It checks: - the bleed is detected and corrected, with calls over 99 % accurate; - calls are the same with Dask chunking; - already-corrected spots aren't corrected again; - xtalk_tol=None turns correction off; - 2-color encodings are skipped. - test_decode_polar_multiple_dark_bases is renamed test_decode_multiple_dark_bases and also covers decode_se. - NIS-seq fixture: dropped the buggy hand-built 2-color baseline. 2-color now uses the 3-color spot-quality threshold (thr_r2 = thr_r3). - test_decoders_2col_nis_seq assertions and docstring values are updated to match: SE 23.9 %, polar 39.7 %. - Notebook references are renamed, and the "(recommended …)" comment is removed. Notebook (docs/notebooks/base_calling_methods.ipynb) - clearer throughout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding options for sign encoding and polar basecalling methods