Conversation
nrminor
force-pushed
the
v3.2.0-rc
branch
3 times, most recently
from
July 9, 2026 15:10
f69837f to
754d2ad
Compare
nrminor
force-pushed
the
v3.2.0-rc
branch
7 times, most recently
from
July 17, 2026 14:22
b857e13 to
005e37c
Compare
Treat the checked-in integration samplesheet as the set of SRA rows requested for an e2e run, while leaving the manifest as metadata for the full fixture collection. The slow e2e harness should assert biological expectations for rows actually selected by the samplesheet, not require outputs for rows removed during local prototyping. The samplesheet itself is therefore excluded from immutable fixture checksum checks; the reference FASTA, indexes, and taxonomy fixtures remain checksum-protected.
Add the first CRUMBS implementation slice: a small Python CLI that turns samtools depth -aa rows into one coverage-support summary per contig. This commit is intentionally limited to the script and its behavior tests. Workflow wiring, taxonomy adaptation, and BioBoxes profile estimation stay in later commits so review can focus on the percentile convention, zero-depth handling, and the CRUMBS cap calculation.
Add the Nextflow process wrapper for CRUMBS contig coverage summarization. This commit should stay narrowly focused on streaming samtools depth -aa over the existing filtered BAM/BAI artifacts into the coverage-summary script. It should not wire the process into reporting yet; that belongs in the later CRUMBS profiling and reporting commits.
Add the NCBI-specific adapter that turns observed adjusted taxids into the explicit profile_taxonomy.tsv contract consumed by later CRUMBS profile estimation. This commit should keep taxonomy lookup out of the future estimator. It reuses the existing py_nvd.taxonomy machinery and focuses on producing taxon_id, taxon_name, rank, taxpath, and taxpathsn rows with behavior tests over the existing miniature taxdump style.
Add the CRUMBS profile estimator that joins final BLAST/LCA assignments, contig coverage summaries, and an explicit profile taxonomy table into contig evidence, taxon evidence, BioBoxes profile, and QC outputs. This commit keeps taxonomy lookup out of the estimator. It focuses on the core profile semantics from the ADRs: collapse repeated BLAST rows to one assignment per contig, compute Taxon CRUMBS as summed p99 Contig CRUMBS divided by assigned contig length, normalize Taxon CRUMBS into BioBoxes percentages, preserve zero-CRUMBS evidence in sidecars, and omit zero-percentage taxa from the BioBoxes profile.
Carry the full rank path alongside CRUMBS taxonomy ID and name paths so downstream tree-shaped exports do not need to infer ancestor ranks from lineage depth. The CRUMBS profile taxonomy adapter already resolves full NCBI lineage objects. This commit keeps the parallel rank sequence as rankpath, validates that it stays aligned with taxpath and taxpathsn, and propagates it into the per-sample taxon evidence table. BioBoxes output remains unchanged because rankpath is CRUMBS-specific metadata rather than part of the CAMI profile format.
Add a terminal CRUMBS report exporter that converts each per-sample taxon evidence table into third-party-friendly taxonomic report formats. The exporter writes a sourmash/taxburst-style Krona TSV with canonical rank columns and a Kraken-style kreport text file. Both formats use CRUMBS relative profile mass rather than read counts or genome-copy abundance. The kreport count-like fields are integerized profile mass with a configurable scale so callers can preserve small positive taxa without inventing a new scientific quantity. Wire the exporter into the CRUMBS profiling subworkflow after taxon profile estimation and publish the new per-sample outputs under the experimental CRUMBS result tree.
Add per-sample CRUMBS Taxburst reports directly from each source taxon evidence table. Extend the existing Krona-style renderer to preserve variable-depth NCBI paths and arbitrary ranks, key children by taxon ID, accumulate direct assignment mass, and aggregate clade mass upward. Emit both interactive HTML and the existing TaxBurst-compatible JSON tree. Routing the renderer from CRUMBS taxa evidence avoids using the intentionally fixed-rank Krona compatibility export as an intermediate representation, where shallow or noncanonical lineages can erase detailed taxonomy. Publish the rendered reports separately from machine-readable taxonomy exports so the experimental CRUMBS result tree retains the established report layout.
…ng DAG display Add a run-level merged CRUMBS Taxburst report directly from the per-sample taxon evidence tables. Reuse the same taxon-ID-keyed renderer as the per-sample reports so variable-depth NCBI paths, arbitrary ranks, duplicate direct observations, and independent sample totals survive in the merged Krona hierarchy. Group CRUMBS taxa artifacts by sample and render them as separate datasets in one interactive report. This avoids routing the primary visualization through the fixed-rank Krona compatibility export. Organize CRUMBS profiling, taxonomy, and report publications by result kind, with per-sample Taxburst reports nested below the run-level report. This commit also places experimental reporting features properly inside an if block so that they are not displayed when `params.experimental == false`.
Taxburst parses every rank column present in a Krona TSV header as an actual lineage node. CRUMBS was exporting fixed strain/genome columns even for species-level rows, causing Taxburst to create duplicate empty-name nodes from trailing blank cells. Write per-file Krona headers only through the deepest contiguous rank populated for every positive row. When a positive lineage has no superkingdom, place it under an explicit unclassified superkingdom so Taxburst receives its required first rank instead of crashing with a KeyError. This keeps taxonomy labels honest, avoids sample-local placeholder ranks, and preserves richer CRUMBS taxonomy evidence in the source taxa TSV and kreport outputs. Cover the rankless case through the installed Taxburst CLI so the exporter-to-renderer compatibility seam remains executable.
Adds RAPID_SCREENING as the experimental pre-assembly k-mer screening stage, with sourmash retained as the current engine rather than embedded in the durable capability name. RAPID_SCREENING_EVAL joins preliminary screening signals with later NVD assembly, BLAST/LCA, and CRUMBS follow-up evidence after REPORTING. The eval normalizes sourmash tax-summary detections into engine-aware screening evidence while preserving sourmash-specific source metrics and parser inputs. It records screening_engine, screening_taxon_name, screening_lineage, and primary_screening_mass, and classifies signals as same_rank_followup_evidence, screening_reference_only, no_same_rank_followup_evidence, or not_comparable. Later evidence is treated as calibration evidence rather than perfect ground truth; the implementation intentionally avoids reciprocal precision/recall, BLAST reference-membership accounting, and arbitrary mass thresholds. Publishes engine artifacts below experimental_rapid_screening/engines/sourmash and eval artifacts below experimental_rapid_screening/eval. The eval surface consists of rapid_screening_eval.duckdb, screening_signal_followup_by_sample_rank.tsv, screening_signals_without_same_rank_followup.tsv, and rapid_screening_eval.html. Adds fixture coverage for all four follow-up bins, tiny positive values, shared WVDB/non-WVDB lineages, source-specific metrics, normalized DuckDB schema, renamed artifacts, Nextflow wiring, and publish hierarchy. Checks run: - just check - just test-one bin/test_rapid_screening_eval.py - just test-one lib/py_nvd/test_nextflow_integrity.py - targeted Ruff checks for renamed bin and integration-test files
Move de novo assembly out of PREPROCESS_CONTIGS so that contig preprocessing now only masks low-complexity sequence and filters short assembled contigs. NVD_MAIN now builds the assembly seam explicitly. Preprocessed virus reads are filtered for the existing minimum-read threshold, branched by sequencing platform, and sent to SHORT_READ_DENOVO_ASSEMBLY or LONG_READ_DENOVO_ENSEMBLY before their contigs are mixed back together for contig preprocessing. The routing is kept local in the Nextflow dataflow with branch and mix operators; both assembly subworkflows may appear in the graph, but only the matching channel receives records. skip_assembly still prevents assembly records from being scheduled. Add the two assembly subworkflows as thin wrappers around RUN_SPADES. Short-read assembly preserves the existing interleaved versus single-end SPAdes behavior, while long-read assembly uses SPAdes' Nanopore mode as a transitional implementation behind the ensembly seam. Update the static Nextflow tripwire to cover the new assembly scheduling boundary. Validation: - just check - pixi run nextflow run . -profile test -preview with assets/test_samplesheet.csv
Removes tests that asserted implementation shape or fixture immutability rather than behavioral invariants. The removed checks covered literal Nextflow source fragments, SHA256 fixture checksum enforcement, exact serialized render formatting, and a redundant SQLite mtime assertion. Keeps the checks that still exercise useful behavior: active Nextflow Python script references compile, the read resolver help path starts, integration fixtures are present, and the remaining renderer/taxonomy tests assert observable outputs directly. Verified with `just test`, which passed with 423 selected tests and 4 deselected.
Add a small ONT SRA fixture row to the generated integration samplesheet so the slow end-to-end integration path can exercise long-read assembly routing instead of only short-read routing. Keep the fixture generator, checked-in CSV, and reference manifest coherent with one another. The manifest now includes the new row metadata and the updated checksum/byte count for the integration samplesheet. This commit does not run the slow networked end-to-end test locally. Focused validation covered the non-slow integration fixture checks and generator lint.
Add the long-read assembler packages required by the accepted ensembly design to the Pixi environment: myloasm, metaMDBG, and Flye/metaFlye. Update the lockfile across the supported Pixi platforms so future workflow commits can assume these commands are available in the runtime environment. Keep MMseqs2 dependency management unchanged in substance; it remains the candidate-generation tool added upstream. Validated command availability with a small Pixi runtime smoke rather than running assembly.
Keep the default long-read assembly behavior on the transitional SPAdes path unless params.experimental is enabled. When experimental mode is on, run myloasm, metaMDBG, and metaFlye independently for each long-read sample, collect their FASTA outputs, and concatenate them into the existing downstream assembly contract. Each experimental assembler process emits a stable tuple carrying sample metadata, assembler identity, and a per-assembler contig FASTA. The long-read subworkflow groups those outputs by sample and emits tuple(sample_id, platform, read_structure, fasta), so PREPROCESS_CONTIGS and downstream BLAST remain unchanged in both modes. This commit intentionally does not implement the conservative union logic yet. Header normalization, exact duplicate collapse, exact containment verification, MMseqs2 candidate parsing, and the SQLite audit sidecar remain Python-owned follow-up work per ADR 0001.
Replace the experimental long-read concat milestone with a Python-owned union step that prepares representative contigs, runs MMseqs2 as an explicit Nextflow process for exact-containment candidates, and finalizes a conservative FASTA for downstream preprocessing. The union logic should own header normalization, exact duplicate and reverse-complement duplicate collapse, exact containment verification, and SQLite sidecar metadata. The existing non-experimental SPAdes fallback remains unchanged. Name the final union FASTA and all published sidecars by sample so concurrent samples cannot overwrite one another in the shared results directory.
Add a small regression test around the exact MMseqs2 candidate-generation behavior used by the experimental long-read contig union. The test protects forward containment, reverse-complement containment, one-mismatch survival, and the explicit max-seq-len setting needed to avoid split-boundary false negatives. This keeps Python as the final deletion authority while checking that the native candidate generator can produce the candidate rows the verifier depends on.
Rename assembler process labels from RUN_* to ASSEMBLE_WITH_* so the Nextflow DAG describes what each process does rather than merely saying that a tool runs. Apply the naming consistently across the short-read SPAdes path, the non-experimental long-read SPAdes fallback, the experimental long-read assembler processes, and the results config selector for SPAdes.
Route the experimental long-read assembler FASTAs and conservative union output into the existing 03_assembled_contigs results stage. This keeps the output hierarchy aligned with the pipeline stage model after adding the long-read ensembly path. The raw long-read assembler outputs are grouped under long_read_assemblers, and the union FASTA plus internal sidecars are grouped under long_read_union. Publishing is gated on params.experimental so the default path remains unchanged. The surrounding results comments now describe de novo assembly generally instead of SPAdes specifically.
Let experimental long-read assembler processes fail normally when the assembler exits nonzero, so Nextflow can apply the project retry-then-ignore error strategy and retain the failure context in the task work directory. The processes now only create empty FASTA outputs for successful assembler runs that do not produce the expected contig file. This keeps a real assembler failure distinct from a successful no-contigs result. Allow the ensembly fan-in to emit partial assembler groups with groupTuple(remainder: true), so samples can still union successful assembler outputs when another assembler has failed and been ignored. If no assembler succeeds for a sample, no union is emitted for that sample. Also remove the unused versions.yml publish filtering from the new long-read assembler and union publish selectors.
Set metaMDBG's minimum read overlap from its default to the pipeline's min_consecutive_bases parameter so experimental long-read assembly uses the same sensitivity floor as downstream contig filtering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release-candidate PR for the v3.2.0 stack (whoopsie, forgot to open before the v3.3.0-rc!).
This is the CRUMBS score release, which introduces a de-biased abundance mass metric that can be used to estimate relative abundance. CRUMBS = Coverage Robust Unified Mapped Base Support.
Highlights: