Conversation
nrminor
force-pushed
the
v3.3.0-rc
branch
17 times, most recently
from
July 14, 2026 22:12
ffef836 to
964587d
Compare
Treat the checked-in integration samplesheet as the set of SRA rows requested for an e2e run, while leaving the manifest as metadata for the full fixture collection. The slow e2e harness should assert biological expectations for rows actually selected by the samplesheet, not require outputs for rows removed during local prototyping. The samplesheet itself is therefore excluded from immutable fixture checksum checks; the reference FASTA, indexes, and taxonomy fixtures remain checksum-protected.
Add the first CRUMBS implementation slice: a small Python CLI that turns samtools depth -aa rows into one coverage-support summary per contig. This commit is intentionally limited to the script and its behavior tests. Workflow wiring, taxonomy adaptation, and BioBoxes profile estimation stay in later commits so review can focus on the percentile convention, zero-depth handling, and the CRUMBS cap calculation.
Add the Nextflow process wrapper for CRUMBS contig coverage summarization. This commit should stay narrowly focused on streaming samtools depth -aa over the existing filtered BAM/BAI artifacts into the coverage-summary script. It should not wire the process into reporting yet; that belongs in the later CRUMBS profiling and reporting commits.
Add the NCBI-specific adapter that turns observed adjusted taxids into the explicit profile_taxonomy.tsv contract consumed by later CRUMBS profile estimation. This commit should keep taxonomy lookup out of the future estimator. It reuses the existing py_nvd.taxonomy machinery and focuses on producing taxon_id, taxon_name, rank, taxpath, and taxpathsn rows with behavior tests over the existing miniature taxdump style.
Add the CRUMBS profile estimator that joins final BLAST/LCA assignments, contig coverage summaries, and an explicit profile taxonomy table into contig evidence, taxon evidence, BioBoxes profile, and QC outputs. This commit keeps taxonomy lookup out of the estimator. It focuses on the core profile semantics from the ADRs: collapse repeated BLAST rows to one assignment per contig, compute Taxon CRUMBS as summed p99 Contig CRUMBS divided by assigned contig length, normalize Taxon CRUMBS into BioBoxes percentages, preserve zero-CRUMBS evidence in sidecars, and omit zero-percentage taxa from the BioBoxes profile.
Carry the full rank path alongside CRUMBS taxonomy ID and name paths so downstream tree-shaped exports do not need to infer ancestor ranks from lineage depth. The CRUMBS profile taxonomy adapter already resolves full NCBI lineage objects. This commit keeps the parallel rank sequence as rankpath, validates that it stays aligned with taxpath and taxpathsn, and propagates it into the per-sample taxon evidence table. BioBoxes output remains unchanged because rankpath is CRUMBS-specific metadata rather than part of the CAMI profile format.
Add a terminal CRUMBS report exporter that converts each per-sample taxon evidence table into third-party-friendly taxonomic report formats. The exporter writes a sourmash/taxburst-style Krona TSV with canonical rank columns and a Kraken-style kreport text file. Both formats use CRUMBS relative profile mass rather than read counts or genome-copy abundance. The kreport count-like fields are integerized profile mass with a configurable scale so callers can preserve small positive taxa without inventing a new scientific quantity. Wire the exporter into the CRUMBS profiling subworkflow after taxon profile estimation and publish the new per-sample outputs under the experimental CRUMBS result tree.
Add per-sample CRUMBS Taxburst reports directly from each source taxon evidence table. Extend the existing Krona-style renderer to preserve variable-depth NCBI paths and arbitrary ranks, key children by taxon ID, accumulate direct assignment mass, and aggregate clade mass upward. Emit both interactive HTML and the existing TaxBurst-compatible JSON tree. Routing the renderer from CRUMBS taxa evidence avoids using the intentionally fixed-rank Krona compatibility export as an intermediate representation, where shallow or noncanonical lineages can erase detailed taxonomy. Publish the rendered reports separately from machine-readable taxonomy exports so the experimental CRUMBS result tree retains the established report layout.
Treat BBMerge's unmerged output as independent single-read queries after pair classification. Force singleton-mode read filtering so BBTools cannot reconstruct mate coupling from /1 and /2 identifiers. Retain each record that independently passes quality and length thresholds, even when its former mate is filtered out. Permit single-read batches to contain odd record counts and make no downstream pairing guarantee. Add regression coverage in which one mate passes QC and the other fails.
Rename the contig-oriented query lookup vocabulary to the generalized query-sequence model before introducing read-derived BLAST queries. This commit keeps BLAST admission behavior contig-only, but migrates the existing assembly-derived query path from the prototype contig-specific terms to the durable evidence vocabulary: the SQLite lookup table becomes query_sequences, source-local provenance is source_id rather than contig_id, and assembly query identifiers use the nvdContigQuery prefix. Keeping this migration separate from read-derived query admission makes the next feature commit additive rather than mixing schema correction with new BLAST behavior.
Add vsearch to the Pixi environment so the read-query normalization path can use a battle-tested exact dereplication tool instead of implementing high-volume per-read deduplication in Python. This commit only changes dependency metadata. The later read-query normalization work will verify and use vsearch to collapse exact duplicate unmapped read sequences within each evidence-class batch while preserving representative IDs and abundance counts.
Route BLAST work through natural evidence-class batches instead of treating each sample as one monolithic query FASTA.
This commit should keep biological behavior contig-only while splitting selected contig query FASTA records into real evidence-class batches such as short_assembly_contig and long_assembly_contig. The BLAST-facing tuple should carry evidence_class, BLAST and adjacent processes should use Nextflow tags like `${sample_id}, ${evidence_class}`, and per-batch TSVs should be stacked back into the existing per-sample reporting shape.
This intentionally does not normalize read FASTQ, use vsearch, add read-derived BLAST batches, change CRUMBS read-query semantics, or update LabKey schemas.
Add an experimental query-batch summary artifact for the BLAST query preparation seam. The new SUMMARIZE_BLAST_QUERY_BATCHES process runs only when params.experimental is true and publishes per-sample TSVs under the blast_query_batches summary directory. The summary records the expected query classes, their query source, whether each class emitted a FASTA and lookup sidecar, and the number of FASTA records that were actually submitted for BLAST. This makes absent query classes explicit without synthesizing empty query FASTAs at process boundaries. Use a small Polars-backed helper to keep the tabular shaping declarative and to leave the expensive BLAST path unchanged. The summary process stages grouped FASTAs and lookup databases into separate directories so reused contig lookup sidecars do not collide when multiple query-class batches share the same SQLite filename.
Adds a Polars LazyFrame-based builder for the query-level Big Table, producing one row per BLASTed query from retained final BLAST rows. The output uses final-artifact vocabulary such as assigned_* and support_tier, preserves retained-hit ambiguity diagnostics without row-exploding hits, avoids second_* columns and numeric confidence scores, and orders columns/rows for left-to-right and top-to-bottom human scanning. Wires the per-sample query Big Table into REPORTING behind the experimental feature gate. When experimental mode is enabled, per-sample audit artifacts are published under results/nvd/query_big_tables/per_sample and the featured all-sample query_big_table.tsv is published at the results/nvd root, leaving room for a peer taxon_big_table.tsv in the next slice. The process currently builds from the final BLAST TSV; the builder accepts CRUMBS inputs for the later CRUMBS/taxon-table wiring slice.
Rename the taxon Big Table CRUMBS percentage column to relative_crumbs_percent so the public artifact avoids CRUMBS-internal “emitted” terminology while still reading percentage_emitted from the CRUMBS taxa TSV. Add PEP 723 script metadata and uv script shebangs to the Big Table command-line entrypoints so direct Nextflow bin-script execution can resolve Polars without depending on the ambient Python environment. Replace duplicated inline Nextflow concatenation logic for query and taxon Big Tables with a shared Polars-based stack_big_tables.py helper that validates exact schema order before stacking per-sample TSVs. Keep only the two all-sample Big Tables as featured files at the NVD results root. Publish per-sample query and taxon Big Tables under 08_experiment_summary/big_tables, move CRUMBS and sourmash outputs under 08_experiment_summary, and keep mapback-unmapped reads with the stage 04 virus contig classification outputs instead of creating root-level experimental directories.
Add a needletail-backed profile_fastx.py command that scans FASTA/FASTQ artifacts once and emits sidecars for sequence count, aggregate profile JSON, length histogram, and per-read mean-quality histogram. The script is directly executable via a uv PEP 723 shebang and streams aggregation instead of materializing per-record observations. Add a reusable PROFILE_FASTX process and stage-specific aliases for preprocessed read shards and filtered contigs. Profile sidecars publish beside the sequence files they describe so profiles, future plots, and FASTX artifacts sort together by name. Propagate profiled read-shard metadata as a meta map containing id, platform, read_structure, query_class, profile_stage, and sequence_count. Assembly, sourmash profiling, and mapback routing now use cached profile counts instead of native Nextflow/Groovy countFastq/countFasta scans. Keep legacy tuple adapters only at existing process boundaries that have not yet been converted to meta maps.
Preserve semantic read-batch identity and per-sample batch counts from pair merging through terminal read profiling. Form each sample's available post-QC batches with a sized Nextflow group key so complete samples can reach assembly, mapback, and sourmash without waiting for every sample in the run. Retain the pipeline's best-effort failure behavior by emitting available remainder batches after upstream closure. Centralize grouping so all downstream consumers operate on the same available post-QC reads, and emit zero-count mapback sidecars for absent query classes so one-batch remainders satisfy the existing process contract.
Extend cached FASTX profiles with exact sequence and base counts at configured thresholds, then use that evidence to decide independently whether metaMDBG, Myloasm, and metaFlye should run for each long-read sample. Preserve complete preprocessed FASTQs as assembler inputs while avoiding allocations for samples with no reads meeting an assembler's minimum useful length. Emit deterministic per-sample eligibility rows and a combined TSV explaining each run or skip decision. Experimental runs with no long-read samples still publish a header-only report instead of silently omitting the artifact. Publish assembler outputs in dedicated stage-03 directories and retain the existing partial-union behavior when individual assemblers fail or produce no contigs.
Construct the final LabKey log collection only when LabKey integration is enabled. Avoid eagerly creating the LabKey upload log directory for runs that cannot produce LabKey artifacts. Preserve the existing final-log collection and storage behavior for LabKey-enabled runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Another brief release-candidate PR, this time for the v3.3.0 stack, that I'll flesh out more soon.
This is a big one. Highlights include:
More soon!