Conversation
nrminor
force-pushed
the
v3.1.0-rc
branch
8 times, most recently
from
June 24, 2026 20:01
07dbf7b to
9f0dcc8
Compare
nrminor
force-pushed
the
v3.1.0-rc
branch
4 times, most recently
from
July 7, 2026 12:28
29429c2 to
9b62f8c
Compare
nrminor
force-pushed
the
v3.1.0-rc
branch
4 times, most recently
from
July 9, 2026 15:10
8c8c9ab to
5ca42e8
Compare
nrminor
force-pushed
the
v3.1.0-rc
branch
4 times, most recently
from
July 14, 2026 02:45
2182062 to
eff55fc
Compare
Replace database-row-weighted dominant assignment and the absolute five-bit LCA window with one coherent reference-consensus policy. Preserve the user-selected BLAST retention scope, include all score ties at its ordinal boundary, and select assignment references whose bitscores are at least 95% of the best retained score. Call an assignment dominant when those references represent one distinct taxid; otherwise compute their strict NCBI lowest common ancestor. Keep LCA annotation separate from quality filtering and rename stale filtering-oriented internals to describe score context and taxonomic consensus. Parse raw BLAST outfmt 6 as delimiter-only text, treating double quotes as literal reference-title data while retaining strict ten-field width validation. Keep selected TSV output conventionally quoted so downstream readers recover exact titles. Cover the motivating close-score edge case, retention-boundary ties, multi-taxid references, assignment methods, literal reference-title quotes, and empty inputs through the command-line seams.
Make setup-installed shell hooks invoke the NVD wrapper by its known absolute path so systems without ~/.local/bin on PATH can initialize correctly. Detect and offer to replace legacy hooks, preserve explicit consent before editing shell RC files, and remove Bash-specific activation guidance from the installer.
Add a standalone read-input resolver for NVD samplesheets. The resolver turns exact FASTQ paths, paired or single FASTQ glob declarations, and SRA accessions into a canonical JSONL manifest that downstream Nextflow wiring can consume later. The resolver keeps the tricky ingress rules localized: exact files take precedence over globs, globs take precedence over SRA accessions, lower-precedence sources produce loud warnings, exact path columns reject glob metacharacters, and all local paths are validated before the pipeline fans out. Add focused tests for platform normalization, duplicate sample rejection, symlink-preserving absolute paths, single-end glob sorting, CASAVA-style paired lane validation, compression consistency, JSONL output shape, and stderr warning behavior. This commit deliberately stops before Deacon streaming or GATHER_READS integration so the canonical declaration contract can be reviewed on its own.
Add stream_fastqs_to_deacon.py as the isolated mechanism for feeding resolved FASTQ bundles into Deacon without materializing full concatenated FASTQs. The helper creates named pipes, streams one or more ordered FASTQ inputs into those pipes, runs Deacon against the pipe paths, and reports failures from both Deacon and the stream producers. Keep the lifecycle in Python rather than embedding a long Bash script in Nextflow. The helper validates executable availability, input presence, paired-list length, compression consistency, and explicitly rejects zstd until a supervised zstd producer is added. Add tests with both fake consumers and real Deacon. The real-Deacon tests build a tiny index and compare FIFO-streamed output byte-for-byte against a tiny direct-control run, while the fake-consumer tests pin failure modes that are difficult to trigger reliably with Deacon itself: producer errors, truncated gzip streams, Deacon exiting before all FIFOs open, Deacon failing after reading from a FIFO, FIFO cleanup, and the guarantee that the consumer sees FIFO inputs rather than regular concatenated files. This follows the resolved read declaration commit and prepares for a later GATHER_READS/PREPROCESS_READS integration commit.
Tighten the read-input resolver contract before wiring it into Nextflow. FASTQ paths and glob patterns must now be absolute so resolution does not depend on the current working directory of a Nextflow task. This keeps the samplesheet interface explicit for cluster and automation use cases, where stable absolute symlink paths are preferable to context-sensitive relative paths. The resolver still preserves symlink paths rather than rewriting them to storage targets. Also remove zstd FASTQ suffixes from the resolver's supported input set until the Deacon streaming helper grows supervised zstd producer support. This keeps the resolver and streaming helper contracts aligned.
Wire the resolved read-input manifest into the Nextflow pipeline. This commit should make GATHER_READS resolve the samplesheet once, route SRA and local read declarations into a single explicit bundle channel shape, and let Deacon filtering choose direct input for single-file bundles or stream_fastqs_to_deacon.py for multi-file bundles. Keep the integration focused on production wiring rather than adding the full lane-glob pipeline integration test. That test coverage will follow in a separate commit once the channel contract and process wiring are in place. The intended internal contract is tuple(meta, r1_files, r2_files), where meta carries sample id, platform, read mode, and source while FASTQ paths remain top-level path collections for Nextflow staging and hashing.
Add a maintainer script that normalizes WVDB FASTA identifiers, projects WVDB annotations into sourmash lineage CSVs, and validates WVDB manifest coverage. The transform is a standalone PEP 723 Python script built around Polars so it can be run with uv without becoming part of the Nextflow runtime path. Add Just recipes for the reference build workflow: one recipe fetches and validates the NCBI Virus sourmash inputs, one builds and validates the WVDB-only sourmash side, and the top-level WVDB alias combines both sides when the upstream NCBI artifacts are available. The NCBI recipe fails early when an upstream URL returns an HTML maintenance page instead of a sourmash artifact. Document the WVDB-to-sourmash lineage policy, including the decision to leave missing genus/species ranks blank rather than inventing false precision.
Remove the temporary repo-local Claude sourmash skill now that the experimental sourmash work is captured in source, tests, and documentation. Tighten the gitignore allowlist so future branch-local agent skills are not accidentally tracked.
Prevent URL-fetched sourmash reference and lineage processes from emitting every non-hidden entry in the task sandbox. On OSPool/HTCondor container runs, broad path globs can include runtime directories such as home, scratch, tmp, var, and var_tmp, contaminating downstream gather and taxonomy input tuples.
Apply an experimental process policy so sourmash profiling and sample-similarity QC failures do not terminate the rest of the pipeline. Experimental tasks ignore failures when params.experimental is enabled, while retaining the standard retry-then-ignore behavior if reused outside the experimental gate.
Restructure the NCBI Virus + WVDB sourmash reference recipes so the build root contains only the publishable combined artifacts, while downloaded inputs and intermediate WVDB/NCBI products live under a hidden .work directory. This makes the artifact to upload and use in NVD obvious and avoids confusing partial or intermediate signature directories with the final sourmash zip database. Name WVDB-derived artifacts after the upstream WVDB v1.0 source release instead of the premature internal v2 label. The previous v2 wording was not a real product version and made it look like the recipe downloaded a different WVDB release than it actually uses.
Normalize WVDB lineage rows so missing intermediate ranks are represented with explicit unclassified-rank placeholders when a lower rank is present. This avoids malformed sourmash lineage paths such as double semicolons in downstream tree consumers while preserving genuinely unknown trailing ranks as blanks.
Extend the experimental sourmash reporting path with two additional outputs from the existing gather-to-taxonomy workflow. First, include sourmash tax metagenome's bioboxes output format so each sample emits a CAMI-compatible taxonomic profile alongside the existing csv_summary, lineage_summary, krona, and kreport artifacts. Second, add a BetterPlot Sankey report built from the existing sourmash csv_summary output. BetterPlot is added as a dependency together with the conda PyQt package needed to satisfy its current hard PyQt requirement without forcing PyPI source builds on platforms such as linux-aarch64. The new visualization remains under the experimental sourmash reporting branch and should inherit the same non-terminal failure behavior as the other experimental sourmash reports.
Visualization tools in the experimental sourmash reporting path require taxon display names to be globally unique, but sourmash csv_summary output can legitimately contain repeated labels such as Unclassified or unclassified genus under different parent lineages. Add a NORMALIZE_TAXONOMIC_PROFILE_SUMMARY process and helper script that builds a private normalized summary CSV. The normalizer preserves row order and non-lineage columns, keeps parent prefixes consistent after renaming, and uses original parent context in disambiguated labels to avoid recursively nested display names. Feed the normalized summary into both the taxburst sunburst and sourmash Sankey renderers, staging the input as the sample ID so plot labels stay readable. Stop publishing intermediate sourmash tax_metagenome outputs, and publish per-sample taxburst HTML/JSON under reports/taxburst/per_sample so a later merged taxburst report can occupy reports/taxburst.
Build one run-level taxonomic profile display-name normalization map, apply it to each sample summary before per-sample reports, and add a merged multi-sample TaxBurst/Krona-style HTML report. The goal is for per-sample and merged visualizations in the same run to use identical normalized labels while preserving sample metadata in Nextflow tuples instead of recovering sample IDs from filenames.
Add just recipes and short aliases for running the slow end-to-end integration test with SPAdes assembly skipped, covering both the normal and experimental e2e paths. Teach the integration harness to honor NVD_INTEGRATION_SKIP_ASSEMBLY by passing --skip_assembly to Nextflow. In skip-assembly mode, assert the expected partial-run BLAST shape while still allowing experimental sourmash assertions to run when NVD_INTEGRATION_EXPERIMENTAL is also enabled.
WVDB annotation fields use sentinel values such as Unclassified, Unknown, and NA to indicate that a rank was not classified. Previously the reference build treated these as ordinary non-empty source values and filled internal gaps with bare labels such as unclassified genus. Those labels are ambiguous in downstream tree visualizations, and they forced the runtime reporting pipeline to repair names after sourmash had already summarized each sample. This change makes placeholder naming part of the WVDB reference-build policy. Source-column selection now prefers the first informative value and lets unknown sentinels fall through to later fallback columns. When no informative source value exists, explicit sentinels and missing internal ranks are converted into contextual placeholder labels such as unclassified family [under Picornavirales]. If a short context would collide with another generated placeholder or with a real source label, the context expands deterministically, with a hash suffix reserved for pathological cases. The WVDB sourmash input tests now cover contextual placeholder labels, sentinel fallback behavior, common sentinel spellings, generated-label collision avoidance, and row-order-independent lineage output. The reference build documentation now describes sentinel fallback and contextual placeholders as stable reference-release behavior rather than per-run reporting normalization.
Remove the runtime TaxBurst summary normalization path now that WVDB placeholder names are handled by the reference build. The reporting subworkflow now selects each sample-specific sourmash `.summarized.csv` directly from `SOURMASH_TAX_METAGENOME` output and feeds that summary to per-sample TaxBurst and Sankey rendering as soon as the sample is ready. The merged multi-sample TaxBurst report remains an all-sample artifact: it still groups profile summaries before rendering, but that grouping no longer blocks per-sample reports. The run-level normalization-map processes, helper script, and tests are removed because display-safe placeholder names are now a reference-build invariant rather than a per-run repair step. A Nextflow integrity tripwire now checks that the reporting workflow no longer references the runtime normalizer and that per-sample report renderers consume `ch_sourmash_profile_summaries` directly. This commit assumes sourmash references provide display-safe placeholder lineages. Older references with repeated bare placeholders such as `Unclassified` may still fail TaxBurst because the runtime normalization repair path has been removed. The updated NCBI Virus + WVDB reference build was run on CHTC, and the resulting TaxBurst plots were inspected successfully.
Add taxpath generation to the WVDB sourmash input builder so custom WVDB lineages can support sourmash BioBoxes and other taxid-bearing outputs. The builder now accepts a reference sourmash lineage CSV, reuses exact unambiguous reference taxids for known lineage prefixes, backfills missing ancestors from unique reference paths, and assigns stable WVDB-local numeric taxids for novel or placeholder nodes. Build reusable reference lineage indexes once per run instead of reconstructing them for every WVDB row. Add Loguru progress logging on stderr for the major build phases, including reference loading and indexing, FASTA normalization, WVDB projection, reference backfill, placeholder contextualization, taxpath generation, combination, row counts, node counts, and elapsed times. Add a preserving combine-lineages subcommand for the NCBI Virus + WVDB reference build. This avoids using sourmash tax prepare as the final lineage writer, because tax prepare validates lineages but strips taxpath from CSV output. Update the reference-build recipe to pass the downloaded NCBI Virus lineages into the WVDB builder and use the preserving combiner for the publishable combined lineage CSV. Extend WVDB builder tests around taxpath emission, reference taxid reuse, ancestor backfill, novel descendants, and preserving combination.
Remove the explicit sourmash_output_bioboxes parameter so experimental sourmash reporting asks for BioBoxes automatically when possible, but guard the tax metagenome process against taxonomy CSVs that lack taxpath. The process now inspects the lineage header, includes bioboxes only when taxpath is available, and otherwise warns while still emitting the non-BioBoxes sourmash reports. Update the mini sourmash lineage fixture generation to include taxpaths and add a fixture test that ensures the integration taxonomy can support BioBoxes output. Keep the model, schema, and config surfaces aligned with the removed parameter.
Thread the existing host/contaminant index resolution through the contig screening subworkflow as an optional depletion index. The public parameter surface stays compatible: users still opt in with the existing host_index, host_index_url, or host_contaminants_fasta settings. When a depletion index is configured, DEACON_FILTER_CONTIGS now enriches contigs against the virus index and pipes those candidates into deacon depletion before writing the final FASTA. When no depletion index is configured, the process keeps the original direct virus-screening behavior and uses a staged sentinel path to avoid relying on unsupported optional path inputs. This keeps read depletion and contig depletion using the same resolved index, avoiding a second fetch/build/union path while leaving future naming work around host versus depletion indexes separate from this compatibility-preserving change.
Allow NVD runs to proceed without a configured target enrichment index by routing read and contig selection through an explicit no-enrichment mode. This keeps the existing Deacon-shaped workflow seam while making the target index optional for users who want to run the heavier unenriched path or disable enrichment from the CLI. The commit also renames the read enrichment process away from human-virus-specific wording toward the target-enrichment terminology.
Use the rolling latest container tag as the default runtime image from v3.1 onward so newly added pipeline tools are available to Docker, Apptainer, and CHTC executions without another stale version pin. Keep the Nextflow parameter, Python model, tests, and v3.1 parameter schema aligned while leaving the frozen v3.0 schema unchanged.
Collaborator
|
I am loving the turnaround time savings from lane concat using deacon. Deacon is such a fantastic tool that has helped us make this uber fast. I have found one issue that is leading to a bloat in the size of the raw_blast_hits. TL;DRComparing one prod sample run through v3.0 and v3.1 (same BLAST DB, same
The two are the same underlying policy ("retain all tied references, resolve Setup (anonymized)
Because the assemblies and reference DBs are identical, all differences are isolated to Headline metrics
Finding — retention count is no longer a cap (the defect)
With Evidence:
Worked example — one contig, identical inputs:
Same contig, same single-value bitscore tie. v3.0 kept an arbitrary 5 of the tied Why it's a defect
|
core_nt carries many redundant reference sequences, so a single contig can match dozens of references at an identical bitscore. Every tied reference then flows into the metagenomic hits list, inflating the production table far beyond the configured retention: a conserved-region contig can insert up to max_blast_targets rows where blast_retention_count was requested. Apply an ordinal cutoff when inserting the list. Keep only the top blast_retention_count reference sequences per (sample_id, qseqid) by descending bitscore, breaking ties deterministically on sseqid and retaining every taxid row of a kept reference. Taxonomic assignment is unaffected because the LCA was already computed upstream over the full reference set. Keep the cutoff in the list insert (labkey_upload_blast_results.py) rather than the shared prepare step, so the raw BLAST result uploaded to WebDAV still contains every hit and only the production list is bounded. LABKEY_UPLOAD_BLAST now passes blast_retention_count through to the script.
Exercise apply_reference_cutoff: a top-bitscore tie larger than the retention count is bounded to K references and truncated deterministically to the lowest sseqids, and the cutoff is scoped per (sample_id, qseqid) so a multi-taxid reference counts as a single hit with all of its rows retained.
Collaborator
|
After some internal discussion we have decided that it is a useful finding to show the top blast hits up to however many are tied for the lowest E-value. The issue with our database being populated with many more rows is handled now at the LIMS integration section. If there are more than the max blast hit cutoff tied at the same lowest E-value that information will stay. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This release candidate PR stacks on top of a few feature branches, including
lane-concatandexperimental-sourmash. It also brings with it a number of commits intended as part of the RC itself.In all, this commits bring a number of new features:
--experimentalparameter and command line argument that can turn on experimental features deployed for testing in main. This is meant to make testing new features without maintaining multiple checkouts of NVD easier.--experimentaland will remain so for the next few releases while we evaluate whether sourmash is the right engine for this rapid screening. As of currently, these features use a reference sketch built from NCBI Virus combined with v1.0 of the Wastewater Virus DataBase/--skip-assemblyand--skip-blastflags that allow users to only run in silico enrichment and rapid screening without the vastly more expensive assembly and BLAST workflows.