diff --git a/.claude/development-notes/README.md b/.claude/development-notes/README.md index 0b0e90a..cd6811e 100644 --- a/.claude/development-notes/README.md +++ b/.claude/development-notes/README.md @@ -2,19 +2,19 @@ Why PoolSeqFlow is built the way it is. This is the home for everything that used to live in the source as commentary: decisions and who made them, what was tried and rejected, what a choice was measured against, and the traps that produced a wrong answer once already. -It is not user documentation — that is `manual/PoolSeqFlow-manual.md`. It is not an explanation of what the code does — that stays in the code. It is the reasoning a person needs before they *change* something, and the reason we moved it here is that the two kinds of writing had grown into each other: files where the argument for a design outweighed the description of it, and a developer had to read three paragraphs of history to find out what a function returned. +It is not user documentation -- that is `manual/PoolSeqFlow-manual.md`. It is not an explanation of what the code does -- that stays in the code. It is the reasoning a person needs before they *change* something, and the reason we moved it here is that the two kinds of writing had grown into each other: files where the argument for a design outweighed the description of it, and a developer had to read three paragraphs of history to find out what a function returned. ## These notes are dated **Every note records the code as it stood when it was written, and says so at the top.** Later work moves things and a note is not rewritten to follow it: these are the record of how the project got here, not a second manual to keep in sync. -So a present-tense description inside a note describes the code *at that note's date*. `manual/PoolSeqFlow-manual.md` is the current answer and always is — where the two disagree, the manual is right and the note is history. +So a present-tense description inside a note describes the code *at that note's date*. `manual/PoolSeqFlow-manual.md` is the current answer and always is -- where the two disagree, the manual is right and the note is history. -**Two files are exceptions and both say so at the top.** This index, and `shell-and-nextflow-gotchas.md` — a reference that is appended to as traps are found, with per-entry dates, rather than a record of one moment. Both are kept current. +**Two files are exceptions and both say so at the top.** This index, and `shell-and-nextflow-gotchas.md` -- a reference that is appended to as traps are found, with per-entry dates, rather than a record of one moment. Both are kept current. ## The rule -**It lives in `CLAUDE.md` at the repository root now, not here.** That file loads into every session; this one does not, which is why the rule was broken twice in a session that had it written down. Read `CLAUDE.md` for the current wording — the clause-level test, the tell-words, the third category, and the exclusions. +**It lives in `CLAUDE.md` at the repository root now, not here.** That file loads into every session; this one does not, which is why the rule was broken twice in a session that had it written down. Read `CLAUDE.md` for the current wording -- the clause-level test, the tell-words, the third category, and the exclusions. In short: every comment *clause* is either what the code does or why we chose it. What it does stays, in one line. Why we chose it comes here. @@ -23,12 +23,12 @@ In short: every comment *clause* is either what the code does or why we chose it | Where | What | |---|---| | **The source** | Only what a reader needs to understand the function below: what it does, what it returns, a contract or coupling they cannot see from here. | -| **`manual/PoolSeqFlow-manual.md`** | Decisions that matter **scientifically** — anything that changes what a result means, or that a person interpreting output needs to know. The manual is the project's real documentation and is what the published site is generated from. | +| **`manual/PoolSeqFlow-manual.md`** | Decisions that matter **scientifically** -- anything that changes what a result means, or that a person interpreting output needs to know. The manual is the project's real documentation and is what the published site is generated from. | | **Here** | The record of how it got here. Design churn, alternatives tried and dropped, what a thing used to be, measurements behind a choice, who decided what and when. | The manual is the one to keep watching. *"Pool size sets the detection limit"* is science and belongs in the manual. *"We tried three bcftools mechanisms and the second one silently returned zero records"* is development history and belongs here. A decision we changed five times is confusing to a reviewer and must not end up in either the source or the manual. -**No source file references these notes**, and no source file references the test suite. Both point the wrong way. A note is dated, so a comment pointing at one imports a description of code as it used to be — which is the exact failure the rule above exists to prevent. And the dependency runs tests → scripts: a test may say what it is testing; a script may not say it is tested. +**No source file references these notes**, and no source file references the test suite. Both point the wrong way. A note is dated, so a comment pointing at one imports a description of code as it used to be -- which is the exact failure the rule above exists to prevent. And the dependency runs tests -> scripts: a test may say what it is testing; a script may not say it is tested. ## Where this lives @@ -38,16 +38,17 @@ The manual is the one to keep watching. *"Pool size sets the detection limit"* i ## Layout -One file per subject, not per source file — the reasoning crosses file boundaries far more than the code does. A note names the source it applies to; a source file does not generally point back, except where a tripwire comment says "see the notes on X". +One file per subject, not per source file -- the reasoning crosses file boundaries far more than the code does. A note names the source it applies to; a source file does not generally point back, except where a tripwire comment says "see the notes on X". | File | Subject | |---|---| +| `sequencer-suffix.md` | why `_S` must stay in `SampleID` and must not reach `RG_Sample`: the two unconnected messages a short name produces, the silent pool split a blank `RG_Sample` produces, and why stripping the suffix would lose a lane | | `metadata-file.md` | the sample metadata CSV, its prefixes, pooling and pool sizes | | `variant-model.md` | how work is shared between runs; the divergence tree | | `parameter-resolution.md` | what is computed vs set, and how a run's parameters are built | | `step-0-verification.md` | the checks, how often each runs, and how the report is assembled | | `change-guards.md` | what stops a project changing configuration mid-analysis | -| `config-migration.md` | carrying a configuration forward a release: the 2.2.0 → 3.0.0 pass | +| `config-migration.md` | carrying a configuration forward a release: the 2.2.0 -> 3.0.0 pass | | `four-roots.md` | installation / launch dir / mainDir / storageDir | | `dictionaries-and-snpeff.md` | step 1, and the three bugs its shape is the fix for | | `promotion.md` | moving artifacts from the working volume to permanent storage | @@ -77,6 +78,18 @@ One file per subject, not per source file — the reasoning crosses file boundar | `host-glibc-floor.md` | v3.1.1's analysis environment installed only on the machine that froze it: virtual packages, the four guards, and why a solve cannot answer it | | `someone-elses-machine.md` | four defects a green suite could not see, because the suite runs where the assumption holds; and the second machine that found three of them in an hour | | `macos-support.md` | the sizing, after the manual was found claiming a platform the pinned environments cannot solve on: three blockers, and why exporting is what makes it single-platform | +| `data-folder-walk.md` | step 0 prunes hidden folders and the read channel only filters them: what 3.2.0's CHANGELOG claimed, the symlink divergence beside it, and why a remote out-of-memory failure was not this | +| `dropzerodepth-coverage.md` | the three separate mechanisms that had to be asserted, why `analysisParams()` being an exclusion list made one case prove less than its comment claimed, and the measurement showing the fixture cannot hold a zero-depth cell | +| `logging.md` | every step lost its log on every failure path, and step 0 alone had the fix: the four designs measured against five failure modes, why writing as it happens beats a trap, and the two hypotheses that were wrong | +| `parameter-rules.md` | the build: one helper, two callers, and every threshold measured. Which rules are in and why `cutadapt.min_length` cannot be, the level corrected by measuring, and why no multi-word phrase is safe to assert against folded output | +| `parameter-validation.md` | PLANNED for before the next release cycle: step 0 validates no parameters at all, the catalogue of settings that produce nothing or silently change what a number means, and why the rules must live in one shell helper that both step 0 and `check_project.sh` call | +| `migration-hole.md` | `dropZeroDepth` arriving needed a `migrate_config` note and nothing else: the shape every later addition copies and why stating the default beats narrating the change, the false conclusion a template key count leads to about the `storageDir` marker, and the two wrong files it was built in first | +| `manual-pass.md` | the four shapes a release's read of the manual keeps finding, moved out of `RELEASING.md` because they are editorial rules an agent needs and Z does not, and what `check-manual-parameters.sh` proves against what the requirement actually is | +| `empty-results.md` | a run that found nothing used to publish a table of column names and report success: the seven stages that all exit 0 over no records, the three refusals and the placeholder trap they avoid, and the measurement showing the destructive `-C` band is set by `varQualMin` rather than being a fixed range | +| `unmeasured-cells.md` | what a pool with no reads at a site publishes, once `vcffilter.dropZeroDepth` made such a cell reachable: the measured shape bcftools emits, why a published `0` was two answers in one character, and the one `basicstats` aggregate that is left open | +| `mapq-downgrade.md` | `scaleMapQ` measured whole-genome on real pools: three regimes (dead at 10 and below, total data loss from 11 to about 20, inverted above 30), why 100 dominates the shipped 50, and the frequency compression that scales with the frequency | +| `cap-bam-cost.md` | SHELVED, measured: capping is the pipeline's second most expensive step and runs on one core of four. Where the time goes, why chromosomes are already independent, and why mpileup cannot take the cap instead. Its diagnosis is SUPERSEDED by `cap-depth-rewrite.md`; its timings are the record of what the step used to cost | +| `cap-depth-rewrite.md` | capping made about seven times faster on Z's own BAMs, byte for byte: why the sorted stream makes all but one position of the check redundant, the difference array, the awk sweep that created the elements it deleted, and the two cases that passed for the wrong reason | | `brainstorming.md` | ideas for later releases: what each would buy, what it would break, and where it sits | | `module-queue.md` | the plan from v3.1.1: thirteen modules at one a week, published without a release, and the two shape questions the roster raised | @@ -86,12 +99,12 @@ Each note says at the top what it was written against. Two changes cut across ma **Two scope renames happened after most of these were written**, in `cc00833`: the option scopes `samtools` and `bcftools` are now `cleanBAM` and `variantCall`. Any note naming `params.samtools.mapq` or `params.bcftools.maxDepth` is describing pre-3.0 code. -**The comment campaign is finished and committed as `7ad02c1`, except `test/`**, whose comments — some 2,500 lines of them across the suite — were never swept. There the filter is different and `CLAUDE.md` says so: a comment recording the bug a case guards *is* its function, and deleting it is how the case later looks arbitrary and gets removed. +**The comment campaign is finished and committed as `7ad02c1`, except `test/`**, whose comments -- some 2,500 lines of them across the suite -- were never swept. There the filter is different and `CLAUDE.md` says so: a comment recording the bug a case guards *is* its function, and deleting it is how the case later looks arbitrary and gets removed. ## Running a comment pass -**Verification, when a pass runs:** `nextflow lint .` at zero errors *and* zero warnings, the suite, and per file a diff of non-comment lines against `HEAD`. For a Python file the line count misleads — a module docstring is not a `#` comment — so prove it with an AST comparison that strips docstrings. **Run `bash -n` over every shell file too**: an apostrophe written into a comment inside an embedded awk program closes the surrounding single-quoted string and stops the file parsing. +**Verification, when a pass runs:** `nextflow lint .` at zero errors *and* zero warnings, the suite, and per file a diff of non-comment lines against `HEAD`. For a Python file the line count misleads -- a module docstring is not a `#` comment -- so prove it with an AST comparison that strips docstrings. **Run `bash -n` over every shell file too**: an apostrophe written into a comment inside an embedded awk program closes the surrounding single-quoted string and stops the file parsing. -**The templates are not source.** `parameters.config.template`, `metadata.csv.template` and `multi-run.csv.example` are shipped to the user and read while editing, so their comments are the only help available at the point of use. Z ruled on them directly (2026-08-30): *"Too many comments make them unreadable… These are all manual material. The guides should be minimal here."* A template's comments say what a setting **is** and what format it takes; everything explanatory goes to the manual. **What must not be cut is not commentary at all:** the commented-out parameter assignments are the knobs. There are 14 in `parameters.config.template`, and `grep -vE '^\s*//'` hides them from an ordinary settings diff, so count them separately. +**The templates are not source.** `parameters.config.template`, `metadata.csv.template` and `multi-run.csv.example` are shipped to the user and read while editing, so their comments are the only help available at the point of use. Z ruled on them directly (2026-08-30): *"Too many comments make them unreadable... These are all manual material. The guides should be minimal here."* A template's comments say what a setting **is** and what format it takes; everything explanatory goes to the manual. **What must not be cut is not commentary at all:** the commented-out parameter assignments are the knobs. There are 14 in `parameters.config.template`, and `grep -vE '^\s*//'` hides them from an ordinary settings diff, so count them separately. -**Tooling:** `dev/scripts/comment-audit.sh` lists tell-word clauses and comment blocks of 4+ lines. It always exits 0 and is deliberately not a test case — a hard failure would train the next pass to reword around the words instead of deleting the decision behind them. +**Tooling:** `dev/scripts/comment-audit.sh` lists tell-word clauses and comment blocks of 4+ lines. It always exits 0 and is deliberately not a test case -- a hard failure would train the next pass to reword around the words instead of deleting the decision behind them. diff --git a/.claude/development-notes/analysis-versioning.md b/.claude/development-notes/analysis-versioning.md index 1bf129c..843407a 100644 --- a/.claude/development-notes/analysis-versioning.md +++ b/.claude/development-notes/analysis-versioning.md @@ -19,7 +19,7 @@ last_change_day() { } ``` -Work sits uncommitted in this project for as long as it is under review — that is the whole point of the working tree being the review surface. So a frame change made on the 7th and bumped on the 7th went BEHIND again at midnight on the 8th, and again on the 9th, each time asking for a fresh stamp from a frame nobody had touched since. A new version for a new day, which is exactly what the rule forbids. +Work sits uncommitted in this project for as long as it is under review -- that is the whole point of the working tree being the review surface. So a frame change made on the 7th and bumped on the 7th went BEHIND again at midnight on the 8th, and again on the 9th, each time asking for a fresh stamp from a frame nobody had touched since. A new version for a new day, which is exactly what the rule forbids. It now takes the newest mtime among the changed paths, falling back to today only when the change removed files and left no mtime to read. @@ -34,7 +34,7 @@ The obvious repair is to drop dates from the frame check entirely and ask what t | moved it | `306dd26`, `c6544d7`, `2915346`, `a06c312` | | did not | `381541e`, `fabdb5c`, `4cb4ae7`, `e76774a` | -Each of the four that did not landed on a day an earlier commit had already bumped — which is the property the day comparison exists to give: **one bump covers a day's frame changes.** `e76774a` is the clearest case for keeping it: a comment-only pass over `analysis/lib/`, which changes nothing a derivation produces and should not need a version of its own. +Each of the four that did not landed on a day an earlier commit had already bumped -- which is the property the day comparison exists to give: **one bump covers a day's frame changes.** `e76774a` is the clearest case for keeping it: a comment-only pass over `analysis/lib/`, which changes nothing a derivation produces and should not need a version of its own. So the day comparison stays for committed history, and only the dirty answer changed. @@ -42,7 +42,7 @@ So the day comparison stays for committed history, and only the dirty answer cha **A change confined to `analysis/modules//test/` does not move that module's manifest version.** Found the same day, by the check reporting `basicstats` as behind after two of its cases were edited. -The reasoning is the rule above applied where it also holds. `analysis/modules/*/test/` carries `export-ignore`, so nothing in it reaches a published module tarball — a case cannot change what a user installs or what the module computes. The manifest version is what `modules install` sorts on and what every published result records the module *by*, so moving it for a test fix says the module changed when it did not. +The reasoning is the rule above applied where it also holds. `analysis/modules/*/test/` carries `export-ignore`, so nothing in it reaches a published module tarball -- a case cannot change what a user installs or what the module computes. The manifest version is what `modules install` sorts on and what every published result records the module *by*, so moving it for a test fix says the module changed when it did not. `dirty "$dir" ":(exclude)${dir}test"` is the whole change. A change to `main.nf`, the module's R, its manifest, its `references.bib` or its `citations.json` still requires the bump. @@ -50,7 +50,7 @@ The reasoning is the rule above applied where it also holds. `analysis/modules/* ## What guards it -`test/suites/00_static.sh`, `test_the_frame_version_moves_with_a_change_and_not_with_the_calendar`. It builds a repository of its own under `TEST_TMPDIR` — a frame, a catalogue and a module with a case of its own — and runs the script against that, because the script takes its root from its own location and the answer depends on whether the tree it reads is dirty. +`test/suites/00_static.sh`, `test_the_frame_version_moves_with_a_change_and_not_with_the_calendar`. It builds a repository of its own under `TEST_TMPDIR` -- a frame, a catalogue and a module with a case of its own -- and runs the script against that, because the script takes its root from its own location and the answer depends on whether the tree it reads is dirty. Two assertions carry it. A frame file whose mtime is in January, a version that says January, and a check that stays quiet although today is September. And a module whose `test/` changed staying quiet while the same module's `main.nf` changing does not. diff --git a/.claude/development-notes/analysis-wrapper.md b/.claude/development-notes/analysis-wrapper.md index d868840..04af634 100644 --- a/.claude/development-notes/analysis-wrapper.md +++ b/.claude/development-notes/analysis-wrapper.md @@ -2,45 +2,45 @@ **Written 2026-08-30 and revised 2026-08-31, against the tree at `7d65893`.** The module store landed that same evening in `d9886c5` and `complete` on 2026-09-01, so both sections below headed PLANNED are built, and the module dispatch has changed shape since. -It was the first landing piece of E4b, when the analysis layer had a wrapper of its own. **It does not any more** — Z merged it into `./PoolSeqFlow` on 2026-08-31, so everything here is `PoolSeqFlow analysis `. `wrapper.md` covers the wrapper as a whole; this covers what is particular to the analysis half. +It was the first landing piece of E4b, when the analysis layer had a wrapper of its own. **It does not any more** -- Z merged it into `./PoolSeqFlow` on 2026-08-31, so everything here is `PoolSeqFlow analysis `. `wrapper.md` covers the wrapper as a whole; this covers what is particular to the analysis half. -## ONE WRAPPER — Z, 2026-08-31 +## ONE WRAPPER -- Z, 2026-08-31 -Z: *"we need to get rid of separate wrappers for the tool. It adds unnecessary complexity and redundancy while a lot of things are already shared. The original wrapper already copies the analysis wrapper in the folder `.local/bin` automatically. This means, the analysis wrapper is part of the installation of main workflow any way. So there is no real reason to maintain them separately… A single wrapper has less surface to cover."* +Z: *"we need to get rid of separate wrappers for the tool. It adds unnecessary complexity and redundancy while a lot of things are already shared. The original wrapper already copies the analysis wrapper in the folder `.local/bin` automatically. This means, the analysis wrapper is part of the installation of main workflow any way. So there is no real reason to maintain them separately... A single wrapper has less surface to cover."* -`PoolSeqFlow-analysis` is deleted. `analysis` is the one subcommand that carries a word of its own; every other subcommand still takes none. **The one-argument contract is amended**, which the separate wrapper had existed to avoid — that trade was reconsidered once the shared surface was visible, and the amendment is cheaper than the second executable. +`PoolSeqFlow-analysis` is deleted. `analysis` is the one subcommand that carries a word of its own; every other subcommand still takes none. **The one-argument contract is amended**, which the separate wrapper had existed to avoid -- that trade was reconsidered once the shared surface was visible, and the amendment is cheaper than the second executable. -It widened again within hours, and again since: a module run is `analysis [nocpp]`, and the store is `analysis modules install []`. What survived is the shape — the top-level subcommands take nothing, and everything variable hangs off `analysis`. `usage_analysis` in the wrapper is the current statement of it, and `four-roots.md` records the same widening from the other side. +It widened again within hours, and again since: a module run is `analysis [nocpp]`, and the store is `analysis modules install []`. What survived is the shape -- the top-level subcommands take nothing, and everything variable hangs off `analysis`. `usage_analysis` in the wrapper is the current statement of it, and `four-roots.md` records the same widening from the other side. What the merge removed, and this is the measure of what a second wrapper cost: an entry in `PAYLOAD_ITEMS`, a second entry in `WRAPPERS` and its two symlinks, a second `VERSION=`/`# Version:` pair for `bump-version.sh` to rewrite and for `release.yml` and `00_static` to police, a duplicate `require_install` / `require_project_config` / `require_env`, and its own installation-root resolution and `wrapper_lib.sh` sourcing. Two sections of this note went with them: the argument about which wrapper owns the symlinks, and the dangling-symlink case that only existed because two executables could be provided by different versions. -**The trap the merge created, and closed.** `install/check_analysis_install.sh` read `ENV_NAME` — and the merged wrapper already exports `ENV_NAME` as the *pipeline's* environment, so `analysis check` would have verified the wrong one and reported success over a missing R. It reads `ANALYSIS_ENV_NAME` now, and the comment in it says why the two are apart. Anything else that shares this wrapper's environment needs the same care. +**The trap the merge created, and closed.** `install/check_analysis_install.sh` read `ENV_NAME` -- and the merged wrapper already exports `ENV_NAME` as the *pipeline's* environment, so `analysis check` would have verified the wrong one and reported success over a missing R. It reads `ANALYSIS_ENV_NAME` now, and the comment in it says why the two are apart. Anything else that shares this wrapper's environment needs the same care. -## What `uninstall` offers — Z's contract, 2026-08-30 +## What `uninstall` offers -- Z's contract, 2026-08-30 -Z: *"PoolSeqFlow uninstall should show PoolSeqFlow (the legacy one), and the versioned ones. If a versioned one also has Analysis installed, PoolSeqFlow choosing that one … should remove both the main and the analysis together. [analysis] uninstall should only uninstall the analysis layer."* +Z: *"PoolSeqFlow uninstall should show PoolSeqFlow (the legacy one), and the versioned ones. If a versioned one also has Analysis installed, PoolSeqFlow choosing that one ... should remove both the main and the analysis together. [analysis] uninstall should only uninstall the analysis layer."* -This forced a change the `-analysis$` filter alone did not make: **the chooser deals in installation NAMES, not version strings.** `installed_versions()` returned bare versions (`2.2.0`), which cannot represent the legacy environment at all — it has no version, so stripping `PoolSeqFlow-` off it produced nothing and it was silently absent from the list. `installed_installations()` returns `PoolSeqFlow`, `PoolSeqFlow-2.2.0`, and the `uninstall` arm derives the version from the name, leaving it empty for the legacy entry, which has no payload. +This forced a change the `-analysis$` filter alone did not make: **the chooser deals in installation NAMES, not version strings.** `installed_versions()` returned bare versions (`2.2.0`), which cannot represent the legacy environment at all -- it has no version, so stripping `PoolSeqFlow-` off it produced nothing and it was silently absent from the list. `installed_installations()` returns `PoolSeqFlow`, `PoolSeqFlow-2.2.0`, and the `uninstall` arm derives the version from the name, leaving it empty for the legacy entry, which has no payload. -Two supporting functions rather than inline printing: `installation_note()` decides what is said about an entry, and `list_installations()` renders the list. The prompt and the non-interactive refusal both call the latter, so they cannot drift into describing the same installations differently — which they previously did. +Two supporting functions rather than inline printing: `installation_note()` decides what is said about an entry, and `list_installations()` renders the list. The prompt and the non-interactive refusal both call the latter, so they cannot drift into describing the same installations differently -- which they previously did. -**Write `installation_note()` with `if`, not `&&`.** `[ x = y ] && notes=…` as a whole statement returns 1 when the test fails, and under `set -e` that exits the wrapper. The project has been bitten by this before — `poolseqflow_envs()` carries a `|| true` for the same reason. +**Write `installation_note()` with `if`, not `&&`.** `[ x = y ] && notes=...` as a whole statement returns 1 when the test fails, and under `set -e` that exits the wrapper. The project has been bitten by this before -- `poolseqflow_envs()` carries a `|| true` for the same reason. ## `uninstall` always confirms, and conda is never asked (Z, 2026-08-30) Z: *"When someone invokes uninstall it should always ask for confirmation to uninstall. Also for `conda remove -n ` can we pass `-y`, because someone saying no to that breaks the uninstall."* -**Always** is the whole point. Three paths previously reached removal with no question at all: the single-installation fast path, the versioned wrapper (`PoolSeqFlow-2.2.0 uninstall`), and the chooser — where *picking which* was silently treated as agreeing to the removal. The confirmation sits after the HAVE_ENV / HAVE_ANALYSIS_ENV / HAVE_PAYLOAD checks, so it can list exactly what exists and will go, and it is skipped only when none of the three is present. +**Always** is the whole point. Three paths previously reached removal with no question at all: the single-installation fast path, the versioned wrapper (`PoolSeqFlow-2.2.0 uninstall`), and the chooser -- where *picking which* was silently treated as agreeing to the removal. The confirmation sits after the HAVE_ENV / HAVE_ANALYSIS_ENV / HAVE_PAYLOAD checks, so it can list exactly what exists and will go, and it is skipped only when none of the three is present. -The wording follows `uninstall_all`'s, including `if ! read -r CONFIRM` → abort. **This makes `uninstall` unattendable**, exactly as `uninstall_all` already was. A CI job that ran `PoolSeqFlow-2.2.0 uninstall` will refuse rather than proceed unasked; that is the intended trade and it is stated in the manual. +The wording follows `uninstall_all`'s, including `if ! read -r CONFIRM` -> abort. **This makes `uninstall` unattendable**, exactly as `uninstall_all` already was. A CI job that ran `PoolSeqFlow-2.2.0 uninstall` will refuse rather than proceed unasked; that is the intended trade and it is stated in the manual. The `-y` half is a real bug, not tidiness: `conda env remove` prompts, and a user answering no left the wrapper carrying on to delete the payload and report success over an environment that was still there. A static case now greps for a `conda env remove` without it, since the next call site added will not have anyone remembering this. -## An analysis environment is not a version — a real bug, found 2026-08-30 +## An analysis environment is not a version -- a real bug, found 2026-08-30 `installed_versions()` built its list by stripping `PoolSeqFlow-` off environment names, so `PoolSeqFlow-2.2.0-analysis` parsed as a version called **`2.2.0-analysis`**. With one release installed plus its analysis environment, `choose_version_to_uninstall` counted two, skipped the single-version fast path, and: -- non-interactively, `PoolSeqFlow uninstall` **refused outright** — "2 versions of PoolSeqFlow are installed" — where before it just worked; +- non-interactively, `PoolSeqFlow uninstall` **refused outright** -- "2 versions of PoolSeqFlow are installed" -- where before it just worked; - interactively, it prompted and offered `2.2.0-analysis` as something to remove. Reproduced against the launcher harness before fixing, not reasoned about. The list now filters `-analysis$`. @@ -51,13 +51,13 @@ Reproduced against the launcher harness before fixing, not reasoned about. The l `choose_installation_to_uninstall` takes a different branch when `[ ! -t 0 ]`, so a herestring reaches the *refusal*, never the prompt. Every existing case tested the refusal. Covering the selection needs a pty. -**`script(1)` is the obvious way and it silently truncates the suite.** With `script -qec` driving one case, `run_tests.sh` stopped after that case: 18 of 49 reported, no failure, no stderr, exit 0, and the summary printed normally — so it read as a passing 18-case suite. The runner feeds its loop `while read -r fn; do … done < <(comm …)`, and `script` reads its caller's stdin as well as its own redirect, draining the remaining test names. A heredoc on `script` does not prevent it. Isolated reproductions did *not* show this, which is why it took bisecting the suite to find. +**`script(1)` is the obvious way and it silently truncates the suite.** With `script -qec` driving one case, `run_tests.sh` stopped after that case: 18 of 49 reported, no failure, no stderr, exit 0, and the summary printed normally -- so it read as a passing 18-case suite. The runner feeds its loop `while read -r fn; do ... done < <(comm ...)`, and `script` reads its caller's stdin as well as its own redirect, draining the remaining test names. A heredoc on `script` does not prevent it. Isolated reproductions did *not* show this, which is why it took bisecting the suite to find. `test/tools/on_a_tty.py` replaces it: `pty.fork`, write the answer, drain the master, exit with the child's status, and **never read fd 0**. Under 60 lines and the failure mode cannot recur. The lesson generalises: **a truncated suite that reports a smaller number of passes looks exactly like a passing suite.** Nothing in the harness notices that fewer cases ran than exist. A guard comparing the discovered count against the reported count would have turned this from a bisect into a message. -## Bare words, not flags — REVERSED 2026-08-30 by Z +## Bare words, not flags -- REVERSED 2026-08-30 by Z It shipped for about an hour as `--install` / `--uninstall` / `--version`. **Z reversed it the same day**: *"we keep the way we use the wrappers consistent. We either add `--` to PoolSeqFlow or lose it from analysis, I say lose it from analysis."* @@ -71,25 +71,25 @@ The module name is passed straight through as `nextflow run analysis.nf --module **`analysis.nf`'s roster is the single authority on what modules exist**, mirroring the main pipeline's step 0. A second copy of the list in the wrapper would drift, and drift silently: the wrapper's copy would refuse a module that works, or accept one that does not exist and let Nextflow produce the error instead. -The price is that a typo starts a JVM before being refused — about 21 seconds. Accepted; the alternative is the drift. +The price is that a typo starts a JVM before being refused -- about 21 seconds. Accepted; the alternative is the drift. -**The dispatch is two Nextflow runs now, and the roster has moved.** A module run is `analysis.nf --module `, which checks the project and clears the results folder, and then `//main.nf`, which produces the results — a module is a pipeline of its own, so it names itself and `--module` is not passed on. A module the frame provides has no directory and the first run is all of it. The roster itself lives in `analysis/lib/nf/modules.nf`, which a module reads too. The claim above survives all of it: the wrapper still keeps no list, and still validates only shape. +**The dispatch is two Nextflow runs now, and the roster has moved.** A module run is `analysis.nf --module `, which checks the project and clears the results folder, and then `//main.nf`, which produces the results -- a module is a pipeline of its own, so it names itself and `--module` is not passed on. A module the frame provides has no directory and the first run is all of it. The roster itself lives in `analysis/lib/nf/modules.nf`, which a module reads too. The claim above survives all of it: the wrapper still keeps no list, and still validates only shape. -**`complete` — planned when this was written, built 2026-09-01 in `f4508f5`.** Moving `Analysis/` to permanent storage sat in the plan file under E4b. It was advertised in the usage line until 2026-08-31 with no arm behind it, so it fell through to the module dispatcher and came back as *"'complete' is not installed"* — and the test that exists to catch exactly that had it excluded by name. It came out of the usage line and out of the exclusion until it was real. The design call held: it runs as `analysis/complete.nf` rather than shell, because promotion in the main pipeline is `PromoteArtifacts` in `9_completion.nf`, a Nextflow process, and the analysis layer's promotion is the same problem, which should not be solved a second way. +**`complete` -- planned when this was written, built 2026-09-01 in `f4508f5`.** Moving `Analysis/` to permanent storage sat in the plan file under E4b. It was advertised in the usage line until 2026-08-31 with no arm behind it, so it fell through to the module dispatcher and came back as *"'complete' is not installed"* -- and the test that exists to catch exactly that had it excluded by name. It came out of the usage line and out of the exclusion until it was real. The design call held: it runs as `analysis/complete.nf` rather than shell, because promotion in the main pipeline is `PromoteArtifacts` in `9_completion.nf`, a Nextflow process, and the analysis layer's promotion is the same problem, which should not be solved a second way. -**`modules list|available|install|uninstall` — planned when this was written, built the same evening in `d9886c5`.** Until they existed a refusal could not tell a user to run them, so the module refusal named the store directory instead. +**`modules list|available|install|uninstall` -- planned when this was written, built the same evening in `d9886c5`.** Until they existed a refusal could not tell a user to run them, so the module refusal named the store directory instead. ## Check order in the module arm -`require_analysis_install` → `require_project_config` → `require_analysis_env`, copied from the main wrapper's `run`. +`require_analysis_install` -> `require_project_config` -> `require_analysis_env`, copied from the main wrapper's `run`. -Standing outside a project is reported before a missing environment. Both orders are defensible — the environment is missing wherever you stand — but matching the pipeline arm matters more than the marginal improvement, and a user in the wrong directory is the commoner case. +Standing outside a project is reported before a missing environment. Both orders are defensible -- the environment is missing wherever you stand -- but matching the pipeline arm matters more than the marginal improvement, and a user in the wrong directory is the commoner case. **This bit the tests immediately.** `run_analysis_launcher_with_envs` originally built a sandbox with no `parameters.config`, and `test_a_module_refuses_to_borrow_the_pipeline_environment` never reached the environment check at all. The harness now makes its sandbox a project, and the outside-a-project case builds its own bare directory. -## `install/environment-analysis.yml` — IT SOLVES. Risk retired 2026-08-30 +## `install/environment-analysis.yml` -- IT SOLVES. Risk retired 2026-08-30 -The stage's flagged release risk is **resolved**. Z installed it the same day, having loosened `python`, `bash`, `coreutils` and `r-base` to bare names first — pin what you must, let the solver do the rest, then export what it chose. Measured in the built environment: +The stage's flagged release risk is **resolved**. Z installed it the same day, having loosened `python`, `bash`, `coreutils` and `r-base` to bare names first -- pin what you must, let the solver do the rest, then export what it chose. Measured in the built environment: | | | |---|---| @@ -98,14 +98,14 @@ The stage's flagged release risk is **resolved**. Z installed it the same day, h | optparse / jsonlite / Matrix | 1.8.2 / 2.0.0 / 1.7-6 | | nextflow / samtools / bcftools / htslib | 26.04.6 / 1.24 / 1.24 / 1.24 | -`openjdk` arrives as a nextflow dependency, so the analysis environment can run Nextflow on its own. **A false alarm worth recording:** calling `$ENV/bin/nextflow` by absolute path with an ambient PATH picks up the wrong JVM and reports "please make sure that Java 17 or later is installed". With the environment's own PATH it is fine. Anything checking this environment must run with it *active*, never by absolute path — which is what `check` does. +`openjdk` arrives as a nextflow dependency, so the analysis environment can run Nextflow on its own. **A false alarm worth recording:** calling `$ENV/bin/nextflow` by absolute path with an ambient PATH picks up the wrong JVM and reports "please make sure that Java 17 or later is installed". With the environment's own PATH it is fine. Anything checking this environment must run with it *active*, never by absolute path -- which is what `check` does. **Still to do before v3.0.0:** 1. `dev/scripts/export-environment.sh PoolSeqFlow--analysis` to replace the hand-written spec with an exact-build export. The script routes an `-analysis` name to the right file and writes the pinned header over the "NOT YET PINNED" one. Not done yet because the versions move again at the release bump anyway. 2. **`dev/scripts/prep-version.sh` still only handles the pipeline environment.** It clones `PoolSeqFlow-` into a scratch env, updates, runs the suite, exports. The analysis env needs the same or a release ships a stale pin. -The package list was a first guess at what the E4c roster needs. It has been revised since: F1 added `r-rcpp` for the compiled path a module may offer, `r-dofuture` for binned per-site work, and `r-rmarkdown`, `r-knitr`, `r-png` and an explicitly named `pandoc` for the PDF report every published analysis carries. Base R still covers more than expected — `cmdscale` (MDS), `prcomp` (PCA), `mantelhaen.test` (CMH), `p.adjust` (FDR) and weighted `glm` (phenotypic association) are all in `stats`. What is actually pulled in is `data.table` for the table sizes, `ggplot2`/`pheatmap` for figures, `optparse` for the module CLIs and `jsonlite` for the provenance sidecars. +The package list was a first guess at what the E4c roster needs. It has been revised since: F1 added `r-rcpp` for the compiled path a module may offer, `r-dofuture` for binned per-site work, and `r-rmarkdown`, `r-knitr`, `r-png` and an explicitly named `pandoc` for the PDF report every published analysis carries. Base R still covers more than expected -- `cmdscale` (MDS), `prcomp` (PCA), `mantelhaen.test` (CMH), `p.adjust` (FDR) and weighted `glm` (phenotypic association) are all in `stats`. What is actually pulled in is `data.table` for the table sizes, `ggplot2`/`pheatmap` for figures, `optparse` for the module CLIs and `jsonlite` for the provenance sidecars. `samtools`/`bcftools`/`htslib` are duplicated from the pipeline environment on purpose: the analysis layer derives per-position depth from BAMs in `Analysis/Main`, and an environment that has to borrow the pipeline's is not opt-in. @@ -123,7 +123,7 @@ sed -n 's/^ *- *r-\([^=]*\).*$/\1/p' "$ENV_FILE" | grep -vx base The `[^=]*` matters: the file is unpinned today (`r-data.table`) and will be an exact-build export tomorrow (`r-data.table=1.18.4=r44h...`). Both parse. -**Do not hand-write a second list.** The first design here was an explicit `R_PACKAGES` list in the checker plus a `00_static` case asserting it matched the yml — a guard against drift that could instead be made impossible. Deriving it removes the guard and the drift together. +**Do not hand-write a second list.** The first design here was an explicit `R_PACKAGES` list in the checker plus a `00_static` case asserting it matched the yml -- a guard against drift that could instead be made impossible. Deriving it removes the guard and the drift together. **conda's name is not R's name.** `r-matrix` provides `Matrix`, capital M, and `library(matrix)` fails. Rather than keep a mapping table, both `check` and `cite` recover the real name case-insensitively from `rownames(installed.packages())`. Verified against the live environment: all six resolve, `Matrix` included. @@ -135,46 +135,46 @@ It **degrades rather than refuses** when the environment is absent: the software ### The citation text lives in `lib/wrapper_lib.sh` -`poolseqflow_citation()` plus `CONCEPT_DOI`. Both `cite` arms call it, so the DOI and the "cite the version you actually ran" argument exist once. Verified by diffing `./PoolSeqFlow cite` against the **deployed pre-refactor 2.2.0 copy** — byte-identical, which is a real before/after rather than a self-comparison. +`poolseqflow_citation()` plus `CONCEPT_DOI`. Both `cite` arms call it, so the DOI and the "cite the version you actually ran" argument exist once. Verified by diffing `./PoolSeqFlow cite` against the **deployed pre-refactor 2.2.0 copy** -- byte-identical, which is a real before/after rather than a self-comparison. The DOI still exists twice: here and in `install/citations.json`, which the per-run `CITATIONS.md` is built from. Collapsing those would mean parsing JSON in the wrapper without a guaranteed `jq`. Left as two, noted here. ## What was NOT done, and why -- **`install/check_install.sh` was left alone**, and `check_analysis_install.sh` is a separate script rather than a mode of it. The pipeline checker resolves its tool list through `nextflow config` from `params.software`; the analysis layer has no such block, and its interesting checks are R packages, which the pipeline knows nothing about. Sharing `lib/tool_version.sh` gets the useful commonality — identical version reporting — without forcing one script to serve two contracts. +- **`install/check_install.sh` was left alone**, and `check_analysis_install.sh` is a separate script rather than a mode of it. The pipeline checker resolves its tool list through `nextflow config` from `params.software`; the analysis layer has no such block, and its interesting checks are R packages, which the pipeline knows nothing about. Sharing `lib/tool_version.sh` gets the useful commonality -- identical version reporting -- without forcing one script to serve two contracts. - **`list` shows `PoolSeqFlow--analysis` as a flat entry.** Z ruled this fine (2026-08-30): it is cosmetic grouping, not a correctness bug. -- **`.gitattributes` gives `analysis/` nothing**, which is correct — it must NOT be `export-ignore`d, and it is not. Verified: `git archive HEAD` carries the directory. It has two entries since, both narrow and neither touching the directory itself: `analysis/modules/*/test/`, because a module's cases travel with the module, and `analysis/modules-index.tsv`, because a frozen copy of the catalogue inside a tarball would be a second answer to what can be installed. +- **`.gitattributes` gives `analysis/` nothing**, which is correct -- it must NOT be `export-ignore`d, and it is not. Verified: `git archive HEAD` carries the directory. It has two entries since, both narrow and neither touching the directory itself: `analysis/modules/*/test/`, because a module's cases travel with the module, and `analysis/modules-index.tsv`, because a frozen copy of the catalogue inside a tarball would be a second answer to what can be installed. -## AMENDMENT 2026-09-09 — item 2 of "Still to do before v3.0.0" is built +## AMENDMENT 2026-09-09 -- item 2 of "Still to do before v3.0.0" is built **`dev/scripts/prep-version.sh` now prepares both environments.** It clones `PoolSeqFlow-` and `PoolSeqFlow--analysis` into `PoolSeqFlow-update` and `PoolSeqFlow-update-analysis`, updates each, runs the full suite with `TEST_CONDA_ENV` and `TEST_ANALYSIS_ENV` pointed at both, and exports both only if it passes. `--from-analysis ` names a different source. Item 1 above is unchanged and is still done by the release itself: the first pinned `environment-analysis.yml` is what step 2 of `dev/RELEASING.md` writes. Three things settled while building it. -**The suite finds an analysis environment by globbing `PoolSeqFlow-*-analysis` and taking the first with an `Rscript`.** That is whichever name sorts first, not the one being prepared, so the scratch environment has to be named explicitly through `TEST_ANALYSIS_ENV` — a glob would have tested the release environment and exported the scratch one, silently. +**The suite finds an analysis environment by globbing `PoolSeqFlow-*-analysis` and taking the first with an `Rscript`.** That is whichever name sorts first, not the one being prepared, so the scratch environment has to be named explicitly through `TEST_ANALYSIS_ENV` -- a glob would have tested the release environment and exported the scratch one, silently. -**The refusal to export an environment carrying module packages had to be askable separately.** `export-environment.sh --check` runs that guard and stops, writing nothing. Without it the question was asked at step 4, after the clone, the two solves and the full suite — an hour on the wrong side of an answer the source environment already determined, because the clone inherits its packages. +**The refusal to export an environment carrying module packages had to be askable separately.** `export-environment.sh --check` runs that guard and stops, writing nothing. Without it the question was asked at step 4, after the clone, the two solves and the full suite -- an hour on the wrong side of an answer the source environment already determined, because the clone inherits its packages. **Both environments carry Nextflow and nothing was comparing them.** `install/environment-analysis.yml`'s own header says it carries the pipeline environment's version, but the two solve independently and no test asserts it. The script reports a divergence as a warning rather than refusing: it is the suite that says whether a pair works, and a hard refusal here would be a release policy nobody has set. **The `-analysis` suffix on the scratch name is load-bearing.** `export-environment.sh` decides which file an environment belongs in by matching `*-analysis` against its name, so `PoolSeqFlow-update-analysis` routes itself and needs no output argument. -Found in passing, and fixed: three fixtures in `test/suites/00_static.sh` wrote `modules-repo/index.tsv` into a directory their `mkdir -p` never created. The redirect failed, the catalogue was absent, and `check-analysis-versions.sh` skips its catalogue check entirely when the file is missing — so all three cases passed while covering nothing there. Fallout from the catalogue move in `e384928`, where the old path's parent (`analysis/`) came for free. Measured after the fix: with the file planted, a row changed without the header moving reports `BEHIND: the catalogue changed and its #!index-version did not`; without it, nothing. +Found in passing, and fixed: three fixtures in `test/suites/00_static.sh` wrote `modules-repo/index.tsv` into a directory their `mkdir -p` never created. The redirect failed, the catalogue was absent, and `check-analysis-versions.sh` skips its catalogue check entirely when the file is missing -- so all three cases passed while covering nothing there. Fallout from the catalogue move in `e384928`, where the old path's parent (`analysis/`) came for free. Measured after the fix: with the file planted, a row changed without the header moving reports `BEHIND: the catalogue changed and its #!index-version did not`; without it, nothing. -## AMENDMENT 2026-09-09 (second) — the unmigrated-config refusal, E7e +## AMENDMENT 2026-09-09 (second) -- the unmigrated-config refusal, E7e **`require_migrated_config` in `./PoolSeqFlow`.** Written against the tree at `ff08f61`. -**What the path actually did before it, measured rather than assumed.** A real v2.2.0 `parameters.config` resolves through `nextflow -C nextflow.config config` with **no error of any kind** — `includeConfig "${launchDir}/parameters.config"` reads the file as given, and only `cores`, `dryRun` and `dryRunDir` have a safety net in `nextflow.config`. Every parameter 3.0 added or renamed comes back empty: `storageDir`, `metadataFile`, `metadataPath`, `ploidy`, `multiRun`, `dir.allLogs`, `dir.allOutputs`. Step 0 then builds `dir_log = "${params.dir.allLogs}/0_verify_environment"`, which becomes `null/0_verify_environment`, so the run starts and writes into a directory named `null`. +**What the path actually did before it, measured rather than assumed.** A real v2.2.0 `parameters.config` resolves through `nextflow -C nextflow.config config` with **no error of any kind** -- `includeConfig "${launchDir}/parameters.config"` reads the file as given, and only `cores`, `dryRun` and `dryRunDir` have a safety net in `nextflow.config`. Every parameter 3.0 added or renamed comes back empty: `storageDir`, `metadataFile`, `metadataPath`, `ploidy`, `multiRun`, `dir.allLogs`, `dir.allOutputs`. Step 0 then builds `dir_log = "${params.dir.allLogs}/0_verify_environment"`, which becomes `null/0_verify_environment`, so the run starts and writes into a directory named `null`. -**The marker is `storageDir` being assigned, and that is the whole test.** The rename table lives in an awk function inside `bin/config_migrate.sh`; a second copy of it in the wrapper would drift from the first. One fact that cannot drift is better than a list that can: every config for this release assigns `storageDir` because the template does and `migrate_config` writes it, and no 2.x config does because that root was `projectDir`. The old names the message prints — `projectDir`, `diploidy`, `rgTagsFile`, `rgTagsPath` — are advisory only; the refusal does not turn on them. +**The marker is `storageDir` being assigned, and that is the whole test.** The rename table lives in an awk function inside `bin/config_migrate.sh`; a second copy of it in the wrapper would drift from the first. One fact that cannot drift is better than a list that can: every config for this release assigns `storageDir` because the template does and `migrate_config` writes it, and no 2.x config does because that root was `projectDir`. The old names the message prints -- `projectDir`, `diploidy`, `rgTagsFile`, `rgTagsPath` -- are advisory only; the refusal does not turn on them. **The cost of that choice, named so it is not rediscovered:** a release that renames `storageDir` must move the marker with it, or the guard silently passes everything. `02_launcher`'s case is what says so, and `dev/RELEASING.md` step 3 carries the warning. -**Which arms guard, and why it is not all of them.** `run|resume`, `dryrun`, `reset`, `analysis complete` and the module dispatch — what runs the pipeline, runs a module, or acts on either's outputs. Not `migrate_config`, which is the fix and would otherwise be unreachable. Not `clean` or `dryclean`, which read `workDir` and `params.dryRunDir`, and the latter has a default in `nextflow.config` regardless. +**Which arms guard, and why it is not all of them.** `run|resume`, `dryrun`, `reset`, `analysis complete` and the module dispatch -- what runs the pipeline, runs a module, or acts on either's outputs. Not `migrate_config`, which is the fix and would otherwise be unreachable. Not `clean` or `dryclean`, which read `workDir` and `params.dryRunDir`, and the latter has a default in `nextflow.config` regardless. -**It broke eight fixtures, and the fixtures were wrong.** `run_analysis_launcher_with_envs` wrote `// stub project marker` as a whole `parameters.config` — enough for `require_project_config`, which only tests existence. Those cases are about which environment is chosen, so the marker now carries a `params { storageDir }` block and looks like a current project. A stub that cannot be told from an unmigrated config was the actual defect. +**It broke eight fixtures, and the fixtures were wrong.** `run_analysis_launcher_with_envs` wrote `// stub project marker` as a whole `parameters.config` -- enough for `require_project_config`, which only tests existence. Those cases are about which environment is chosen, so the marker now carries a `params { storageDir }` block and looks like a current project. A stub that cannot be told from an unmigrated config was the actual defect. -**The manual had a stale paragraph and it was deleted, not corrected.** *"Skip it and the failure is not a clean one. An absent parameter interpolates as the literal string `null`, so a later step dies with `.command.sh: line 17: null: command not found`"* — true until this landed, and exactly the "warning about a bug that is fixed" class `dev/RELEASING.md` step 1 tells you to look for. +**The manual had a stale paragraph and it was deleted, not corrected.** *"Skip it and the failure is not a clean one. An absent parameter interpolates as the literal string `null`, so a later step dies with `.command.sh: line 17: null: command not found`"* -- true until this landed, and exactly the "warning about a bug that is fixed" class `dev/RELEASING.md` step 1 tells you to look for. -**Verified end to end** in a scratch project holding the real v2.2.0 template: `run` refused and named `migrate_config`; `migrate_config` was not refused and wrote a config with `storageDir`; `run` then got past the guard and failed on the template's own `/path/to/working/directory` placeholder, which is the honest error for a config nobody has filled in. Mutation-tested by removing the guard from the `run` arm — the case fails on the placement assertion as well as the message, so moving it later would not pass either. +**Verified end to end** in a scratch project holding the real v2.2.0 template: `run` refused and named `migrate_config`; `migrate_config` was not refused and wrote a config with `storageDir`; `run` then got past the guard and failed on the template's own `/path/to/working/directory` placeholder, which is the honest error for a config nobody has filled in. Mutation-tested by removing the guard from the `run` arm -- the case fails on the placement assertion as well as the message, so moving it later would not pass either. diff --git a/.claude/development-notes/association-math.md b/.claude/development-notes/association-math.md index 9e64c26..2ae6923 100644 --- a/.claude/development-notes/association-math.md +++ b/.claude/development-notes/association-math.md @@ -1,41 +1,41 @@ # F2's math, and the three things that were measured rather than reasoned -**Written 2026-09-07, against the tree at `15d4c34` plus the uncommitted config reshaping. Nothing in `analysis/modules/association/` exists yet** — this is the record of what was settled before any of it was written, so that the module is built against measurements rather than against intuition. +**Written 2026-09-07, against the tree at `15d4c34` plus the uncommitted config reshaping. Nothing in `analysis/modules/association/` exists yet** -- this is the record of what was settled before any of it was written, so that the module is built against measurements rather than against intuition. The plan file's F2 section predates F0f and F0g and is superseded on scales, covariates and degrees of freedom by what is below. ## Z's two calls on scales and covariates - **An ordinal phenotype is fitted linearly on the level index; a nominal one is refused by name.** F0f shipped four scales where the plan covered two. The manual has to say an ordinal slope is *per level-step* and not per unit of anything, because fitting a line asserts the equal spacing that "ordinal" explicitly disclaims. -- **Covariates are fitted as extra predictors**, not reported only. `df = n − 2 − q`. +- **Covariates are fitted as extra predictors**, not reported only. `df = n - 2 - q`. ## The permutation scheme, and the candidate that doubles the false-positive rate Permuting a phenotype freely stops being valid once covariates are in the model, and there are three plausible repairs. **Measured at six pools with one covariate, 20,000 replicates against the exhaustive 720-permutation null, nominal α = 0.05:** -| scheme | p ≤ .05 | p ≤ .10 | p ≤ .50 | mean p | +| scheme | p <= .05 | p <= .10 | p <= .50 | mean p | |---|---|---|---|---| | permute the RAW phenotype, re-residualise at each site | 0.0482 | 0.0992 | 0.5072 | 0.4984 | | **permute the RESIDUALISED phenotype** | **0.0986** | **0.1682** | **0.5794** | **0.4393** | -| Freedman–Lane | 0.0512 | 0.0986 | 0.4968 | 0.4997 | +| Freedman-Lane | 0.0512 | 0.0986 | 0.4968 | 0.4997 | -**The middle one is the natural implementation and it is wrong**, at roughly twice its nominal rate, with the whole p-distribution shifted — a mean p of 0.44 where it must be 0.5. +**The middle one is the natural implementation and it is wrong**, at roughly twice its nominal rate, with the whole p-distribution shifted -- a mean p of 0.44 where it must be 0.5. **The cause is geometric, not confounding.** The observed residualised phenotype lies in the same subspace as the residualised frequency; its permutations do not. So the observed correlation sits systematically high inside its own null. Two controls settle it: making the phenotype **independent** of the covariate gave the same 0.1002, which rules out confounding; and a **constant** covariate made all three schemes numerically identical, which rules out the harness. -**So: permute the raw phenotype and re-residualise it at each site.** It is the cheaper of the two valid schemes — re-residualising one permuted phenotype is a single *n*-vector per site, where Freedman–Lane residualises per **allele**. With no covariates it degenerates to plain label permutation, so there is one code path and not two. +**So: permute the raw phenotype and re-residualise it at each site.** It is the cheaper of the two valid schemes -- re-residualising one permuted phenotype is a single *n*-vector per site, where Freedman-Lane residualises per **allele**. With no covariates it degenerates to plain label permutation, so there is one code path and not two. -## Weighted Frisch–Waugh–Lovell is exact, so covariates keep the closed form +## Weighted Frisch-Waugh-Lovell is exact, so covariates keep the closed form -Verified against `lm(f ~ y + z, weights = w)`: residualising both response and predictor on the covariates and running the simple weighted fit reproduces **b1 and its standard error to 12 significant figures** — provided the degrees of freedom come from the *full* model and not from the residual regression. That is the whole reason covariates do not cost F2 its vectorized closed form. Per-site weights make the residualisation per site, but it stays one small solve per site, independent of allele count. +Verified against `lm(f ~ y + z, weights = w)`: residualising both response and predictor on the covariates and running the simple weighted fit reproduces **b1 and its standard error to 12 significant figures** -- provided the degrees of freedom come from the *full* model and not from the residual regression. That is the whole reason covariates do not cost F2 its vectorized closed form. Per-site weights make the residualisation per site, but it stays one small solve per site, independent of allele count. ## Degrees of freedom, and the refusal that falls out of them -`design.units`, per module rule 17c — never `design.series`, and never the column count. F0g made that answer correct for an untimed project too, so **the "pools are the units" fallback that was written down while `units` was empty is dead and must not come back.** +`design.units`, per module rule 17c -- never `design.series`, and never the column count. F0g made that answer correct for an untimed project too, so **the "pools are the units" fallback that was written down while `units` was empty is dead and must not come back.** -**And the degrees of freedom are not the whole of it: the FIT is on units too.** Collapse each unit's pools onto one weighted value first — weights summed, frequency their weighted mean — and fit those. Re-reading a pool-level statistic against the unit count controls nothing; `calibration.md`, written the same day against measurements this note predates, has the numbers. With covariates, `df = n_observed − 2 − q` counted over units after the roll-up. +**And the degrees of freedom are not the whole of it: the FIT is on units too.** Collapse each unit's pools onto one weighted value first -- weights summed, frequency their weighted mean -- and fit those. Re-reading a pool-level statistic against the unit count controls nothing; `calibration.md`, written the same day against measurements this note predates, has the numbers. With covariates, `df = n_observed - 2 - q` counted over units after the roll-up. -**A refusal falls straight out and has to be loud rather than silent:** a timed design of three units gives `df = 3 − 2 − q`, so **any covariate at all makes a repeated-measures design unfittable.** That is the design being honest about itself, not a defect — but it stops a run, so it must say so in one sentence instead of emitting a column of NA. +**A refusal falls straight out and has to be loud rather than silent:** a timed design of three units gives `df = 3 - 2 - q`, so **any covariate at all makes a repeated-measures design unfittable.** That is the design being honest about itself, not a defect -- but it stops a run, so it must say so in one sentence instead of emitting a column of NA. ## The weight belongs to the frequency, never to the label @@ -45,8 +45,8 @@ The plan says to *"permute the phenotype labels with their weights attached"*. * Committed with the corpus at `e3eb887`: -- **`assocb.smallest_p` = 0.1** — the 3-vs-3 floor, reached by a site whose *t* is literally infinite. The complementary labeling always ties, so 2/20 is the floor and no site in such a design can be genome-wide significant. The quantitative floor is 1/720 and `chr2:550` reaches it. +- **`assocb.smallest_p` = 0.1** -- the 3-vs-3 floor, reached by a site whose *t* is literally infinite. The complementary labeling always ties, so 2/20 is the floor and no site in such a design can be genome-wide significant. The quantitative floor is 1/720 and `chr2:550` reaches it. - **`chr2:550`** is triallelic with |t_REF| = 22.2 against 7.65 for the best alternate: minimising over alternates alone gives p = 0.0016 where maximising over every allele gives 2.4e-5. It is the case that would catch a dropped REF row. - **`chr1:700`**: weighted t = 5.35 against unweighted 4.74, so a fit that ignores `n_eff` fails it. - **`chr10:1600` and `chr10:1800`** give identical statistics from 740 alternate reads and from 3. -- **`zero_variance` exists because t and p are not reproducible there.** Two alleles of one perfectly separated site are algebraically one test, but one residual sum lands on exactly 0 and the other on 1e-18, so t is `Inf` against `8e15` — and raising one pool's depth from 20 to 60 moved which site got which. **A case must assert that the module trapped those rows, never that it reproduced their numbers.** +- **`zero_variance` exists because t and p are not reproducible there.** Two alleles of one perfectly separated site are algebraically one test, but one residual sum lands on exactly 0 and the other on 1e-18, so t is `Inf` against `8e15` -- and raising one pool's depth from 20 to 60 moved which site got which. **A case must assert that the module trapped those rows, never that it reproduced their numbers.** diff --git a/.claude/development-notes/bin-and-lib.md b/.claude/development-notes/bin-and-lib.md index 22def8c..26e2cac 100644 --- a/.claude/development-notes/bin-and-lib.md +++ b/.claude/development-notes/bin-and-lib.md @@ -14,18 +14,18 @@ Before it, two separate verifiers each carried a hand-kept exclusion list: SOURCED="tool_version.sh wrapper_lib.sh" ``` -in `install/check_install.sh` and again in `dev/scripts/verify-archive.sh`, so that a sourced library was not failed for missing an executable bit. Two copies of the same list, in different files, updated by hand whenever a sourced file was added — and one of them had already been wrong once before (`verify-archive.sh` asserted a hardcoded module count that was wrong by five, failing every PR until 2026-08-30). +in `install/check_install.sh` and again in `dev/scripts/verify-archive.sh`, so that a sourced library was not failed for missing an executable bit. Two copies of the same list, in different files, updated by hand whenever a sourced file was added -- and one of them had already been wrong once before (`verify-archive.sh` asserted a hardcoded module count that was wrong by five, failing every PR until 2026-08-30). With the split the rule needs no list at all. **Every file in `bin/` must be executable, full stop**, and `lib/` is checked for presence instead. The directory a file is in states its contract, so there is nothing to keep in step. ## What it touched -`params.dir.lib` is new in the template beside `dir.bin`, and is deliberately **not** on `PATH` — `nextflow.config` prepends `dir.bin` so process scripts can call helpers by bare name, and a sourced library is never called that way. `scripts/citations.nf` sources `${run.dir.lib}/tool_version.sh`. `PAYLOAD_ITEMS` gained `lib`, so it deploys and ships. +`params.dir.lib` is new in the template beside `dir.bin`, and is deliberately **not** on `PATH` -- `nextflow.config` prepends `dir.bin` so process scripts can call helpers by bare name, and a sourced library is never called that way. `scripts/citations.nf` sources `${run.dir.lib}/tool_version.sh`. `PAYLOAD_ITEMS` gained `lib`, so it deploys and ships. ## The trap it sprang, which is the part worth remembering -`test/lib/sandbox.sh`'s launcher stub fabricates its payload as **empty placeholder directories** — deliberately, because nothing in those cases runs Nextflow. That worked only while the wrapper needed no payload *contents*. The moment `PoolSeqFlow` began sourcing a file out of the payload, 18 of 22 launcher cases failed at once, on a message about an incomplete installation that had nothing to do with what any of them was testing. +`test/lib/sandbox.sh`'s launcher stub fabricates its payload as **empty placeholder directories** -- deliberately, because nothing in those cases runs Nextflow. That worked only while the wrapper needed no payload *contents*. The moment `PoolSeqFlow` began sourcing a file out of the payload, 18 of 22 launcher cases failed at once, on a message about an incomplete installation that had nothing to do with what any of them was testing. That is the same failure mode the comment above that loop already warned about, from when `multi-run.csv.example` was added to the payload. The placeholder strategy is fine; what it cannot survive is a payload file the wrapper *reads*. -**Anything added to `lib/` in future must be copied into every harness site that fabricates an installation**, or every launcher case fails together and says nothing useful about why. There were two when this was written and there are five now — `grep -rn wrapper_lib test/` finds them, which is the check to run rather than trusting a count here. +**Anything added to `lib/` in future must be copied into every harness site that fabricates an installation**, or every launcher case fails together and says nothing useful about why. There were two when this was written and there are five now -- `grep -rn wrapper_lib test/` finds them, which is the check to run rather than trusting a count here. diff --git a/.claude/development-notes/bin-helpers.md b/.claude/development-notes/bin-helpers.md index ab43ad5..720b4c9 100644 --- a/.claude/development-notes/bin-helpers.md +++ b/.claude/development-notes/bin-helpers.md @@ -2,43 +2,43 @@ **Written 2026-08-31, against the tree at `7d65893`.** The helpers themselves are unchanged; the citation chain below gained a BibTeX front end since, and the analysis layer a citations file of its own. -Small programs the pipeline shells out to. Several exist because logic in a Nextflow process costs a JVM start to test, while a script in here is unit-testable in milliseconds — that cost model is the single biggest influence on what lives here. +Small programs the pipeline shells out to. Several exist because logic in a Nextflow process costs a JVM start to test, while a script in here is unit-testable in milliseconds -- that cost model is the single biggest influence on what lives here. Not all of `bin/` is covered here. `cap_depth.awk` and `depth_cutoff.py` are in `depth-cutoff.md`, and `config_migrate.sh` in `config-migration.md`. **`MajorAlleleToRef.py` has no note anywhere**, which is a gap worth closing: it re-polarizes every site on cohort totals, which is why a distance or a frequency from a six-pool run is not the same quantity as one from a twelve-pool run. -## `atomic_mv.sh` — why staging exists +## `atomic_mv.sh` -- why staging exists -The work directory and `storageDir` are normally on **different filesystems**, so `mv` is a copy followed by an unlink rather than a `rename()`. A job killed mid-copy leaves a truncated file at the destination — and the pipeline's skip logic only asks whether a file **exists**, so the next run treats that fragment as a completed step. +The work directory and `storageDir` are normally on **different filesystems**, so `mv` is a copy followed by an unlink rather than a `rename()`. A job killed mid-copy leaves a truncated file at the destination -- and the pipeline's skip logic only asks whether a file **exists**, so the next run treats that fragment as a completed step. Staging through a `.part` suffix avoids it: the cross-filesystem copy lands on a name the skip logic cannot match, and the final step is a rename *within* the destination filesystem, which is atomic. Either the artifact is complete and correctly named, or it is not there at all. -**The trailing slash is the caller's intent, never inferred from `[ -d "$DEST" ]`.** Inference cannot tell "put it in this directory" apart from "a directory is sitting on the artifact's name" — and the second is a collision that has to fail. Obeying it silently would bury the artifact one level down under a name the skip logic still matches, so the step would look complete on every later run. +**The trailing slash is the caller's intent, never inferred from `[ -d "$DEST" ]`.** Inference cannot tell "put it in this directory" apart from "a directory is sitting on the artifact's name" -- and the second is a collision that has to fail. Obeying it silently would bury the artifact one level down under a name the skip logic still matches, so the step would look complete on every later run. **It has no lock.** That is safe only because the variant analysis fixes the shape up front: one task per (variant, stage, key), so nothing races for a path. Never make sharing depend on a check-then-act existence test. Two concurrent callers would stage through the same `${DEST}.part`, and one's EXIT trap would delete the other's staged copy between its two `mv` calls. -## `find_artifact.sh` — why it exists +## `find_artifact.sh` -- why it exists -Artifacts **move during a run**: a step writes its output to the working volume, and it is promoted to permanent storage once the step consuming it has succeeded. A skip check therefore cannot ask a single directory whether its work is done — the honest question is "is it in permanent storage, or still waiting to be promoted, or neither". +Artifacts **move during a run**: a step writes its output to the working volume, and it is promoted to permanent storage once the step consuming it has succeeded. A skip check therefore cannot ask a single directory whether its work is done -- the honest question is "is it in permanent storage, or still waiting to be promoted, or neither". Roots are given permanent-first. If an artifact exists in both, the promoted copy is the finished one and the other is residue from a move that did not complete. That is **reported** rather than passed over, because a stale copy silently winning would go on winning for every later run. -**Both roots take the same relative path**, which is the whole reason the working tree mirrors the output tree exactly instead of inventing its own layout — it makes the lookup one path against N roots rather than a translation between two schemes. +**Both roots take the same relative path**, which is the whole reason the working tree mirrors the output tree exactly instead of inventing its own layout -- it makes the lookup one path against N roots rather than a translation between two schemes. Exit codes are three-way on purpose: 0 found, 1 absent *silently* (the ordinary answer on a first run), 2 usage mistake. A caller has to be able to tell "absent" from "you called me wrong". -## `classify_manifest.sh` — why CHANGED, ADDED and REMOVED are distinguished +## `classify_manifest.sh` -- why CHANGED, ADDED and REMOVED are distinguished -A **CHANGED** value means the user altered a setting and the outputs on disk were produced with the old one. **ADDED** and **REMOVED** mean the set of parameters itself moved, which is what a release does — the outputs predate the parameter existing, so there is no earlier value to conflict with. +A **CHANGED** value means the user altered a setting and the outputs on disk were produced with the old one. **ADDED** and **REMOVED** mean the set of parameters itself moved, which is what a release does -- the outputs predate the parameter existing, so there is no earlier value to conflict with. A plain `diff` shows both as a pair of `+`/`-` lines, which is why every release that introduced a parameter used to fail every existing project and tell it to delete its results. -**Since the version block landed, that distinction no longer changes any verdict** — a release change is refused outright before this runs, so every kind of difference fails. It still decides how the difference *reads*. +**Since the version block landed, that distinction no longer changes any verdict** -- a release change is refused outright before this runs, so every kind of difference fails. It still decides how the difference *reads*. -Writing this as a unit-testable script rather than inline in the process **found a real bug**: the inline version silently skipped lines without `=`, which hides a key and can make a real change look like none. Its edge cases — a value containing `=`, an empty value, no trailing newline, an unparseable line — are milliseconds in the helpers suite and a JVM start each end to end. +Writing this as a unit-testable script rather than inline in the process **found a real bug**: the inline version silently skipped lines without `=`, which hides a key and can make a real change look like none. Its edge cases -- a value containing `=`, an empty value, no trailing newline, an unparseable line -- are milliseconds in the helpers suite and a JVM start each end to end. -## `parse_multirun.py` and `parse_metadata.py` — why Python +## `parse_multirun.py` and `parse_metadata.py` -- why Python -The values in both files are **free text**. `readPattern` defaults to `*_R{1,2}.fq.gz`, which contains a comma; a description field reading `Pop1, replicate 2` is ordinary. Splitting on commas would cut either in half and report a row with the wrong field count — a mistake that looks like a completely different mistake. `csv` implements the real quoting rules. +The values in both files are **free text**. `readPattern` defaults to `*_R{1,2}.fq.gz`, which contains a comma; a description field reading `Pop1, replicate 2` is ordinary. Splitting on commas would cut either in half and report a row with the wrong field count -- a mistake that looks like a completely different mistake. `csv` implements the real quoting rules. Both **report every problem at once, with line numbers**: a hand-written table with four mistakes should take one fix-and-rerun cycle, not four. @@ -48,7 +48,7 @@ Both exit **2 for a usage mistake and 1 for a bad file**, so a caller can tell " ## An apostrophe in a comment can break the script -`filterFalsePositives.sh` embeds its awk program in a **single-quoted** shell string. Any apostrophe inside it — including one in an awk `#` comment — closes that string and the file stops parsing as shell. Writing *"the VCF's own header"* in a comment there broke it during the 2026-08-30 pass; the existing text at the `-p` error message uses the `'"'"'` dance for exactly this reason. +`filterFalsePositives.sh` embeds its awk program in a **single-quoted** shell string. Any apostrophe inside it -- including one in an awk `#` comment -- closes that string and the file stops parsing as shell. Writing *"the VCF's own header"* in a comment there broke it during the 2026-08-30 pass; the existing text at the `-p` error message uses the `'"'"'` dance for exactly this reason. Never rely on spotting it. `bash -n` catches it instantly, which is why every comment pass ends with a syntax check over all of `bin/`, `install/`, `dev/scripts/` and the wrapper. @@ -58,24 +58,24 @@ See `false-positive-filter.md`. ## `depth2freq.awk` and `createDepthFile.sh` -The existing awk-over-`bcftools query` idiom: bcftools does the VCF surgery, awk touches only numbers. Worth knowing when adding anything numeric to steps 6–7 — it is the pattern the rest of the pipeline already uses. +The existing awk-over-`bcftools query` idiom: bcftools does the VCF surgery, awk touches only numbers. Worth knowing when adding anything numeric to steps 6-7 -- it is the pattern the rest of the pipeline already uses. ## Citations, and the two traps in writing them -`install/citations.json` → `bin/write_citations.py` → `CITATIONS.md` + `references.bib`, beside the results. Run by `scripts/citations.nf` once per invocation. +`install/citations.json` -> `bin/write_citations.py` -> `CITATIONS.md` + `references.bib`, beside the results. Run by `scripts/citations.nf` once per invocation. -**The chain gained a front end since, and the authored file moved.** `install/references.bib` is what you edit now — its own header says so — and `dev/scripts/bib2citations.py` compiles it into `install/citations.json`, which `00_static` fails on when the two disagree. Editing the JSON is overwritten and caught rather than silently kept. Beware that **`references.bib` names two different files**: the authored input under `install/`, and the per-run output written beside a user's results. An analysis run merges three sources — `install/citations.json` for PoolSeqFlow and Nextflow, `analysis/citations.json` for the layer itself, and the module's own. +**The chain gained a front end since, and the authored file moved.** `install/references.bib` is what you edit now -- its own header says so -- and `dev/scripts/bib2citations.py` compiles it into `install/citations.json`, which `00_static` fails on when the two disagree. Editing the JSON is overwritten and caught rather than silently kept. Beware that **`references.bib` names two different files**: the authored input under `install/`, and the per-run output written beside a user's results. An analysis run merges three sources -- `install/citations.json` for PoolSeqFlow and Nextflow, `analysis/citations.json` for the layer itself, and the module's own. **Versions are probed at run time, not read from `environment.yml`.** `params.software` can point a tool at a system installation, and then the pinned version is not what ran. `lib/tool_version.sh` is shared with `install/check_install.sh` so the version a run reports and the version its citations claim cannot disagree. -**`| tee` swallows the exit status.** `python3 … | tee citations.txt` made the task's status tee's, so a Python traceback produced an empty file and a green run — COMPLETED in the trace, the failure only in the log. Write then `cat`. Mutation-tested by pointing `--data` at a nonexistent file. +**`| tee` swallows the exit status.** `python3 ... | tee citations.txt` made the task's status tee's, so a Python traceback produced an empty file and a green run -- COMPLETED in the trace, the failure only in the log. Write then `cat`. Mutation-tested by pointing `--data` at a nonexistent file. -**A shared `key` deduplicates the bibliography**; `tools: {key: "Display Name"}` lets one entry cover several tools, because SAMtools and BCFtools are one paper (Z's call). Versions are keyed by the DATA key — a display name does not map back to it, which silently lost the versions for `Trim Galore` and `snpEff` until it was fixed. +**A shared `key` deduplicates the bibliography**; `tools: {key: "Display Name"}` lets one entry cover several tools, because SAMtools and BCFtools are one paper (Z's call). Versions are keyed by the DATA key -- a display name does not map back to it, which silently lost the versions for `Trim Galore` and `snpEff` until it was fixed. **Only what the run invoked is listed.** `when: "annotate"` gates SnpEff; citing it on a run that did not annotate would claim a step that did not happen. This is the rule the analysis layer's conditional tools will follow. -**The node is built from `params`, not from a run.** The file describes the *invocation* and belongs at the base `Output/` root, which a run that set its own `storageDir` would not name. `annotate` is the one exception — it is per run, and a tool that any run invoked was invoked. `collect()` over the frequency tables makes one emission however many runs there are, so this is always one task. +**The node is built from `params`, not from a run.** The file describes the *invocation* and belongs at the base `Output/` root, which a run that set its own `storageDir` would not name. `annotate` is the one exception -- it is per run, and a tool that any run invoked was invoked. `collect()` over the frequency tables makes one emission however many runs there are, so this is always one task. **`citations.json` is resolved to an absolute path in the workflow body**, not named inside the process: `projectDir` in a process script is discouraged, and the reference data is part of the installation rather than an input. -**The references have not been verified by Z.** Written from knowledge; the DOIs and page ranges need checking against the papers before 3.0 ships. This covers the *software* citations here. A module's *method* citations are a separate set, live in the module's own `references.bib`, and were verified during F1 — which is where the `n_eff` attribution was found wrong twice before it was right. +**The references have not been verified by Z.** Written from knowledge; the DOIs and page ranges need checking against the papers before 3.0 ships. This covers the *software* citations here. A module's *method* citations are a separate set, live in the module's own `references.bib`, and were verified during F1 -- which is where the `n_eff` attribution was found wrong twice before it was right. diff --git a/.claude/development-notes/brainstorming.md b/.claude/development-notes/brainstorming.md index 03c50fb..1ea10bd 100644 --- a/.claude/development-notes/brainstorming.md +++ b/.claude/development-notes/brainstorming.md @@ -1,4 +1,4 @@ -# Brainstorming — ideas for later releases +# Brainstorming -- ideas for later releases **Written 2026-09-01, against the tree at `ebf08a2`.** The rsync entry graduated eight hours later. Integrity-by-hash has not, and one of the things it planned to build on has gone since. @@ -8,7 +8,7 @@ When an entry graduates it moves to the plan and leaves a line behind saying whe --- -## Integrity by hash, everywhere — Z, 2026-09-01 +## Integrity by hash, everywhere -- Z, 2026-09-01 The idea, in Z's words: *"record sha256 of every file created. Then check the existence of the file against a hash especially for resume logic. And instead of creating dummy files we symlink every file that needs to be there. So hash is checked against the real file all the time."* @@ -16,15 +16,15 @@ Three separate changes that only make sense together. Z's own framing: *"This wo ### What it would buy -**Resume currently trusts a filename.** Every step's skip branch asks `find_artifact.sh` whether a path exists and treats presence as "this stage is done". A truncated, half-written or corrupted artifact passes that test, and everything downstream is built on it. This is the same class of failure as the two `atomic_mv.sh` defects — the difference is that those were about how a file gets into place, and this is about whether the file that is already in place is the right one. +**Resume currently trusts a filename.** Every step's skip branch asks `find_artifact.sh` whether a path exists and treats presence as "this stage is done". A truncated, half-written or corrupted artifact passes that test, and everything downstream is built on it. This is the same class of failure as the two `atomic_mv.sh` defects -- the difference is that those were about how a file gets into place, and this is about whether the file that is already in place is the right one. **It would make the check total rather than incidental.** `atomic_mv.sh` now proves a cross-filesystem copy byte-identical before it removes the source, but that guarantee ends the moment the move returns. Nothing revisits the artifact afterwards. A recorded hash extends the same guarantee across the whole life of the project. -**It composes with what the analysis layer already does.** `.provenance` beside every intermediate records the identity of the results it was derived from, and a mismatch refuses. That is this idea one level up — the sidecar answers *"were these the right inputs"*, a hash answers *"is this the file we wrote"*. Same shape, same failure philosophy: refuse and name, never silently rebuild. +**It composes with what the analysis layer already does.** `.provenance` beside every intermediate records the identity of the results it was derived from, and a mismatch refuses. That is this idea one level up -- the sidecar answers *"were these the right inputs"*, a hash answers *"is this the file we wrote"*. Same shape, same failure philosophy: refuse and name, never silently rebuild. ### What it would break, and this is the hard part -**The dummy files are not a shortcut — they stand where no file exists.** Step 7's skip branches `touch` an empty file when a *downstream* artifact is already present: +**The dummy files are not a shortcut -- they stand where no file exists.** Step 7's skip branches `touch` an empty file when a *downstream* artifact is already present: ``` if [ -f ]; then touch ${sorted_vcf} # nothing to point at @@ -35,11 +35,11 @@ elif [ -f ${target_sorted_vcf} ]; then ln -s ${target_sorted_vcf} . # the real The last branch already does what Z is asking for. The earlier ones cannot, because **the file they would symlink to does not exist and is not supposed to**: every step-7 VCF is an intermediate, and `_sort_fp_dq.vcf` deliberately does not survive a run. The empty file is there only to satisfy Nextflow's `output:` declaration so the DAG can proceed past a stage whose product has been superseded. -So "symlink instead of a dummy" cannot be done by substitution. It needs the DAG to be able to say *"this output is not required, because a later one exists"* — optional outputs, or a differently shaped channel, or a stage that is not run at all rather than run to produce a placeholder. That is the overhaul Z anticipated, and it is the whole cost of the idea. The hashing is the easy half. +So "symlink instead of a dummy" cannot be done by substitution. It needs the DAG to be able to say *"this output is not required, because a later one exists"* -- optional outputs, or a differently shaped channel, or a stage that is not run at all rather than run to produce a placeholder. That is the overhaul Z anticipated, and it is the whole cost of the idea. The hashing is the easy half. **Reading every artifact costs what the artifacts cost.** Hashing is proportional to total bytes written, and the BAMs dominate. Cheap at write time, where the data is already in hand; expensive if a resume re-hashes everything before deciding what to skip. -**A hash record is new state that can go stale and can refuse a run.** It needs the care `.poolseqflow_params` gets — and the same question about what happens when it disagrees. +**A hash record is new state that can go stale and can refuse a run.** It needs the care `.poolseqflow_params` gets -- and the same question about what happens when it disagrees. ### What already exists to build on @@ -50,9 +50,9 @@ So "symlink instead of a dummy" cannot be done by substitution. It needs the DAG ### Open questions, none of them answered - **Sidecar or manifest?** A `.sha256` beside each artifact travels with it through a granular move between roots, the way `.provenance` does. One manifest per run is easier to read whole and harder to keep true across two roots. -- **When is it checked?** Every resume, or only when asked? Re-hashing a 30 GB BAM set to decide what to skip would cost more than some of the steps it skips. A cheap pre-filter — size and mtime — with the hash as the tiebreak is the obvious compromise and needs thinking about. +- **When is it checked?** Every resume, or only when asked? Re-hashing a 30 GB BAM set to decide what to skip would cost more than some of the steps it skips. A cheap pre-filter -- size and mtime -- with the hash as the tiebreak is the obvious compromise and needs thinking about. - **What happens on a mismatch?** The layer's habit is refuse-and-name. Rebuilding silently would be the opposite of the point, but refusing means a corrupted artifact needs a documented way out. -- **Do hashes join the recorded manifest** the change guard compares, or stay separate? They describe the outputs, not the settings, so probably separate — but then there are two records beside `Output/` with different rules. +- **Do hashes join the recorded manifest** the change guard compares, or stay separate? They describe the outputs, not the settings, so probably separate -- but then there are two records beside `Output/` with different rules. - **Does this replace `-resume`, or sit beside it?** Nextflow's own resume already hashes task inputs. This is about artifacts that outlive the work directory, which Nextflow does not track. ### Where it sits @@ -61,37 +61,37 @@ After 3.0.0. It touches every step's skip branch, which is the highest-traffic c --- -## Try Strelka2 as a caller — Z, 2026-09-05 +## Try Strelka2 as a caller -- Z, 2026-09-05 -Prompted by Pinto, Sousa & Silva 2026, *Variant calling in genomics: a comparative performance analysis and decision guide*, PLOS ONE 21(2):e0339891, `10.1371/journal.pone.0339891`. Seven callers on human WGS (NA12878, ~50×): GATK, FreeBayes, DeepVariant, samtools, Strelka2, Octopus, VarScan2. +Prompted by Pinto, Sousa & Silva 2026, *Variant calling in genomics: a comparative performance analysis and decision guide*, PLOS ONE 21(2):e0339891, `10.1371/journal.pone.0339891`. Seven callers on human WGS (NA12878, ~50x): GATK, FreeBayes, DeepVariant, samtools, Strelka2, Octopus, VarScan2. -**What it says.** On whole genome, **Strelka2 has the highest precision (0.8326) and the best F1 (0.9009)**; Octopus has the best recall (0.9838) at a heavy compute cost; DeepVariant led on a chromosome-20 subset but **could not finish the whole genome** within their compute budget. samtools is reported as fast and low-memory — *"short runtimes and low memory usage, making it highly efficient"* — which is an efficiency result, not an accuracy win. +**What it says.** On whole genome, **Strelka2 has the highest precision (0.8326) and the best F1 (0.9009)**; Octopus has the best recall (0.9838) at a heavy compute cost; DeepVariant led on a chromosome-20 subset but **could not finish the whole genome** within their compute budget. samtools is reported as fast and low-memory -- *"short runtimes and low memory usage, making it highly efficient"* -- which is an efficiency result, not an accuracy win. -Z had already eliminated DeepVariant and Octopus on speed, independently, and this supports that. **Strelka2 was never tried and should be.** Z, 2026-09-05: *"there is strelka2, I never tried it but we should give it a shot… it is important we try it for this purpose."* +Z had already eliminated DeepVariant and Octopus on speed, independently, and this supports that. **Strelka2 was never tried and should be.** Z, 2026-09-05: *"there is strelka2, I never tried it but we should give it a shot... it is important we try it for this purpose."* ### The measurement this needs is not the one in the paper -The benchmark is **individual genotype calling against a truth set on one diploid human**. This pipeline does not want genotypes, it wants allele frequencies from a pool, and the two are not the same question — precision and recall on a genotype call say nothing directly about the bias in a frequency estimated from allele depths. A caller that wins on F1 could still be worse here if its `AD` handling is worse. +The benchmark is **individual genotype calling against a truth set on one diploid human**. This pipeline does not want genotypes, it wants allele frequencies from a pool, and the two are not the same question -- precision and recall on a genotype call say nothing directly about the bias in a frequency estimated from allele depths. A caller that wins on F1 could still be worse here if its `AD` handling is worse. -So a trial has to measure **what this pipeline actually reads**: the per-allele depths, on pooled data, against a known composition. The right shape is the corpus method used for the depth cutoff — build pools of known composition, call them with both, compare the recovered frequencies to the truth rather than comparing the callers to each other. +So a trial has to measure **what this pipeline actually reads**: the per-allele depths, on pooled data, against a known composition. The right shape is the corpus method used for the depth cutoff -- build pools of known composition, call them with both, compare the recovered frequencies to the truth rather than comparing the callers to each other. ### What it would break -- **Strelka2 does not emit the same `FORMAT` fields.** Everything after step 6 reads `AD` — `depth2freq.awk`, `MajorAlleleToRef.py`, the false-positive filter. If Strelka2's output does not carry per-allele depths in the same shape, this is a converter and not a swap. +- **Strelka2 does not emit the same `FORMAT` fields.** Everything after step 6 reads `AD` -- `depth2freq.awk`, `MajorAlleleToRef.py`, the false-positive filter. If Strelka2's output does not carry per-allele depths in the same shape, this is a converter and not a swap. - **It is a germline caller built around diploid genotype models.** The same objection that applies to any genotype-first tool applies here: pooled samples are not diploid, and a model that assumes they are may be doing something unhelpful to the counts even when the counts are the only part being used. - **A second caller is a second identity.** Step 6's identity names `variantCall.*`; a caller choice would have to join it, or two runs using different callers would share a results directory. ### Where it sits -After 3.0.0. It is a genuine question rather than a preference, and the corpus that would answer it does not exist yet — building that is most of the work, and it is worth having regardless of what the answer turns out to be. +After 3.0.0. It is a genuine question rather than a preference, and the corpus that would answer it does not exist yet -- building that is most of the work, and it is worth having regardless of what the answer turns out to be. --- -## rsync instead of hand-rolled copy-and-verify — Z, 2026-09-01 +## rsync instead of hand-rolled copy-and-verify -- Z, 2026-09-01 -> **GRADUATED the same morning, in `77fdbd7`** — eight hours after "not now" was written here. `rsync=3.4.4=hffd6c76_1` is pinned in `install/environment.yml`, unpinned in `install/environment-analysis.yml`, and `rsync` is a slot in `params.software`. `bin/atomic_mv.sh` stages with `rsync -a`. +> **GRADUATED the same morning, in `77fdbd7`** -- eight hours after "not now" was written here. `rsync=3.4.4=hffd6c76_1` is pinned in `install/environment.yml`, unpinned in `install/environment-analysis.yml`, and `rsync` is a slot in `params.software`. `bin/atomic_mv.sh` stages with `rsync -a`. > -> **The doubt below was upheld, which is the part worth keeping.** The section headed *Where it is weaker* argued that rsync verifies what it wrote against what it read, and would not catch a source that changed mid-copy. The implementation does not rely on rsync's verification at all: it compares the staged copy against the source with `diff -qr --no-dereference` before anything is removed, and the comment above that line — *"THE COPY IS NOT TAKEN ON TRUST"* — is this argument in the code. The two `sha256sum` passes became that one `diff`, and `test_atomic_mv_refuses_a_copy_that_does_not_match_its_source` still guards it. +> **The doubt below was upheld, which is the part worth keeping.** The section headed *Where it is weaker* argued that rsync verifies what it wrote against what it read, and would not catch a source that changed mid-copy. The implementation does not rely on rsync's verification at all: it compares the staged copy against the source with `diff -qr --no-dereference` before anything is removed, and the comment above that line -- *"THE COPY IS NOT TAKEN ON TRUST"* -- is this argument in the code. The two `sha256sum` passes became that one `diff`, and `test_atomic_mv_refuses_a_copy_that_does_not_match_its_source` still guards it. > > **`--remove-source-files` was not adopted**, for the same reason: it would delete the source on rsync's own say-so, exactly where the independent check has to sit. The source is removed after the verified rename instead. `--partial` was not adopted either. @@ -99,16 +99,16 @@ Z, raised while `atomic_mv.sh` was being rewritten: *"I'm also starting to lean ### The availability worry does not apply here -rsync is not in coreutils and a minimal container image often lacks it — so the worry is right in general, and wrong for this project. **PoolSeqFlow ships its own conda environment and already pins the tools it needs**, `coreutils=9.11` and `bash=5.2.37` among them. Whether a server has rsync is not the question; whether `install/environment.yml` has a line for it is. conda-forge carries it, and `bin/atomic_mv.sh` runs inside a task with that environment active. +rsync is not in coreutils and a minimal container image often lacks it -- so the worry is right in general, and wrong for this project. **PoolSeqFlow ships its own conda environment and already pins the tools it needs**, `coreutils=9.11` and `bash=5.2.37` among them. Whether a server has rsync is not the question; whether `install/environment.yml` has a line for it is. conda-forge carries it, and `bin/atomic_mv.sh` runs inside a task with that environment active. -So this is a dependency decision, not a portability one. The cost is a line in the environment, a name in step 0's software check, and a re-export — the same shape as any other tool the pipeline calls. +So this is a dependency decision, not a portability one. The cost is a line in the environment, a name in step 0's software check, and a re-export -- the same shape as any other tool the pipeline calls. ### What it would buy - **`--remove-source-files` implements the invariant directly.** rsync deletes a source file only after it has transferred successfully. That is the exact property the September fix was written by hand to get. -- **Transfer verification is built in**, and would replace the two `sha256sum` passes. *To confirm before adopting*: rsync's whole-file checksum verification is part of the delta-transfer path, and local-to-local transfers default to `--whole-file`, which skips the delta algorithm. Whether the end-to-end check still runs in that mode needs testing rather than assuming — it is the entire reason to switch. +- **Transfer verification is built in**, and would replace the two `sha256sum` passes. *To confirm before adopting*: rsync's whole-file checksum verification is part of the delta-transfer path, and local-to-local transfers default to `--whole-file`, which skips the delta algorithm. Whether the end-to-end check still runs in that mode needs testing rather than assuming -- it is the entire reason to switch. - **`--partial` makes an interrupted transfer resumable**, which for a 24 GB per-position depth file is worth more than it sounds. -- Directories, sparse files (`-S`), ACLs and xattrs (`-AX`), hardlinks (`-H`) — all native, where the hand-rolled version handles them by getting `cp -a` right and hoping. +- Directories, sparse files (`-S`), ACLs and xattrs (`-AX`), hardlinks (`-H`) -- all native, where the hand-rolled version handles them by getting `cp -a` right and hoping. ### What it would not change @@ -116,7 +116,7 @@ So this is a dependency decision, not a portability one. The cost is a line in t ### Where it is weaker than what is there now -rsync verifies *what it wrote against what it read*. The current digest compares *the destination against the source as it stands afterwards*, which is a different question and catches a source that changed while it was being copied — reproduced, and the case that guards it is `test_atomic_mv_refuses_a_copy_that_does_not_match_its_source`. rsync would copy a moving source happily and report success. Whether that matters depends on whether a caller can ever hand over a source still being written; in the pipeline the producing task has finished, so it is defensive rather than load-bearing. +rsync verifies *what it wrote against what it read*. The current digest compares *the destination against the source as it stands afterwards*, which is a different question and catches a source that changed while it was being copied -- reproduced, and the case that guards it is `test_atomic_mv_refuses_a_copy_that_does_not_match_its_source`. rsync would copy a moving source happily and report success. Whether that matters depends on whether a caller can ever hand over a source still being written; in the pipeline the producing task has finished, so it is defensive rather than load-bearing. ### Where it sits @@ -124,15 +124,15 @@ rsync verifies *what it wrote against what it read*. The current digest compares --- -## Covariate adjustment as a layer-wide question — Z, 2026-09-07 +## Covariate adjustment as a layer-wide question -- Z, 2026-09-07 -Deferred out of F2 deliberately, and not because the arithmetic is hard. The arithmetic is settled: weighted Frisch–Waugh–Lovell reproduces `lm(f ~ y + z, weights = w)` to twelve significant figures provided the degrees of freedom come from the full model, and the permutation scheme that works with covariates was measured — permute the RAW phenotype and re-residualise at each site, where permuting the already-residualised one runs at twice its nominal rate. `association-math.md` holds both. +Deferred out of F2 deliberately, and not because the arithmetic is hard. The arithmetic is settled: weighted Frisch-Waugh-Lovell reproduces `lm(f ~ y + z, weights = w)` to twelve significant figures provided the degrees of freedom come from the full model, and the permutation scheme that works with covariates was measured -- permute the RAW phenotype and re-residualise at each site, where permuting the already-residualised one runs at twice its nominal rate. `association-math.md` holds both. ### Why it is not an F2 setting Z, 2026-09-07: *"this requires a constant discussion where we would need covariate analysis in the whole analysis layer and then design tools to address it after we have a much more complete set."* -A covariate is not a property of one test. If `association` adjusts for a covariate and `mds` does not, two published folders from one project disagree about what the data was, and a reader has no way to see it. The same question reaches diversity, FST, every trajectory statistic — each needs its own answer to *what does adjusting mean here*, and several of those answers are not "put it in the model" at all. +A covariate is not a property of one test. If `association` adjusts for a covariate and `mds` does not, two published folders from one project disagree about what the data was, and a reader has no way to see it. The same question reaches diversity, FST, every trajectory statistic -- each needs its own answer to *what does adjusting mean here*, and several of those answers are not "put it in the model" at all. So the decision is that the layer gets one answer, not that F2 gets one first. The frame already declares covariates and their scales and prints them; what it does not do is fit them, and nothing should until there are enough modules to see the shape of the problem. @@ -142,23 +142,23 @@ So the decision is that the layer gets one answer, not that F2 gets one first. T ### What it would break -Every degree of freedom counted so far. `df = n_observed − 2 − q` over units means a timed design of three units cannot fit a single covariate — that refusal has to be loud rather than a column of NA, and it is a design being honest rather than a defect. And a covariate correlated with the phenotype changes what a published effect size means, which is a manual problem before it is a code problem. +Every degree of freedom counted so far. `df = n_observed - 2 - q` over units means a timed design of three units cannot fit a single covariate -- that refusal has to be loud rather than a column of NA, and it is a design being honest rather than a defect. And a covariate correlated with the phenotype changes what a published effect size means, which is a manual problem before it is a code problem. ### Where it sits -After the roster is fuller — F4 onwards, once there are enough modules that "what does this layer do about covariates" has a real answer instead of one module's opinion. Z, 2026-09-07, on the interim: *"since the results and metadata will be aligned, the user might be able to handle it themselves."* +After the roster is fuller -- F4 onwards, once there are enough modules that "what does this layer do about covariates" has a real answer instead of one module's opinion. Z, 2026-09-07, on the interim: *"since the results and metadata will be aligned, the user might be able to handle it themselves."* --- -## Plotting as a published, re-runnable script — Z, 2026-09-08 +## Plotting as a published, re-runnable script -- Z, 2026-09-08 The idea, in Z's words: *"What we can do is maybe move the plot code outside of mds (and all modules), print the code and let the user make the plot they want to make by manually editing the code. We output the scripts anyway."* -Raised while deciding whether `mds` should take a color palette setting. The answer to that was no — how many levels a variable has, whether it is ordered, and whether a reader needs the categories distinguishable or merely grouped are properties of the experiment, and every way of getting a palette wrong is quiet. This is the idea that makes the refusal generous instead of merely restrictive. +Raised while deciding whether `mds` should take a color palette setting. The answer to that was no -- how many levels a variable has, whether it is ordered, and whether a reader needs the categories distinguishable or merely grouped are properties of the experiment, and every way of getting a palette wrong is quiet. This is the idea that makes the refusal generous instead of merely restrictive. ### What it would buy -**It ends a settings surface that otherwise grows forever.** `mds` already carries `colorBy`, `shapeBy` and `dimensions`; `basicstats` carries `chromosomes` for its depth panels; `association` carries `chromosomes` again plus `reportBelow` and `reportTop`. Every one of those exists because a plot needed a decision made for it, and the next request is always another one — facet by this, log that axis, drop that pool, order the levels my way. A script the user owns answers all of them at once and adds no setting. +**It ends a settings surface that otherwise grows forever.** `mds` already carries `colorBy`, `shapeBy` and `dimensions`; `basicstats` carries `chromosomes` for its depth panels; `association` carries `chromosomes` again plus `reportBelow` and `reportTop`. Every one of those exists because a plot needed a decision made for it, and the next request is always another one -- facet by this, log that axis, drop that pool, order the levels my way. A script the user owns answers all of them at once and adds no setting. **The escape hatch is already half there and nobody can use it.** Each module publishes its own `.R` beside the result with the shared library folded in, so the plotting code ships today. What stops anyone editing it is that re-running it recomputes everything: for `mds` that is the whole distance matrix over every depth table. A plot script reading the published TSVs would redraw in seconds. @@ -166,7 +166,7 @@ Raised while deciding whether `mds` should take a color palette setting. The ans ### What it would break, and this is the part that decides the shape -**The PDF report is built from the PNGs.** `analysis/lib/rmd/report.Rmd:54` walks the published files and embeds every `.png` it finds. A module that publishes no figure contributes none, and the report every analysis carries — F1's, and one of the reasons the analysis layer is worth running at all — becomes a page of tables. +**The PDF report is built from the PNGs.** `analysis/lib/rmd/report.Rmd:54` walks the published files and embeds every `.png` it finds. A module that publishes no figure contributes none, and the report every analysis carries -- F1's, and one of the reasons the analysis layer is worth running at all -- becomes a page of tables. So the version that survives is **not** "modules stop plotting". It is: the plot becomes a standalone script that reads the published TSVs, and the module runs it once. The default figure still exists, the report is unchanged, and the script beside it is re-runnable against `mds.tsv` without recomputing a distance. Z's idea, with the report constraint folded in. @@ -178,7 +178,7 @@ Whether the published plot script carries its settings substituted in, or reads ### What already exists to build on -Every module already publishes its own `.R` and its compiled sources, `PublishResults` already takes whatever lands in `published/`, and `report.Rmd` already discovers files rather than being told about them — so a new `*_plot.R` needs no wiring. `mds.tsv` and `eigenvalues.tsv` are already sufficient to redraw `mds.png` without touching a depth table, and the manual's `mds` section carries a base-R snippet that does exactly that. +Every module already publishes its own `.R` and its compiled sources, `PublishResults` already takes whatever lands in `published/`, and `report.Rmd` already discovers files rather than being told about them -- so a new `*_plot.R` needs no wiring. `mds.tsv` and `eigenvalues.tsv` are already sufficient to redraw `mds.png` without touching a depth table, and the manual's `mds` section carries a base-R snippet that does exactly that. ### Where it sits @@ -186,11 +186,11 @@ At or just before **E5b**, which is already the pass that makes the three module --- -## A module repository served from the site — Z, 2026-09-09 +## A module repository served from the site -- Z, 2026-09-09 Z's framing: *"we make a repo with tarballs into the website. The 'official' repo... third-parties would need to make their own repos and additional repos are allowed."* -**Not an index page — a package repository.** The site serves the catalogue AND the tarballs it points at, the way a distribution's archive does. Third parties stand up their own, and PoolSeqFlow can be pointed at more than one. (An earlier draft of this entry described only the index; that was a misreading and the difference is the whole engineering problem below.) +**Not an index page -- a package repository.** The site serves the catalogue AND the tarballs it points at, the way a distribution's archive does. Third parties stand up their own, and PoolSeqFlow can be pointed at more than one. (An earlier draft of this entry described only the index; that was a misreading and the difference is the whole engineering problem below.) **Deferred deliberately, and the repository stays exactly as it is.** Z, 2026-09-09: *"We won't work on it yet. It might be a future date task (nothing urgent, nobody besides me is developing this yet). No third-party apps for now, but our infrastructure allows expansion."* @@ -204,14 +204,14 @@ Z's framing: *"we make a repo with tarballs into the website. The 'official' rep ### The hard problem: a site deploy replaces everything, and a package repository may not -A published version's bytes have to be identical forever. The catalogue pins them by sha256, and `install ` promises the same code every time. A GitHub Pages deploy is a wholesale replacement of build output, so a tarball regenerated on each deploy either breaks its digest — installs start failing — or has its digest regenerated with it, in which case a pinned version silently means something new. That is worse, because nothing reports it. +A published version's bytes have to be identical forever. The catalogue pins them by sha256, and `install ` promises the same code every time. A GitHub Pages deploy is a wholesale replacement of build output, so a tarball regenerated on each deploy either breaks its digest -- installs start failing -- or has its digest regenerated with it, in which case a pinned version silently means something new. That is worse, because nothing reports it. **Measured, 2026-09-09.** `git archive` is reproducible only from a commit: | what is archived | mtime it stamps | two builds seconds apart | |---|---|---| -| a tree — `HEAD:analysis/modules/mds` | **now** | different digests | -| a commit — `HEAD -- analysis/modules/mds` | the commit date | **identical digests** | +| a tree -- `HEAD:analysis/modules/mds` | **now** | different digests | +| a commit -- `HEAD -- analysis/modules/mds` | the commit date | **identical digests** | The catch is the shape: the installer requires the archive to unpack to `/`, and only the tree form gives that. `git archive --mtime` settles both at once, measured byte-identical across builds and with the right root: @@ -219,15 +219,15 @@ The catch is the shape: the installer requires the archive to unpack to `/ git archive --format=tar --mtime="@" --prefix=/ :analysis/modules/ | gzip -n ``` -`--mtime` is a recent `git archive` flag — present in 2.55 here. A build using it should assert a minimum git version rather than discover its absence as silently drifting digests. +`--mtime` is a recent `git archive` flag -- present in 2.55 here. A build using it should assert a minimum git version rather than discover its absence as silently drifting digests. ### What the repository is actually FOR, which the note first got wrong -**No module is downloadable today, at any version.** The catalogue has zero rows and nothing hosts a tarball; the three shipped modules arrive inside the release and that is all. An earlier draft of this entry said multiple versions "already work" because the catalogue format carries a version column and `install ` pins on it — that is the format being *able to express* it, not the thing existing. Z, 2026-09-09: *"nobody can download a different version of any module today."* +**No module is downloadable today, at any version.** The catalogue has zero rows and nothing hosts a tarball; the three shipped modules arrive inside the release and that is all. An earlier draft of this entry said multiple versions "already work" because the catalogue format carries a version column and `install ` pins on it -- that is the format being *able to express* it, not the thing existing. Z, 2026-09-09: *"nobody can download a different version of any module today."* -So the repository is not a convenience over an existing mechanism. **It is what would make module distribution exist at all, and multiple versions and backwards compatibility with it** — someone on an older pipeline being able to fetch a module version that still runs there. Z: *"it changes if we have a repo. We can start having backwards compatibility and all."* +So the repository is not a convenience over an existing mechanism. **It is what would make module distribution exist at all, and multiple versions and backwards compatibility with it** -- someone on an older pipeline being able to fetch a module version that still runs there. Z: *"it changes if we have a repo. We can start having backwards compatibility and all."* -**Publishing has to be a deliberate act.** Deriving the set of published versions from git history was raised and rejected — Z, 2026-09-09: *"I don't want you to derive it from the history."* Every intermediate version bump would become a published release, including ones made mid-development and never meant to leave. Whatever marks a version as published, it is something a person does on purpose. +**Publishing has to be a deliberate act.** Deriving the set of published versions from git history was raised and rejected -- Z, 2026-09-09: *"I don't want you to derive it from the history."* Every intermediate version bump would become a published release, including ones made mid-development and never meant to leave. Whatever marks a version as published, it is something a person does on purpose. A module's own repository does not have this problem, because the module is at the root there and the commit form gives the right shape directly. It is only the first-party monorepo case that needs the re-rooting. @@ -235,31 +235,31 @@ A module's own repository does not have this problem, because the module is at t ### What it would break, and the one bug it would ship with -**`.github/workflows/docs.yml` has a `paths:` filter** — `manual/**`, `mkdocs.yml`, `CHANGELOG.md`, `build_docs.py`, itself. `analysis/modules-index.tsv` is not on it. Wired naively, adding a module row would not rebuild the site, the catalogue would silently stay stale, and the symptom is a module that cannot be installed with nothing saying why. One line, easy to miss. +**`.github/workflows/docs.yml` has a `paths:` filter** -- `manual/**`, `mkdocs.yml`, `CHANGELOG.md`, `build_docs.py`, itself. `analysis/modules-index.tsv` is not on it. Wired naively, adding a module row would not rebuild the site, the catalogue would silently stay stale, and the symptom is a module that cannot be installed with nothing saying why. One line, easy to miss. -**Publishing a module becomes a merge plus a site deploy** rather than a merge alone. Still not a release, so the property that matters — a module appearing without a PoolSeqFlow release — survives. +**Publishing a module becomes a merge plus a site deploy** rather than a merge alone. Still not a release, so the property that matters -- a module appearing without a PoolSeqFlow release -- survives. ### Two facts that bound when this can be decided **The URL becomes permanent at 3.0.0.** No released version has the module system: v2.2.0 carries neither `MODULE_INDEX_URL` nor `analysis/modules-index.tsv`. So 3.0.0 is the first release that will ever read a catalogue, and from then on every released copy has the address compiled into its own `lib/wrapper_lib.sh` and asks for it forever. Changing it later means serving both addresses indefinitely. **Keeping the raw URL is therefore a decision with a permanent consequence, not a deferral of one.** -**`POOLSEQFLOW_MODULE_INDEX` substitutes, it does not add.** `module_index_source()` is `${POOLSEQFLOW_MODULE_INDEX:-$MODULE_INDEX_URL}` — one value. So "additional repositories are allowed" is true of the *shape* and not yet of the code: today you can point a machine at a different catalogue, not at several at once. Several would need a list, a precedence rule for a name published in two of them, and a decision about whether a row may be shadowed. None of that is written. +**`POOLSEQFLOW_MODULE_INDEX` substitutes, it does not add.** `module_index_source()` is `${POOLSEQFLOW_MODULE_INDEX:-$MODULE_INDEX_URL}` -- one value. So "additional repositories are allowed" is true of the *shape* and not yet of the code: today you can point a machine at a different catalogue, not at several at once. Several would need a list, a precedence rule for a name published in two of them, and a decision about whether a row may be shadowed. None of that is written. -What third-party repositories need beyond that is nothing: `fetch_url` already takes any URL, the digest already makes the host a question of integrity rather than trust, and `#!index-format` already refuses a layout this release cannot read. A third-party repository is a catalogue and some tarballs at an address — the download machinery is done. +What third-party repositories need beyond that is nothing: `fetch_url` already takes any URL, the digest already makes the host a question of integrity rather than trust, and `#!index-format` already refuses a layout this release cannot read. A third-party repository is a catalogue and some tarballs at an address -- the download machinery is done. ### What already exists to build on -The catalogue's two headers already separate layout from content — `#!index-format` is refused on, `#!index-version` never is, precisely so a release can read a catalogue newer than itself. `fetch_url` already takes a URL, a path or `file://`. The digest in each row already makes the URL's host a matter of integrity rather than trust. And the site is already generated wholly from one source, which is the pattern a repository section would follow. +The catalogue's two headers already separate layout from content -- `#!index-format` is refused on, `#!index-version` never is, precisely so a release can read a catalogue newer than itself. `fetch_url` already takes a URL, a path or `file://`. The digest in each row already makes the URL's host a matter of integrity rather than trust. And the site is already generated wholly from one source, which is the pattern a repository section would follow. ### Where it sits After 3.0.0, and nothing about it is urgent while Z is the only person developing modules. -**One thing gets harder by waiting, and the cost of getting it wrong is bounded.** `MODULE_INDEX_FORMAT="1"` compiles into every released copy and `require_module_index_format` is an exact-match refusal, so catalogue columns added later — `frame` and `environment`, which is what would let the repository hand an older pipeline a version that still runs on it — are unreadable by 3.0.0. The consequence is not data loss or a broken install: a 3.0.0 user who wants a module published later is told to upgrade. That may simply be acceptable for the first release of the module system, and it is Z's call rather than a deadline anyone has to meet. +**One thing gets harder by waiting, and the cost of getting it wrong is bounded.** `MODULE_INDEX_FORMAT="1"` compiles into every released copy and `require_module_index_format` is an exact-match refusal, so catalogue columns added later -- `frame` and `environment`, which is what would let the repository hand an older pipeline a version that still runs on it -- are unreadable by 3.0.0. The consequence is not data loss or a broken install: a 3.0.0 user who wants a module published later is told to upgrade. That may simply be acceptable for the first release of the module system, and it is Z's call rather than a deadline anyone has to meet. --- -## The catalogue's columns are matched by name and read by position — Z, 2026-09-10 +## The catalogue's columns are matched by name and read by position -- Z, 2026-09-10 Added when the `kind` column landed and cost thirteen failing cases. **Z deferred it explicitly: "We can look at the name-position thing later. Add it to the list of post v3.1.0 tasks."** @@ -273,16 +273,16 @@ A column addition that costs one line instead of thirteen cases. More to the poi ### The shape it probably wants -An accessor rather than a destructuring — `row_field "$row" version` reading the same `MODULE_INDEX_COLUMNS` list that built the row. Then a column's position is knowledge held in exactly one place instead of at every call site, and `MODULE_INDEX_COLUMNS` becomes the single declaration it was always meant to be. +An accessor rather than a destructuring -- `row_field "$row" version` reading the same `MODULE_INDEX_COLUMNS` list that built the row. Then a column's position is knowledge held in exactly one place instead of at every call site, and `MODULE_INDEX_COLUMNS` becomes the single declaration it was always meant to be. The cost is a subshell per field per row where there is now one `read` per row. `available` renders every row in the catalogue, so this is measurable and worth measuring before committing to it; a catalogue is tens of rows today and the answer may simply be that it does not matter. ### What it would break -Nothing published. The wire format is unchanged — this is entirely how the wrapper reads a row it already has. It does not touch `#!index-format`, so no released copy notices. +Nothing published. The wire format is unchanged -- this is entirely how the wrapper reads a row it already has. It does not touch `#!index-format`, so no released copy notices. **It does touch every consumer at once**, which is exactly the change that wants a full suite behind it rather than a `--case` run. That is the argument for after a release rather than during one. ### Where it sits -**After v3.1.0.** Nothing is wrong today: the columns and the destructurings agree, and `00_static` checks every row against the file it advertises. This is paying down the next addition's cost, not fixing a defect — and the next addition is not scheduled. +**After v3.1.0.** Nothing is wrong today: the columns and the destructurings agree, and `00_static` checks every row against the file it advertises. This is paying down the next addition's cost, not fixing a defect -- and the next addition is not scheduled. diff --git a/.claude/development-notes/calibration.md b/.claude/development-notes/calibration.md index 28423b0..7e878f9 100644 --- a/.claude/development-notes/calibration.md +++ b/.claude/development-notes/calibration.md @@ -1,24 +1,24 @@ # What was measured against a known truth, and the four things it changed -**Written 2026-09-07, against the tree at `15d4c34` plus the uncommitted `analysis.design` scope move. `dev/validation/` and `analysis/modules/association/` are both new — the harness exists, the module does not.** That order is the point: these are measurements taken while the design was still free to change, and three of them changed it. +**Written 2026-09-07, against the tree at `15d4c34` plus the uncommitted `analysis.design` scope move. `dev/validation/` and `analysis/modules/association/` are both new -- the harness exists, the module does not.** That order is the point: these are measurements taken while the design was still free to change, and three of them changed it. The harness is `dev/validation/calibrate.R`, four sections, twenty-seven seconds for the cheap three. `dev/validation/README.md` says how to run it and what discipline makes a number there evidence. This note is what the numbers said. ## Why it is not in `test/` -`test/` asserts exact values over eleven hand-computed sites and fails when one moves. That answers *is the arithmetic what we said it was*. A false-positive rate is a rate, a power curve is a curve, and the limit where an assumption stops holding is found by pushing until it breaks — so the assertions here are distributional, the replicate counts are in the hundreds of thousands, and a run is minutes rather than seconds. Nothing here gates a commit. +`test/` asserts exact values over eleven hand-computed sites and fails when one moves. That answers *is the arithmetic what we said it was*. A false-positive rate is a rate, a power curve is a curve, and the limit where an assumption stops holding is found by pushing until it breaks -- so the assertions here are distributional, the replicate counts are in the hundreds of thousands, and a run is minutes rather than seconds. Nothing here gates a commit. **The generator never calls `analysis/lib/R/`.** Every draw is `rbinom` and arithmetic. Had the simulation used our own `n_eff` to build its read counts, a wrong `n_eff` would cancel on both sides and the harness would report agreement to twelve figures while being wrong about the world. ## `n_eff` is exactly the variance of a two-stage pooled draw -A pool of `n_chrom` chromosomes at true frequency *p*, sequenced to depth *d*, is two samplings — which chromosomes entered the pool, then which of those the reads found. Conditioning on the first and adding the variances gives `Var(f) = p(1-p)(n_chrom + d - 1)/(n_chrom·d)`, which is `p(1-p)/n_eff` exactly. +A pool of `n_chrom` chromosomes at true frequency *p*, sequenced to depth *d*, is two samplings -- which chromosomes entered the pool, then which of those the reads found. Conditioning on the first and adding the variances gives `Var(f) = p(1-p)(n_chrom + d - 1)/(n_chrom·d)`, which is `p(1-p)/n_eff` exactly. -**Measured over 54 cells at 200,000 replicates each, the largest deviation is 2.87 standard errors** — what you expect as the maximum of 54 standard normals. The form `n·d/(n + d)`, also in circulation, misses by **27.81**. So the comment in `n_eff.R` saying the −1 is not a slip is now a measurement. The grid reaches `n_chrom = 100` by three routes — ploidy 1×100, 2×50, 4×25 — and they agree, which is what ploidy-generality looks like when it is exercised rather than asserted. +**Measured over 54 cells at 200,000 replicates each, the largest deviation is 2.87 standard errors** -- what you expect as the maximum of 54 standard normals. The form `n·d/(n + d)`, also in circulation, misses by **27.81**. So the comment in `n_eff.R` saying the -1 is not a slip is now a measurement. The grid reaches `n_chrom = 100` by three routes -- ploidy 1x100, 2x50, 4x25 -- and they agree, which is what ploidy-generality looks like when it is exercised rather than asserted. ## The closed-form p is calibrated under its own assumptions, which was not what I expected -Across nine cells at 200,000 null sites each — depths 30, 100 and 1000 against frequencies 0.05, 0.25 and 0.50 — the weighted *t* rejects at 0.0494 to 0.0515 for α = 0.05, 0.0085 to 0.0103 for 0.01, and 0.00090 to 0.00142 for 0.001. Including depth 30 with a 5% allele, where the frequencies are nearly discrete. +Across nine cells at 200,000 null sites each -- depths 30, 100 and 1000 against frequencies 0.05, 0.25 and 0.50 -- the weighted *t* rejects at 0.0494 to 0.0515 for α = 0.05, 0.0085 to 0.0103 for 0.01, and 0.00090 to 0.00142 for 0.001. Including depth 30 with a 5% allele, where the frequencies are nearly discrete. **So the argument for publishing a permutation p is not that the parametric one is miscalibrated in the ordinary case.** It is exactness at multiallelic sites where the union bound measured 38% inflated, the floor being visible so six pools cannot claim p = 1e-9, and robustness when the assumptions fail. Only the first two were established before this; the third is what the last two sections went looking for. @@ -26,7 +26,7 @@ The far tail is **not** measured. At 200,000 sites the α = 1e-4 column counts t ## The permutation p is conservative over discrete counts, by a factor worth printing -Over the exhaustive 720 relabelings at six pools, no cell rejects more often than its α. **At depth 30 with a 5% allele it runs at 0.36 of the α it was asked for** — 0.0036 against a nominal 0.01. Shallow reads of a rare allele put identical counts in several pools; tied relabelings then give tied statistics, and every tie counts toward the p-value. +Over the exhaustive 720 relabelings at six pools, no cell rejects more often than its α. **At depth 30 with a 5% allele it runs at 0.36 of the α it was asked for** -- 0.0036 against a nominal 0.01. Shallow reads of a rare allele put identical counts in several pools; tied relabelings then give tied statistics, and every tie counts toward the p-value. That is the test being safe, not broken, so the gate on it is one-sided. But a user who asks for 0.01 and gets 0.0036 is owed the number, and it belongs in the manual beside the floor. @@ -41,13 +41,13 @@ Six units of two pools, identical sites, five estimators differing only in what | 0.05 | 0.1599 | 0.0916 | 0.0468 | 0.1555 | 0.0427 | | 0.10 | 0.1895 | 0.1145 | 0.0486 | 0.1895 | 0.0454 | -**Taking rule 17c literally — compute *t* from twelve pools, read it against four degrees of freedom — is wrong in both directions.** 0.0183 when the pools really are independent, 0.1145 when they are not. That is the signature of an adjustment applied to the wrong quantity: it never controls the rate and it costs power when there was nothing to correct. +**Taking rule 17c literally -- compute *t* from twelve pools, read it against four degrees of freedom -- is wrong in both directions.** 0.0183 when the pools really are independent, 0.1145 when they are not. That is the signature of an adjustment applied to the wrong quantity: it never controls the rate and it costs power when there was nothing to correct. **Averaging within unit before fitting holds at 0.0468 to 0.0493 across the whole range.** The plan named this as *"the better closed-form route, and the one to build if the unit count bites"*. It bites, and it is not an alternative held in reserve. -**Permuting pool labels is as broken as no correction at all** — 0.1895 against 0.1895 at dispersion 0.10. The plan said "permute the phenotype labels" without saying at what level, and the level is the whole thing. Permuting over units holds flat. +**Permuting pool labels is as broken as no correction at all** -- 0.1895 against 0.1895 at dispersion 0.10. The plan said "permute the phenotype labels" without saying at what level, and the level is the whole thing. Permuting over units holds flat. -**None of this costs a branch.** At one pool per unit the weighted mean of that pool is its own frequency and the unit weight is its own `n_eff`, so the roll-up is the identity and the unit permutation is the pool permutation. A 3-vs-3 design with no replication runs the same code and gets the same answer. The rule to write is *roll up to units first, always* — the degrees of freedom then fall out of the fit instead of being applied to it. +**None of this costs a branch.** At one pool per unit the weighted mean of that pool is its own frequency and the unit weight is its own `n_eff`, so the roll-up is the identity and the unit permutation is the pool permutation. A 3-vs-3 design with no replication runs the same code and gets the same answer. The rule to write is *roll up to units first, always* -- the degrees of freedom then fall out of the fit instead of being applied to it. ## The weight model: it is `cor(depth, phenotype)`, and it breaks the permutation p rather than the closed form @@ -55,8 +55,8 @@ Six pools of 50 diploids, ten independent poolings per cell, 10,000 sites each. | depth pattern | cor(y,d) | perm, even | perm, conc 5 | perm, conc 1 | param, conc 1 | |---|---|---|---|---|---| -| flat | — | 0.0536 | 0.0487 | 0.0499 | 0.0504 | -| mixed | −0.032 | 0.0333 | 0.0344 | 0.0376 | 0.0591 | +| flat | -- | 0.0536 | 0.0487 | 0.0499 | 0.0504 | +| mixed | -0.032 | 0.0333 | 0.0344 | 0.0376 | 0.0591 | | tilted | 0.458 | 0.0454 | 0.0505 | 0.0503 | 0.0571 | | aligned | 0.891 | 0.0605 | 0.0575 | 0.0558 | 0.0513 | @@ -64,17 +64,17 @@ Six pools of 50 diploids, ten independent poolings per cell, 10,000 sites each. **The thing permuting was supposed to protect against is the thing permuting is worse at.** Weighted least squares handles heteroscedasticity when the weights are right; the permutation null shuffles the phenotype across pools carrying different amounts of information, so the observed high-correlation assignment is compared against a null built mostly out of low-correlation ones. -Uneven pooling is second order. Its worst effect is on the closed form at heterogeneous depth — 0.0591, an 18% inflation — and `flat` is clean at every concentration, which isolates the mechanism to depth heterogeneity and not to skew alone. +Uneven pooling is second order. Its worst effect is on the closed form at heterogeneous depth -- 0.0591, an 18% inflation -- and `flat` is clean at every concentration, which isolates the mechanism to depth heterogeneity and not to skew alone. ## The weights are not what is wrong, and this is the turn the whole campaign hinges on -The obvious repair is a better variance model. The fit asserts `Var(fᵢ) = σ²/n_effᵢ`; uneven pooling leaves the truth at `p(1-p)[(1 - kᵢ/n_chrom)/dᵢ + kᵢ/n_chrom]` for a skew factor `kᵢ ≥ 1`, so the excess is `p(1-p)(kᵢ-1)/n_chrom` — depth-free and additive. Write it as `Var = p(1-p)(θ + 1/n_effᵢ)` and the weight `1/(θ + 1/n_effᵢ)` reduces to `n_eff` where read sampling dominates and flattens toward equal where it does not, so low-and-uneven, deep-and-even and deep-and-uneven become limits of one expression instead of a menu of corrections. +The obvious repair is a better variance model. The fit asserts `Var(fᵢ) = σ²/n_effᵢ`; uneven pooling leaves the truth at `p(1-p)[(1 - kᵢ/n_chrom)/dᵢ + kᵢ/n_chrom]` for a skew factor `kᵢ >= 1`, so the excess is `p(1-p)(kᵢ-1)/n_chrom` -- depth-free and additive. Write it as `Var = p(1-p)(θ + 1/n_effᵢ)` and the weight `1/(θ + 1/n_effᵢ)` reduces to `n_eff` where read sampling dominates and flattens toward equal where it does not, so low-and-uneven, deep-and-even and deep-and-uneven become limits of one expression instead of a menu of corrections. -**θ is estimable and the estimator is good.** Method of moments over sites — the scatter a site shows, less the sampling variance the weights predict, in units of `p(1-p)` — recovers the analytic truth across two orders of magnitude: 0.00194 against 0.00195, 0.01904 against 0.01885, 0.04334 against 0.04455, 0.25987 against 0.24500. +**θ is estimable and the estimator is good.** Method of moments over sites -- the scatter a site shows, less the sampling variance the weights predict, in units of `p(1-p)` -- recovers the analytic truth across two orders of magnitude: 0.00194 against 0.00195, 0.01904 against 0.01885, 0.04334 against 0.04455, 0.25987 against 0.24500. **And re-weighting by it fixes nothing.** At the failing cell, 0.0609 under `n_eff` becomes 0.0622 under the two-term weights and 0.0607 under equal weights. Three weightings, one answer. -The row that explains it is `aligned` with *even* pooling: θ̂ = 0.00008, no excess dispersion at all, `n_eff` exact, **the weights already right** — and the permutation still rejects at 0.0609. Pools at 30×, 200× and 1000× carry different precision. Weighted least squares handles that for estimation, which is why the closed form sits at 0.0500 in the same cell. But a permutation test needs the observations to be **interchangeable**, and unequal variances are not made equal by weighting them correctly. +The row that explains it is `aligned` with *even* pooling: θ̂ = 0.00008, no excess dispersion at all, `n_eff` exact, **the weights already right** -- and the permutation still rejects at 0.0609. Pools at 30x, 200x and 1000x carry different precision. Weighted least squares handles that for estimation, which is why the closed form sits at 0.0500 in the same cell. But a permutation test needs the observations to be **interchangeable**, and unequal variances are not made equal by weighting them correctly. **So `depthPhenotypeCor` was the wrong repair for the right problem, and the failure is exchangeability rather than variance.** @@ -91,22 +91,22 @@ One permutation σ serves every site in a draw, as the label scheme does, so cor | depth | concentration | cor(y,d) | labels | residuals | resid+θ | |---|---|---|---|---|---| -| flat | even | — | 0.0536 | 0.0496 | 0.0518 | -| flat | 1 | — | 0.0465 | 0.0461 | 0.0496 | -| mixed | even | −0.032 | 0.0348 | 0.0512 | 0.0506 | -| mixed | 1 | −0.032 | 0.0392 | **0.0621** | 0.0497 | +| flat | even | -- | 0.0536 | 0.0496 | 0.0518 | +| flat | 1 | -- | 0.0465 | 0.0461 | 0.0496 | +| mixed | even | -0.032 | 0.0348 | 0.0512 | 0.0506 | +| mixed | 1 | -0.032 | 0.0392 | **0.0621** | 0.0497 | | aligned | even | 0.891 | **0.0609** | 0.0535 | 0.0509 | | aligned | 1 | 0.891 | 0.0503 | 0.0474 | 0.0489 | **`resid+θ` is calibrated in all six cells.** Three things sit in that table and the middle one is the evidence: -- the failing cell is repaired, 0.0609 → 0.0509 -- **plain residual permutation breaks in exactly one cell, and it is the cell where the weights are wrong.** Skewed pooling means `e·√w` is not equal-variance after all, so the scheme's own assumption fails — and it fails nowhere else. θ̂ repairs that cell and only that cell, 0.0621 → 0.0497. The mechanism confirms itself rather than a number coming out right -- the label scheme's conservatism at uneven depth, 0.035–0.039, is **sensitivity given away**, not safety +- the failing cell is repaired, 0.0609 -> 0.0509 +- **plain residual permutation breaks in exactly one cell, and it is the cell where the weights are wrong.** Skewed pooling means `e·√w` is not equal-variance after all, so the scheme's own assumption fails -- and it fails nowhere else. θ̂ repairs that cell and only that cell, 0.0621 -> 0.0497. The mechanism confirms itself rather than a number coming out right +- the label scheme's conservatism at uneven depth, 0.035-0.039, is **sensitivity given away**, not safety So θ earns its place as what makes the residuals exchangeable, not as a reweighting of the fit. -**No breakdown was found.** Pushing the pooling from even to an effective pool of two individuals out of fifty — θ from 0 to 0.245 — leaves `resid+θ` between 0.0485 and 0.0597 with no trend, excursions consistent with noise at eight poolings. There is no θ threshold for a guard to sit on within any range a real experiment reaches. +**No breakdown was found.** Pushing the pooling from even to an effective pool of two individuals out of fifty -- θ from 0 to 0.245 -- leaves `resid+θ` between 0.0485 and 0.0597 with no trend, excursions consistent with noise at eight poolings. There is no θ threshold for a guard to sit on within any range a real experiment reaches. ## What it costs the floor, and the answer is better than expected @@ -121,23 +121,23 @@ Enumerated exhaustively, both schemes, binary phenotype: | aligned | labels | 0.0500 | 0.10000 | 0.0000 | 0.1057 | | aligned | residuals | 0.00139 | 0.00556 | 0.0524 | 0.1027 | -**At equal precision the residual scheme reproduces the label scheme exactly** — same smallest p, same rates, to four figures. With equal weights *t* depends only on the two group means, so rearranging residuals within a phenotype group changes nothing and the 720 collapse to the same 20 distinct values, ties included. The design-based guarantee survives untouched wherever it applies. +**At equal precision the residual scheme reproduces the label scheme exactly** -- same smallest p, same rates, to four figures. With equal weights *t* depends only on the two group means, so rearranging residuals within a phenotype group changes nothing and the 720 collapse to the same 20 distinct values, ties included. The design-based guarantee survives untouched wherever it applies. -**The extra resolution appears only when the pools carry unequal precision, and it is calibrated there** — FPR 0.0524 at α = 0.05. A deep pool's frequency genuinely does carry more information than a shallow one's, so the finer granularity is earned rather than assumed. What a reader must be told is that a smaller p in an unbalanced design partly reflects the imbalance, which is a disclosure and not a defect. +**The extra resolution appears only when the pools carry unequal precision, and it is calibrated there** -- FPR 0.0524 at α = 0.05. A deep pool's frequency genuinely does carry more information than a shallow one's, so the finer granularity is earned rather than assumed. What a reader must be told is that a smaller p in an unbalanced design partly reflects the imbalance, which is a disclosure and not a defect. ## Arity, where the design departs from what other tools do -`S = max|t|` over every allele of a site, read against a null built from the same site — same *k*, same alleles, same sum-to-one constraint. Nothing is bounded, so nothing is corrected. +`S = max|t|` over every allele of a site, read against a null built from the same site -- same *k*, same alleles, same sum-to-one constraint. Nothing is bounded, so nothing is corrected. -| k | residual p | min over all × (k−1) | min over alternates × (k−1) | +| k | residual p | min over all x (k-1) | min over alternates x (k-1) | |---|---|---|---| | 2 | 0.0463 | 0.0494 | 0.0494 | | 3 | 0.0513 | **0.0684** | 0.0488 | | 4 | 0.0450 | **0.0640** | 0.0499 | -**Exact at every arity**, and the selection rate relative to biallelic is 1.00×, 1.11×, 0.97× — no enrichment of multiallelic sites at the top of a table. +**Exact at every arity**, and the selection rate relative to biallelic is 1.00x, 1.11x, 0.97x -- no enrichment of multiallelic sites at the top of a table. -**The union bound inflates by 1.37× at k = 3**, which independently reproduces the 38% recorded before this harness existed. And **dropping the reference is calibrated** — it is not a false-positive problem at all, which is why it survived testing once. It is a power problem, and the power section is where it shows. +**The union bound inflates by 1.37x at k = 3**, which independently reproduces the 38% recorded before this harness existed. And **dropping the reference is calibrated** -- it is not a false-positive problem at all, which is why it survived testing once. It is a power problem, and the power section is where it shows. ## Power, which none of the above was evidence of @@ -156,24 +156,24 @@ A test that never rejects never lies, so every section above is only half an arg At even depth the two are equivalent. At uneven depth `resid+θ` recovers the sensitivity the label scheme's conservatism was costing, in the mid-power range where the difference decides a finding. -**At very large effects the residual scheme is slightly weaker** — 0.9153 against 0.9524 — and the reason is structural rather than incidental: the residuals are taken about the null model, so at a strong alternative they carry the signal, and permuting them builds a null that is itself inflated. It matters little where both are above 0.9, and it is a real property to state rather than discover. +**At very large effects the residual scheme is slightly weaker** -- 0.9153 against 0.9524 -- and the reason is structural rather than incidental: the residuals are taken about the null model, so at a strong alternative they carry the signal, and permuting them builds a null that is itself inflated. It matters little where both are above 0.9, and it is a real property to state rather than discover. And the triallelic site the multiallelic decision exists for, where both alternates rise and the reference falls by their sum: -| slope | max over all | min over alternates × 2 | min over all × 2 | +| slope | max over all | min over alternates x 2 | min over all x 2 | |---|---|---|---| | 0.04 | **0.4668** | 0.3337 | 0.5854 | | 0.08 | 0.9408 | 0.8646 | 0.9871 | -**Dropping the reference costs 40% of the power** at the effect size where power is decided. The third column is higher only because it is mis-sized — the same union bound measured at 1.37× its nominal rate above — so it buys detections with false positives and is not a comparison at matched size. +**Dropping the reference costs 40% of the power** at the effect size where power is decided. The third column is higher only because it is mis-sized -- the same union bound measured at 1.37x its nominal rate above -- so it buys detections with false positives and is not a comparison at matched size. ## Three corrections to the harness, and one of them is a trap the module inherits -**`count/B` is not a permutation p-value.** The raw fraction can return 0, and no permutation p can be 0 — the observed labeling is always one of the labelings. On data satisfying every assumption of the model it rejected at 0.0569 against a nominal 0.05 at B = 300. **The valid form is `(1 + count)/(1 + B)`, and a module that samples permutations rather than enumerating them inherits this directly.** An enumerated null needs no correction: the observed labeling is already in the set being counted. Probed to be sufficient — exhaustive 720 gives 0.0508, sampled 300, 1000 and 3000 give 0.0466, 0.0517 and 0.0508. +**`count/B` is not a permutation p-value.** The raw fraction can return 0, and no permutation p can be 0 -- the observed labeling is always one of the labelings. On data satisfying every assumption of the model it rejected at 0.0569 against a nominal 0.05 at B = 300. **The valid form is `(1 + count)/(1 + B)`, and a module that samples permutations rather than enumerating them inherits this directly.** An enumerated null needs no correction: the observed labeling is already in the set being counted. Probed to be sufficient -- exhaustive 720 gives 0.0508, sampled 300, 1000 and 3000 give 0.0466, 0.0517 and 0.0508. **A binomial margin over sites is too tight, because sites are not independent trials.** Every site inside one run shares a phenotype and the same permutation draws. At 100,000 sites that margin called the harness's own control a failure. The gate now uses the spread measured across independent poolings, which assumes nothing about what happens inside one. -**A nuisance drawn once is a realisation and not a measurement.** The first version of the weight section drew each pool's contribution vector once per cell, so a row said what happened to one pooling. The same cell read 0.0716 at 10,000 sites and 0.0521 at 50,000, and the difference was which vectors the stream handed out — not sampling error on the rate. Every cell is now averaged over ten poolings, and the spread across them is reported beside the mean, because a study is one draw from that spread rather than the average of ten. +**A nuisance drawn once is a realisation and not a measurement.** The first version of the weight section drew each pool's contribution vector once per cell, so a row said what happened to one pooling. The same cell read 0.0716 at 10,000 sites and 0.0521 at 50,000, and the difference was which vectors the stream handed out -- not sampling error on the rate. Every cell is now averaged over ten poolings, and the spread across them is reported beside the mean, because a study is one draw from that spread rather than the average of ten. ## Six pools is not a general claim, so the number of them was swept @@ -184,31 +184,31 @@ Depths in ascending blocks over a phenotype that rises with the pool index, pool | 4 | 0.0833 | 0.0155 | 0.0000 | 0.0306 | 0.0004 | | 6 | 0.00278 | **0.0611** | 0.0481 | 0.1530 | 0.1366 | | 8 | 0.00005 | **0.0578** | 0.0486 | 0.2260 | 0.2190 | -| 12 | — | **0.0566** | 0.0520 | 0.3625 | 0.3512 | -| 20 | — | 0.0496 | 0.0512 | 0.5886 | 0.5945 | -| 30 | — | 0.0503 | 0.0514 | 0.7904 | 0.7979 | +| 12 | -- | **0.0566** | 0.0520 | 0.3625 | 0.3512 | +| 20 | -- | 0.0496 | 0.0512 | 0.5886 | 0.5945 | +| 30 | -- | 0.0503 | 0.0514 | 0.7904 | 0.7979 | -**The label scheme's failure is a small-design problem and it decays with n** — 0.0611, 0.0578, 0.0566, 0.0496, 0.0503. By twenty pools it is gone: more pools means the null averages over more assignments and the observed one stops being special. So the whole exchangeability failure lives exactly where pool-seq lives, six to twelve pools, and disappears where nobody would have worried. +**The label scheme's failure is a small-design problem and it decays with n** -- 0.0611, 0.0578, 0.0566, 0.0496, 0.0503. By twenty pools it is gone: more pools means the null averages over more assignments and the observed one stops being special. So the whole exchangeability failure lives exactly where pool-seq lives, six to twelve pools, and disappears where nobody would have worried. `resid+θ` is calibrated at every n from 6 to 30 and matches or beats the label scheme on power throughout. -**Four pools is not an analysis.** The floor is 2/4! = 0.083, so α = 0.05 cannot be reached by relabeling at all, and the power column is incoherent — 0.0306 one way and 0.0004 the other. That is a refusal. +**Four pools is not an analysis.** The floor is 2/4! = 0.083, so α = 0.05 cannot be reached by relabeling at all, and the power column is incoherent -- 0.0306 one way and 0.0004 the other. That is a refusal. **And the n = 4 row exposes a trap the module has to keep avoiding.** The label scheme shows 0.0155 there, which should be impossible against a floor of 0.083. It happens because the harness *sampled* 150 draws from a set of only 24 orderings, and `(1 + r)/(1 + B)` then reaches below the true floor by luck alone. **Sampling a permutation set small enough to enumerate silently breaks the floor guarantee**, so enumerating when the set fits is a correctness requirement and not an optimisation, and it wants a case that would notice if someone changed it. At a slope of 0.04 with even depth, power runs 0.17 at six pools, 0.28 at eight, 0.46 at twelve, 0.75 at twenty, 0.91 at thirty. Uneven depth costs roughly a fifth throughout. That table is what a researcher wants before they spend money. -**Why this arm holds pooling even, which a first pass did not.** Run at Dirichlet concentration 1, the sweep showed the label scheme's failure largely *gone* — and that is not a contradiction of the table above, it is a second effect on top of it. Heavy pooling skew makes every pool noisy, and that noise swamps the depth differences that break exchangeability in the first place. So concentration and pool count interact, and a sweep that moved both would have reported their sum as if it were the effect of n. Pooling is held even here for that reason, and concentration is `breakdown`'s question instead. The interaction is real and is not measured anywhere: nothing establishes how much skew it takes to mask a given depth-phenotype correlation. +**Why this arm holds pooling even, which a first pass did not.** Run at Dirichlet concentration 1, the sweep showed the label scheme's failure largely *gone* -- and that is not a contradiction of the table above, it is a second effect on top of it. Heavy pooling skew makes every pool noisy, and that noise swamps the depth differences that break exchangeability in the first place. So concentration and pool count interact, and a sweep that moved both would have reported their sum as if it were the effect of n. Pooling is held even here for that reason, and concentration is `breakdown`'s question instead. The interaction is real and is not measured anywhere: nothing establishes how much skew it takes to mask a given depth-phenotype correlation. ## Against an independently written tool -`dev/validation/external.R`, against **BayPass 3.1** (Gautier 2015) — Fortran, Bayesian, MCMC, models population structure through an Ω matrix and returns a Bayes factor. Nothing about it resembles what we do except the question it answers. +`dev/validation/external.R`, against **BayPass 3.1** (Gautier 2015) -- Fortran, Bayesian, MCMC, models population structure through an Ω matrix and returns a Bayes factor. Nothing about it resembles what we do except the question it answers. **The two are never compared to each other directly.** Different statistics on different scales cannot agree numerically and it would mean nothing if they did. Both are scored against the sites that were *planted*, which neither can see, at 20 pools and 1,200 sites of which 100 carry an effect. -**Biallelic, which BayPass reads natively:** PoolSeqFlow recovers **46%** of the planted sites in its top 100, BayPass **44%**, and the Spearman correlation between our −log₁₀(p) and its BF over all 1,200 sites is **0.764**. Two unrelated implementations of two unrelated statistics, ranking the same genome the same way. +**Biallelic, which BayPass reads natively:** PoolSeqFlow recovers **46%** of the planted sites in its top 100, BayPass **44%**, and the Spearman correlation between our -log₁₀(p) and its BF over all 1,200 sites is **0.764**. Two unrelated implementations of two unrelated statistics, ranking the same genome the same way. -**Triallelic, which its input format cannot express.** BayPass takes two counts per pool, so the site has to be reduced first — and the reduction is the user's choice: +**Triallelic, which its input format cannot express.** BayPass takes two counts per pool, so the site has to be reduced first -- and the reduction is the user's choice: | | both alternates rise together | alternates move against each other | |---|---|---| @@ -216,9 +216,9 @@ At a slope of 0.04 with even depth, power runs 0.17 at six pools, 0.28 at eight, | BayPass, first alternate | 44% | 16% | | BayPass, alternates summed | **53%** | 10% | -Random guessing recovers 8.3%. **Neither reduction is right for both shapes.** Summing the alternates is the *best* answer when the reference carries the signal — better than ours, because it is one test where ours pays a small multiplicity cost — and it is very nearly the *worst possible* answer when the alternates carry it between them, collapsing to 10% because summing them cancels the signal outright. Reading one alternate does the reverse, less dramatically. +Random guessing recovers 8.3%. **Neither reduction is right for both shapes.** Summing the alternates is the *best* answer when the reference carries the signal -- better than ours, because it is one test where ours pays a small multiplicity cost -- and it is very nearly the *worst possible* answer when the alternates carry it between them, collapsing to 10% because summing them cancels the signal outright. Reading one alternate does the reverse, less dramatically. -A genome holds both shapes and nothing in the data says which a site is. **The module is never asked, because it reads every allele** — which is the whole content of the multiallelic decision, now measured against an outside tool rather than argued. +A genome holds both shapes and nothing in the data says which a site is. **The module is never asked, because it reads every allele** -- which is the whole content of the multiallelic decision, now measured against an outside tool rather than argued. ## What the campaign settled, in one place @@ -230,10 +230,10 @@ A genome holds both shapes and nothing in the data says which a site is. **The m 6. That preserves the combinatorial floor exactly where precision is equal, and earns finer resolution only where it is not. 7. `max|t|` over every allele is exact at any arity; the union bound is not, and dropping the reference costs power rather than calibration. -**Which is why the module is model-based, and says so.** Z, 2026-09-07: *"This tool becomes model-based. We detail when to use which model in the manual… if someone has a different use I will sit down and develop it as soon as possible, so they can come to me instead of trying to fit their data into an analysis when it does not fit."* The price of that is that failure gets quieter, so the diagnostics are **published and not merely checked** — a guard that passes tells a reader nothing, while a printed θ̂ says what the model had to absorb on their behalf. +**Which is why the module is model-based, and says so.** Z, 2026-09-07: *"This tool becomes model-based. We detail when to use which model in the manual... if someone has a different use I will sit down and develop it as soon as possible, so they can come to me instead of trying to fit their data into an analysis when it does not fit."* The price of that is that failure gets quieter, so the diagnostics are **published and not merely checked** -- a guard that passes tells a reader nothing, while a printed θ̂ says what the model had to absorb on their behalf. ## What is still not established -Everything above is six pools at `n_chrom = 100`, base frequency 0.5, one quantitative phenotype shape apart from the floor section. There is **no sweep over the number of pools**, no covariate arm — so the interaction between weighted FWL and residual permutation is unmeasured — nothing in the far tail below α = 1e-4, and no comparison against an independently written tool. The module implements label permutation as this is written and not the residual scheme. +Everything above is six pools at `n_chrom = 100`, base frequency 0.5, one quantitative phenotype shape apart from the floor section. There is **no sweep over the number of pools**, no covariate arm -- so the interaction between weighted FWL and residual permutation is unmeasured -- nothing in the far tail below α = 1e-4, and no comparison against an independently written tool. The module implements label permutation as this is written and not the residual scheme. Those are the next tiers. None of them is done, and the manual cannot claim what they would establish. diff --git a/.claude/development-notes/callable-sites.md b/.claude/development-notes/callable-sites.md index 33e8b20..55f32a2 100644 --- a/.claude/development-notes/callable-sites.md +++ b/.claude/development-notes/callable-sites.md @@ -1,4 +1,4 @@ -# The callable-sites track — built, measured, and reverted +# The callable-sites track -- built, measured, and reverted **Written 2026-08-31, against the tree at `7d65893`.** Nothing of the reverted work has come back, and the replacement it argues for is still not built. The three lessons at the end all still hold. @@ -8,13 +8,13 @@ The removed work is in `callable-sites-reverted.patch`, beside this note, if any ## What it was -`CallableSites` in `scripts/5_reports.nf`: one `samtools depth` pass per sample through `bin/callable_bins.awk`, gzipped to `Output/Reports/Callable/_callable.tsv.gz`. Per 1 kb window it recorded how many positions reached each of several depth thresholds, so anything measured per window would have a denominator. Thresholds were several rather than one because "callable" is a question about the analysis, not the alignment — a diversity estimate might want 10x, a presence/absence question 1x. +`CallableSites` in `scripts/5_reports.nf`: one `samtools depth` pass per sample through `bin/callable_bins.awk`, gzipped to `Output/Reports/Callable/_callable.tsv.gz`. Per 1 kb window it recorded how many positions reached each of several depth thresholds, so anything measured per window would have a denominator. Thresholds were several rather than one because "callable" is a question about the analysis, not the alignment -- a diversity estimate might want 10x, a presence/absence question 1x. ## Why it went Z: *"all of this needs to be wired to the analysis layer, because we don't know which analysis is needed by the user."* -The main pipeline was producing an artifact on the guess that someone would want it. Of the fifteen tools in the E4 roster, exactly two would have read it — π/θ_W/Tajima's D, and the SFS sweep scan — and only for windows that are a multiple of the bin size. Z's sharper objection: it is a **derivable** table. A full per-position depth file yields it at any resolution, and yields it at analysis time rather than freezing the resolution at run time. +The main pipeline was producing an artifact on the guess that someone would want it. Of the fifteen tools in the E4 roster, exactly two would have read it -- π/θ_W/Tajima's D, and the SFS sweep scan -- and only for windows that are a multiple of the bin size. Z's sharper objection: it is a **derivable** table. A full per-position depth file yields it at any resolution, and yields it at analysis time rather than freezing the resolution at run time. So the analysis layer derives it on demand into `Analysis/Main`, and the pipeline publishes only what a person reading results actually wants. @@ -38,7 +38,7 @@ A 360x ratio. Extrapolated, bgzipped: | 1 Gb plant | 8.1 GB | 25 GB | | 3 Gb vertebrate | 24.4 GB | 75 GB | -Sample count scales sub-linearly — 4x the samples is ~3x the size, because the `chrom`/`pos` columns amortise. +Sample count scales sub-linearly -- 4x the samples is ~3x the size, because the `chrom`/`pos` columns amortise. **`bgzip` + `tabix` is the finding that makes the full file usable.** Same size as plain gzip, and an arbitrary 10 kb region comes back in ~6 ms, so a module never reads the file whole. Any per-position artifact the analysis layer builds should be bgzipped and tabix-indexed for this reason. @@ -46,8 +46,8 @@ For scale: those figures are roughly **5% of the ready BAMs** the project alread ## Two things learned that outlived the revert -**`samtools depth` takes multiple BAMs in one pass** and emits one column per BAM — `chrom pos d1 d2 ...` — with `-H` writing a header naming each file. So a joint cohort depth file is ONE process, not one per sample. The per-sample design that was reverted was more expensive than the alternative. +**`samtools depth` takes multiple BAMs in one pass** and emits one column per BAM -- `chrom pos d1 d2 ...` -- with `-H` writing a header naming each file. So a joint cohort depth file is ONE process, not one per sample. The per-sample design that was reverted was more expensive than the alternative. -**The flags are inverted between the two tools.** `samtools depth -q` is BASE quality and `-Q` is MAPPING quality; `bcftools mpileup -q` is MAPPING quality and `-Q` is BASE quality. Same two letters, swapped. This project has already shipped a bug from exactly that confusion — in 2.2.0 `baseQualMin` and `varQualMin` were supplying each other's mpileup flags, and because both default to 30 nothing looked wrong. Whatever builds a depth track in the analysis layer must swap the flags, not the values. +**The flags are inverted between the two tools.** `samtools depth -q` is BASE quality and `-Q` is MAPPING quality; `bcftools mpileup -q` is MAPPING quality and `-Q` is BASE quality. Same two letters, swapped. This project has already shipped a bug from exactly that confusion -- in 2.2.0 `baseQualMin` and `varQualMin` were supplying each other's mpileup flags, and because both default to 30 nothing looked wrong. Whatever builds a depth track in the analysis layer must swap the flags, not the values. -**And a design error worth not repeating:** the reverted `CallableSites` read the **ready** BAM in step 5, while `VariantCall` piles up the **capped** BAM in step 6. Denominator and numerator would have described different read sets, diverging precisely at the anomalous pile-ups — the regions anyone would scrutinise. Any depth artifact must be built from the same BAMs the calls came from. +**And a design error worth not repeating:** the reverted `CallableSites` read the **ready** BAM in step 5, while `VariantCall` piles up the **capped** BAM in step 6. Denominator and numerator would have described different read sets, diverging precisely at the anomalous pile-ups -- the regions anyone would scrutinise. Any depth artifact must be built from the same BAMs the calls came from. diff --git a/.claude/development-notes/cap-bam-cost.md b/.claude/development-notes/cap-bam-cost.md new file mode 100644 index 0000000..e82065a --- /dev/null +++ b/.claude/development-notes/cap-bam-cost.md @@ -0,0 +1,62 @@ +# What `CapBAM` costs, measured + +**Written 2026-09-28, against the tree at `c5bacb4` plus uncommitted work. SHELVED: measured, not fixed.** Noticed by Z during the first end-to-end run of the early-development *Drosophila* project on 3.2.0. Nothing was changed as a result. + +## The measurement + +From that run's own trace, `Output/Reports/PoolSeqFlow_pipeline_trace.txt`, three pools of pooled *D. melanogaster* reads at roughly 300-400 MB of gzipped FASTQ per mate, `threads = 4`: + +| process | realtime | %cpu | +|---|---|---| +| `AlignReads:Align` (Sample1) | 14.9 min | 401 | +| `AlignReads:Align` (Sample2) | 11.8 min | 383 | +| `AlignReads:Align` (Sample3) | 11.7 min | 384 | +| **`VariantCalling:CapBAM` (Sample2)** | **10.6 min** | **110** | +| **`VariantCalling:CapBAM` (Sample3)** | **10.3 min** | **111** | +| `SortCleanBams:SortCleanBam` (Sample3) | 6.7 min | 137 | +| `BuildDictionaries:BuildSnpEffDb` | 4.4 min | 106 | +| `TrimQcClip:TrimReads` (Sample1) | 3.6 min | 98 | + +**Capping is the second most expensive step in the pipeline**, at about seventy percent of what alignment costs, and it is the only expensive one that does no biology. Alignment earns its fifteen minutes; this is bookkeeping. + +**It runs on one core while the run is given four.** Alignment sits at 400% and capping at 110%, so on a four-core machine three cores idle for ten minutes per sample. The wall-clock cost is therefore close to the whole of its CPU cost, where alignment's is a quarter of its own. + +The cap chosen for Sample1 on that run was 563, derived per sample by step 5 from its own depth histogram. + +## Where the time goes, from `bin/cap_depth.awk` and its caller + +`scripts/6_variant_call.nf` runs, per sample: + +```sh +samtools view -h ready.bam | cap_depth.awk -v cap=N | samtools view -b -o capped.bam - +``` + +Three costs, none of them the algorithm: + +- **A full BAM to SAM to BAM round trip.** The whole alignment is decompressed to text and recompressed, to decide which records to keep. +- **Neither `samtools view` is threaded.** No `-@`, so both the decompression and the recompression are serial. That alone accounts for much of the gap between 110% and 400%. +- **The awk walks every reference position of every read, twice.** A read is kept only if *every* position it covers is still under the cap, so the check pass has to finish before the increment pass begins. At 150 bp that is up to three hundred array operations per read, in an interpreter. `reflen()` additionally walks the CIGAR string one character at a time per read. + +## The property that makes this parallelizable, which is already in the code + +`cap_depth.awk` resets its state at every new reference sequence: + +```awk +if ($3 != chrom) { delete depth; chrom = $3; low = $4 + 0 } +``` + +**So chromosomes are already independent.** Capping each one separately and merging gives a byte-identical read set, because no decision on one sequence can depend on another. A *Drosophila* assembly has a handful of arms carrying nearly all the reads, so the parallelism available is roughly the core count, and it needs no change to the capping rule at all. That is the obvious first thing to try and it costs no correctness argument. + +## Why the cap is not simply handed to mpileup, which would cost nothing + +`bcftools mpileup -d` caps depth at pileup time with no BAM rewritten, and `variantCall.maxDepth` already passes it. It cannot replace this, and the reason is worth recording so nobody spends the afternoon rediscovering it: + +**The cap is per sample.** Step 5 measures a ceiling for each sample from that sample's own depth histogram, and `capBAM.maxDepth = -1` is what asks for it. `mpileup -d` applies one number to every input file in the cohort call, and step 6 pileups the whole cohort in a single invocation. One number cannot carry seven samples' ceilings. + +Passing the maximum of the per-sample caps would leave every shallower sample uncapped, which is not the same filter and would change the frequencies. + +## What is not measured + +Only three samples, one organism, one machine, one cap value, and `%cpu` is the trace's own figure rather than an instrumented profile. **Nothing here separates the two `samtools view` ends from the awk in the middle**, so the split between "serial compression" and "interpreter" is inferred from the shape of the pipeline rather than timed. A run with `-@` added to both ends and nothing else changed would settle that in one measurement, and should be the first thing done if anyone picks this up. + +See [[depth-cutoff.md]] for why the per-sample ceiling exists at all and what the alternatives were measured against. diff --git a/.claude/development-notes/cap-depth-rewrite.md b/.claude/development-notes/cap-depth-rewrite.md new file mode 100644 index 0000000..8c516ba --- /dev/null +++ b/.claude/development-notes/cap-depth-rewrite.md @@ -0,0 +1,99 @@ +# Capping, made about seven times faster on real data + +**Written 2026-09-29, against the tree at `772db1f` plus uncommitted work.** Picked up from `cap-bam-cost.md`, which measured the problem on 2026-09-28 and shelved it. That note's diagnosis was wrong on both counts and this one replaces it; it is kept because its timings are the record of what the step used to cost. + +## What the old note got wrong + +It said: *"Neither `samtools view` is threaded. No `-@`, so both the decompression and the recompression are serial. That alone accounts for much of the gap between 110% and 400%."* + +Measured, on 1M synthetic reads: + +| stage | time | +|---|---| +| decompress, `-@ 0` | 0 s | +| decompress, `-@ 3` | 0 s | +| gawk pass-through, `{print}` | 1 s | +| **`cap_depth.awk`** | **46 s** | +| recompress, `-@ 0` | 3 s | +| recompress, `-@ 3` | 1 s | +| full pipe as shipped | 47 s | +| full pipe with `-@ 3` on both ends | **48 s** | + +The two `samtools` ends cost about two seconds in forty-seven, and threading them made it marginally *slower*. **The whole cost is the awk**, and the old note's second proposal -- parallelising by chromosome -- would have been real work aimed at the right bottleneck with the wrong instrument. + +The tell was available and was not read: `gawk '{print}'` over the same file is 1 second. The script was running at 1/33rd of awk's own throughput, which is a statement about the script, not about awk. + +## Three redundancies, each removed + +### The check loop never needed more than one position + +```awk +end = pos + span - 1 +for (p = pos; p <= end; p++) { + if (depth[p] + 0 >= cap) { dropped++; next } +} +``` + +The stream is coordinate-sorted, so every read already processed starts at or before `pos`. Such a read covers position `p >= pos` if and only if its end reaches `p`. So + + depth[p] = #{ kept reads S : end_S >= p } + +which is **non-increasing in p**. The maximum over `[pos, end]` is therefore always `depth[pos]`, and 150 lookups collapse to one. + +The proof needs all four of: the sorted input, the trim-from-behind that never deletes a position at or above `pos`, the bypass for reads consuming no reference, and the reset at a new sequence. All four were already there. + +### The increment loop never needed to touch every position + +A difference array records `+1` at the read's first position and `-1` one past its last, and a running total carried along the leading edge gives `depth[pos]` for nothing. O(span) becomes O(1) per read, with one sweep step per reference position. + +### The sweep was creating the elements it then deleted + +```awk +while (low < pos) { low++; cur += diff[low]; delete diff[low] } +``` + +**Referencing `diff[low]` in awk creates that element.** Most reference positions hold no entry, so this created and destroyed one array element per position of the genome. Guarding with `if (low in diff)`, which does not create, took the sweep-dominated synthetic case from 14 s to 6 s on its own -- more than the difference array had bought there. + +## Measured, on Z's own *D. melanogaster* pools + +Whole BAMs, at the caps those runs actually chose, output hashed end to end: + +| sample | reads | cap | before | after | | md5 | +|---|---|---|---|---|---|---| +| Sample1 | 8,632,916 | 563 | 686 s | 93 s | 7.4x | identical | +| Sample2 | 6,243,115 | 502 | 493 s | 74 s | 6.7x | identical | +| Sample3 | 6,543,737 | 502 | 540 s | 82 s | 6.6x | identical | + +**28.6 minutes to 4.1 across the cohort.** Every kept/dropped tally matches to the record: `kept 8632916, dropped 60297` and so on, identical on both sides. + +Note how little is dropped -- 0.7%, 0.65%, 0.55%. Almost every read passes, so the old check loop almost never exited early and ran its full span on nearly every read. That is why real data lands near 7x where the dense synthetic reached 11x. + +## What this does to the pipeline's shape + +From the trace that started the investigation: + +| process | was | now | +|---|---|---| +| `AlignReads:Align` | 14.9 min | 14.9 min | +| **`VariantCalling:CapBAM`** | **10.6 min** | **~1.6 min** | +| `SortCleanBams:SortCleanBam` | 6.7 min | 6.7 min | +| `BuildDictionaries:BuildSnpEffDb` | 4.4 min | 4.4 min | + +**Capping goes from the second most expensive step to near the cheapest**, which removes the case for the two things the old note pointed at. Parallelising by chromosome is unnecessary. So is reaching for a tool outside awk: the BAM to SAM to BAM round trip is still there, and about three quarters of every line is `SEQ` and `QUAL` that the decision never reads, but a dependency bought to save another minute on a step that no longer costs one is a bad trade. + +## Coverage, which did not exist + +**`cap_depth.awk` shipped from 1.0 with no unit coverage at all.** The rewrite was verified against three real BAMs before a single case existed. Seven now sit in `03_helpers` (static, no JVM). + +**They assert the contract, not the implementation.** The first recomputes per-position depth from the *output* and requires that no position exceeds the cap -- which is what the helper promises and holds however the inside is written. Checked both ways: all seven pass against the old implementation and the new one, and each fails when the guard it covers is broken (off-by-one on the cap, a CIGAR walk that ignores `D`, the sequence reset, the header bypass, the no-reference-consuming bypass). + +Two cases were passing for the wrong reason first time round and both are recorded where they sit: + +- The fixture was built by concatenating `$(...)` substitutions, which **strip trailing newlines**, gluing the last record of one stack onto the first of the next. It is appended to a file now. +- The no-reference-consuming case put an unmapped record *between* the stack and the soft-clipped read. An unmapped record's `RNAME` is `*`, which triggers a new-sequence reset and clears the depth before the soft-clipped read is judged -- so the case passed with the guard deleted outright. The unmapped record is last now, where a sorted BAM puts it anyway. + +## Still open + +- **The input is never checked for being sorted.** Both the old code and the new go silently wrong on an unsorted stream, and the new one goes wrong differently: `while (low < pos)` simply does not run, leaving a stale total. `if (pos < low)` is one integer comparison per read and turns a silent wrong answer into a loud failure. Not built; not asked for. +- **`#!/usr/bin/awk -f` hardcodes the system awk.** On the machine these numbers came from that is gawk 5.4.1. The release's conda environment ships its own `awk` and it is never used, so this step's speed is set by whatever the host provides -- mawk on Debian-family hosts, gawk on RHEL -- and sits outside the pinned environment. A cluster may be getting quite different performance from anything measured here. +- **`reflen()` walks the CIGAR one character per read.** CIGAR strings repeat enormously, so memoizing by string would replace the loop with one lookup. Pure awk, no dependency, unmeasured. diff --git a/.claude/development-notes/change-guards.md b/.claude/development-notes/change-guards.md index e251db8..6c15c69 100644 --- a/.claude/development-notes/change-guards.md +++ b/.claude/development-notes/change-guards.md @@ -1,12 +1,12 @@ # The change guards -**Written 2026-08-31, against the tree at `7d65893`.** Both guards still work as described. The metadata file has gained columns since, and two of the additions sit awkwardly with what is below — noted in place. +**Written 2026-08-31, against the tree at `7d65893`.** Both guards still work as described. The metadata file has gained columns since, and two of the additions sit awkwardly with what is below -- noted in place. What stops a project from producing results under one configuration and then quietly continuing under another. Two guards, both in `scripts/0_verify_environment.nf`. -> **For the manual — and it is there now** (verified 2026-08-31). The files beside the results, what each one freezes, and what to delete are all documented, and `.poolseqflow_version` is described as *"A mismatch is a hard stop, not a warning"* in two places. This note used to say the documentation asserted the opposite; that was true of the pre-3.0 pages and is not true of the manual. +> **For the manual -- and it is there now** (verified 2026-08-31). The files beside the results, what each one freezes, and what to delete are all documented, and `.poolseqflow_version` is described as *"A mismatch is a hard stop, not a warning"* in two places. This note used to say the documentation asserted the opposite; that was true of the pre-3.0 pages and is not true of the manual. -## The reproducibility guard — one task for the whole project +## The reproducibility guard -- one task for the whole project Z, 2026-08-28: *"Copy the parameters.config and multirun.csv to .parameters.config and .multirun.csv, if they don't exist it is the first run, if they exist they can be compared in terms of what they contain."* And the rule those copies enforce: *"the parameter file being the same with what it was in the beginning and the parameters that are set for each run being kept as they are."* @@ -14,28 +14,28 @@ Z, 2026-08-28: *"Copy the parameters.config and multirun.csv to .parameters.conf | | How | Why | |---|---|---| -| `.multirun.csv` | **as written** | It is the user's own file, nothing in a release touches it, and every column is a deliberate divergence — so any edit is a change to the run set, full stop. | +| `.multirun.csv` | **as written** | It is the user's own file, nothing in a release touches it, and every column is a deliberate divergence -- so any edit is a change to the run set, full stop. | | `.parameters.config` | **by resolved value** | It also carries settings that cannot change a number. | The table being compared as written is also what makes a **regrouping** visible. `Shared_` numbers are assigned in order of appearance, so an edited table can leave `Shared_1` naming a different pair than the one whose results are in it. The table copy sees that directly instead of inferring it from a member list. -For the config, Z, 2026-08-28: *"Ignore resources and paths."* Moving a project to another disk or running it on a bigger node must not invalidate finished results — `mainDir`, `storageDir`, `threads`, `memory`, `cores.*`, `software.*`, `java.*`. That is exactly what `analysisParams()` excludes, so the comparison runs on its output and the raw file is stored beside it as the record of what was actually written. +For the config, Z, 2026-08-28: *"Ignore resources and paths."* Moving a project to another disk or running it on a bigger node must not invalidate finished results -- `mainDir`, `storageDir`, `threads`, `memory`, `cores.*`, `software.*`, `java.*`. That is exactly what `analysisParams()` excludes, so the comparison runs on its output and the raw file is stored beside it as the record of what was actually written. -The two together freeze every run's effective configuration: the base from the config, the per-run overrides from the table. **That is why this needs no per-run task and no per-directory manifest** — both of which it replaced. Two earlier attempts at this were wrong before the right one: a per-run manifest and then a per-directory one. +The two together freeze every run's effective configuration: the base from the config, the per-run overrides from the table. **That is why this needs no per-run task and no per-directory manifest** -- both of which it replaced. Two earlier attempts at this were wrong before the right one: a per-run manifest and then a per-directory one. -**The version is a block of its own**, checked first and short-circuiting everything else. Z, 2026-08-28: *"Nobody should ever resume to a pipeline using a different version. That needs a block on its own. Reset and re-run."* The reasoning is a citation argument — *"If they change their version midway, which version will they cite?"* This **reverses** the old rule, under which a version was recorded and never enforced. A project can no longer span two releases, and the whole added-by-a-release classification that existed to let it is gone with it. +**The version is a block of its own**, checked first and short-circuiting everything else. Z, 2026-08-28: *"Nobody should ever resume to a pipeline using a different version. That needs a block on its own. Reset and re-run."* The reasoning is a citation argument -- *"If they change their version midway, which version will they cite?"* This **reverses** the old rule, under which a version was recorded and never enforced. A project can no longer span two releases, and the whole added-by-a-release classification that existed to let it is gone with it. -## The metadata guard — keyed to the step-6 variant +## The metadata guard -- keyed to the step-6 variant What it asks is whether the file has been edited since the things that absorbed it were produced: the read group values, which step 4 bakes into each BAM, and the row order, which step 6 turns into the VCF's sample column order. So its answer depends on the **content of two directories**, and two runs may share the file and still have different ones. Step 6's key is the finer of the two and contains step 4's, so runs sharing it share both artifacts and therefore share one answer. -**Anything coarser breaks the guard itself.** The no-BAMs-and-no-VCF branch treats the situation as "nothing has consumed the file yet" and **records a new baseline** — so a run with no BAMs deciding for a run that has them would adopt the edit while the BAMs on disk still carried the old read groups. Every other wrong existence answer in this pipeline costs redundant work; this one costs the guard. +**Anything coarser breaks the guard itself.** The no-BAMs-and-no-VCF branch treats the situation as "nothing has consumed the file yet" and **records a new baseline** -- so a run with no BAMs deciding for a run that has them would adopt the edit while the BAMs on disk still carried the old read groups. Every other wrong existence answer in this pipeline costs redundant work; this one costs the guard. -**The probe looks in four places, not two.** Permanent storage as well as the working volume, and the producing *variant's* directories rather than the member's own. Once sharing was turned on the ready BAMs are promoted to `Output/All_Runs/Ready`, which no member's own `Output/` contains — so a guard looking only there answered 0 on every invocation after promotion and **had already stopped guarding**. That was a real dead guard, found by E1y. +**The probe looks in four places, not two.** Permanent storage as well as the working volume, and the producing *variant's* directories rather than the member's own. Once sharing was turned on the ready BAMs are promoted to `Output/All_Runs/Ready`, which no member's own `Output/` contains -- so a guard looking only there answered 0 on every invocation after promotion and **had already stopped guarding**. That was a real dead guard, found by E1y. -**What it projects, and therefore what it cannot see.** `metadataGuardLines()` writes the `RG_*` columns and the `param_*` columns, one line per sample. Everything else in the file is outside the comparison — which now includes the `exp_`, `pt_` and `cov_` columns the analysis layer reads. The failing branch tells the user *"Columns you added of your own are not compared, so this is a change to something the pipeline acted on"*, which was exactly true when the only things outside the projection were a user's own arbitrary columns. It is now also true of columns the project itself defines: a design column can be rewritten after a run, and a module will publish a design the results were never produced under while the identity check passes. Observed 2026-09-05 and recorded as a risk, not fixed. +**What it projects, and therefore what it cannot see.** `metadataGuardLines()` writes the `RG_*` columns and the `param_*` columns, one line per sample. Everything else in the file is outside the comparison -- which now includes the `exp_`, `pt_` and `cov_` columns the analysis layer reads. The failing branch tells the user *"Columns you added of your own are not compared, so this is a change to something the pipeline acted on"*, which was exactly true when the only things outside the projection were a user's own arbitrary columns. It is now also true of columns the project itself defines: a design column can be rewritten after a run, and a module will publish a design the results were never produced under while the identity check passes. Observed 2026-09-05 and recorded as a risk, not fixed. ## Three kinds of edit, three different remedies @@ -43,18 +43,18 @@ The guard distinguishes them because the fix is proportionate to what the edit i | Edit | What survives | What must go | |---|---|---| -| Row order only | the BAMs — read groups are matched by ID, not position | the called VCF and everything derived from it | +| Row order only | the BAMs -- read groups are matched by ID, not position | the called VCF and everything derived from it | | A tag or adapter value | nothing downstream of cleaning | the ready BAMs too | -| A pool size | the BAMs **and** the called VCF — it is a step 7 parameter | only what step 7 derived | +| A pool size | the BAMs **and** the called VCF -- it is a step 7 parameter | only what step 7 derived | The pool-size branch was added with E3b. Routing it through the tag branch would have told someone to delete every BAM and realign an entire project to change one number. -**A fourth override column arrived since, and it lands in exactly that trap.** `paramColumns()` holds four now — `param_poolSize`, `param_capMaxDepth`, `param_adapter1`, `param_adapter2` — and `metadataGuardLines()` projects all of them, so a `param_capMaxDepth` edit is seen. But there is no branch for it: it is not a reordering, it is not equal after `drop_size`, so it falls through to *"Read group or adapter values changed… Every BAM carries the old ones."* The capped BAM is transient — built at step 6 from the ready BAM and written to neither root — so changing a cap invalidates the called VCF and step 7's output and nothing before them. The remedy offered is the one the pool-size branch exists to avoid. Observed 2026-09-05, not fixed. +**A fourth override column arrived since, and it lands in exactly that trap.** `paramColumns()` holds four now -- `param_poolSize`, `param_capMaxDepth`, `param_adapter1`, `param_adapter2` -- and `metadataGuardLines()` projects all of them, so a `param_capMaxDepth` edit is seen. But there is no branch for it: it is not a reordering, it is not equal after `drop_size`, so it falls through to *"Read group or adapter values changed... Every BAM carries the old ones."* The capped BAM is transient -- built at step 6 from the ready BAM and written to neither root -- so changing a cap invalidates the called VCF and step 7's output and nothing before them. The remedy offered is the one the pool-size branch exists to avoid. Observed 2026-09-05, not fixed. -The row-order case is reported as **two orderings, not a line diff** — a diff of a permutation shows the same text as both removed and added, which reads as nonsense. The pool-size case is reported per pool for the same reason: a line diff would print the whole read group twice for every row and leave the reader to spot the one number that differs. +The row-order case is reported as **two orderings, not a line diff** -- a diff of a permutation shows the same text as both removed and added, which reads as nonsense. The pool-size case is reported per pool for the same reason: a line diff would print the whole read group twice for every row and leave the reader to spot the one number that differs. ## A portability trap in the pool-size branch -The condition asks whether the two baselines are equal once the pool size field is removed. That was written as `sed 's/\tparam_poolSize=[^\t]*//'` — and **`\t` in a sed PATTERN is a GNU extension**. BSD sed reads it as a literal `t`, without complaining. On macOS the comparison would never hold, the pool-size branch would never be reached, and a size edit would be reported as a read group change — telling that user, and only that user, to delete every BAM. +The condition asks whether the two baselines are equal once the pool size field is removed. That was written as `sed 's/\tparam_poolSize=[^\t]*//'` -- and **`\t` in a sed PATTERN is a GNU extension**. BSD sed reads it as a literal `t`, without complaining. On macOS the comparison would never hold, the pool-size branch would never be reached, and a size edit would be reported as a read group change -- telling that user, and only that user, to delete every BAM. Replaced with awk, which is portable and was already doing the rendering. `sed -i` without a suffix is also GNU-only but fails *loudly*, which is why the codebase gets away with it elsewhere. diff --git a/.claude/development-notes/check-split-and-layout.md b/.claude/development-notes/check-split-and-layout.md index df253f9..178800f 100644 --- a/.claude/development-notes/check-split-and-layout.md +++ b/.claude/development-notes/check-split-and-layout.md @@ -4,11 +4,11 @@ ## The shape problem Z named -`install/check_install.sh` had grown to check three things: the tools, the `bin/` helpers, and whether `parameters.config` parses — resolving the tool list out of `params.software` when a project was present and falling back to a canonical list when it was not. Taking `parameters.config` out of it (Z: *"We changed how install worked entirely. Checking parameters.config is just going to cause confusion"*) left a hardcoded `CANONICAL` list that duplicated the template's `software` block, and left nothing checking a project at all. +`install/check_install.sh` had grown to check three things: the tools, the `bin/` helpers, and whether `parameters.config` parses -- resolving the tool list out of `params.software` when a project was present and falling back to a canonical list when it was not. Taking `parameters.config` out of it (Z: *"We changed how install worked entirely. Checking parameters.config is just going to cause confusion"*) left a hardcoded `CANONICAL` list that duplicated the template's `software` block, and left nothing checking a project at all. Z, on being shown that: ***"It is not the error I'm concerned about it is the shape. The tools were checked against the params.config."*** -Moving `params.software` into `nextflow.config` was tried and **reverted** — Z: ***"Tools remain in params.config."*** The answer was two commands instead: ***"We need a separate check install and check project."*** +Moving `params.software` into `nextflow.config` was tried and **reverted** -- Z: ***"Tools remain in params.config."*** The answer was two commands instead: ***"We need a separate check install and check project."*** ## What each one is @@ -17,19 +17,19 @@ Moving `params.software` into `nextflow.config` was tried and **reverted** — Z | `check install` | the tools a release is built to run, **and that each comes from the release's own conda environment**; every helper in `bin/`. Reads no `parameters.config` and needs no project. | | `check project` | `parameters.config` is current for this release and parses; `metadata.csv` and the run table parse, through the parsers step 0 uses; and every command **as `params.software` names it**. | -**A bare `check` is refused.** Z chose this over keeping it as an alias: whichever one it picked would leave the other unchecked while reporting success. `check` is the second subcommand after `analysis` to carry a word of its own, which the argument block at the top of the wrapper had to be taught — every other subcommand takes none, so `check install` was rejected before reaching its arm. +**A bare `check` is refused.** Z chose this over keeping it as an alias: whichever one it picked would leave the other unchecked while reporting success. `check` is the second subcommand after `analysis` to carry a word of its own, which the argument block at the top of the wrapper had to be taught -- every other subcommand takes none, so `check install` was rejected before reaching its arm. **`run_check()` is a function and not a nested `case`.** Two suite cases read the wrapper's own case arms and compare them against its usage line; a nested `install)` at the same indentation is read as a top-level subcommand. The top-level usage says `check `, following the `analysis ` convention, because a nested `{install|project}` breaks a flat split on `|`. ## The environment check, which is the part that was silently wrong -`check_tool` asked `command -v` and accepted whatever `PATH` returned. **Every tool in `CANONICAL` is pinned in `install/environment.yml`** — verified, all fourteen including `gawk` and `python` — so one resolving from anywhere else means the environment is missing a package and the machine's own copy is standing in, at another version, on this machine only. The run works and reproduces nowhere. +`check_tool` asked `command -v` and accepted whatever `PATH` returned. **Every tool in `CANONICAL` is pinned in `install/environment.yml`** -- verified, all fourteen including `gawk` and `python` -- so one resolving from anywhere else means the environment is missing a package and the machine's own copy is standing in, at another version, on this machine only. The run works and reproduces nowhere. It now compares each resolved path against `CONDA_PREFIX` and reports `OUTSIDE THE ENVIRONMENT` as a failure. When the environment is not active there is nothing to compare against, and the header says so rather than checking `PATH` and calling that an answer. **`check project` deliberately does NOT apply that rule.** Repointing a tool at a system binary is a thing a project is allowed to do, and that check is where you see the result. -**A fixture that disabled the check under test.** The first version of these cases named the fake environment directory `env` while passing `ENV_NAME=check-install-env`; the script only compares paths when `basename $CONDA_PREFIX` matches `ENV_NAME`, so the comparison was off and both cases passed over nothing. The fixture directory is named for the environment now, and the passing case asserts the `from ` header line — which appears only when the script decided it knows which environment it is in, and is therefore what stops the case going vacuous again. +**A fixture that disabled the check under test.** The first version of these cases named the fake environment directory `env` while passing `ENV_NAME=check-install-env`; the script only compares paths when `basename $CONDA_PREFIX` matches `ENV_NAME`, so the comparison was off and both cases passed over nothing. The fixture directory is named for the environment now, and the passing case asserts the `from ` header line -- which appears only when the script decided it knows which environment it is in, and is therefore what stops the case going vacuous again. ## The directory move @@ -42,14 +42,14 @@ install/ the two pinned environment files, and nothing else citations/ the pipeline's own references.bib and the citations.json generated from it ``` -`analysis/` keeps its own `citations.json` and `references.bib`, and so does each module — **every separately publishable unit keeps its bib beside it**. The pipeline's pair had no home but `install/`, which is what `citations/` fixes. +`analysis/` keeps its own `citations.json` and `references.bib`, and so does each module -- **every separately publishable unit keeps its bib beside it**. The pipeline's pair had no home but `install/`, which is what `citations/` fixes. Two things the move broke and how: - **`check_install.sh` enumerates `bin/*` as pipeline helpers**, so it would have reported itself and its two siblings as helpers a run depends on. The three are skipped by name. -- **`07_analysis_frame`'s one-way-dependency case** greps `bin/` for any mention of the analysis layer, and `check_analysis_install.sh` is full of them. It is excluded **by name**, not by pattern, and a second case asserts exactly one file matches `bin/check_analysis*` — otherwise a rename or a second script widens the exclusion silently, since `--exclude` takes a glob and a name matching nothing is not an error. +- **`07_analysis_frame`'s one-way-dependency case** greps `bin/` for any mention of the analysis layer, and `check_analysis_install.sh` is full of them. It is excluded **by name**, not by pattern, and a second case asserts exactly one file matches `bin/check_analysis*` -- otherwise a rename or a second script widens the exclusion silently, since `--exclude` takes a glob and a name matching nothing is not an error. -`citations` was added to `PAYLOAD_ITEMS`, to the pipeline sandbox's copy list, and to `00_static`'s archive case — which now also asserts four **named files** and not only their directories, because a directory traveling empty satisfies every `assert_contains` on a path prefix. +`citations` was added to `PAYLOAD_ITEMS`, to the pipeline sandbox's copy list, and to `00_static`'s archive case -- which now also asserts four **named files** and not only their directories, because a directory traveling empty satisfies every `assert_contains` on a path prefix. ## State when this was written diff --git a/.claude/development-notes/concurrency.md b/.claude/development-notes/concurrency.md index 14f98d3..d98b1da 100644 --- a/.claude/development-notes/concurrency.md +++ b/.claude/development-notes/concurrency.md @@ -1,6 +1,6 @@ # Concurrency: two analysis modules deriving one intermediate -**Written 2026-09-01, describing work committed 48 minutes later in `77fdbd7`** — so the overhaul below is in that commit, not in the tree the note was written against. Both items under *Still owed before release* have been done since. Suite numbers are updated to the post-split names; the case names never changed. +**Written 2026-09-01, describing work committed 48 minutes later in `77fdbd7`** -- so the overhaul below is in that commit, not in the tree the note was written against. Both items under *Still owed before release* have been done since. Suite numbers are updated to the post-split names; the case names never changed. The analysis layer shares intermediates across **separate Nextflow invocations**. `Analysis/Main` is skip-by-existence: a module checks whether a derivation is already there, and builds it if not. Two modules started at the same time therefore race for the same output path, and nothing in Nextflow arbitrates between two processes it did not start. @@ -8,11 +8,11 @@ Until 2026-08-31 the plan recorded this as *"reasoned safe (.part then mv), NOT ## What was measured -`dev/`-free probes in a session scratchpad, against `bin/atomic_mv.sh` as it stands: 8 concurrent callers, one destination, 64 MB each, over 5–8 rounds. Two filesystem cases, because `mv` is a different operation in each — `rename(2)` within one filesystem, copy-then-unlink across two. Each caller wrote a file of one repeated byte, so a destination holding more than one distinct byte would be proof of interleaving rather than an inference from timing. A second probe ran a watcher sampling the destination's size throughout the race. +`dev/`-free probes in a session scratchpad, against `bin/atomic_mv.sh` as it stands: 8 concurrent callers, one destination, 64 MB each, over 5-8 rounds. Two filesystem cases, because `mv` is a different operation in each -- `rename(2)` within one filesystem, copy-then-unlink across two. Each caller wrote a file of one repeated byte, so a destination holding more than one distinct byte would be proof of interleaving rather than an inference from timing. A second probe ran a watcher sampling the destination's size throughout the race. | | as it stands | with the fix below | |---|---|---| -| callers that failed | **5–8 of 8, every round** | 0 of 8, every round | +| callers that failed | **5-8 of 8, every round** | 0 of 8, every round | | destination absent after all callers finished | **1 of 5 cross-filesystem rounds** | 0 | | destination present and **incomplete**, seen by a watcher | **10 sightings across 5 rounds**, sizes from 49 MB of 64 | 0 across 8 rounds | | content interleaved (two bytes in one file) | never observed | never observed | @@ -20,7 +20,7 @@ Until 2026-08-31 the plan recorded this as *"reasoned safe (.part then mv), NOT ## Why -`atomic_mv.sh` stages through `${DEST}.part` — a name derived **only from the destination**, so every concurrent caller uses the same temp file — and installs `trap 'rm -rf -- "${DEST}.part"' EXIT`. +`atomic_mv.sh` stages through `${DEST}.part` -- a name derived **only from the destination**, so every concurrent caller uses the same temp file -- and installs `trap 'rm -rf -- "${DEST}.part"' EXIT`. Two consequences, both measured: @@ -29,7 +29,7 @@ Two consequences, both measured: The second is the dangerous one. It is silent, it produces a wrong number rather than an error, and the artifact it leaves behind is wrong for good. -None of this contradicts the script's own docstring: *"NO LOCKING. The caller is responsible for one task per artifact path."* Within the pipeline Nextflow guarantees that. **The analysis layer's cross-invocation sharing does not**, which is the gap — the script is being asked for something it never promised. +None of this contradicts the script's own docstring: *"NO LOCKING. The caller is responsible for one task per artifact path."* Within the pipeline Nextflow guarantees that. **The analysis layer's cross-invocation sharing does not**, which is the gap -- the script is being asked for something it never promised. ## The fix, and what it is worth @@ -44,11 +44,11 @@ mv "$STAGE/item" "$DEST" Measured against the same probes: **0 failures out of 8 callers, 0 partial sightings across 8 rounds, no leftovers.** Every caller copies to a name nobody else touches, and the final `rename(2)` is atomic within the destination filesystem, so the destination is only ever absent or complete. -**Landed 2026-08-31**, ahead of E6 — Z lifted the freeze for it, because it blocks the intermediates E4b-mod is about to build. +**Landed 2026-08-31**, ahead of E6 -- Z lifted the freeze for it, because it blocks the intermediates E4b-mod is about to build. **`mktemp` and not `$$`.** A PID is unique among live processes on one machine, and two nodes of a cluster sharing `mainDir` can hold the same one. A guarantee was available, so a near-certainty was not worth taking. -**A staging DIRECTORY and not a temp file, and this is the part that bit.** The first fix used `mktemp "${DEST}.part.XXXXXXXX"`, which passed every probe, every helper test and `--fast` — and then failed all 40 cases of `03_pipeline`, because **`atomic_mv.sh` also moves directories**: step 1 moves a whole snpEff database with it. `mv` refuses to overwrite a file with a directory, and the script's own usage line said ``, which is how the assumption survived. The usage line now says the source may be either, and `test_atomic_mv_moves_a_directory` in `05_helpers` covers it. +**A staging DIRECTORY and not a temp file, and this is the part that bit.** The first fix used `mktemp "${DEST}.part.XXXXXXXX"`, which passed every probe, every helper test and `--fast` -- and then failed all 40 cases of `03_pipeline`, because **`atomic_mv.sh` also moves directories**: step 1 moves a whole snpEff database with it. `mv` refuses to overwrite a file with a directory, and the script's own usage line said ``, which is how the assumption survived. The usage line now says the source may be either, and `test_atomic_mv_moves_a_directory` in `05_helpers` covers it. The lesson is narrower than "run the tests": the probes and the unit tests were all built around **files**, because the bug being fixed was about files, and a fix verified only against the shape of its own bug report is a fix verified against half the callers. @@ -57,7 +57,7 @@ The lesson is narrower than "run the tests": the probes and the unit tests were ## What the fix does not buy - **Duplicated work stays.** Two modules deriving the same intermediate both derive it. That is accepted: the alternative is a lock, and a stale lock file is worse than a repeated computation. -- **Last writer wins.** Safe only because a shared derivation is byte-identical whoever produced it — which is exactly why the library takes decisions as required arguments and `Analysis/Main` keys the filename on them. `freqmatrix_pairwise.tsv` is safe to race for; `freqmatrix.tsv` would not be, because two modules could legitimately produce different bytes for one name. The naming rule and the concurrency guarantee are the same rule. +- **Last writer wins.** Safe only because a shared derivation is byte-identical whoever produced it -- which is exactly why the library takes decisions as required arguments and `Analysis/Main` keys the filename on them. `freqmatrix_pairwise.tsv` is safe to race for; `freqmatrix.tsv` would not be, because two modules could legitimately produce different bytes for one name. The naming rule and the concurrency guarantee are the same rule. - **The measurement is local.** tmpfs against tmpfs, and tmpfs against tmpfs on another mount. `rename(2)` atomicity is a POSIX guarantee within one filesystem, but NFS and some network filesystems weaken it. A user running `mainDir` on NFS is outside what was measured, and nothing here detects that. ## Reproducing it @@ -69,7 +69,7 @@ The probes were written in a session scratchpad and are not tracked; they are tw --- -## 2026-09-01 — a SECOND defect in the same file, and this one destroyed the source +## 2026-09-01 -- a SECOND defect in the same file, and this one destroyed the source Found while scoping `PoolSeqFlow analysis complete`, by an adversarial pass over the design rather than by a test. The August fix bought *"the destination is only ever absent or complete"* and its "What the fix does not buy" section never mentioned the source. It should have. @@ -84,73 +84,73 @@ mv "$STAGE/item" "$DEST" Between those two lines the staged copy is **the only copy**, and the `EXIT` trap deletes it. -### Measured, on this machine, `/dev/shm` → `/tmp` (two tmpfs mounts, so `mv` takes its cross-device path) +### Measured, on this machine, `/dev/shm` -> `/tmp` (two tmpfs mounts, so `mv` takes its cross-device path) | probe | before | after | |---|---|---| -| the final rename fails — **no signal, no race** | source GONE, destination GONE, exit 1 | source intact, destination absent | +| the final rename fails -- **no signal, no race** | source GONE, destination GONE, exit 1 | source intact, destination absent | | SIGTERM to the script, 900 MB tree, mid-copy | source **124 of 300 files**, destination GONE | source 300 of 300 | | SIGTERM to the process group, same | source 300 (safe by luck: `mv` was killed too) | source 300 | -| SIGTERM to the script, 900 MB single file, ×5 | **2 of 5 TOTAL LOSS** | 0 of 5 | +| SIGTERM to the script, 900 MB single file, x5 | **2 of 5 TOTAL LOSS** | 0 of 5 | -**The no-signal case is the one that matters.** A destination that cannot be written when the rename runs — a full volume, a quota, a read-only mount — destroys the artifact and exits 1. Nothing about it is a race. +**The no-signal case is the one that matters.** A destination that cannot be written when the rename runs -- a full volume, a quota, a read-only mount -- destroys the artifact and exits 1. Nothing about it is a race. -The signal cases have a mechanism worth recording: bash defers a trapped signal until the current foreground command returns, so a SIGTERM arriving during `mv "$SRC" "$STAGE/item"` **guarantees** that `mv` completes — source unlinked — before the `EXIT` trap runs and deletes the staged copy. Killing the whole process group is safer only because `mv` dies too. +The signal cases have a mechanism worth recording: bash defers a trapped signal until the current foreground command returns, so a SIGTERM arriving during `mv "$SRC" "$STAGE/item"` **guarantees** that `mv` completes -- source unlinked -- before the `EXIT` trap runs and deletes the staged copy. Killing the whole process group is safer only because `mv` dies too. -### The fix, as first written — superseded the same day by the redesign below +### The fix, as first written -- superseded the same day by the redesign below The invariant is **the source is removed only once the destination exists, complete, under its final name.** Worst case is the artifact in **both** roots, which `find_artifact.sh` already reports and `PromoteArtifacts` already asserts against. That invariant survives into the final version; the machinery around it did not. Two paths, and which one runs is asked of the kernel: -- **Within one filesystem** — a single `rename(2)`. The artifact keeps its inode and acquires the new name, so no second copy exists that could be wrong and there is nothing to verify. Confirmed by inode: 50 MB in 10 ms, unchanged inode, nothing staged. -- **Across filesystems** — `cp -a` into the caller's own staging directory, **both sides hashed and compared**, rename into place, `trap - EXIT`, then the source. +- **Within one filesystem** -- a single `rename(2)`. The artifact keeps its inode and acquires the new name, so no second copy exists that could be wrong and there is nothing to verify. Confirmed by inode: 50 MB in 10 ms, unchanged inode, nothing staged. +- **Across filesystems** -- `cp -a` into the caller's own staging directory, **both sides hashed and compared**, rename into place, `trap - EXIT`, then the source. -**`mv` is not used for either.** Given two names on different filesystems `mv` answers by copying **onto the destination's own name**, and an interrupted write there is a partial file every skip check reads as finished — the August bug, reintroduced through the back door. The first version of this fix predicted the filesystem with `stat -c %d` and then called `mv`, which is a guess about what the kernel will do; `os.rename` raises `EXDEV` and never falls back, so the fast path is taken only when it is genuinely a rename. +**`mv` is not used for either.** Given two names on different filesystems `mv` answers by copying **onto the destination's own name**, and an interrupted write there is a partial file every skip check reads as finished -- the August bug, reintroduced through the back door. The first version of this fix predicted the filesystem with `stat -c %d` and then called `mv`, which is a guess about what the kernel will do; `os.rename` raises `EXDEV` and never falls back, so the fast path is taken only when it is genuinely a rename. -**The hash comparison is Z's, 2026-09-01**, on the grounds that data integrity is worth the cost. Its reach is narrower than it looks and worth stating: `cp` already fails loudly on ENOSPC, so `set -e` catches a full volume before the hash runs, and reading the copy back immediately usually comes from page cache rather than the platter — so this does not certify storage. What it does catch, reproduced: **a source that changed while it was being copied** (exit 1, destination never written, source intact), and a `cp` that returns 0 with wrong bytes. A directory is reduced to one digest over every entry's type and path, every symlink's target and every file's contents, so a dropped empty directory and a truncated member are both differences. +**The hash comparison is Z's, 2026-09-01**, on the grounds that data integrity is worth the cost. Its reach is narrower than it looks and worth stating: `cp` already fails loudly on ENOSPC, so `set -e` catches a full volume before the hash runs, and reading the copy back immediately usually comes from page cache rather than the platter -- so this does not certify storage. What it does catch, reproduced: **a source that changed while it was being copied** (exit 1, destination never written, source intact), and a `cp` that returns 0 with wrong bytes. A directory is reduced to one digest over every entry's type and path, every symlink's target and every file's contents, so a dropped empty directory and a truncated member are both differences. -**The cost is real and was accepted deliberately.** A cross-filesystem move now makes four passes over the data — read source, write copy, read source again, read copy — where `mv` made two. Same-filesystem promotion is unaffected, which is why the rename path is worth keeping rather than folding into one uniform code path: a rename creates no copy, so verification there would compare a file with itself. +**The cost is real and was accepted deliberately.** A cross-filesystem move now makes four passes over the data -- read source, write copy, read source again, read copy -- where `mv` made two. Same-filesystem promotion is unaffected, which is why the rename path is worth keeping rather than folding into one uniform code path: a rename creates no copy, so verification there would compare a file with itself. -The same commit closes a second hole in the same file: the trailing-slash form resolved `dst/` to `dst/` and never re-tested it, so an existing directory of that name swallowed the source as `dst//item` — exit 0, source gone, artifact under a name no skip check looks for. The `-d` refusal now runs after the resolution, so both spellings reach it. +The same commit closes a second hole in the same file: the trailing-slash form resolved `dst/` to `dst/` and never re-tested it, so an existing directory of that name swallowed the source as `dst//item` -- exit 0, source gone, artifact under a name no skip check looks for. The `-d` refusal now runs after the resolution, so both spellings reach it. ### Why no test covers the cross-device path -`TEST_TMPDIR` is a single filesystem and `guard_path` refuses anything outside it, so the suite can only reach the same-filesystem path. The two new `05_helpers` cases cover what is reachable: the directory-destination refusal (deterministic, and it loses the source on the old code) and the inode assertion that the same-filesystem path is a rename. **The cross-device evidence is the table above and nothing in CI reproduces it.** Closing that needs either a second mount point in the harness or a documented `guard_path` exception — a decision, not an oversight. +`TEST_TMPDIR` is a single filesystem and `guard_path` refuses anything outside it, so the suite can only reach the same-filesystem path. The two new `05_helpers` cases cover what is reachable: the directory-destination refusal (deterministic, and it loses the source on the old code) and the inode assertion that the same-filesystem path is a rename. **The cross-device evidence is the table above and nothing in CI reproduces it.** Closing that needs either a second mount point in the harness or a documented `guard_path` exception -- a decision, not an oversight. ### The method note, which is the same one as last time -August measured the destination because the bug report was about the destination. This defect was in the same four lines the whole time. *"A fix verified only against the shape of its own bug report is verified against half the callers"* — and the half that went unexamined was the half where the data comes from. +August measured the destination because the bug report was about the destination. This defect was in the same four lines the whole time. *"A fix verified only against the shape of its own bug report is verified against half the callers"* -- and the half that went unexamined was the half where the data comes from. --- -## 2026-09-01, later — the overhaul, and the diagnosis that made it small +## 2026-09-01, later -- the overhaul, and the diagnosis that made it small -Z stopped the patching: *"I believe the script is prone for errors… we are now patching it for every edge case. This means the main logic does not cover edge cases… When this happens, I overhaul the whole script. Simplicity is the key."* +Z stopped the patching: *"I believe the script is prone for errors... we are now patching it for every edge case. This means the main logic does not cover edge cases... When this happens, I overhaul the whole script. Simplicity is the key."* Three guards had accreted in two days: a destination that is already a directory, a directory source onto a file destination, and the trailing-slash form resolving a destination without re-testing it. ### The cause was not the contract -My diagnosis was that the destination argument was ambiguous — file or directory to move into — and that the ambiguity generated the guards. That is the symptom. **The cause is that two different things wrote the destination name:** `rename(2)` on the fast path, and `mv` on the staged one. `mv` carries a userspace convention the kernel does not — *a destination that is a directory means put the source inside it* — and every guard was a hand-rolled re-implementation of a rule the kernel already enforces. +My diagnosis was that the destination argument was ambiguous -- file or directory to move into -- and that the ambiguity generated the guards. That is the symptom. **The cause is that two different things wrote the destination name:** `rename(2)` on the fast path, and `mv` on the staged one. `mv` carries a userspace convention the kernel does not -- *a destination that is a directory means put the source inside it* -- and every guard was a hand-rolled re-implementation of a rule the kernel already enforces. Measured, with both sides checked afterwards: | | `rename(2)` | `mv` | |---|---|---| -| file → existing file | replaces | replaces | -| file → existing directory | **EISDIR**, both sides untouched | nests it inside, exit 0 | -| directory → existing file | **ENOTDIR**, both sides untouched | refuses | -| directory → existing non-empty directory | **ENOTEMPTY**, both sides untouched | refuses | -| directory → existing empty directory | replaces | refuses | +| file -> existing file | replaces | replaces | +| file -> existing directory | **EISDIR**, both sides untouched | nests it inside, exit 0 | +| directory -> existing file | **ENOTDIR**, both sides untouched | refuses | +| directory -> existing non-empty directory | **ENOTEMPTY**, both sides untouched | refuses | +| directory -> existing empty directory | replaces | refuses | -Make `rename(2)` the only writer and there is nothing left to enumerate. **The trailing-slash form was never the problem** — it was a guard sitting *below* the resolution. With no guard downstream, both spellings traverse identical code, which `test_atomic_mv_both_destination_spellings_agree` now asserts. +Make `rename(2)` the only writer and there is nothing left to enumerate. **The trailing-slash form was never the problem** -- it was a guard sitting *below* the resolution. With no guard downstream, both spellings traverse identical code, which `test_atomic_mv_both_destination_spellings_agree` now asserts. ### What the file became -`place()` — `os.rename` via `python3`, exit 9 for EXDEV — is the only writer, used for the direct move and for the staged one. Across filesystems: `rsync -a` into the caller's own staging directory, `diff -qr --no-dereference` against the source, `place()` into the destination, then the source. +`place()` -- `os.rename` via `python3`, exit 9 for EXDEV -- is the only writer, used for the direct move and for the staged one. Across filesystems: `rsync -a` into the caller's own staging directory, `diff -qr --no-dereference` against the source, `place()` into the destination, then the source. -- **Destination or source type tests: 3 → 0.** Code lines 79 → 47. +- **Destination or source type tests: 3 -> 0.** Code lines 79 -> 47. - **Call sites changed: 0.** The trailing-slash form stays; Z's earlier decision to drop it was taken from the wrong diagnosis and reversed once the right one landed. - **`trap - EXIT` and the paired `rmdir` are gone.** The trap now runs unconditionally, because after the final rename the staging directory is empty and removing it costs nothing. The arm/disarm coordination that hosted the September data-loss bug has no replacement rather than a safer version. - **`digest()` is gone**, and with it a dependency on `findutils` (`find -printf`, `xargs`) that was never pinned, and a carried-over bug where a symlink source failed across filesystems because the digest followed the link. `test_atomic_mv_moves_a_symlink_across_filesystems` guards the fix. @@ -160,27 +160,27 @@ Make `rename(2)` the only writer and there is nothing left to enumerate. **The t Z's call, measured before adopting. **rsync gives the verification and not the atomicity:** - A single file: writes `.name.XXXXXX` and renames. Destination name never partial. -- **A directory: no whole-tree atomicity — 101 of 200 files were visible under the final path mid-transfer.** So the staging directory and the final rename stay. rsync writes into the stage, where partial state is invisible. -- **`--remove-source-files` must never be used.** It deletes the source when rsync's transfer succeeds, which is before the rename into place — exactly the September bug. +- **A directory: no whole-tree atomicity -- 101 of 200 files were visible under the final path mid-transfer.** So the staging directory and the final rename stay. rsync writes into the stage, where partial state is invisible. +- **`--remove-source-files` must never be used.** It deletes the source when rsync's transfer succeeds, which is before the rename into place -- exactly the September bug. -**RSYNC DOES NOT VERIFY, and I claimed it did.** The claim came from a probe that used `--remove-source-files`, where a source mutated mid-transfer produced exit 23 — that is rsync declining to *delete* a changed source, not detecting a bad transfer. Isolated afterwards, flag against version: +**RSYNC DOES NOT VERIFY, and I claimed it did.** The claim came from a probe that used `--remove-source-files`, where a source mutated mid-transfer produced exit 23 -- that is rsync declining to *delete* a changed source, not detecting a bad transfer. Isolated afterwards, flag against version: | | exit | copy vs source | |---|---|---| | `rsync -a`, 3.4.4 and 3.5.0 alike | **0, silent** | **DIFFERENT** | | `rsync -a --remove-source-files`, both | 23 | DIFFERENT | -Plain `rsync -a` copies a source that is being written, reports success and says nothing. **So `diff -qr --no-dereference` is not a second opinion on top of rsync — it is the only verification there is**, and without it the source would be removed after an unchecked copy. Z asked for it on instinct; it is load-bearing. Measured to catch a changed file, a missing file, an extra file, a dropped empty directory and a retargeted symlink. It does not compare file modes, which `rsync -a` owns. +Plain `rsync -a` copies a source that is being written, reports success and says nothing. **So `diff -qr --no-dereference` is not a second opinion on top of rsync -- it is the only verification there is**, and without it the source would be removed after an unchecked copy. Z asked for it on instinct; it is load-bearing. Measured to catch a changed file, a missing file, an extra file, a dropped empty directory and a retargeted symlink. It does not compare file modes, which `rsync -a` owns. The general lesson is the one this note already carries twice: **a tool's exit code answers the question the tool was asked, not the question you had in mind.** The flag under test was not the flag whose behavior was being attributed. ### What it gives up -**Cross-filesystem type refusals now happen after the copy.** `EXDEV` outranks `EISDIR` in the kernel, so no trial rename can report the type problem first, and moving a directory onto a file across volumes copies the whole tree before failing. The outcome is identical — exit 1, source kept, destination untouched, nothing staged — only the wasted work differs, on a path that always aborts. `test_atomic_mv_refusals_reach_the_same_outcome_across_filesystems` pins it. +**Cross-filesystem type refusals now happen after the copy.** `EXDEV` outranks `EISDIR` in the kernel, so no trial rename can report the type problem first, and moving a directory onto a file across volumes copies the whole tree before failing. The outcome is identical -- exit 1, source kept, destination untouched, nothing staged -- only the wasted work differs, on a path that always aborts. `test_atomic_mv_refusals_reach_the_same_outcome_across_filesystems` pins it. **A directory now replaces an existing EMPTY directory** instead of being refused. Nothing is lost and no caller reaches it, but it is a behavior change and `test_atomic_mv_replaces_an_empty_directory` asserts it rather than leaving it to be discovered. -### Still owed before release — both done since +### Still owed before release -- both done since - **`rsync` was in neither `install/environment.yml` nor `parameters.config.template`'s `software` block**, so step 0 did not verify it and a missing rsync failed deep in a task at promotion time rather than at step 0, which is what step 0 exists to prevent. It is pinned as `rsync=3.4.4=hffd6c76_1` now and has a `software` slot, so step 0 checks it. - **`findutils` and `diffutils` were unpinned** while `coreutils`, `gawk`, `grep` and `sed` were. Both carry exact pins now. `diffutils` is the one that mattered: `diff` is load-bearing here and in the analysis layer's provenance check. diff --git a/.claude/development-notes/config-migration.md b/.claude/development-notes/config-migration.md index c2c5fbf..aba979b 100644 --- a/.claude/development-notes/config-migration.md +++ b/.claude/development-notes/config-migration.md @@ -1,6 +1,6 @@ -# `bin/config_migrate.sh` — the 2.2.0 → 3.0.0 pass +# `bin/config_migrate.sh` -- the 2.2.0 -> 3.0.0 pass -**Written 2026-08-31, against the tree at `7d65893`.** The pass it describes is done and the five DROPPED keys are still five. The template has grown since — 97 live assignments then, 101 now — so the diff figures below are a snapshot of 2026-08-30 rather than a current count. +**Written 2026-08-31, against the tree at `7d65893`.** The pass it describes is done and the five DROPPED keys are still five. The template has grown since -- 97 live assignments then, 101 now -- so the diff figures below are a snapshot of 2026-08-30 rather than a current count. Done 2026-08-30, the last pipeline-side work before the analysis layer. `change-guards.md` covers what stops a project changing configuration mid-analysis; this covers carrying a configuration *forward* across a release. @@ -15,52 +15,52 @@ git show v2.2.0:parameters.config.template > /tmp/t220.config # awk that tracks `name {` / `}` depth and emits scope.scope.keyvalue ``` -**90 keys → 97. 27 gone, 34 new.** Doing it by eye would have missed the two that mattered, because neither shows up as a changed value. +**90 keys -> 97. 27 gone, 34 new.** Doing it by eye would have missed the two that mattered, because neither shows up as a changed value. ## What the diff said -**Every parameter whose default differs is a `${...}` expression** — the whole `dir.*` tree, `gffPath`, `reference`, `referencePath`, and `mainDir`'s placeholder. Those already take the COMPUTED branch, where the template wins and the user is told. That is most of the diff and it needed nothing. +**Every parameter whose default differs is a `${...}` expression** -- the whole `dir.*` tree, `gffPath`, `reference`, `referencePath`, and `mainDir`'s placeholder. Those already take the COMPUTED branch, where the template wins and the user is told. That is most of the diff and it needed nothing. **One genuine `reformatted()` case: `variantCall.maxDepth`.** 2.2.0 shipped `bcftools.maxDepth = 2000`; the prefix rename would carry it into 3.0, where the knob means something else. Details below. **One misnamed parameter, left alone: `varQualMin`.** Details below. -## `variantCall.maxDepth` — the value is not the change +## `variantCall.maxDepth` -- the value is not the change -In 2.2.0 this was the only depth control in the pipeline: one number, every pileup. From 3.0, step 5 measures a ceiling per sample from its own depth histogram and step 6 applies it to the BAM *before* calling, so `variantCall.maxDepth` is a second ceiling over that one and ships as `0` — which `mpileup` reads as **no limit at all**, measured. +In 2.2.0 this was the only depth control in the pipeline: one number, every pileup. From 3.0, step 5 measures a ceiling per sample from its own depth histogram and step 6 applies it to the BAM *before* calling, so `variantCall.maxDepth` is a second ceiling over that one and ships as `0` -- which `mpileup` reads as **no limit at all**, measured. Carrying `2000` forward therefore leaves a project capped at a number the release never chose, on top of a per-sample cap that cannot see it. The prefix rename would have done exactly that silently, which is what `reformatted()` exists to stop. -The report entry alone was not enough — the mechanism moved, not the value — so the epilogue explains it and gives the exact way back: `variantCall.maxDepth = 2000` with `capBAM.maxDepth = 0`. Someone reproducing old numbers needs that, and no report line fits it. +The report entry alone was not enough -- the mechanism moved, not the value -- so the epilogue explains it and gives the exact way back: `variantCall.maxDepth = 2000` with `capBAM.maxDepth = 0`. Someone reproducing old numbers needs that, and no report line fits it. -## `varQualMin` is misnamed, and stays that way — REVERTED 2026-08-30 +## `varQualMin` is misnamed, and stays that way -- REVERTED 2026-08-30 -`bcftools mpileup -q` is the **mapping** quality minimum and `-Q` is base quality. 2.2.0 fixed half of this: the CHANGELOG records `baseQualMin` and `varQualMin` being "the wrong way round", after which `baseQualMin` correctly fed `-Q`. But `varQualMin` was left feeding `-q`, so it sets mapping quality under a name that says variant quality — and there is no variant-quality option in `mpileup` at all. The manual documents it truthfully as mapping quality; only the name lies. +`bcftools mpileup -q` is the **mapping** quality minimum and `-Q` is base quality. 2.2.0 fixed half of this: the CHANGELOG records `baseQualMin` and `varQualMin` being "the wrong way round", after which `baseQualMin` correctly fed `-Q`. But `varQualMin` was left feeding `-q`, so it sets mapping quality under a name that says variant quality -- and there is no variant-quality option in `mpileup` at all. The manual documents it truthfully as mapping quality; only the name lies. It was renamed to `mapQualMin` and **Z reversed that within the hour**: *"Don't the variable name change. I worry that there will be something will be there in the documents stray."* -Worth recording, because the reasoning generalises past this parameter. A rename of a shipped parameter touches the template, `resolve_parameters.nf`, the manual in several unrelated sections, the migration table, and the tests — and the manual is 3000 lines. The cost is not the edit, it is the confidence that nothing was missed, and it buys only a better name for something already documented correctly. **On the eve of freezing the pipeline that trade is wrong**, whatever it would be worth at the start of a release cycle. +Worth recording, because the reasoning generalises past this parameter. A rename of a shipped parameter touches the template, `resolve_parameters.nf`, the manual in several unrelated sections, the migration table, and the tests -- and the manual is 3000 lines. The cost is not the edit, it is the confidence that nothing was missed, and it buys only a better name for something already documented correctly. **On the eve of freezing the pipeline that trade is wrong**, whatever it would be worth at the start of a release cycle. The revert also took out machinery: `renamed()` had grown to return a space-separated candidate list, because `mapQualMin` was the first parameter renamed twice and needed both `variantCall.varQualMin` and `bcftools.varQualMin`. With no double rename left, that generality is unused and speculative, so the single-name lookup came back with it. Four lines to add again when a real double rename arrives. **If it is ever revisited**, the options are: rename to `mapQualMin` (behavior-neutral, the flags already match the manual); or swap the flags so the names become true, which **changes results** for anyone who tuned the two apart. Only the first is safe. -## The KNOB category — 13 of 27 "losses" were not losses +## The KNOB category -- 13 of 27 "losses" were not losses -A 2.2.0 config has `cores.*` (eight) and five option strings (`bwa.options`, `fastqc.options`, `trim_galore.options`, `trim_galore.adapterOptions`, `bcftools.mpileupOptions`) as **live assignments**. In 3.0 all thirteen are computed and ship **commented out** in the template — still there, still settable. +A 2.2.0 config has `cores.*` (eight) and five option strings (`bwa.options`, `fastqc.options`, `trim_galore.options`, `trim_galore.adapterOptions`, `bcftools.mpileupOptions`) as **live assignments**. In 3.0 all thirteen are computed and ship **commented out** in the template -- still there, still settable. -Reported as DROPPED they read as gone for good, and the worst thing a migration report can do is send someone looking for a setting that is sitting in their new config. Pass 2 now records commented-out assignments as knobs, and `forward()` maps an old key onto the current scope names (`bcftools.` → `variantCall.`) so `bcftools.mpileupOptions` finds `variantCall.mpileupOptions`. +Reported as DROPPED they read as gone for good, and the worst thing a migration report can do is send someone looking for a setting that is sitting in their new config. Pass 2 now records commented-out assignments as knobs, and `forward()` maps an old key onto the current scope names (`bcftools.` -> `variantCall.`) so `bcftools.mpileupOptions` finds `variantCall.mpileupOptions`. -`DROPPED` is now five: `dir.scripts`, `gff`, `dir.output.temp`, `rgTagsFile`, `rgTagsPath` — all genuinely gone. +`DROPPED` is now five: `dir.scripts`, `gff`, `dir.output.temp`, `rgTagsFile`, `rgTagsPath` -- all genuinely gone. -## RGTags.csv → metadata.csv stays a DROP, with a message +## RGTags.csv -> metadata.csv stays a DROP, with a message -Tempting to add `metadataFile` ← `rgTagsFile` to `renamed()`. **Do not.** The formats are incompatible, so carrying the filename would point `metadataFile` at a file that cannot parse, and `reformatted()` on top would report it as handled when the user has real work to do. +Tempting to add `metadataFile` <- `rgTagsFile` to `renamed()`. **Do not.** The formats are incompatible, so carrying the filename would point `metadataFile` at a file that cannot parse, and `reformatted()` on top would report it as handled when the user has real work to do. The script already reads `rgTagsFile` from the *backup* to emit a `mv` for the file itself. So the parameter drops, the file is moved, and the epilogue explains that metadata.csv is not a rename: it decides which rows merge into one pool and can set pool size and sensitivity per sample. Verified before writing it that step 0 fails with `METADATA FILE CHECK: FAIL` when the file is absent (`0_verify_environment.nf`), so the note can say so. ## Traps -**No apostrophes in the awk program.** It is single-quoted inside the shell script, and one `'` in a comment ends it — `bash -n` then fails somewhere else entirely. Cost a debugging round here on the word `key's`, and it is already recorded as a project-wide trap. +**No apostrophes in the awk program.** It is single-quoted inside the shell script, and one `'` in a comment ends it -- `bash -n` then fails somewhere else entirely. Cost a debugging round here on the word `key's`, and it is already recorded as a project-wide trap. -**`migrated_value` in the test suite takes the FIRST match.** `maxDepth` is defined in `capBAM` *and* `variantCall`, and `capBAM` comes first in the template. An existing case asserted `bcftools.maxDepth should land in variantCall.maxDepth` while actually reading `capBAM`'s — its `sed` set both to the same number, so it passed either way. `migrated_value_in ` now exists and that case was moved onto `baseQualMin`. +**`migrated_value` in the test suite takes the FIRST match.** `maxDepth` is defined in `capBAM` *and* `variantCall`, and `capBAM` comes first in the template. An existing case asserted `bcftools.maxDepth should land in variantCall.maxDepth` while actually reading `capBAM`'s -- its `sed` set both to the same number, so it passed either way. `migrated_value_in ` now exists and that case was moved onto `baseQualMin`. diff --git a/.claude/development-notes/dag-wiring.md b/.claude/development-notes/dag-wiring.md index cf64e7c..af72268 100644 --- a/.claude/development-notes/dag-wiring.md +++ b/.claude/development-notes/dag-wiring.md @@ -1,4 +1,4 @@ -# The DAG — `poolseqflow.nf` +# The DAG -- `poolseqflow.nf` **Written 2026-08-31, against the tree at `7d65893`.** The nine Nextflow behaviors and the wiring below all still hold. One detail has moved: which variables are declared without `def`, corrected in place. @@ -10,23 +10,23 @@ Every one of these was learned by getting it wrong. ### A workflow cannot be invoked twice -Nextflow answers *"Process 'X' has been already used"*. Aliasing is the supported way round it whenever the number of call sites is known while the script is read — which is the case here: the DAG's shape is fixed, and only multi-run's N comes from data. Hence one alias per attachment point. +Nextflow answers *"Process 'X' has been already used"*. Aliasing is the supported way round it whenever the number of call sites is known while the script is read -- which is the case here: the DAG's shape is fixed, and only multi-run's N comes from data. Hence one alias per attachment point. ### A `def` local of the workflow body is invisible to a closure The closure resolves its names against the **script binding**, finds nothing, and dies on `Cannot get property 'dir' on null object`. So a name a closure has to see is declared **without** `def`, deliberately, which makes it a binding variable. -Which names those are has changed since: today it is `plan` and `log_dirs`, and `log_dirs` is the one that proves the rule, because `workflow.onComplete { assembleCombinedLogs(log_dirs) }` is the closure. `run_defs` carries a `def` and is fine — nothing outside the workflow body reads it. The rule is about where a name is read from, not about which names happen to need it today. +Which names those are has changed since: today it is `plan` and `log_dirs`, and `log_dirs` is the one that proves the rule, because `workflow.onComplete { assembleCombinedLogs(log_dirs) }` is the closure. `run_defs` carries a `def` and is fine -- nothing outside the workflow body reads it. The rule is about where a name is read from, not about which names happen to need it today. ### Nothing in an `onComplete` handler may throw When one fails, Nextflow reports *"Failed to invoke `workflow.onComplete` event handler"* **instead of** the error that actually stopped the run. A fault in the logging replaces the diagnosis with a line about logging. -It cost the multi-run dictionary conflict its entire message once: a GString reaching a String parameter turned a precise explanation into a pointer at this file. That is why the handler does nothing but call one guarded function — anything evaluated in the handler's *argument list* is outside that function's own try/catch, which is exactly where that failure was. +It cost the multi-run dictionary conflict its entire message once: a GString reaching a String parameter turned a precise explanation into a pointer at this file. That is why the handler does nothing but call one guarded function -- anything evaluated in the handler's *argument list* is outside that function's own try/catch, which is exactly where that failure was. ### `combine` spreads a List item -`combine` treats a channel item that is a List as a tuple and spreads it, so combining a `collect()`ed list of N reports produces an N+1 element tuple rather than a pair, and every closure downstream is called with the wrong arity. A scalar cannot be spread — which is why step 1's gate is a `count` and not the reports themselves. Nothing reads the gate's value anyway. +`combine` treats a channel item that is a List as a tuple and spreads it, so combining a `collect()`ed list of N reports produces an N+1 element tuple rather than a pair, and every closure downstream is called with the wrong arity. A scalar cannot be spread -- which is why step 1's gate is a `count` and not the reports themselves. Nothing reads the gate's value anyway. ### `combine(by: 0)` versus `join` @@ -38,17 +38,17 @@ It cost the multi-run dictionary conflict its entire message once: a GString rea ### Positional matching is the recurring hazard -With one run there was exactly one of each singleton and implicit value channels broadcast them for free, so "the reference index" and "the sample" could be two separate process inputs. With N of each, positional matching pairs whichever arrived first — and the result is one analysis run against another's reference, which no later check could detect. +With one run there was exactly one of each singleton and implicit value channels broadcast them for free, so "the reference index" and "the sample" could be two separate process inputs. With N of each, positional matching pairs whichever arrived first -- and the result is one analysis run against another's reference, which no later check could detect. So: singletons are combined onto their samples **by key**, and two per-sample channels are joined on the work item **and** the sample. ## Ordering that produces wrong numbers rather than errors -**`runDefinitions()` must be called before `resolveParameters()`.** The first takes each run's copy of the parameters while "absent" still means "the user did not set this". The second destroys that distinction by filling the computed values in, and `fill` will not overwrite — so a run changing an input to a derivation would silently keep the base run's derived value. Reversing the two lines produces wrong numbers, not an error. +**`runDefinitions()` must be called before `resolveParameters()`.** The first takes each run's copy of the parameters while "absent" still means "the user did not set this". The second destroys that distinction by filling the computed values in, and `fill` will not overwrite -- so a run changing an input to a derivation would silently keep the base run's derived value. Reversing the two lines produces wrong numbers, not an error. ## The step-0 gate is the only one the pipeline needs -A shared step must wait for **every** run that shares it, not just the one whose parameters it carries. `VerifyAll` is `errorStrategy 'finish'`, which lets already-submitted tasks complete — so a step gated on one member's report could finish and promote while another member's `CheckRunParameters` was still deciding to fail. +A shared step must wait for **every** run that shares it, not just the one whose parameters it carries. `VerifyAll` is `errorStrategy 'finish'`, which lets already-submitted tasks complete -- so a step gated on one member's report could finish and promote while another member's `CheckRunParameters` was still deciding to fail. Gating **step 2** covers every step after it, because a variant's members are always a subset of its parent's: a step-2 variant that waited for all of its members has already waited for all of every variant descended from it. @@ -56,13 +56,13 @@ Gating **step 2** covers every step after it, because a variant's members are al ## Sharing was built switched off -While `sharingEnabled()` returned false, every run was its own variant at every step, so each tree was a straight line and this file described exactly the DAG it described before multi-run. That was deliberate: it made the rewiring provable by running old and new code over one fixture and showing nothing moved. **It returns a constant `true` now** — see `variant-model.md` for what that leaves unreachable. +While `sharingEnabled()` returned false, every run was its own variant at every step, so each tree was a straight line and this file described exactly the DAG it described before multi-run. That was deliberate: it made the rewiring provable by running old and new code over one fixture and showing nothing moved. **It returns a constant `true` now** -- see `variant-model.md` for what that leaves unreachable. ## Promotion attachment points Each hangs off a step's output **alongside** that output's real consumer rather than in front of it, so nothing upstream changes shape and no value channel can be turned into a queue channel. -The signal is the consuming step **having finished**, not the artifact — several processes take an input purely for ordering and read an absolute path instead, so holding the file proves nothing about who is done with it. +The signal is the consuming step **having finished**, not the artifact -- several processes take an input purely for ordering and read an absolute path instead, so holding the file proves nothing about who is done with it. **The gate is keyed by the PRODUCING variant, not the consuming one.** Once a producer is shared its consumers may not be: two runs can share step 2 and diverge at step 3, so two step-3 work items read one set of trimmed reads, and releasing on the first to finish would delete a file the second still needs. @@ -70,14 +70,14 @@ The FastQC zips are the one artifact produced and consumed inside a single step, Two artifacts have more than one consuming **step**, so their gates are assembled at the call site: -- **Ready BAMs** — step 5 per sample, step 6 for the cohort. Both are gathered onto their step-4 producer first, which collapses calling to one signal per producer however many step-6 variants read it. `combine(by: 0)` then re-emits each of the producer's samples' own signals, so the result is still one task per sample: the sample identity comes from step 5's side and calling contributes only its completion. -- **The called VCF** — step 7 always, step 8 only where annotation is on. Both shapes can be in flight at once, since a step-6 variant may feed an annotating branch and a non-annotating one simultaneously. +- **Ready BAMs** -- step 5 per sample, step 6 for the cohort. Both are gathered onto their step-4 producer first, which collapses calling to one signal per producer however many step-6 variants read it. `combine(by: 0)` then re-emits each of the producer's samples' own signals, so the result is still one task per sample: the sample identity comes from step 5's side and calling contributes only its completion. +- **The called VCF** -- step 7 always, step 8 only where annotation is on. Both shapes can be in flight at once, since a step-6 variant may feed an annotating branch and a non-annotating one simultaneously. Whether an annotation signal is ever coming is a property of the producer's **children**, not of the producer's own `annotate`: the variant carries its lead member's parameters, and `annotate` is not part of step 6's identity, so the lead cannot answer for the rest. ## The combined log -Each task appends to `Logs//*_nextflow.log`, one writer per file, so tasks never contend — but a run ends up scattered across dozens of files. By the time `onComplete` runs there are no writers left, so gathering is safe there in a way it would not be mid-run. +Each task appends to `Logs//*_nextflow.log`, one writer per file, so tasks never contend -- but a run ends up scattered across dozens of files. By the time `onComplete` runs there are no writers left, so gathering is safe there in a way it would not be mid-run. Only the current run is collected: every block carries the session id that wrote it, so blocks from earlier runs in the same file are skipped. The combined file is overwritten each run; the full history stays in the per-process logs. diff --git a/.claude/development-notes/data-folder-walk.md b/.claude/development-notes/data-folder-walk.md new file mode 100644 index 0000000..c3f11c2 --- /dev/null +++ b/.claude/development-notes/data-folder-walk.md @@ -0,0 +1,57 @@ +# What walks `Data/`, and what only filters it + +**Written 2026-09-27, against the tree at `b96bfca` plus uncommitted work, after 3.2.0 shipped.** 3.2.0 made reads findable in subfolders of `Data/` by changing one glob, and its CHANGELOG says hidden folders "are not searched". Half of that is true. The half that is not is a cost rather than a wrong result, which is why the code was left as it is - and why the claim outlived the review. + +## Two walks over the same directory, under two rules + +**Step 0 genuinely prunes.** Three `find` invocations, at [`0_verify_environment.nf`](../../scripts/0_verify_environment.nf) lines 363, 537 and 591, all of the shape: + +```sh +find ${dataDir} -mindepth 1 -name '.*' -type d -prune -o ${readPattern} -print +``` + +A dot directory is never entered. `-mindepth 1` is load-bearing: the data root itself may sit under a dot - a project under `~/.local` would otherwise find nothing. + +**The read channel does not prune. It filters afterwards.** [`2_trim_reads.nf:25-26`](../../scripts/2_trim_reads.nf): + +```groovy +channel.fromFilePairs("${variant.reads}", checkIfExists: true) + .filter { _id, files -> !hiddenBelow(dataRoot, "${files[0]}".toString()) } +``` + +`.filter` is a channel operator. It sees what `fromFilePairs` has already emitted, so the glob has walked everything the pattern can reach - dot directories included - before one match is discarded. + +**The two agree on the outcome and disagree on the cost.** No sample under a hidden folder reaches the pipeline either way. But `p.reads` is `"${p.dir.data}/**${p.readPattern}"` and `**` is depth-unbounded, so on NetApp storage - where `.snapshot` is exposed read-only inside directories and holds a copy of every file per snapshot - the walk pays for every copy and then throws it away. Step 0 does not pay. + +## The claim that shipped + +The 3.2.0 CHANGELOG's Added section reads *"Hidden folders are skipped. Anything beginning with a dot is not searched."* True of step 0, false of the channel. It was written while the filter was being added and describes the intent rather than the mechanism. + +**Z's ruling, 2026-09-27: a released CHANGELOG's text is not edited.** The section had already shipped under the `v3.2.0` tag, so an in-place correction would make the repository and the published release disagree in silence. Only a mechanical ASCII conversion was applied to that file (em dash, arrow and multiplication sign, verified as byte-exact apart from those three substitutions). The correction belongs in a later release's own section, which is why it is recorded here in the meantime. + +## A third divergence, and it is not about dot folders + +**Symlinks.** No `find` in `scripts/` is invoked with `-L`, so step 0 does not descend into a symlinked directory. The channel's glob passes only `checkIfExists: true` - no `followLinks: false` - so Nextflow's walker follows them. + +Measured 2026-09-26 with `Files.walkFileTree` and `FileVisitOption.FOLLOW_LINKS` against the composed pattern: with `Data/link -> `, the walk entered `link/work/ab/` and matched a file there, which is the exact name shape Nextflow stages into `work/`. `hiddenBelow()` does not reject it, because such a path has no dot component. A cycle back into `Data/` was reported once as `FileSystemLoopException` and the walk then continued and finished, so a loop is not a runaway. + +Not observed on a real project. Recorded because the two walks are easy to assume identical, and step 0's pair check - the thing that now refuses a missing mate - cannot see what the channel would find through a symlink. + +## What this is not, and the reasoning is the point + +A remote out-of-memory failure on a collaborator's cluster (reported 2026-09-25, 3.2.0, an installed release) looked exactly like this walk's fault, and was not. Recording the refutation because the shape of this code invites the mistake: + +- **The glob never ran.** The workflow body reaches `readPairChannel` at [`poolseqflow.nf:128`](../../poolseqflow.nf), after `VerifyEnvironment(` at :102 and `BuildDictionaries(` at :113. Nextflow logs one `<< taskConfig executor` line per process, 1:1 and never cached; a local full run logs 36. The failing run logged **one**. It died creating process 1 of 36, ten lines of body before the glob. +- **The magnitude was wrong regardless.** Measured: 1,000,000 retained `java.nio.file.Path` objects cost 119 MB, about 125 bytes each. Exhausting the 32 GB that JVM was granted needs on the order of 10^8 paths; a cold NFS walk sustains 10^3 to 10^4 entries per second, so the 38 seconds available yields 10^4 to 10^5. +- **The whole pipeline runs in 128 MB of head heap.** Measured by sweeping `NXF_OPTS=-Xmx` against the committed fixture: 128m completes all 36 processes and writes all four reports; 64m dies with none configured; 16m will not start a JVM. So "PoolSeqFlow needs more memory" was never the answer to anything. + +That failure is still open at this note's date and belongs to a different subject. Every frame it produced is inside `java.lang.invoke` - `LambdaForm.compileToBytecode`, `InvokerBytecodeGenerator`, `IndyInterface` - which is Groovy's invokedynamic machinery rather than any data structure of ours, and the same install failed to *compile* Nextflow's own static report template in 30 seconds on a run with zero tasks. See [[someone-elses-machine]] for the class. + +## The change not made + +**Bounding the walk instead of filtering its output.** The filter cannot do it: a channel operator runs after the walk completes, so the cost is already paid by the time it is reached. It would have to be bounded at the source, in the `fromFilePairs` call. + +It was not done because it fixes a cost and not a result, and because the failure that raised the question turned out to be something else. Two things to know before doing it: + +- **`**` matches across directories including none. `**/` matches only the nested ones** and would silently stop every flat project, which is every project before 3.2.0. `{,**/}` is unparseable by Nextflow. The composed pattern is actually `***` - `**` from the code plus the leading `*` of `readPattern` - and the JDK glob matcher treats that identically to `**`, measured. +- **The case that proves it already exists and had to be moved to earn that.** `test_hidden_folders_are_excluded_from_the_read_channel` in `04_pipeline` plants `Data/.snapshot/nightly/` copies and **runs step 2**, because a case that runs only step 0 passes with the channel's filter deleted outright - measured, it did. That vacuous version is the same failure as [[gates-that-stopped-checking]]: the assertion was aimed at the half that was never changing. diff --git a/.claude/development-notes/depth-cutoff.md b/.claude/development-notes/depth-cutoff.md index 96cd2ed..707050b 100644 --- a/.claude/development-notes/depth-cutoff.md +++ b/.claude/development-notes/depth-cutoff.md @@ -6,13 +6,13 @@ How `bin/depth_cutoff.py` came to be shaped the way it is, and the three designs ## What it is for -`bcftools.maxDepth` was **2000**, one hand-set number applied to every sample of every run. bcftools' own default is 250; 2000 was chosen because it was comfortably larger, and nothing in the output ever said whether it had bitten. Pool coverage is non-homogeneous by design, so the right ceiling is a property of the sample's own depth distribution, and the thing worth truncating is a collapsed repeat or a PCR hill — a *second population* at high depth, not a high mean. One global number either cuts legitimate coverage in a deep sample or lets a pile-up through in a shallow one. +`bcftools.maxDepth` was **2000**, one hand-set number applied to every sample of every run. bcftools' own default is 250; 2000 was chosen because it was comfortably larger, and nothing in the output ever said whether it had bitten. Pool coverage is non-homogeneous by design, so the right ceiling is a property of the sample's own depth distribution, and the thing worth truncating is a collapsed repeat or a PCR hill -- a *second population* at high depth, not a high mean. One global number either cuts legitimate coverage in a deep sample or lets a pile-up through in a shallow one. ## The method Z insisted on, and it is the reason this works Z, 2026-08-30, after two candidate algorithms had been argued about in the abstract: *"before jumping into the math, how about we create ~20 distributions ~5 of them clean, rest of them odd distributions then look at the data to find the right mathematical approach."* -That is what `test/tools/depth_corpus.py` is. It killed two designs in about ten minutes that argument had not dented, and it is now a committed test rather than a scratch file, so the evidence stays executable. **The bounds are derived from how each case was built** — `lo` is the 99.9th percentile of that case's real coverage component, `hi` is the mean of its planted anomaly — so the suite measures the detector against the corpus's intent, never against its own previous output. They are written into an `expected.tsv` the script emits beside the histograms; nothing is committed but the generator, so there is no file to open and no expectation to quietly edit. Recording the detector's current answers as the expectation would have made the test a changelog. +That is what `test/tools/depth_corpus.py` is. It killed two designs in about ten minutes that argument had not dented, and it is now a committed test rather than a scratch file, so the evidence stays executable. **The bounds are derived from how each case was built** -- `lo` is the 99.9th percentile of that case's real coverage component, `hi` is the mean of its planted anomaly -- so the suite measures the detector against the corpus's intent, never against its own previous output. They are written into an `expected.tsv` the script emits beside the histograms; nothing is committed but the generator, so there is no file to open and no expectation to quietly edit. Recording the detector's current answers as the expectation would have made the test a changelog. ## Three designs, and what the corpus did to them @@ -20,15 +20,15 @@ That is what `test/tools/depth_corpus.py` is. It killed two designs in about ten The smoothing window was a fraction of the mode. Z: *"If we had a super deep file with depth reaching up to 20000 at times, window size would be 2000 which feels incorrect in a scenario where the cutoff was at ~500 then the peak at 20000."* A window of 2000 erases a trough at 500 completely. -The deeper fault is the anchor. **In linear depth a deep lobe is always spread thinner** — a lobe of mean `m` and dispersion `r` has width about `m/√r`, so its peak height falls as it gets deeper. The argmax is therefore biased towards shallow lobes: on `bad-reference` it lands on the depth-1 junk, and any walk rightward from there crosses the real coverage and cuts into it. +The deeper fault is the anchor. **In linear depth a deep lobe is always spread thinner** -- a lobe of mean `m` and dispersion `r` has width about `m/√r`, so its peak height falls as it gets deeper. The argmax is therefore biased towards shallow lobes: on `bad-reference` it lands on the depth-1 junk, and any walk rightward from there crosses the real coverage and cuts into it. -**2. Log-depth bins, anchored at the median covered position.** Fixed bins per decade fixes the window problem properly — coverage is multiplicative, so a hill at fifty times the mode is the same shape at 40x and at 4000x, and a constant number of log bins gives every part of the range the same relative resolution. That part survived into the shipped version. +**2. Log-depth bins, anchored at the median covered position.** Fixed bins per decade fixes the window problem properly -- coverage is multiplicative, so a hill at fifty times the mode is the same shape at 40x and at 4000x, and a constant number of log bins gives every part of the range the same relative resolution. That part survived into the shipped version. -The median anchor did not, and Z called it before it was measured: *"think about a scenario people used some system where the reference is not good enough, say ticks. There are a lot of sites with zero coverage."* `samtools stats` does not report depth 0 at all — verified — so uncovered sites are genuinely absent from the histogram, and the first instinct was that the objection did not apply. It does. A poor reference does not leave sites at depth 0; it leaves them at depth **1 to 5**, and those are in the histogram. On `bad-reference`, two thirds junk, the median covered position is **depth 3** against real coverage at 60x. +The median anchor did not, and Z called it before it was measured: *"think about a scenario people used some system where the reference is not good enough, say ticks. There are a lot of sites with zero coverage."* `samtools stats` does not report depth 0 at all -- verified -- so uncovered sites are genuinely absent from the histogram, and the first instinct was that the objection did not apply. It does. A poor reference does not leave sites at depth 0; it leaves them at depth **1 to 5**, and those are in the histogram. On `bad-reference`, two thirds junk, the median covered position is **depth 3** against real coverage at 60x. **3. Log-depth bins, anchored at the deeper of the position median and the base median.** Shipped. -The base median weights each position by its depth, so it counts reads rather than sites. Junk at depth 1-3 holds essentially no reads, so it cannot move the base median — `bad-reference` gives **63**, squarely in the 60x lobe. The two medians fail on opposite cases, and the measurement is the whole argument: +The base median weights each position by its depth, so it counts reads rather than sites. Junk at depth 1-3 holds essentially no reads, so it cannot move the base median -- `bad-reference` gives **63**, squarely in the 60x lobe. The two medians fail on opposite cases, and the measurement is the whole argument: | case | position median | base median | the real coverage | |---|---|---|---| @@ -36,11 +36,11 @@ The base median weights each position by its depth, so it counts reads rather th | `hill-large` | 128 ✓ | **4117** ✗ | 120x | | `hill-dominant` | **17428** ✗ | **21316** ✗ | 200x | -Taking the deeper of the two is not a compromise, it is the safe direction: **an anchor that is too deep can only miss an anomaly and report the sample uncapped, while an anchor that is too shallow cuts into real coverage.** Every other test in the detector is oriented the same way. This is why the `max()` of two medians has a comment in the source explaining itself — it is exactly the kind of line a later reader "simplifies" to one median. +Taking the deeper of the two is not a compromise, it is the safe direction: **an anchor that is too deep can only miss an anomaly and report the sample uncapped, while an anchor that is too shallow cuts into real coverage.** Every other test in the detector is oriented the same way. This is why the `max()` of two medians has a comment in the source explaining itself -- it is exactly the kind of line a later reader "simplifies" to one median. ## The one-character fix that mattered most -The trough walk originally advanced on `height <= trough_height`. In a wide empty trough every height is zero, so the trough kept advancing to the *last* empty bin — the far side. The caps that produced were arithmetically correct and useless: +The trough walk originally advanced on `height <= trough_height`. In a wide empty trough every height is zero, so the trough kept advancing to the *last* empty bin -- the far side. The caps that produced were arithmetically correct and useless: | case | cap with `<=` | cap with `<` | the anomaly | |---|---|---|---| @@ -52,37 +52,37 @@ Truncating a 30000x organelle to 15849 does nothing. The cap belongs where cover ## Two cases that are deliberately left uncapped -`hill-large` (the pile-up holds a fifth of the covered genome) and `hill-dominant` (four fifths — Z's own scenario) are recorded in the corpus with verdict `none`, and that is a **policy decision, not a detector limitation**. Neither is distinguishable from a library that simply ran deep, and capping either truncates a large fraction of a real genome on a guess. They are reported uncapped, and `param_capMaxDepth` is how a user overrules that for one sample. This follows Z's standing rule: fail loudly or document, never automate away a decision. +`hill-large` (the pile-up holds a fifth of the covered genome) and `hill-dominant` (four fifths -- Z's own scenario) are recorded in the corpus with verdict `none`, and that is a **policy decision, not a detector limitation**. Neither is distinguishable from a library that simply ran deep, and capping either truncates a large fraction of a real genome on a guess. They are reported uncapped, and `param_capMaxDepth` is how a user overrules that for one sample. This follows Z's standing rule: fail loudly or document, never automate away a decision. -The corresponding trap is that "uncapped" must be *visible*. A sample the detector declines to cut is the one case where the pipeline decided to do nothing, and the user cannot tell that from the output — which is why the sentence explaining the decision is published for every sample whether it was capped or not. +The corresponding trap is that "uncapped" must be *visible*. A sample the detector declines to cut is the one case where the pipeline decided to do nothing, and the user cannot tell that from the output -- which is why the sentence explaining the decision is published for every sample whether it was capped or not. -**The process publishing that sentence declares only one of the three files it writes.** `DepthProfile` produces `_depth_histogram.tsv`, `_depth_cap.txt` and `_depth_report.txt`, and its `output:` block names the decision file alone — so the report carrying the sentence above is outside Nextflow's tracking and outside the skip test, which asks about the decision file. A pipeline run cannot reach the bad state: the `atomic_mv` calls put the decision file last, so an interruption leaves the decision missing, the skip does not fire, and all three are rebuilt. Only deleting the histogram by hand gets there. Open as E6h and triaged low. Observed 2026-09-05. +**The process publishing that sentence declares only one of the three files it writes.** `DepthProfile` produces `_depth_histogram.tsv`, `_depth_cap.txt` and `_depth_report.txt`, and its `output:` block names the decision file alone -- so the report carrying the sentence above is outside Nextflow's tracking and outside the skip test, which asks about the decision file. A pipeline run cannot reach the bad state: the `atomic_mv` calls put the decision file last, so an interruption leaves the decision missing, the skip does not fire, and all three are rebuilt. Only deleting the histogram by hand gets there. Open as E6h and triaged low. Observed 2026-09-05. -**Fixed 2026-09-08 (E6h).** The histogram and the report are a second emit, `report`, keyed on run and sample the way `AlignmentReport`'s and `CoverageReport`'s are — separate from `profile` because step 6 reads that tuple by position, so widening it would have broken the consumer. The skip now tests all three files and both branches symlink all three back into the work directory; without that second half, declaring the outputs would have made every resume against an existing profile fail on a missing declared output. `optional: true` would have been the wrong escape, since it hides a genuine miss. +**Fixed 2026-09-08 (E6h).** The histogram and the report are a second emit, `report`, keyed on run and sample the way `AlignmentReport`'s and `CoverageReport`'s are -- separate from `profile` because step 6 reads that tuple by position, so widening it would have broken the consumer. The skip now tests all three files and both branches symlink all three back into the work directory; without that second half, declaring the outputs would have made every resume against an existing profile fail on a missing declared output. `optional: true` would have been the wrong escape, since it hides a genuine miss. **The ordering of the three `atomic_mv` calls stops being load-bearing with that change**, which is worth knowing before anyone tidies it: whichever of the three is missing, the skip no longer fires. Before, only the decision file going last kept the partial state unreachable. ## Everything is oriented towards not capping -Each of the detector's three tests — the trough fraction, the rise factor, the minimum rise mass — can only fail in the direction of leaving the sample alone, and the anchor rule is chosen the same way. An unrecognizable histogram therefore comes out reported rather than guessed at. This is deliberate and it is the property to preserve if the constants are ever retuned: a false cap silently truncates real coverage and the frequency tables that come out of it look perfectly ordinary, while a missed cap is visible in the published report. +Each of the detector's three tests -- the trough fraction, the rise factor, the minimum rise mass -- can only fail in the direction of leaving the sample alone, and the anchor rule is chosen the same way. An unrecognizable histogram therefore comes out reported rather than guessed at. This is deliberate and it is the property to preserve if the constants are ever retuned: a false cap silently truncates real coverage and the frequency tables that come out of it look perfectly ordinary, while a missed cap is visible in the published report. ## How the capping itself is done -`bin/cap_depth.awk`, a greedy one-pass truncator over a coordinate-sorted SAM stream: a read is kept only if every position it covers is still below the cap, so the depth afterwards can never exceed it. Coordinate order is what makes one pass enough — once the stream has moved past a position nothing can add to it again, so the running depth array is trimmed from behind and never holds more than one read's span. Measured on a synthetic BAM: 4000x → exactly 100x at `cap=100`, and **no position that was already under the cap changed**. +`bin/cap_depth.awk`, a greedy one-pass truncator over a coordinate-sorted SAM stream: a read is kept only if every position it covers is still below the cap, so the depth afterwards can never exceed it. Coordinate order is what makes one pass enough -- once the stream has moved past a position nothing can add to it again, so the running depth array is trimmed from behind and never holds more than one read's span. Measured on a synthetic BAM: 4000x -> exactly 100x at `cap=100`, and **no position that was already under the cap changed**. **There is no tool for this in the environment.** `samtools view` has `--subsample FLOAT` and nothing depth-aware; a first draft of `CapBAM` called a `--max-depth` flag that does not exist, with a `|| fallback` that would have silently done nothing. No pysam, no sambamba, no picard. Hence the awk. -**Reads are dropped, not pairs, and that is safe** — checked rather than assumed, because the opposite would have been a silent halving. `bcftools mpileup` excludes anomalous pairs by the PROPER_PAIR flag *in the record*; removing one mate from the file does not clear the survivor's flag, so the survivor is still counted. If a future change starts rewriting flags, this stops being true. +**Reads are dropped, not pairs, and that is safe** -- checked rather than assumed, because the opposite would have been a silent halving. `bcftools mpileup` excludes anomalous pairs by the PROPER_PAIR flag *in the record*; removing one mate from the file does not clear the survivor's flag, so the survivor is still counted. If a future change starts rewriting flags, this stops being true. -**The alternative was region exclusion** — a depth-derived BED plus `samtools view -L` — and it was not chosen. Excluding a region makes that sample depth-0 there, and `vcffilter.minDP` removes a site when *any* sample is under-covered, so one sample's collapsed repeat would delete those sites for the whole cohort. Truncation keeps the site and reduces one sample's weight in it. +**The alternative was region exclusion** -- a depth-derived BED plus `samtools view -L` -- and it was not chosen. Excluding a region makes that sample depth-0 there, and `vcffilter.minDP` removes a site when *any* sample is under-covered, so one sample's collapsed repeat would delete those sites for the whole cohort. Truncation keeps the site and reduces one sample's weight in it. ## The figures in the manual -`dev/scripts/plot_depth_cutoff.py` writes SVG directly. matplotlib is not in the environment, and adding it would put a large dependency into `install/environment.yml` for something no run needs; hand-written SVG is text, so it diffs, scales, and carries the site's own palette. The figures are generated once and committed — the docs build copies `manual/assets/` wholesale and never runs this. +`dev/scripts/plot_depth_cutoff.py` writes SVG directly. matplotlib is not in the environment, and adding it would put a large dependency into `install/environment.yml` for something no run needs; hand-written SVG is text, so it diffs, scales, and carries the site's own palette. The figures are generated once and committed -- the docs build copies `manual/assets/` wholesale and never runs this. -Both axes are logarithmic and both have to be. Depth is multiplicative, and the anomalous population holds a thousandth of the positions, so on a linear count axis it is simply invisible — which is the difficulty the figure exists to show. The bins are log-spaced but never narrower than one integer depth: below depth 17 a pure log bin is thinner than the gap between consecutive integers, and the first draft grew a comb of false zeros there. +Both axes are logarithmic and both have to be. Depth is multiplicative, and the anomalous population holds a thousandth of the positions, so on a linear count axis it is simply invisible -- which is the difficulty the figure exists to show. The bins are log-spaced but never narrower than one integer depth: below depth 17 a pure log bin is thinner than the gap between consecutive integers, and the first draft grew a comb of false zeros there. -`build_docs.py` gained an image-path rewrite for these. The manual is authored as a single file and read directly in the IDE, so images are written `assets/…` as the manual itself reads them; every generated page but the home page is emitted one directory down, so the path moves with it. +`build_docs.py` gained an image-path rewrite for these. The manual is authored as a single file and read directly in the IDE, so images are written `assets/...` as the manual itself reads them; every generated page but the home page is emitted one directory down, so the path moves with it. ## Measurements worth not repeating @@ -93,12 +93,12 @@ COV [20-20] 20 2 COV [100<] 100 128 <- the open top bin, present only when it is non-empty ``` -Only non-zero bins are emitted, so the histogram is sparse; depth 0 is not reported at all. The default `-c` ceiling of **1000** collapses everything above it into that open bin — on the trial BAM it swallowed 25 positions silently — so the ceiling has to be raised and the open bin has to be a loud failure, not a warning. +Only non-zero bins are emitted, so the histogram is sparse; depth 0 is not reported at all. The default `-c` ceiling of **1000** collapses everything above it into that open bin -- on the trial BAM it swallowed 25 positions silently -- so the ceiling has to be raised and the open bin has to be a loud failure, not a warning. `samtools depth` has no cap in 1.24 (`-d` is accepted and ignored) and is a usable but strictly more expensive fallback. `samtools coverage` is per-contig means, which is the wrong instrument. -**`bcftools mpileup -d 0` means no limit**, measured on 1.24 with a synthetic 600-read pile-up: default → 250, `-d 250` → 250, `-d 0` → **600**, `-d 100000` → 600, `-d 5` → 5. This is what makes `variantCall.maxDepth = 0` safe to ship as the default. It had been asserted from memory twice before anyone measured it; had `0` meant literally zero, the shipped configuration would have produced empty coverage everywhere, silently, at full pipeline cost. +**`bcftools mpileup -d 0` means no limit**, measured on 1.24 with a synthetic 600-read pile-up: default -> 250, `-d 250` -> 250, `-d 0` -> **600**, `-d 100000` -> 600, `-d 5` -> 5. This is what makes `variantCall.maxDepth = 0` safe to ship as the default. It had been asserted from memory twice before anyone measured it; had `0` meant literally zero, the shipped configuration would have produced empty coverage everywhere, silently, at full pipeline cost. ## The constants, and what would move them -`BINS_PER_DECADE = 20`, `SMOOTH_BINS = 3`, `TROUGH_FRACTION = 0.10`, `RISE_FACTOR = 2.0`, `RISE_MIN_FRACTION = 1e-5`. All twenty corpus cases pass with margin, so none of these is currently at a boundary — but `RISE_MIN_FRACTION` is the one to watch. It is what stops a sparse upper tail being read as a second population (`heavy-tail` is the case that would break first), and it is a fraction of the covered genome, so a very small reference makes the absolute floor of one position bind instead. If a real dataset ever needs one of these moved, add it to the corpus first and change the number second. +`BINS_PER_DECADE = 20`, `SMOOTH_BINS = 3`, `TROUGH_FRACTION = 0.10`, `RISE_FACTOR = 2.0`, `RISE_MIN_FRACTION = 1e-5`. All twenty corpus cases pass with margin, so none of these is currently at a boundary -- but `RISE_MIN_FRACTION` is the one to watch. It is what stops a sparse upper tail being read as a second population (`heavy-tail` is the case that would break first), and it is a fraction of the covered genome, so a very small reference makes the absolute floor of one position bind instead. If a real dataset ever needs one of these moved, add it to the corpus first and change the number second. diff --git a/.claude/development-notes/dictionaries-and-snpeff.md b/.claude/development-notes/dictionaries-and-snpeff.md index 66e37b5..f33f0e8 100644 --- a/.claude/development-notes/dictionaries-and-snpeff.md +++ b/.claude/development-notes/dictionaries-and-snpeff.md @@ -1,20 +1,20 @@ -# Step 1 — dictionaries and the snpEff database +# Step 1 -- dictionaries and the snpEff database **Written 2026-08-31, against the tree at `7d65893`.** Step 1 itself is unchanged. One sentence below leans on a rule in `atomic_mv.sh` that the `77fdbd7` overhaul narrowed, corrected in place. `scripts/1_build_dictionaries.nf`. The one step whose artifacts were shared between runs before multi-run existed. -> **For the manual, not here.** That dictionaries live under `mainDir/Reference/Dictionaries` and are **reused across projects** is user-facing — it is why a second project against the same genome starts fast, and why `reset` does not delete them. +> **For the manual, not here.** That dictionaries live under `mainDir/Reference/Dictionaries` and are **reused across projects** is user-facing -- it is why a second project against the same genome starts fast, and why `reset` does not delete them. ## Why step 1 groups runs, and why it is not part of the variant model -Dictionaries live under `mainDir/Reference/Dictionaries`, and runs **share** `mainDir` — so two runs naming the same reference resolve to exactly the same output paths. Left alone under multi-run they would both build it, at the same time, into one place. `atomic_mv.sh` has no locking, so that is a **race**, not merely duplicated work. +Dictionaries live under `mainDir/Reference/Dictionaries`, and runs **share** `mainDir` -- so two runs naming the same reference resolve to exactly the same output paths. Left alone under multi-run they would both build it, at the same time, into one place. `atomic_mv.sh` has no locking, so that is a **race**, not merely duplicated work. This is therefore not an optimisation and does not belong with the sharing work: threading runs through step 1 without grouping them would have broken something that already worked. -`dictionaryKey()` groups on **output paths**. `dictionarySettings()` **throws** when two runs on one key disagree, rather than quietly splitting them — "build it once" would otherwise hand one run a dictionary built to the other's settings, which is the silent-wrong-result failure this project keeps finding. Reported rather than resolved: which of two conflicting settings should win is the user's decision and there is no safe default. +`dictionaryKey()` groups on **output paths**. `dictionarySettings()` **throws** when two runs on one key disagree, rather than quietly splitting them -- "build it once" would otherwise hand one run a dictionary built to the other's settings, which is the silent-wrong-result failure this project keeps finding. Reported rather than resolved: which of two conflicting settings should win is the user's decision and there is no safe default. -Resource settings are deliberately absent from that comparison — heap size and core counts change how long the build takes, not what it produces, and a shared dictionary cannot have per-run resources anyway. +Resource settings are deliberately absent from that comparison -- heap size and core counts change how long the build takes, not what it produces, and a shared dictionary cannot have per-run resources anyway. `dictionaryRuns()` deliberately **merges** runs that disagree about `annotate`: one database serves them all, and a run that does not annotate is not harmed by its existence. Splitting that group would have two of them build concurrently into one directory. @@ -22,15 +22,15 @@ Resource settings are deliberately absent from that comparison — heap size and ### The snpEff marker was not keyed by genome -The marker lived at the top of the snpEff folder, so a single top-level file answered *"has any database been built here?"* when the question is *"has THIS genome's database been built?"* Building against a second reference found the marker, skipped the build, and inherited the first genome's database. The failure did not surface until **step 8**, after alignment and calling, as `Genome download failed!` — snpEff falls through to `-download` when the config names a database it cannot find. Every index file in this step was already keyed by reference name; this was the one artifact that was not. +The marker lived at the top of the snpEff folder, so a single top-level file answered *"has any database been built here?"* when the question is *"has THIS genome's database been built?"* Building against a second reference found the marker, skipped the build, and inherited the first genome's database. The failure did not surface until **step 8**, after alignment and calling, as `Genome download failed!` -- snpEff falls through to `-download` when the config names a database it cannot find. Every index file in this step was already keyed by reference name; this was the one artifact that was not. A project built by an earlier release has its marker in the old place, so the first run after upgrading rebuilds once. Deliberate: the old marker records that *a* database was built, not which genome it was for, so it cannot be adopted without guessing. ### The reference had to be gzipped, and nothing said so -`gunzip -c` was unconditional, so a plain `reference.fasta` died with *"not in gzip format"* — while nothing anywhere required the `.gz`: not step 0, and not the template, whose own `referenceFa = referenceFile.replace('.gz', '')` reads as though a plain name is expected to work. The GFF has always accepted both, which is exactly what made the asymmetry invisible. +`gunzip -c` was unconditional, so a plain `reference.fasta` died with *"not in gzip format"* -- while nothing anywhere required the `.gz`: not step 0, and not the template, whose own `referenceFa = referenceFile.replace('.gz', '')` reads as though a plain name is expected to work. The GFF has always accepted both, which is exactly what made the asymmetry invisible. -It is **copied rather than symlinked** in the uncompressed case, so what lands under `Dictionaries` is a real file either way and `atomic_mv.sh`'s guarantee still holds. That costs one copy of the genome — which the gzipped path has always cost too. +It is **copied rather than symlinked** in the uncompressed case, so what lands under `Dictionaries` is a real file either way and `atomic_mv.sh`'s guarantee still holds. That costs one copy of the genome -- which the gzipped path has always cost too. ### `ls | wc -l` as an existence test @@ -40,19 +40,19 @@ It is **copied rather than symlinked** in the uncompressed case, so what lands u The marker is what every later run reads to decide the database is usable, so it must not be able to appear beside a half-written one. Writing it **inside** the staged directory and moving that directory atomically makes "marker present" mean "database complete". It used to be a second atomic move after a non-atomic `cp -r`, so a kill in between left a partial database and no marker. -**The unit is the genome's directory, not `data/`** — and that is what makes it possible at all. Several genomes share `data/`, and `mv` of a directory onto an existing directory *nests* it rather than replacing it, so publishing `data/` wholesale could only ever have been the merge that `cp -r` was doing. +**The unit is the genome's directory, not `data/`** -- and that is what makes it possible at all. Several genomes share `data/`, and `mv` of a directory onto an existing directory *nests* it rather than replacing it, so publishing `data/` wholesale could only ever have been the merge that `cp -r` was doing. -A database with no marker is incomplete by definition — exactly what a kill during an earlier publish leaves — so it is discarded rather than merged into. `atomic_mv.sh` refuses a destination directory with anything in it, which is the same rule seen from the other side. +A database with no marker is incomplete by definition -- exactly what a kill during an earlier publish leaves -- so it is discarded rather than merged into. `atomic_mv.sh` refuses a destination directory with anything in it, which is the same rule seen from the other side. -It refused *every* existing directory when this was written. The `77fdbd7` overhaul made `rename(2)` the only writer, and `rename(2)` replaces an existing **empty** directory rather than refusing it — pinned by `test_atomic_mv_replaces_an_empty_directory`. A half-written database is never empty, so the argument above is unaffected; only the rule it cites is narrower than it was. +It refused *every* existing directory when this was written. The `77fdbd7` overhaul made `rename(2)` the only writer, and `rename(2)` replaces an existing **empty** directory rather than refusing it -- pinned by `test_atomic_mv_replaces_an_empty_directory`. A half-written database is never empty, so the argument above is unaffected; only the rule it cites is narrower than it was. ## Two snpEff invocation traps -- **`-c` must name the config explicitly.** Without it snpEff finds a config in the working directory only when it is called exactly `snpEff.config`, and otherwise silently falls back to the one bundled with the install — which has no entry for this genome, so the build dies with `Property: '.genome' not found`. That made `params.snpEff.config` a parameter that broke the run if it was ever changed from its default. +- **`-c` must name the config explicitly.** Without it snpEff finds a config in the working directory only when it is called exactly `snpEff.config`, and otherwise silently falls back to the one bundled with the install -- which has no entry for this genome, so the build dies with `Property: '.genome' not found`. That made `params.snpEff.config` a parameter that broke the run if it was ever changed from its default. - **The config is merged, not replaced.** It sits next to the shared `data/` directory and snpEff reads every entry, so it has to name every genome built here. A plain `cp` left it holding only the most recently built one, making every earlier genome unannotatable even though its database was still on disk. ## Channel details - Step 0's completion arrives as a **pure ordering barrier**: nothing in this step reads it, and it is `val` rather than `path` because N runs each publish a report called `0_verify_environment.txt`, and staging N files of one name is a collision for no purpose. - **`annotate` filters the channel**, rather than an `if (params.annotate)` that could only read the base config and would have built for every reference or none. -- **`UngzipReference`'s output is not re-exported.** It is consumed inside the workflow by the two index builders; re-exporting published a channel `poolseqflow.nf` has never read, in any release. `BuildSnpEffDb` does not want it either — it runs *alongside* `UngzipReference` and copies the reference the user placed. +- **`UngzipReference`'s output is not re-exported.** It is consumed inside the workflow by the two index builders; re-exporting published a channel `poolseqflow.nf` has never read, in any release. `BuildSnpEffDb` does not want it either -- it runs *alongside* `UngzipReference` and copies the reference the user placed. diff --git a/.claude/development-notes/dropzerodepth-coverage.md b/.claude/development-notes/dropzerodepth-coverage.md new file mode 100644 index 0000000..2ba9c88 --- /dev/null +++ b/.claude/development-notes/dropzerodepth-coverage.md @@ -0,0 +1,69 @@ +# Covering `vcffilter.dropZeroDepth`, and the fixture that cannot hold the case + +**Written 2026-09-29, against the tree at `ad85fb1` plus uncommitted work.** `dropZeroDepth` shipped in `8d0447c` with no case anywhere setting it to any value. Three things needed asserting and they turned out to be governed by three different mechanisms, which is the reason this note exists. + +## The three mechanisms, which are not one mechanism + +Before writing anything I assumed a single "is the parameter tracked" property. There are three, and each needs its own case: + +| what | where it lives | what covers it | +|---|---|---| +| step 7 **reads** the parameter and **declares** it | `stepParameterMap()` in `scripts/variants.nf:44` | `00_static`'s `step parameter map covers what each step reads`, which already existed | +| the parameter is **analysis-affecting**, so flipping it stops a project | `analysisParams()` in `scripts/0_verify_environment.nf:20` | new case in `05_guards` | +| the **expression** it builds is the right way round | the ternary in `scripts/7_vcf2freq.nf:257` | new case in `04_pipeline` | + +**`analysisParams()` is an EXCLUSION list**, and that is the part that misled me: *"Everything not named here counts as analysis-affecting."* So the change guard fires on `dropZeroDepth` whether or not `stepParameterMap()` mentions it. Measured: deleting the parameter from `variants.nf:44` leaves the `05_guards` case passing. The first draft of that case carried a comment claiming it proved the map was read, which was simply false. + +What DOES break the guard case is an edit to the exclusion list -- `vcffilter.` added to `skipPrefix`, or the key added to `skipKey`. Verified by adding it: all five assertions fail. That is a plausible mistake rather than a theoretical one, because `capBAM.histogramMax` is already excluded there on deliberate and correct reasoning. + +## The behavior case is not achievable in this fixture + +The thing worth asserting is a site dropped at `true` and surviving as `NA` at `false`. It needs a cell where one sample has no reads, and **the fixture cannot hold one.** + +Every sample carries about 75x across the whole 20 kb reference -- measured from the raw VCF of a real run, mean DP 75.2, 75.0, 76.6, 65.1, 57.1 -- and no sample has `DP=0` at any of the 135 called sites. `test/data/vcf/README.md` says the same of the called VCF it was cut from. + +**Thinning one sample was tried and fails in an instructive way.** Read names are a sequential fragment index and both mates are written in step, so truncating the two files to the same count keeps every pair intact -- that part works. At 250 pairs the result is not low coverage, it is **none**: + +``` +[M::mem_pestat] skip orientation FF as there are not enough pairs +[M::mem_pestat] skip orientation FR as there are not enough pairs +samtools markdup: Read pairs 0 should be greater than duplicate pairs 0 +``` + +bwa cannot estimate an insert size distribution from 250 pairs, so nothing is flagged properly paired, and step 4's `0x2` required flag discards every read. Mean DP for that sample: **0.0, at all 135 sites.** + +This is the trap `test/tools/make_fixture.py`'s own docstring warns about, in a different dress: *"it gave every fragment the same length, which left bwa with a zero-variance insert size distribution, which meant every indel-bearing pair fell outside the proper-pair window and was discarded by the 0x2 filter in step 4."* + +**And the two requirements are in direct conflict.** bwa needs thousands of pairs to pair-estimate; a zero-depth cell needs almost none locally. No amount of tuning the thinning reaches both. + +The shape that satisfies both is a sample with a **regional** coverage gap: many pairs, concentrated in part of the genome. That is an option in `make_fixture.py` restricting one sample's fragments to a sub-region, plus a committed fixture of its own beside `base/`, about 2 MB. **Not built.** The toggle is redundant above `minDP 1`, so nothing a default run publishes depends on it. Recorded in `test/README.md`'s known gaps so the thinning route is not attempted again. + +## What was built instead, and what it cost + +Step 7 now logs the filter it applied -- `Depth Filtering VCF, excluding FMT/DP<20 || FMT/DP==0` -- which makes the ternary assertable and is worth having for anyone reading a log. + +The case then **rides the existing multi-run** rather than adding a pipeline run. `runs.csv` gained a `vcffilter.dropZeroDepth` column with `false` on the `plain` run, which already diverges at step 7 through `poolSize`, so no sharing group moved and no other case's numbers changed -- the toggle is redundant at `minDP 20`, which is exactly why it is safe to set on a run whose output other cases assert. Cost: zero extra pipeline runs. + +Inverting the ternary fails both of its assertions. + +## A red test found on the way, and why nobody was sent to look + +Running `05_guards` in full -- owed because the multi-run fixture is shared -- turned up a failure that had nothing to do with this work. The case `run definitions resolve each kind of divergence` expected the DERIVED `mpileupOptions` for a run that sets only `maxDepth`: + + RUN depth variantCall.mpileupOptions=-B -C 50 -q 30 -Q 30 -d 4000 ... + +`-C` is the template's `scaleMapQ`, which that row does not set. **It moved from 50 to 100 in `eedf598` and this expectation did not**, so the case had been failing in the committed tree from then until now. + +**The routing is why it went unseen.** `dev/scripts/select-tests.py parameters.config.template` returned `01_migrate` alone -- yet `write_sandbox_config` at `test/lib/sandbox.sh:115` builds EVERY sandbox from that template, so `04_pipeline`, `05_guards`, `06_dryrun` and `10_analysis_verify` all depend on it and **none of the four declared it**. The selector derives its graph from imports and cannot see a file read inside a `sed`; `test/README.md` says exactly that about the half which must be declared by hand. + +So both halves are fixed: the expectation, and the four missing `# covers:` lines. The selector now returns all five suites for a template change. + +The `pinned` row's own `-C 50` was left alone deliberately. It is a literal that must survive verbatim, and now that it differs from the default it can no longer pass by coincidence -- a better test than it was. + +This is the family `gates-that-stopped-checking.md` is about, in its purest form: nothing was broken by the change, the checker was simply never asked. + +## Two things this session's earlier work did for itself + +**The emptiness guard caught the pathological fixture.** The first attempt did not fail obscurely: `dropZeroDepth = true` on a sample with no coverage removed every site, and the step-7 refusal added hours earlier stopped the run and named the filter. Without it the run would have published a header-only table and reported success. + +**The new logging is what diagnosed the thinning.** Four steps' logs from a FAILED run showed exactly where 250 read pairs went. Under the design that shipped the same morning, those logs would have been empty, because the copy at the end of each script is never reached when a task fails. diff --git a/.claude/development-notes/dry-run.md b/.claude/development-notes/dry-run.md index cd3f973..9a6f75e 100644 --- a/.claude/development-notes/dry-run.md +++ b/.claude/development-notes/dry-run.md @@ -2,14 +2,14 @@ **Written 2026-08-31, against the tree at `7d65893`.** `dryrun.nf` is unchanged. One forward-looking clause about `members.txt` did not come true and is corrected below. -`dryrun.nf` and the preview it builds. The wrapper side — vetting the directory before `rm -rf`, why that one is the risky one — is in `wrapper.md`. +`dryrun.nf` and the preview it builds. The wrapper side -- vetting the directory before `rm -rf`, why that one is the risky one -- is in `wrapper.md`. ## Why it is a separate entry point AND a flag Two halves of one job, and both are needed: - **A separate entry point, not `-entry` on `poolseqflow.nf`.** The strict parser refuses `-entry` outright. `dryrun.nf` is what stops the pipeline running. -- **`params.dryRun` on top of it.** The flag is what tells step 0's stages to record *nothing*. An entry point without the flag would stamp the project's baseline from a preview — a real `.poolseqflow_params` for results that will never exist. So the workflow body refuses to start when the flag is absent, rather than assuming the wrapper set it. +- **`params.dryRun` on top of it.** The flag is what tells step 0's stages to record *nothing*. An entry point without the flag would stamp the project's baseline from a preview -- a real `.poolseqflow_params` for results that will never exist. So the workflow body refuses to start when the flag is absent, rather than assuming the wrapper set it. ## Everything the dry run is lives in one file, process included @@ -19,21 +19,21 @@ The one place the pipeline departs from *entry points at the top, processes in ` Every path in the preview is one a process will actually write to, because they are enumerated from the variant plan rather than from a template of the layout. A preview cannot therefore describe a layout the run will not produce. Same reason `sharedMemberFiles` and `sharingGroups` are *included* from `variants.nf` rather than reimplemented: the preview writes exactly the members file a real run would write, in exactly the directories it would write it in, and the preview and step 0's partition report cannot disagree about the grouping. -The results tree is enumerated in full. The working volume and the logs get one entry per directory rather than a mirror of their contents, because those trees are built and emptied while the run is in flight — a preview of them would be a preview of a moment. +The results tree is enumerated in full. The working volume and the logs get one entry per directory rather than a mirror of their contents, because those trees are built and emptied while the run is in flight -- a preview of them would be a preview of a moment. What it enumerates is what the **pipeline** writes. `Analysis/` comes from a separate invocation with its own entry script and is not previewed here. ## No directory is ever marked as "already exists" -Tried and rejected. By the time anything can look, Nextflow has already created `workDir` and the session-reports directory, and step 0 has written its report — so a listing marked from disk reports that most of the tree exists, **when what put it there was the preview itself**. The question a reader actually has is whether the *project* has run before, and `.poolseqflow_version` answers exactly that. So the preview reads the stamp and says so in one sentence, and marks nothing. +Tried and rejected. By the time anything can look, Nextflow has already created `workDir` and the session-reports directory, and step 0 has written its report -- so a listing marked from disk reports that most of the tree exists, **when what put it there was the preview itself**. The question a reader actually has is whether the *project* has run before, and `.poolseqflow_version` answers exactly that. So the preview reads the stamp and says so in one sentence, and marks nothing. ## Flattened root names -A storage root becomes one directory in the preview — `home_user_2026_experiments_pool_store` — so that a deeply nested project does not open onto ten levels of empty directories. +A storage root becomes one directory in the preview -- `home_user_2026_experiments_pool_store` -- so that a deeply nested project does not open onto ten levels of empty directories. Flattening is **not injective**, so two roots can collide. That is a label problem, not a layout one, so it is disambiguated with a numeric suffix rather than by refusing to preview. Roots are named in path order so the legend is stable and the suffix always lands on whichever root sorts later; the count of roots already sharing a base name *is* the suffix, so no loop is needed. -**A path belongs to the DEEPEST root containing it.** `mainDir` and `storageDir` are allowed to nest — only identity between them is refused — and attributing a contained root's tree to the outer one would file the results under the working volume. +**A path belongs to the DEEPEST root containing it.** `mainDir` and `storageDir` are allowed to nest -- only identity between them is refused -- and attributing a contained root's tree to the outer one would file the results under the working volume. If a planned directory falls under neither root the preview **throws** rather than dropping it. A preview that quietly omitted a directory would show a layout the run does not produce, which is the one thing it must never do. @@ -47,10 +47,10 @@ The nested `while read` loops each redirect from their own file, so the inner on `printf '%s\n' $who` writing `members.txt` is **unquoted on purpose**: the names split back onto one line each, which is the format a real run writes. -**The clause that used to follow — *"and the analysis layer will read"* — did not come true.** `members.txt` is a record, rewritten every run, and nothing under `analysis/` reads it. The analysis layer recomputes `variantPlan()` rather than parsing what the pipeline left on disk, which is the better answer for the reason this file already gives about marking directories from disk: a record can be edited after the fact, and a recomputation cannot disagree with the run it describes. +**The clause that used to follow -- *"and the analysis layer will read"* -- did not come true.** `members.txt` is a record, rewritten every run, and nothing under `analysis/` reads it. The analysis layer recomputes `variantPlan()` rather than parsing what the pipeline left on disk, which is the better answer for the reason this file already gives about marking directories from disk: a record can be edited after the fact, and a recomputation cannot disagree with the run it describes. ## Labeling folders by step -A folder can belong to several steps — `VCF/` is written by 6, rewritten by 7, annotated by 8 — so the steps that fill a folder are accumulated across variants rather than taken from the first one seen. Each folder is labeled with the steps that fill *it*, not with its group's whole step list. +A folder can belong to several steps -- `VCF/` is written by 6, rewritten by 7, annotated by 8 -- so the steps that fill a folder are accumulated across variants rather than taken from the first one seen. Each folder is labeled with the steps that fill *it*, not with its group's whole step list. Descriptions resolve **first-wins**: the specific entries are added before the general ones, so a general entry can never overwrite a folder's own line. diff --git a/.claude/development-notes/empty-results.md b/.claude/development-notes/empty-results.md new file mode 100644 index 0000000..7e39ff3 --- /dev/null +++ b/.claude/development-notes/empty-results.md @@ -0,0 +1,84 @@ +# A run that found nothing used to report success + +**Written 2026-09-29, against the tree at `85eb380` plus uncommitted work.** The guard `mapq-downgrade.md` said should be built whatever happened to the default. Building it turned up a second finding that made the shipped documentation wrong, so both are here. + +## The hole, measured end to end + +A header-only VCF - exactly what `bcftools mpileup` emits when the pileup reaches it empty - walked through every stage of step 7 by hand: + +| stage | exit | rows out | +|---|---|---| +| `MajorAlleleToRef.py` | 0 | 0 | +| `filterFalsePositives.sh` | 0 | 0 | +| `bcftools view -e` (depth) | 0 | 0 | +| `vcftools --minQ --recode` | **0** | 0 | +| `vcftools --remove-indels --recode` | **0** | 0 | +| `createDepthFile.sh` | 0 | header only | +| `depth2freq.awk` | 0 | header only | + +**Every stage succeeds.** The two `vcftools` calls were the links nobody had checked and they exit 0 as well. What the user was left with was a frequency table holding one line of column names, from a run that reported success at every step. + +Nothing in step 7 could have caught it: all seventeen of its conditionals are `[ -f ]`, and there is no `test -s`, no row count and no `exit 1` anywhere in the file. The two helpers that look like they might check only look at headers - `createDepthFile.sh` refuses when `#CHROM` carries no sample columns, `filterFalsePositives.sh` when `bcftools query -l` returns none - and an empty VCF satisfies both, because bcftools preserves the header. + +## The trap a naive guard falls into + +**Eleven `touch` statements in step 7 write deliberate zero-byte placeholders.** Every process checks whether anything LATER in the chain already exists and emits a stub if so, which is what makes a partially completed step 7 resumable. A guard written as "the file is empty, refuse" fires on every one of those skips. + +So all three guards sit **inside the `else` branch only**, where real work happened. A skip cannot reach them. + +## Three guards, one per door + +Matching the `capBAM.histogramMax` refusal in `5_reports.nf`, which is the house shape: tagged stderr, the parameter named, whether it costs anything already produced, `exit 1`. + +| where | catches | names | +|---|---|---| +| `6_variant_call.nf`, after `bcftools call` | the call set is empty | `variantCall.scaleMapQ`, and prints the composed `mpileupOptions` | +| `7_vcf2freq.nf`, after the cross-sample filter | every site filtered out | `filterFalsePositives.sampleThreshold`, `poolSize` | +| `7_vcf2freq.nf`, after depth and quality | every site filtered out | `minDP`, `minQUAL`, `dropZeroDepth` | + +**`00_static`'s parameter-map check caught the first draft.** The step 6 message interpolated `variantCall.scaleMapQ` and `variantCall.varQualMin`, which step 6 had never read - it reads only the composed `mpileupOptions`. Declaring them in `stepParameterMap()` would have been the wrong fix: a project that pins `mpileupOptions` by hand makes both inert, and two runs would then be held apart by settings that did nothing, which is exactly the trap step 7's own comment warns about in the other direction. Printing the composed string, already declared, is correct AND more informative. + +## The second finding: the destructive band is not a range + +The template said *"Below 11 it does nothing; 11 to about 20 discards every read."* The second half is wrong. + +`-C` caps each read's adjusted mapping quality near its own value, and `-q` (`variantCall.varQualMin`) then rejects anything below the minimum. **So the band is wherever `scaleMapQ` sits below `varQualMin`** - a relationship between two settings, not a fixed range. "11 to about 20" was an artifact of measuring only at the shipped `varQualMin = 30`. + +Sites called, three pools, `2L:5000000-5050000`, cohort call with the release's options, against 695 with the adjustment off: + +| `scaleMapQ` | `-q` 10 | 20 | 30 | 40 | +|---|---|---|---|---| +| off, 0, 5, 10 | 695 | 695 | 695 | 695 | +| 11 | 17 | **0** | **0** | **0** | +| 15 | 65 | **0** | **0** | **0** | +| 20 | 112 | 32 | **0** | **0** | +| 25 | 205 | 163 | **0** | **0** | +| 29 | 279 | 203 | **0** | **0** | +| 30 | 297 | 217 | 44 | **0** | +| 35 | 374 | 368 | 251 | **0** | +| 40 | 460 | 441 | 305 | 52 | +| 50 | 545 | 545 | 490 | 316 | +| 70 | 620 | 617 | 612 | 573 | +| 100 | 648 | 644 | 644 | 641 | + +**The `-q 10` column is the proof: no dead band at all**, because 11 already clears the minimum. Then checked directly at six thresholds - `varQualMin` 12, 15, 20, 30, 40, 50. `scaleMapQ` one below returned zero every time; `scaleMapQ` equal to it returned sites every time. + +**Clearing the minimum is not being safe.** At the default `-q 30`, `-C 30` returns 44 of 695 sites - 6%. That reproduces the whole-genome figure in `mapq-downgrade.md` (94,681 against `-C 100`'s 1,716,020, 5.5%), so the region is representative and the two measurements corroborate each other. + +A synthetic grid pointed at the mechanism first, but its reads were identical clones at MAPQ 60, so it was all-or-nothing and put the boundary in the wrong place. **Every number above is from real pools.** The synthetic is recorded here only because it is what suggested looking at `-q` at all. + +## The check before the run + +`bin/check_project.sh` now reports the relationship, so a project is told before it spends an afternoon. It reads the **composed** `mpileupOptions` rather than the two settings, so a project that pinned the option string by hand is judged on what will actually run, and one that pinned it without a `-C` is reported as unjudged rather than guessed at. + +Verdicts: `0` is off; at or below 10 is inert, with a note to write 0 if that was the intent; below `varQualMin` is a failure; below twice `varQualMin` is a warning carrying the measured 6%; above that passes. + +## Coverage + +- Three cases in `04_pipeline`, one per door, each a full run entering the empty state a different way. **Proven to bite**: with the guards reverted all three fail, and the assertion that fails alongside the exit status is `no frequency table should have been written` - the run completed and published the empty table. +- Seven cases in `02_launcher` for the `check_project.sh` verdicts, against a **stubbed `nextflow`** so they need no JVM. Proven to bite by deleting the check outright: all seven fail. +- The case that matters most for the future is `check project accepts a scale that clears a lower minimum`: the same `-C 15` that is fatal at `-q 30` is fine at `-q 5`. A check written against a constant would call it broken, which is the mistake the old template comment encoded. + +## Still open + +`0_verify_environment.nf` validates no parameters at all. The `scaleMapQ` relationship is caught by `check project`, which is optional, and by step 6, which is thirteen minutes in. Step 0 is where it would cost nothing - but step 0 has no parameter-validation section to add to, and building one is its own piece of work. diff --git a/.claude/development-notes/experimental-design.md b/.claude/development-notes/experimental-design.md index 6c42f49..14d5475 100644 --- a/.claude/development-notes/experimental-design.md +++ b/.claude/development-notes/experimental-design.md @@ -4,7 +4,7 @@ ## The defect -`designSummary()` computed `roles`, `units` and `conditions` inside `if (checkTimeSettings(...))`. They are not derived from time — they were merely *written* there, because the whole block arrived with the time-series work. A project with no time course therefore reported: +`designSummary()` computed `roles`, `units` and `conditions` inside `if (checkTimeSettings(...))`. They are not derived from time -- they were merely *written* there, because the whole block arrived with the time-series work. A project with no time course therefore reported: ``` UNITS : [] CONDITIONS : [] ROLES : [condition:[], biological:[], technical:[]] @@ -13,11 +13,11 @@ EXPERIMENTAL DESIGN: exp_cage (3 levels), exp_lane (2 levels) TIME VARIABLE: none - no time column, so nothing is a trajectory ``` -Three cages sequenced on two lanes, and the report never mentions that `exp_lane` might be a technical replicate. Degrees of freedom come from `design.units` (module rule 17c), so the commonest design there is — a one-off comparison — handed every module a zero it could not tell from an unanswered question. +Three cages sequenced on two lanes, and the report never mentions that `exp_lane` might be a technical replicate. Degrees of freedom come from `design.units` (module rule 17c), so the commonest design there is -- a one-off comparison -- handed every module a zero it could not tell from an unanswered question. **Why it survived.** `12_analysis_design` had 29 cases and not one of them touched units, conditions or roles; every case that did lived in `14_analysis_series`, behind a `timeVar` block. The untimed path was never asserted on, so it was never wrong out loud. -**The time-series half was sound and is unchanged.** Verified before touching anything, by running the 16-pool replicate fixture through `designSummary()`: 8 series, 2 units, 1 condition, 4 technical — identical before and after. +**The time-series half was sound and is unchanged.** Verified before touching anything, by running the 16-pool replicate fixture through `designSummary()`: 8 series, 2 units, 1 condition, 4 technical -- identical before and after. ## Z's rule @@ -33,7 +33,7 @@ The rule has to be a partition, which rules out the obvious statements of it. **"Two pools are one unit when they agree on the condition and biological columns AND differ in at least one technical column"** is not transitive. With `technicalRep = ['exp_lane']` and pools A/L1, A/L1', A/L2: the first merges with the third, the second merges with the third, and the first two do not merge with each other. No partition exists. -**What works** is to give the untimed case the same shape the timed case already had. The timed path rolls up *series*, and two series can never share a key — so with `technicalRep` empty each series is already its own unit. Generalise the atom: +**What works** is to give the untimed case the same shape the timed case already had. The timed path rolls up *series*, and two series can never share a key -- so with `technicalRep` empty each series is already its own unit. Generalise the atom: | | the atom | rolled up by | |---|---|---| @@ -42,26 +42,26 @@ The rule has to be a partition, which rules out the obvious statements of it. `designMembers()` returns one or the other. `unitsOf()` then groups members by the condition-plus-biological key and, **within each group**, buckets by the technical key: -- every member has its own technical key → one unit (they are repeats of one material) -- every member shares one technical key → one unit per member (nothing tells them apart, so nothing merges them) -- partly one and partly the other → **refuse**, because no partition exists +- every member has its own technical key -> one unit (they are repeats of one material) +- every member shares one technical key -> one unit per member (nothing tells them apart, so nothing merges them) +- partly one and partly the other -> **refuse**, because no partition exists The second case is what `technicalRep = []` degenerates to, and it is why an undeclared project gets one unit per pool. The third is unreachable when there is a time axis, because series keys are unique; it is the untimed twin of the existing "two pools at one timepoint in one series" refusal and offers the same remedies. ## The settings split -`by`, `biologicalRep` and `technicalRep` describe the experiment and were living in `analysis.metadata.series`, which asserted that an experiment without a time course has no design. They moved to `analysis.metadata.design`. `analysis.metadata.series` keeps `incomplete` alone — only a time axis can leave a trajectory ragged. +`by`, `biologicalRep` and `technicalRep` describe the experiment and were living in `analysis.metadata.series`, which asserted that an experiment without a time course has no design. They moved to `analysis.metadata.design`. `analysis.metadata.series` keeps `incomplete` alone -- only a time axis can leave a trajectory ragged. -Z had picked `analysis.design` when asked; it became `analysis.metadata.design` because every sibling setting lives under `analysis.metadata`, and a top-level scope would collide with a module named `design` — the collision `paths.nf` keeps the nesting for. +Z had picked `analysis.design` when asked; it became `analysis.metadata.design` because every sibling setting lives under `analysis.metadata`, and a top-level scope would collide with a module named `design` -- the collision `paths.nf` keeps the nesting for. -**Corrected the same day, by Z, and my reason was wrong.** A module's settings live at `analysis.modules.`, one level below the frame's own scopes, which is exactly what stops a module named `design` reading a frame setting — so there was never a collision to avoid. It moved to a top-level **`analysis.design`**, with `series` nested inside it, on Z's call: *"I think design should move out of metadata (after metadata) but inherit series inside it."* +**Corrected the same day, by Z, and my reason was wrong.** A module's settings live at `analysis.modules.`, one level below the frame's own scopes, which is exactly what stops a module named `design` reading a frame setting -- so there was never a collision to avoid. It moved to a top-level **`analysis.design`**, with `series` nested inside it, on Z's call: *"I think design should move out of metadata (after metadata) but inherit series inside it."* That gives three scopes with a real distinction, and it is worth stating because it decides where a future setting goes: | scope | question it answers | |---|---| -| `analysis.metadata` | how the file is **read** — what a blank means, how a date parses, what scale a column carries | -| `analysis.design` | what the file **describes** — which pools are independent, which repeats are which | +| `analysis.metadata` | how the file is **read** -- what a blank means, how a date parses, what scale a column carries | +| `analysis.design` | what the file **describes** -- which pools are independent, which repeats are which | | `analysis.modules.` | what **this analysis** does with either | `series` sits inside `design` rather than beside it because a series is what a time axis makes of the design, not a thing of its own. `timeVar` stays in `metadata`: it says how the time column is *parsed and ordered*, which is reading. @@ -72,7 +72,7 @@ That gives three scopes with a real distinction, and it is worth stating because ## What a module now reads -`design.units` and `design.conditions` each carry their own `pools`, sorted as `design.pools` is, so a module indexes the published columns with them instead of joining back through `design.series` — which is empty when there is no time axis. A unit carries `members` (its series, or its pool); a condition carries `units`. +`design.units` and `design.conditions` each carry their own `pools`, sorted as `design.pools` is, so a module indexes the published columns with them instead of joining back through `design.series` -- which is empty when there is no time axis. A unit carries `members` (its series, or its pool); a condition carries `units`. `design.seriesBy` became `design.keyColumns`, because it is what identifies the design and not only what identifies a series. @@ -80,25 +80,25 @@ That gives three scopes with a real distinction, and it is worth stating because Z, the same day: *"I think we should also open a covariates parameter under design. automatically detected from metadata but configurable if the user wants to keep or leave some out?"* -Same split as the `exp_` columns: `analysis.metadata.covariates.` says **what the column holds** (its scale), `analysis.metadata.design.covariates` says **what part it plays** (whether a module may fit it). Empty means every `cov_` column that has a declared scale — Z's "automatically detected". An undeclared column cannot be in the set because it has no typed value, and naming one refuses with the declaration that would fix it. +Same split as the `exp_` columns: `analysis.metadata.covariates.` says **what the column holds** (its scale), `analysis.metadata.design.covariates` says **what part it plays** (whether a module may fit it). Empty means every `cov_` column that has a declared scale -- Z's "automatically detected". An undeclared column cannot be in the set because it has no typed value, and naming one refuses with the declaration that would fix it. -**Every covariate stays in `design.covariates`**, each carrying `inDesign`, rather than the excluded ones being filtered out. The confounding argument the covariate prefix exists for — a reader seeing that the high-phenotype pools were also the warm ones — needs the values of the excluded ones as much as the included ones. The cost is a flag a module could forget to check, which is why it is module rule 17e and why the report marks every line `[in the design]` or `[on the record only]`. +**Every covariate stays in `design.covariates`**, each carrying `inDesign`, rather than the excluded ones being filtered out. The confounding argument the covariate prefix exists for -- a reader seeing that the high-phenotype pools were also the warm ones -- needs the values of the excluded ones as much as the included ones. The cost is a flag a module could forget to check, which is why it is module rule 17e and why the report marks every line `[in the design]` or `[on the record only]`. **An exclusion is a warning, not silence.** Leaving a covariate out is as much a decision as putting one in, and neither is visible from the values. -This makes the manual's `#covariates-not-adjusted` section wrong in its old form — *"PoolSeqFlow does not correct any result for a covariate, and will not in 3.0.0"* — because F0f settled that F2 fits them as extra predictors. Rewritten to say what is true at both times: the frame adjusts for nothing, a module says what it fits, and the arithmetic of `n_units - 2` is why the choice is the user's. +This makes the manual's `#covariates-not-adjusted` section wrong in its old form -- *"PoolSeqFlow does not correct any result for a covariate, and will not in 3.0.0"* -- because F0f settled that F2 fits them as extra predictors. Rewritten to say what is true at both times: the frame adjusts for nothing, a module says what it fits, and the arithmetic of `n_units - 2` is why the choice is the user's. ## Phenotypes became plural, on the covariate shape -Z, the same day, proposing the shape themselves: `metadata.phenotypes { pt_wingspan { kind = … } pt_wingpattern { kind = …; levels = … } }`. +Z, the same day, proposing the shape themselves: `metadata.phenotypes { pt_wingspan { kind = ... } pt_wingpattern { kind = ...; levels = ... } }`. -`analysis.metadata.phenotype` was one scope with a `column` key, which asserted that a project has one phenotype. It is now `analysis.metadata.phenotypes`, an open scope with one block per column — the shape `covariates` already had — so the block name *is* the column and the `column` key is gone. `checkScaleDeclaration()` was already shared between the two, so the four kinds and their level rules needed no new code; `checkPhenotypeSettings`/`resolvePhenotype` became the plural forms of the covariate pair, and gained the same undeclared-column warning. +`analysis.metadata.phenotype` was one scope with a `column` key, which asserted that a project has one phenotype. It is now `analysis.metadata.phenotypes`, an open scope with one block per column -- the shape `covariates` already had -- so the block name *is* the column and the `column` key is gone. `checkScaleDeclaration()` was already shared between the two, so the four kinds and their level rules needed no new code; `checkPhenotypeSettings`/`resolvePhenotype` became the plural forms of the covariate pair, and gained the same undeclared-column warning. **Which phenotype an analysis tests against is a module setting**, `analysis.modules..phenotypes`, and it lands with F2 rather than now: `moduleSettings()` checks a scope against *that module's* declared list, so there is nowhere to put it until the module exists. What the frame does today is resolve every declared phenotype and report all of them. That retires the rule that a project analyzes one phenotype at a time under its own `folderName`. `folderName` goes back to being only about where output lands. -**One thing measured rather than assumed:** an empty config block reaches `params` as *nothing*, so `pt_wingspan { }` cannot be refused — the frame never sees the key. Recorded in `shell-and-nextflow-gotchas.md`, with a case that asserts the behavior rather than a case that asserts a refusal which cannot happen. The undeclared-column warning is what covers it. +**One thing measured rather than assumed:** an empty config block reaches `params` as *nothing*, so `pt_wingspan { }` cannot be refused -- the frame never sees the key. Recorded in `shell-and-nextflow-gotchas.md`, with a case that asserts the behavior rather than a case that asserts a refusal which cannot happen. The undeclared-column warning is what covers it. ## The report @@ -117,8 +117,8 @@ SERIES: 8 series over 2 timepoints ## What is still undefended, and it is the same gap as before -A technical column left out of `technicalRep` is read as a condition and the unit count doubles. No check can catch it — both readings are internally consistent, exactly as `dd/MM` and `MM/dd` are. Printing every key column under exactly one role is the whole defense, and it is now printed for untimed projects too, which is where it was missing. +A technical column left out of `technicalRep` is read as a condition and the unit count doubles. No check can catch it -- both readings are internally consistent, exactly as `dd/MM` and `MM/dd` are. Printing every key column under exactly one role is the whole defense, and it is now printed for untimed projects too, which is where it was missing. ## A fixture that describes an impossible state, not fixed here -`test/tools/freq_corpus.py`'s untimed `design.json` claims `time: null` while the corpus metadata has an `exp_time` column. `checkTimeSettings()` refuses that combination — a project with an `exp_time` column and no declared `kind` stops the run — so the frame cannot emit that summary. It is a planted artifact and no module consults the frame for it, so nothing fails; but it is a fixture asserting against a shape the frame never produces, which is the trap hand-written VCFs already sprang once. Reachable only via `timeVar { column = 'exp_something_absent' }`. Left for F2, whose fixture it is. +`test/tools/freq_corpus.py`'s untimed `design.json` claims `time: null` while the corpus metadata has an `exp_time` column. `checkTimeSettings()` refuses that combination -- a project with an `exp_time` column and no declared `kind` stops the run -- so the frame cannot emit that summary. It is a planted artifact and no module consults the frame for it, so nothing fails; but it is a fixture asserting against a shape the frame never produces, which is the trap hand-written VCFs already sprang once. Reachable only via `timeVar { column = 'exp_something_absent' }`. Left for F2, whose fixture it is. diff --git a/.claude/development-notes/false-positive-filter.md b/.claude/development-notes/false-positive-filter.md index 1d904aa..096a690 100644 --- a/.claude/development-notes/false-positive-filter.md +++ b/.claude/development-notes/false-positive-filter.md @@ -2,43 +2,43 @@ **Written 2026-08-31, against the tree at `7d65893`.** The mechanism and the contract are unchanged. `diploidy` has been renamed `ploidy` since, and the parameter names below are updated to the live ones; the old name survives only in `config_migrate.sh`, which maps it forward. -`bin/filterFalsePositives.sh`, called from step 7. The only place in the pipeline where a per-sample number decides which variants survive, so a mistake here is a wrong **result** rather than a failed run — nothing downstream looks odd, the frequency tables are simply computed over the wrong set of sites. +`bin/filterFalsePositives.sh`, called from step 7. The only place in the pipeline where a per-sample number decides which variants survive, so a mistake here is a wrong **result** rather than a failed run -- nothing downstream looks odd, the frequency tables are simply computed over the wrong set of sites. -> **For the manual, not here.** `s = 1 / (2 * ploidy * poolSize)` is the smallest allele fraction a pool of that size can produce — one chromosome out of all of them — so anything below it is noise rather than a rare allele. A reader interpreting a frequency table needs that, and needs to know it is now **per pool** rather than one number for the whole VCF. +> **For the manual, not here.** `s = 1 / (2 * ploidy * poolSize)` is the smallest allele fraction a pool of that size can produce -- one chromosome out of all of them -- so anything below it is noise rather than a rare allele. A reader interpreting a frequency table needs that, and needs to know it is now **per pool** rather than one number for the whole VCF. ## Choosing the mechanism, 2026-08-29 -The threshold used to be uniform: `COUNT(FORMAT/AD[:1]/FORMAT/DP[:] >= s) >= MINSAMPLES`, one `s` for the whole file — so a pool of 10 was judged at a pool of 100's resolution. +The threshold used to be uniform: `COUNT(FORMAT/AD[:1]/FORMAT/DP[:] >= s) >= MINSAMPLES`, one `s` for the whole file -- so a pool of 10 was judged at a pool of 100's resolution. -Four mechanisms were measured on a real 135-record, 6-sample VCF (now `test/data/vcf/called.vcf`), with `sampleThreshold = 0.8` and pools 4–6 shrunk to one individual so per-pool and uniform genuinely differ — **118 records versus 102**: +Four mechanisms were measured on a real 135-record, 6-sample VCF (now `test/data/vcf/called.vcf`), with `sampleThreshold = 0.8` and pools 4-6 shrunk to one individual so per-pool and uniform genuinely differ -- **118 records versus 102**: | | Correct? | Cost | Residue | Binds by | |---|---|---|---|---| -| M1 — annotate a FORMAT tag, filter, strip | yes, 102 | +5 processes, 2 temp files, tabix index | tag must be stripped | column position | -| M2 — boolean summation of subscripted clauses | **no — 0 records, silently** | | | | -| M3 — awk → `INFO/FPKEEP` → `view -i` → `annotate -x` | yes, 102 | +5 processes, 2 temp files, tabix index | 3 `##bcftools_*Command` lines naming FPKEEP survive the strip | column position | -| **M4 — in-stream awk** | yes, 102 | **+1 awk in the existing pipeline** | none | **sample name** | +| M1 -- annotate a FORMAT tag, filter, strip | yes, 102 | +5 processes, 2 temp files, tabix index | tag must be stripped | column position | +| M2 -- boolean summation of subscripted clauses | **no -- 0 records, silently** | | | | +| M3 -- awk -> `INFO/FPKEEP` -> `view -i` -> `annotate -x` | yes, 102 | +5 processes, 2 temp files, tabix index | 3 `##bcftools_*Command` lines naming FPKEEP survive the strip | column position | +| **M4 -- in-stream awk** | yes, 102 | **+1 awk in the existing pipeline** | none | **sample name** | -**M2 is the finding worth keeping.** It parses and returns zero records with no error at all. The bcftools expression engine will accept an expression it evaluates differently than you expect, which is also the argument against trusting M1's three-vector arithmetic — that one does work, verified, but it is trusted rather than seen. +**M2 is the finding worth keeping.** It parses and returns zero records with no error at all. The bcftools expression engine will accept an expression it evaluates differently than you expect, which is also the argument against trusting M1's three-vector arithmetic -- that one does work, verified, but it is trusted rather than seen. -**Name-binding is why M4 wins on correctness, not just cost.** M1 and M3 bind each threshold to the sample's column *position*, and this pipeline deliberately treats the metadata file's row order as a separate identity from its values — precisely because order decides the VCF's columns. A positional binding is one permutation away from applying the wrong pool's threshold and silently changing which variants survive. Measured: the same sizes assigned to samples 1–3 instead of 4–6 give **99** records, not 102. Matching on the VCF's own sample names cannot do that, and lets the filter **refuse** a column it was given no size for — which no positional mechanism can. +**Name-binding is why M4 wins on correctness, not just cost.** M1 and M3 bind each threshold to the sample's column *position*, and this pipeline deliberately treats the metadata file's row order as a separate identity from its values -- precisely because order decides the VCF's columns. A positional binding is one permutation away from applying the wrong pool's threshold and silently changing which variants survive. Measured: the same sizes assigned to samples 1-3 instead of 4-6 give **99** records, not 102. Matching on the VCF's own sample names cannot do that, and lets the filter **refuse** a column it was given no size for -- which no positional mechanism can. -M3 was the front-runner before the measurement. Z confirmed M4 after seeing the comparison, noting it crosses the recorded constraint *"bcftools does all VCF surgery; awk only touches numbers"* — though the two awk passes already in the script rewrite the ALT column, which is more invasive than dropping a line. +M3 was the front-runner before the measurement. Z confirmed M4 after seeing the comparison, noting it crosses the recorded constraint *"bcftools does all VCF surgery; awk only touches numbers"* -- though the two awk passes already in the script rewrite the ALT column, which is more invasive than dropping a line. ## The contract -`-p "Name=count,..."` carries **sizes, not sensitivities**, with `-d `. That keeps the equation in one place on the per-pool path and puts the number the user actually wrote into the `##PoolSeqFlowPool` header line. `-s` remains the flat default for anyone running the script by hand, and `-p` without `-d` is refused rather than defaulting to 2 — a wrong ploidy silently halves or doubles every threshold. +`-p "Name=count,..."` carries **sizes, not sensitivities**, with `-d `. That keeps the equation in one place on the per-pool path and puts the number the user actually wrote into the `##PoolSeqFlowPool` header line. `-s` remains the flat default for anyone running the script by hand, and `-p` without `-d` is refused rather than defaulting to 2 -- a wrong ploidy silently halves or doubles every threshold. Verified byte-identical to the pre-E3b filter when every pool is the same size. ## Provenance that does not survive -The `##PoolSeqFlowPool` header lines record what was applied per column — but **every VCF step 7 touches is transient by design**, so they never reach a published file. The durable record is `.poolseqflow_metadata`, which carries `param_poolSize` because it is a guarded column. Putting the sizes somewhere that survives would mean a header on the frequency tables, which is an E4 question about the TSV contract. +The `##PoolSeqFlowPool` header lines record what was applied per column -- but **every VCF step 7 touches is transient by design**, so they never reach a published file. The durable record is `.poolseqflow_metadata`, which carries `param_poolSize` because it is a guarded column. Putting the sizes somewhere that survives would mean a header on the frequency tables, which is an E4 question about the TSV contract. **E4 answered it, and not that way.** No header was added to the tables. The analysis layer recomputes the figures onto its own target instead: `analysis/lib/nf/pools.nf` holds `poolSensitivity(ploidy, size)`, and a module reads `size`, `ploidy`, `nChrom` and `sensitivity` per pool off `target.pools`. What makes the recomputation safe rather than a second opinion is that **step 7's identity names the pool sizes and the ploidy**, so a results directory can only ever hold runs that were filtered under both. ## Testing -Fifteen cases in the helpers suite, each mutation-tested: `>=`→`>`, the zero-depth guard, the FORMAT lookup and the strict refusal each break exactly one case. The exact-boundary case uses `chr1:8105`, where `TestSample2` has AD=1 and DP=72 — a fraction of exactly 1/72, which is exactly the sensitivity of a pool of 18, since `1/(2*2*18)` is the same division. With the sample threshold at 1.0 every pool must pass, so that one sample decides the record: pool 18 keeps it, pool 17 drops it. That is what proves the comparison is `>=` and not `>`. +Fifteen cases in the helpers suite, each mutation-tested: `>=`->`>`, the zero-depth guard, the FORMAT lookup and the strict refusal each break exactly one case. The exact-boundary case uses `chr1:8105`, where `TestSample2` has AD=1 and DP=72 -- a fraction of exactly 1/72, which is exactly the sensitivity of a pool of 18, since `1/(2*2*18)` is the same division. With the sample threshold at 1.0 every pool must pass, so that one sample decides the record: pool 18 keeps it, pool 17 drops it. That is what proves the comparison is `>=` and not `>`. -**`pipefail` is what matters in this script.** Its real work is one long pipeline whose last stage is awk, and awk succeeds on empty input — so without it every bcftools in the chain could fail while the script exited 0, emitting an empty VCF that the caller publishes to permanent storage, where the existence-based skip logic reuses it on every later run. +**`pipefail` is what matters in this script.** Its real work is one long pipeline whose last stage is awk, and awk succeeds on empty input -- so without it every bcftools in the chain could fail while the script exited 0, emitting an empty VCF that the caller publishes to permanent storage, where the existence-based skip logic reuses it on every later run. diff --git a/.claude/development-notes/four-roots.md b/.claude/development-notes/four-roots.md index a54ab3f..5e57a7e 100644 --- a/.claude/development-notes/four-roots.md +++ b/.claude/development-notes/four-roots.md @@ -15,23 +15,23 @@ A run stands in four directories, and confusing any two of them is a real failur ## The correction that started the campaign -`mainDir` is a **per-project working directory, not the checkout**. `dir.scripts` and `dir.bin` encoded the wrong model from the first working version in March 2026: they pointed at the source tree, which conflated "where the code is" with "where this project's work happens". Everything in the four-roots block (E1j–E1n) follows from unpicking that. Of the two names, only `dir.bin` still exists — `dir.scripts` is gone, so grepping for it finds nothing. +`mainDir` is a **per-project working directory, not the checkout**. `dir.scripts` and `dir.bin` encoded the wrong model from the first working version in March 2026: they pointed at the source tree, which conflated "where the code is" with "where this project's work happens". Everything in the four-roots block (E1j-E1n) follows from unpicking that. Of the two names, only `dir.bin` still exists -- `dir.scripts` is gone, so grepping for it finds nothing. ## Why the rules are what they are -**`mainDir` and `storageDir` are two tiers, not two names for one place.** Outputs are written to the working volume and promoted to permanent storage once whatever consumes them has succeeded. Pointing both at one directory makes every promotion a no-op that moves a file onto itself, and makes `clean` and `reset` — which treat the two roots differently — impossible to reason about. So they must differ. +**`mainDir` and `storageDir` are two tiers, not two names for one place.** Outputs are written to the working volume and promoted to permanent storage once whatever consumes them has succeeded. Pointing both at one directory makes every promotion a no-op that moves a file onto itself, and makes `clean` and `reset` -- which treat the two roots differently -- impossible to reason about. So they must differ. **Neither storage root may BE the installation.** The installation is a tool: one copy serves any number of projects, it is replaced wholesale on upgrade, and from 3.0 it may be read-only and shared. A project working directory kept inside it would be destroyed by an upgrade and would make two projects impossible; permanent storage kept inside it would take the results and their provenance with it. -**Compared as resolved paths.** A string comparison passes happily on `/data/x` versus `/data/x/`, versus `/data/y/../x`, versus a symlink to the same directory — each of those is the same directory with a different spelling. +**Compared as resolved paths.** A string comparison passes happily on `/data/x` versus `/data/x/`, versus `/data/y/../x`, versus a symlink to the same directory -- each of those is the same directory with a different spelling. -**Containment is warned about, not rejected** (Z). One root inside another is harmless in itself — the managed subdirectories still do not collide — but the lifetimes differ sharply enough to be worth saying out loud. The check that actually matters for collisions is that no two computed directories in the `dir` block resolve alike, and that belongs where the block is built. +**Containment is warned about, not rejected** (Z). One root inside another is harmless in itself -- the managed subdirectories still do not collide -- but the lifetimes differ sharply enough to be worth saying out loud. The check that actually matters for collisions is that no two computed directories in the `dir` block resolve alike, and that belongs where the block is built. -**`mainDir` must be durable.** It holds the configs, `Data/` and `Reference/` — the only irreplaceable things in a project. The pre-3.0 documentation describing it as node scratch is wrong from 3.0 on. +**`mainDir` must be durable.** It holds the configs, `Data/` and `Reference/` -- the only irreplaceable things in a project. The pre-3.0 documentation describing it as node scratch is wrong from 3.0 on. ## The tiering problem this solved -On a cluster with node SSDs and a SATA archive, the pipeline never used the fast disk: every `dir.*` resolved to `storageDir`, so `mainDir` held only `work/`, which holds symlinks. Z raised it alongside "run the same reads against several references", and the two turned out to share one root cause — which is how storage tiering and multi-run became one block of work. +On a cluster with node SSDs and a SATA archive, the pipeline never used the fast disk: every `dir.*` resolved to `storageDir`, so `mainDir` held only `work/`, which holds symlinks. Z raised it alongside "run the same reads against several references", and the two turned out to share one root cause -- which is how storage tiering and multi-run became one block of work. ## The one contract amendment that held, and the one that did not @@ -40,7 +40,7 @@ Z, 2026-08-24 wanted both avoided, to keep the wrapper at exactly one subcommand - **The install prefix is an environment variable** (`POOLSEQFLOW_PREFIX`), not a flag. **This held.** - The analysis layer was given a wrapper of its own rather than a flag-taking subcommand. **REVERSED, 2026-08-31.** Z merged it back into `./PoolSeqFlow`, and the contract IS amended: every subcommand takes no arguments except `analysis`, which takes exactly one. The separate executable is deleted. (It shipped briefly as `PoolSeqFlow-analysis`; the `PoolSeqFlow-analyze` spelling this note used never existed at all.) See `analysis-wrapper.md`. -**And it widened again the same evening.** The module store arrived in `d9886c5`, so `analysis` no longer takes exactly one word: a module run is `analysis [nocpp]` and the store is `analysis modules install []`. The amendment that survives is the *shape* — the top-level subcommands still take nothing, and everything variable hangs off `analysis`. `usage_analysis` in the wrapper is the current statement of it. +**And it widened again the same evening.** The module store arrived in `d9886c5`, so `analysis` no longer takes exactly one word: a module run is `analysis [nocpp]` and the store is `analysis modules install []`. The amendment that survives is the *shape* -- the top-level subcommands still take nothing, and everything variable hangs off `analysis`. `usage_analysis` in the wrapper is the current statement of it. ## The wrapper's install prefix, and why it is an environment variable @@ -48,9 +48,9 @@ Z, 2026-08-24 wanted both avoided, to keep the wrapper at exactly one subcommand **An environment variable rather than a `--prefix` flag.** The wrapper takes exactly one subcommand and rejects everything else, and that contract was worth keeping. `make install PREFIX=...` has the same shape. -**When neither is given, an installed wrapper works it out from its own stamp.** A payload always sits at `/opt/PoolSeqFlow-`, so two components back is the prefix. That is what makes `list` and `uninstall` correct without the user having to remember which prefix they installed into — otherwise both would look under `~/.local`, find nothing, and report that nothing is installed while several copies sat elsewhere. A wrapper run from a clone has no stamp and falls back to the default, which is the case where a prefix is being chosen rather than recalled. +**When neither is given, an installed wrapper works it out from its own stamp.** A payload always sits at `/opt/PoolSeqFlow-`, so two components back is the prefix. That is what makes `list` and `uninstall` correct without the user having to remember which prefix they installed into -- otherwise both would look under `~/.local`, find nothing, and report that nothing is installed while several copies sat elsewhere. A wrapper run from a clone has no stamp and falls back to the default, which is the case where a prefix is being chosen rather than recalled. -**`PAYLOAD_ITEMS` must match what `git archive` produces**, which `.gitattributes` defines by exclusion — so installing from a clone and from a download deploy identical trees. A static case checks the two agree, so a file added to the release without being added here fails a test rather than going missing from every install. +**`PAYLOAD_ITEMS` must match what `git archive` produces**, which `.gitattributes` defines by exclusion -- so installing from a clone and from a download deploy identical trees. A static case checks the two agree, so a file added to the release without being added here fails a test rather than going missing from every install. **`POOLSEQFLOW_HOME`** overrides the pipeline a wrapper uses, for running a checkout without installing it. It wins over the deployed stamp, so a developer can point an installed wrapper at a working copy. diff --git a/.claude/development-notes/gates-that-stopped-checking.md b/.claude/development-notes/gates-that-stopped-checking.md index 5d030d9..adb7b9e 100644 --- a/.claude/development-notes/gates-that-stopped-checking.md +++ b/.claude/development-notes/gates-that-stopped-checking.md @@ -10,18 +10,18 @@ Six separate instances in one session, each independently discovered, each the s | what changed | what stopped checking | |---|---| -| `analysis/modules/` became the gitignored install store | `check-analysis-versions.sh` looped over an empty directory — **zero modules examined**, and a release could ship any manifest unbumped | +| `analysis/modules/` became the gitignored install store | `check-analysis-versions.sh` looped over an empty directory -- **zero modules examined**, and a release could ship any manifest unbumped | | the same | `bib2citations.py --check` reported *"every citations.json matches (2 files)"* with three module bibliographies unread | -| the same | `export-environment.sh`'s guard — half of what `RELEASING.md` calls the sharpest trap in the feature | +| the same | `export-environment.sh`'s guard -- half of what `RELEASING.md` calls the sharpest trap in the feature | | the same | `check-module-packages.sh`, the only real conda solve, asking it of an empty set | | the same | five `00_static` globs, `07_analysis_frame`, and `08_analysis_rlib`'s no-package grep, which printed `grep: ... No such file` and passed | -| `conda_install_packages` learned to skip satisfied pins | `check-module-packages.sh` again — every declared pin was already in the baseline, so the filter emptied the list, the function returned 0 without asking conda anything, and the gate printed `ok` | +| `conda_install_packages` learned to skip satisfied pins | `check-module-packages.sh` again -- every declared pin was already in the baseline, so the filter emptied the list, the function returned 0 without asking conda anything, and the gate printed `ok` | The last one is the sharpest because it was *mine*, made an hour after fixing the first five, in full knowledge of the pattern. ## Two ways a case hides it -**A fixture that agrees with the bug.** Both version-gate cases in `00_static` planted their `demo` module into `analysis/modules/` — the store — so the gate found it there and the case went green over a loop that finds nothing in reality. A fixture matching the defect is worse than no fixture: it is a standing assertion that the wrong thing is right. +**A fixture that agrees with the bug.** Both version-gate cases in `00_static` planted their `demo` module into `analysis/modules/` -- the store -- so the gate found it there and the case went green over a loop that finds nothing in reality. A fixture matching the defect is worse than no fixture: it is a standing assertion that the wrong thing is right. **A fixture that disables the check under test.** The first version of the `check install` environment cases named the fake environment directory `env` while passing `ENV_NAME=check-install-env`. The script only compares paths when `basename $CONDA_PREFIX` matches `ENV_NAME`, so the comparison was off and both cases passed over nothing. @@ -29,14 +29,14 @@ The last one is the sharpest because it was *mine*, made an hour after fixing th `make_pipeline_sandbox` copied `analysis/` whole, so a module installed into the checkout arrived in every sandbox and the sandbox stopped being a fresh installation. It surfaced only because Z installed `mds` by hand between the two releases: the roster case that asserts a fresh release offers `verify` alone then failed on Z's machine and would have passed on a clean one. The store is removed after the copy now, matching the tarball, which carries no `analysis/modules` at all. -The same case also asserted *"mds must not be sitting in the store of a checkout"* — which is not a property anyone should hold. Installing a module into the copy you are developing against is ordinary. It asserted the opposite and passed until somebody did it. +The same case also asserted *"mds must not be sitting in the store of a checkout"* -- which is not a property anyone should hold. Installing a module into the copy you are developing against is ordinary. It asserted the opposite and passed until somebody did it. ## What actually finds these **Not reasoning.** Every one was found by running something and reading the output, and twice the reasoning was confidently wrong: - I asserted that `conda_install_packages`' conflict check *had* to precede the filter or the refusal would break, and wrote a comment saying so. The mutation test disproved it: a spec held at another version is not satisfied, so it survives the filter either way. The comment was deleted. -- I argued 3.1.0 had to be 4.0.0 from `CHANGELOG.md`'s semver claim. Z asked *"How is this not adhering semver?"* — and the answer was that semver's own first clause requires a declared public API, which the project had never written down. Declaring it (**the public API is what a result depends on**) settled it and the semver claim stayed. +- I argued 3.1.0 had to be 4.0.0 from `CHANGELOG.md`'s semver claim. Z asked *"How is this not adhering semver?"* -- and the answer was that semver's own first clause requires a declared public API, which the project had never written down. Declaring it (**the public API is what a result depends on**) settled it and the semver claim stayed. **Mutation testing, every time.** Revert the fix, watch the case fail, restore. It caught the two vacuous fixtures above and disproved the ordering claim. A case written after the fact that does not fail on the original bug is worth nothing. @@ -44,4 +44,4 @@ The same case also asserted *"mds must not be sitting in the store of a checkout ## The counting habit that makes it visible -`RELEASING.md` already said *"a number in a message is worth reading twice"*. Today added a second number: **read the SKIP count, not only the failures.** A full run reported `549 passed, 3 skipped` and exit 0; the three that skipped were the PDF report, the compiled hot path, and the compiled-and-parallel agreement — and F1's Rcpp worker bug was caught by that last combination and by nothing else. They skipped because the shell had no working `conda`, so the suite found no analysis environment and said so quietly. That is in `RELEASING.md`'s post-release triage now, with the note that the suite should **refuse** rather than skip when it was asked for a full run. +`RELEASING.md` already said *"a number in a message is worth reading twice"*. Today added a second number: **read the SKIP count, not only the failures.** A full run reported `549 passed, 3 skipped` and exit 0; the three that skipped were the PDF report, the compiled hot path, and the compiled-and-parallel agreement -- and F1's Rcpp worker bug was caught by that last combination and by nothing else. They skipped because the shell had no working `conda`, so the suite found no analysis environment and said so quietly. That is in `RELEASING.md`'s post-release triage now, with the note that the suite should **refuse** rather than skip when it was asked for a full run. diff --git a/.claude/development-notes/host-glibc-floor.md b/.claude/development-notes/host-glibc-floor.md index 52d7426..c204323 100644 --- a/.claude/development-notes/host-glibc-floor.md +++ b/.claude/development-notes/host-glibc-floor.md @@ -1,10 +1,10 @@ -# The glibc floor — a shipped environment that only installed on the machine that froze it +# The glibc floor -- a shipped environment that only installed on the machine that froze it **Written 2026-09-21, against the tree at `d4c4028`.** Reported by Z from a cluster: `PoolSeqFlow analysis install` failed with `nothing provides __glibc >=2.39 needed by sysroot_linux-64-2.39-hc4b9eeb_6`. v3.1.1's analysis layer could not be installed on any host below glibc 2.39, which is most HPC, and every gate in the project passed. ## What it was -`install/environment-analysis.yml` pinned `sysroot_linux-64=2.39=hc4b9eeb_6`. That package declares `__glibc >=2.39` — a dependency on a **virtual package**, conda's name for a property of the machine rather than of anything installable. So the file described a requirement on the host, and the requirement was whatever the host that produced the file happened to have. +`install/environment-analysis.yml` pinned `sysroot_linux-64=2.39=hc4b9eeb_6`. That package declares `__glibc >=2.39` -- a dependency on a **virtual package**, conda's name for a property of the machine rather than of anything installable. So the file described a requirement on the host, and the requirement was whatever the host that produced the file happened to have. The maintainer's machine reports `__glibc=2.44`. It solved there, every release, invisibly. @@ -12,13 +12,13 @@ The maintainer's machine reports `__glibc=2.44`. It solved there, every release, **But the pipeline environment was never floor-free either**, which is what the first draft of this note and of the manual both said. `rsync` is in both files and requires `__glibc >=2.28`. So before the fix the pipeline environment needed 2.28 and the analysis environment needed 2.39, and afterwards both need 2.28. -**Z's observation is what settled the causality**, 2026-09-21: *"I could install PoolSeqFlow but not analysis to the cluster."* That brackets the machine. `rsync` is in both environments, so if it had been the obstacle the pipeline install would have failed too — it did not. The cluster therefore clears 2.28 and not 2.39, which is consistent with RHEL 9 at glibc 2.34. **The failure was `sysroot_linux-64=2.39` and nothing else**, exactly as the error said, and `rsync` is an unrelated lower bound that happened to surface later. Worth recording because the two were briefly conflated here: the package that sets the *declared floor* and the package that *broke the install* are different, and only the second one is the defect. +**Z's observation is what settled the causality**, 2026-09-21: *"I could install PoolSeqFlow but not analysis to the cluster."* That brackets the machine. `rsync` is in both environments, so if it had been the obstacle the pipeline install would have failed too -- it did not. The cluster therefore clears 2.28 and not 2.39, which is consistent with RHEL 9 at glibc 2.34. **The failure was `sysroot_linux-64=2.39` and nothing else**, exactly as the error said, and `rsync` is an unrelated lower bound that happened to surface later. Worth recording because the two were briefly conflated here: the package that sets the *declared floor* and the package that *broke the install* are different, and only the second one is the defect. ## Nothing asked for it Measured before choosing a fix. `gcc_impl_linux-64`, `gxx_impl_linux-64` and `binutils_impl_linux-64` all depend on a **bare** `sysroot_linux-64` with no version constraint, and conda-forge publishes 2.12, 2.17, 2.28, 2.34 and 2.39. The solver was free to take the newest the host allowed and did; `conda update --all` in `prep-version.sh` would have done it again every release. -`libsanitizer=16.2.0` requires `__glibc >=2.17,<3.0.a0`, and measuring a surviving environment showed conda-forge's whole base stack — `libgcc`, `libstdcxx`, `python`, `openssl`, `numpy` and twenty others — sitting at exactly `>=2.17`. So 2.17 is not a conservative choice, it is where the ecosystem already is, and the sysroot was pinned there. +`libsanitizer=16.2.0` requires `__glibc >=2.17,<3.0.a0`, and measuring a surviving environment showed conda-forge's whole base stack -- `libgcc`, `libstdcxx`, `python`, `openssl`, `numpy` and twenty others -- sitting at exactly `>=2.17`. So 2.17 is not a conservative choice, it is where the ecosystem already is, and the sysroot was pinned there. **The floor was set to 2.17 on that reasoning and it was wrong**, because the reasoning was about the toolchain and the floor is about the whole environment. See *What the release-time check found* below: the answer is 2.28, and it is forced rather than chosen. @@ -49,13 +49,13 @@ The build hash is worth noting: `conda search` listed 2.17 builds as `h4a8ded7_* rsync (__glibc >=2.28,<3.0.a0) ``` -**This is the entire argument for the script existing, made on the day it was written.** `rsync` carries its constraint in conda metadata, not in its version, so no amount of reading `environment-analysis.yml` could see it — the static case had passed, correctly, over a file whose real floor was 2.28. +**This is the entire argument for the script existing, made on the day it was written.** `rsync` carries its constraint in conda metadata, not in its version, so no amount of reading `environment-analysis.yml` could see it -- the static case had passed, correctly, over a file whose real floor was 2.28. -Measured across both environments: of 134 packages in the pipeline environment, 100 declare `__glibc >=2.17` and exactly one declares more; of 190 in the analysis environment, 121 and one. The one is `rsync`, in both. And every `rsync` build published on conda-forge — 3.4.3 and both 3.4.4 builds — requires 2.28, so there is no lower build to fall back to. +Measured across both environments: of 134 packages in the pipeline environment, 100 declare `__glibc >=2.17` and exactly one declares more; of 190 in the analysis environment, 121 and one. The one is `rsync`, in both. And every `rsync` build published on conda-forge -- 3.4.3 and both 3.4.4 builds -- requires 2.28, so there is no lower build to fall back to. **So 2.28 is forced, not chosen.** Dropping to 2.17 would mean removing `rsync`, and `bin/atomic_mv.sh` stages every promoted artifact with it. Z's call, 2026-09-21: hold at 2.28. The cluster in use is RHEL 9 at glibc 2.34, and the machine below 2.28 is CentOS 7, end of life since June 2024. -**`rsync` decides the floor; `sysroot` broke the cluster.** Keeping those apart is the point — a floor is the highest bound anything imposes, and a defect is a bound that moved for no reason. Only the second one was a bug. +**`rsync` decides the floor; `sysroot` broke the cluster.** Keeping those apart is the point -- a floor is the highest bound anything imposes, and a defect is a bound that moved for no reason. Only the second one was a bug. ## The update reintroduces the defect immediately, and the pin was measured rather than trusted @@ -64,8 +64,8 @@ Asked directly by Z, 2026-09-21: does `conda update --all` break the floor? Meas | | unpinned | with `sysroot_linux-64 <=2.28` in `conda-meta/pinned` | |---|---|---| | packages moved | 28 | 26 | -| `sysroot_linux-64` | `2.17` → **`2.39`** | unchanged | -| `kernel-headers_linux-64` | `3.10.0` → **`6.12.0`** | unchanged | +| `sysroot_linux-64` | `2.17` -> **`2.39`** | unchanged | +| `kernel-headers_linux-64` | `3.10.0` -> **`6.12.0`** | unchanged | **The update restores the exact v3.1.1 defect**, as two of twenty-eight ordinary-looking build bumps, with the suite green and every static check passing. So the pin in `prepare_env` is load-bearing rather than defensive. @@ -75,22 +75,22 @@ Asked directly by Z, 2026-09-21: does `conda update --all` break the floor? Meas ## Why four guards and not one -Z, 2026-09-21: *"Possibly both and more. Because when we prepare the release, we upgrade conda. It needs to change. We are pinning this library for now but more might come later."* The sysroot is one instance of a class — any package constraining a host virtual package freezes a property of the build machine into a file that ships everywhere — and the release cycle's update step keeps producing new instances. +Z, 2026-09-21: *"Possibly both and more. Because when we prepare the release, we upgrade conda. It needs to change. We are pinning this library for now but more might come later."* The sysroot is one instance of a class -- any package constraining a host virtual package freezes a property of the build machine into a file that ships everywhere -- and the release cycle's update step keeps producing new instances. One number, declared in `export-environment.sh` and written into both files' headers, read by four things: 1. **`export-environment.sh` refuses** to write a file that breaks it. Checked on the generated content, one line before the `mv`, so it answers about the file that is about to exist rather than about something adjacent to it. -2. **`00_static`** refuses to ship one. Cheap, textual, needs no network — this is the case that would have caught v3.1.1. +2. **`00_static`** refuses to ship one. Cheap, textual, needs no network -- this is the case that would have caught v3.1.1. 3. **`prep-version.sh`** writes `sysroot_linux-64 <=` into the scratch clone's `conda-meta/pinned` **before** `conda update --all`, so the solver never proposes the raise. Stated up front rather than corrected afterwards: a correction would be a second solve that can itself fail, and it would automate away a decision that belongs to a person. Only where the package is already present, so the pipeline environment cannot gain a sysroot from a pin written on its behalf. 4. **`check-host-floor.sh`**, at release time against a real installed environment. ### The fourth one exists because the second cannot be enough -`00_static` checks the one package whose **version is the glibc it targets** — `sysroot_linux-64=2.39` says `__glibc >=2.39` in its own name, so a text file is enough to see it. Every other package carries its constraint in conda metadata instead: `libsanitizer=16.2.0` requires `__glibc >=2.17` and nothing in the string `16.2.0` says so. A release that picks up such a package with a higher bound ships this defect again with the static check passing — which is [[gates-that-stopped-checking]] exactly, a gate still running and no longer aimed at the thing. +`00_static` checks the one package whose **version is the glibc it targets** -- `sysroot_linux-64=2.39` says `__glibc >=2.39` in its own name, so a text file is enough to see it. Every other package carries its constraint in conda metadata instead: `libsanitizer=16.2.0` requires `__glibc >=2.17` and nothing in the string `16.2.0` says so. A release that picks up such a package with a higher bound ships this defect again with the static check passing -- which is [[gates-that-stopped-checking]] exactly, a gate still running and no longer aimed at the thing. So the general answer needs the real constraints, and an installed environment has them: every package writes `$PREFIX/conda-meta/.json` carrying its own `depends`. Reading those is exact, offline and instant. -**Measured, and it is why the check needs an install rather than a solve:** `conda env create --dry-run --json` returns name, version, build string, channel and platform per package and **no `depends` key** — 0 of 190 records carried one. The solve says what would be installed, never what any of it requires. +**Measured, and it is why the check needs an install rather than a solve:** `conda env create --dry-run --json` returns name, version, build string, channel and platform per package and **no `depends` key** -- 0 of 190 records carried one. The solve says what would be installed, never what any of it requires. ## What this cost to find, and what would have found it sooner diff --git a/.claude/development-notes/logging.md b/.claude/development-notes/logging.md new file mode 100644 index 0000000..f4edb44 --- /dev/null +++ b/.claude/development-notes/logging.md @@ -0,0 +1,84 @@ +# Every step lost its log on every failure path + +**Written 2026-09-29, against the tree at `85eb380` plus uncommitted work.** Found by Z asking one question about the emptiness guards added the same day: *"The error messages you added in variant call and vcf2freq, do they show up in the last run log?"* They did not, and neither did anything else, in any step, on any failure. + +## The defect + +Every process ended with the same six lines: + +```sh +mkdir -p ${dir_log} +{ + echo "" + echo "===== run=... | session=... | attempt=... | $(date -Is) =====" + cat .command.log +} >> ${dir_log}/.log +``` + +At the **bottom** of the script. Under `set -eo pipefail` any failing command exits immediately, so that block was reached only when the task succeeded. A tool crash, a refusal, an out-of-memory kill: the log was written to `.command.log` in the task directory and never copied out. `Logs//` held nothing for exactly the runs a person needs a log for. + +Nextflow still printed the text to the terminal and `.nextflow.log`, which is why this survived: it looked fine from the console and was empty on disk. On a cluster, where the console is a scheduler's output file somewhere else, `Logs/` is what a person has. + +`0_verify_environment.nf` had the fix from the start -- an `archive_logs` function called on both paths, with a comment saying why. Nothing else did. + +## What was measured + +Four designs, five failure modes, each one a real Nextflow run in an isolated project, counting lines that actually landed in `Logs/`: + +| design | success | tool failure (`set -e`) | explicit `exit 1` | SIGTERM | SIGKILL | +|---|---|---|---|---|---| +| **A**, the shipped tail copy | written | **lost** | **lost** | **lost** | **lost** | +| **B**, `trap archive_log EXIT` | written | written | written | written | **lost** | +| **D-merged**, `exec > >(stdbuf -oL tee -a LOG) 2>&1` | written | written | written | written | **written** | +| D-split, a tee per stream | written | written | written | written | written, **reordered** | + +Every surviving cell carried the header, the process's own messages **and** tool output (a `bcftools` version line and an `ls` error), so tool chatter is captured in all three of B and D. + +**D-merged is the only design green in every column**, and the column that separates it from B is the one that matters most here: a cgroup OOM kill is SIGKILL, which no trap can catch. That is the failure a collaborator hit on an HPC cluster, reporting that the pipeline "is not writing anything to output folder". + +D-split was measured only to rule it out. It reorders lines, because two `tee` processes race, and its one advantage -- keeping `.command.err` populated -- Z ruled unnecessary. + +## Two hypotheses that were wrong, both mine + +**"Removing `>&2` is needed so the message reaches the log."** It is not. `.command.log` is the COMBINED stream. Restored the redirects, re-ran, and the text reached `Logs/` identically. `archive_log` alone was the fix at that point, and the `>&2` removal Z asked for changes only which section Nextflow files it under in the terminal. The redirects are left off because every other `echo` in those scripts writes to stdout. + +**"Generalising step 0's `log_message` would fix it."** It would not. `log_message` writes each line to a **task-local** report file, which `archive_logs` moves at the end -- so under SIGKILL it scores **0 lines in `Logs/`**, exactly as the trap does. The shape that survives is writing straight to the `Logs/` path. Z proposed `log_message` and was right about the principle (write as it happens) while that specific mechanism would have measured as nothing. + +## The change + +21 processes across 10 files. The tail block and the `archive_log` added earlier the same day both removed, replaced at the **top** of each script by: + +```sh +mkdir -p ${dir_log} +{ + echo "" + echo "===== run=... | session=... | attempt=... | $(date -Is) =====" +} >> ${dir_log}/.log +exec > >(stdbuf -oL tee -a ${dir_log}/.log) 2>&1 +``` + +The header and its separating blank line are byte-identical to before, so run-to-run separation inside each log file does not change. + +**`9_completion.nf` was already writing its block twice**, on two different paths -- someone had met this problem there and solved it locally. Both copies are replaced by the one prologue. + +**Step 0 is untouched.** `log_message`, `archive_logs` and its thirteen `cat .command.log` sites stay, because step 0 is building the verification **report** that ships to `Output/Reports/` as a deliverable a user reads. That is an artifact, not a log, and none of the above applies to it. + +## Why nothing blocked it + +| checked | result | +|---|---| +| `.command.out` still captured | yes -- `tee` inherits the original stdout | +| the user still sees a failure message | yes -- Nextflow printed it under "Command error:" with `.command.err` empty | +| concurrent tasks appending to one file | no risk: every log name is per-task wherever tasks run in parallel. The four shared names belong to processes that run once | +| log volume | unchanged. The old tail already copied tool output; this writes the same bytes incrementally. Z's real run is 2.1 MB of logs, largest file 523 KB | +| `exec` disturbing anything reading stdin | no: every `read`/stdin use is inside step 0 | +| `dryrun` reusing these processes | no: it includes only step 0's `VerifyEnvironment` and pure functions | +| `stdbuf` availability | pinned in the release environment, coreutils 9.12. Plain `tee` also survived the kill, so `stdbuf` is insurance rather than a dependency | + +`04_pipeline` passes 51 of 51 afterwards, which matters because **nine of its assertions read step logs and grep their content** -- the content is unchanged, only the timing is. + +## What is still true + +**SIGKILL is not fully solved and cannot be.** The tee flushes per line, so what the task had reached is on disk -- but a cgroup OOM kill kills `tee` too, so the final partial line can be lost. There is no design that survives having its whole process group killed; writing incrementally is the best available and it is what this now does. + +**`set -eo pipefail`'s reason is unrelated**, and it came up because Z reasoned from it. The recorded reason is in `createDepthFile.sh`: *"pipefail is load-bearing: the SAMPLENAMES pipeline below ends in `cut`, which succeeds whatever bcftools did."* Not about logs. diff --git a/.claude/development-notes/manual-pass.md b/.claude/development-notes/manual-pass.md new file mode 100644 index 0000000..7217fdd --- /dev/null +++ b/.claude/development-notes/manual-pass.md @@ -0,0 +1,27 @@ +# Reading the manual for a release: the four shapes that keep coming back + +**Written 2026-09-30, against the tree at `8e8a078` plus uncommitted work.** Lifted out of `dev/RELEASING.md` step 1, where it read as instructions to an agent rather than to Z, who writes the manual and does not need to be told how he writes. + +The step itself keeps only what it asks of a person: read in document order, verify every claim against the code, and run `dev/scripts/check-manual-parameters.sh`. + +## The four + +**A count that moves with every commit.** Case counts, file counts, running totals. **The fix is to remove the number, not to correct it** -- a corrected count is wrong again on the next commit, and nothing will prompt anyone. A count tied to a tag or to a closed list is fine, because both stop moving. + +**A warning about a bug that is fixed.** Nothing prompts you to delete one when the bug goes. The Variant Calling page warned that `filterFalsePositives.sh -h` prints a wrong formula long after it printed the right one. + +**Developer shorthand in user-facing prose.** "The pipeline refuses", not "step 0 refuses". A reader following the manual has no model of which step is which, and a step number is also a thing that moves. + +**A capability written as the only path.** "Each pool *can be set to* get its own", not "each pool gets its own". The manual is read as a promise about what the tool does, so a describable option stated as a default makes the tool sound less configurable than it is -- and advice built on the misreading follows. "Use your smallest pool" and "run separate projects" both survived in several places on exactly this. + +## The parameter audit, and the half it cannot do + +`dev/scripts/check-manual-parameters.sh` is silent when every settable key is named somewhere in the manual. + +**It reads `nextflow.config` as well as the template, and that is the whole reason it exists.** The 2026-08-31 sweep found `dryRun` and `dryRunDir` undocumented because a template-driven audit cannot see a key defined anywhere else; `bwa.options` went the same way, and the whole tool had no row. + +**It proves a name appears, not that the manual says what the parameter does.** The requirement is what it is, what it does, and how to set it. A substring match answers none of that, and it is loose on purpose: a short leaf like `align` is satisfied by the word appearing anywhere, and tightening it produces a false positive on every parameter documented in prose rather than as a literal, which is most of them. The commented-out knobs count as settable, because uncommenting one is what a user does. + +## What the step no longer says, and why that is right + +Step 1 used to point at "the extraction and the grep" in `.claude/development-notes/` **under the manual pass**. No such note existed, no script did it, and no case asserted it -- the step asked for an audit and pointed at a file that was not there. This note is not that pointer restored: the step now carries a command that runs, and a procedure that points into a dated note imports a description of the code as it used to be. diff --git a/.claude/development-notes/mapq-downgrade.md b/.claude/development-notes/mapq-downgrade.md new file mode 100644 index 0000000..51f1758 --- /dev/null +++ b/.claude/development-notes/mapq-downgrade.md @@ -0,0 +1,122 @@ +# What `-C` does to a pool, measured on real data + +**Written 2026-09-28, rewritten 2026-09-29 against the tree at `c5bacb4` plus uncommitted work.** Raised by a collaborator's 3.2.0 run on a cluster and then measured properly against Z's own *D. melanogaster* project: three pools, whole genome, the release's own option string. The first version of this note was synthetic and said so; everything below replaces it. + +`scaleMapQ` reaches every call as `-C`, composed at `scripts/resolve_parameters.nf:85`. + +## The parameter has three regimes and the name suggests none of them + +Mean pileup depth, Sample1, `2L:5000000-5050000`, every option but `-C` as the release composes it: + +| `-C` | mean DP | positions | +|---|---|---| +| 0 | 9.43 | 50,057 | +| 10 | 9.43 | 50,057 | +| **11** | **no output** | **0** | +| **15** | **no output** | **0** | +| **20** | **no output** | **0** | +| 30 | 6.49 | 47,947 | +| 40 | 8.10 | 49,209 | +| 50 | 8.76 | 49,898 | +| 70 | 9.17 | 49,957 | +| 100 | 9.27 | 50,007 | + +**At or below 10 it does nothing at all.** Not "less" - nothing. Whole-genome cohort runs at 0, 5 and 10 are byte-identical: 1,854,369 sites each, md5 `5c089144d201c0c9`, zero differing lines. Three independent thirteen-minute runs. + +**Between 11 and about 20 every read is discarded.** mpileup emits no records, exits 0, and prints no error. A whole-genome `-C 20` run produced 0 sites in 24 seconds against the usual thirteen minutes, with clean stderr at all three stages. + +**From 30 up it is usable and INVERTED**: a higher number is milder. 30 costs 31% of depth, the shipped default of 50 costs about 7% here, 100 costs almost nothing. + +So it reads like a scale and behaves like a threshold, with a dead zone below it and a destructive band just above. + +## Which value is right, measured + +Six whole-genome cohort call sets, all three ready BAMs in one mpileup because that is what step 6 does: + +| `-C` | sites called | +|---|---| +| 0, 5, 10 | 1,854,369 | +| 50 | 1,191,066 | +| 100 | 1,723,859 | + +**Every site below is scored from the `-C 0` rows**, so both groups are measured under identical pileup settings. This matters: `-C` downgrades any read carrying a mismatch, and an ALT read carries one by construction, so `-C` manufactures the very mapping-quality difference a naive across-setting comparison would read as evidence. `MQBZ` sits at -2.9 to -4.9 on ordinary sites under `-C 50` for exactly this reason, which is why it is not used here. + +Grouping every called site by which settings kept it: + +| group | n | Ti/Tv | \|SCBZ\| | QUAL | +|---|---|---|---|---| +| kept by both 50 and 100 | 1,178,812 | **1.168** | 0.32 | 212.7 | +| dropped by 50, kept by 100 | 537,942 | **1.015** | 0.49 | 133.1 | +| dropped by both | 137,615 | **0.847** | 1.31 | 81.1 | + +A clean gradient, and it decides the question. **The 537,942 sites that `-C 50` discards and `-C 100` keeps look like variants, not artifacts**: Ti/Tv 1.015 against the kept group's 1.168 and the junk group's 0.847, and a clip bias of 0.49 against 0.32 and 1.31. Alignment noise cannot manufacture a transition bias, because eight of the twelve possible substitutions are transversions. + +Ti/Tv here is low in absolute terms because this is the RAW step-6 call set, before step 7's depth and quality filters and before the false-positive filter. The bar is not a literature value but the pipeline's own accepted output. + +## And `-C 50` biases the frequencies of the sites it keeps + +At the 1.18M sites both settings call, the alternate frequency is systematically lower under `-C 50`, across 3.4M sample-cells and all three pools: + +| `-C 0` frequency | cells | mean shift | relative | down:up | +|---|---|---|---|---| +| 0.00-0.05 | 455,309 | +0.0099 | +39.8% | 0.01:1 | +| 0.05-0.10 | 32,267 | +0.0014 | +1.9% | 0.47:1 | +| 0.10-0.25 | 510,945 | -0.0012 | -0.7% | 0.70:1 | +| 0.25-0.50 | 854,152 | -0.0302 | -8.1% | 2.00:1 | +| **0.50-0.75** | **741,897** | **-0.0808** | **-12.9%** | **4.05:1** | +| 0.75-1.01 | 831,971 | -0.0470 | -5.3% | 5.88:1 | + +**Rare alleles are untouched** - below 0.10 essentially nothing moves down, so the detection limit the false-positive filter rests on is safe. **The damage is where the pool is most non-reference**, peaking at a 12.9% relative compression around 0.5-0.75. A true frequency of 0.60 publishes as about 0.52. + +That is mechanistically what `-C` must do: it penalizes reads carrying mismatches, so the more alternate-allele reads a site has, the more reads are downgraded and the harder the frequency is pulled toward the reference. **It is not a flat offset. It scales with the quantity being measured**, and those frequencies feed `basicstats`, `fst`, `association` and `mds`. + +Whole-genome, `-C 100` shifts by -0.0076 at 1.73:1 against `-C 50`'s -0.0353 at 2.33:1: **4.6 times less bias**. + +## Where this leaves the default + +The whole curve, every statistic read from the `-C 0` rows: + +| `-C` | sites kept | kept Ti/Tv | dropped Ti/Tv | dropped \|SCBZ\| | freq shift | down:up | +|---|---|---|---|---|---|---| +| 30 | 94,681 | **0.608** | **1.090** | 0.45 | +0.0612 | 0.52:1 | +| 40 | 738,393 | 1.190 | 1.036 | 0.54 | -0.0479 | 2.19:1 | +| 50 | 1,178,812 | 1.168 | 0.978 | 0.67 | -0.0353 | 2.33:1 | +| 70 | 1,568,464 | 1.134 | 0.890 | 0.99 | -0.0154 | 1.97:1 | +| 100 | 1,716,020 | 1.113 | **0.848** | **1.34** | **-0.0076** | 1.73:1 | + +**`-C 30` is inverted and actively harmful**: the set it KEEPS has Ti/Tv 0.608 while the set it DROPS has 1.090. A setting this harsh only passes a site where nearly every read matches the reference, so what survives is dominated by sequencing error rather than by polymorphism. Anything near the bottom of the usable range is worse than useless. + +Above 30 the trend is monotonic and **100 is the best value tested, on every axis**. The set it drops is the most clearly artifactual (Ti/Tv 0.848 and a clip bias of 1.34 against its own kept set's 0.36), it retains the most real sites, and its frequency bias is a twentieth of a percentage point where the shipped 50 costs three and a half. + +**100 also beats turning the adjustment off.** It still removes 138,349 sites that are genuinely bad by every statistic here, and `-C 0` keeps all of them. The objection that motivated this measurement is correct; the shipped value is simply set far too harsh. + +`-C 0` is not right either: the 137,615 sites both settings reject are genuinely bad, and turning the adjustment off keeps them. + +**DECIDED 2026-09-29, Z: the default becomes 100.** `parameters.config.template` carries it, and the analysis above is published in the manual under the filter chain at `#scalemapq-measured`, with the variant-calling settings section pointing at it. The manual's own account is the authoritative one; this note is how it was arrived at. + +Two consequences to carry: + +- **It moves every published number**, so it moves the version and invalidates existing results directories. A project that has already run will be stopped by step 0's change guard naming `variantCall.mpileupOptions`, which is correct: `scaleMapQ` feeds it and that string is in step 6's artifact identity, so the guard was never going to miss this. +- **Nothing was changed about the dead zone or the destructive band.** A user can still write `scaleMapQ = 15` and get an empty run that reports success. The manual now warns, which is documentation rather than a gate, and the emptiness guard below is still the fix. + +## The landmine, and the gate that would catch it + +`scaleMapQ = 15` is a plausible edit by someone who thinks they are being gentler than 50. It produces an **empty VCF, empty frequency tables, and a run that reports success**. + +Nothing in step 7 refuses an empty VCF or an empty frequency table - grepped, there is no such check. So this is not a hypothetical: a one-character configuration change silently destroys every result, and the pipeline says it worked. See [[gates-that-stopped-checking]]; it is the same family as the missing-mate drop 3.2.0 fixed, and the emptiness guard should be built whatever happens to the default. + +## How the measurement was got wrong, three times + +Recorded because the failure mode is instructive and cost an hour. + +`-C 20` returning zero sites was read as a resource failure on three separate occasions. It was the parameter working at its most severe setting. **The tell was in the first event and ignored: 24 seconds against thirteen minutes.** A crash is slow; an empty stream is fast. + +Three command-construction errors produced a table of `-C` values that was non-monotonic and entirely spurious, and it was nearly believed: + +- `-r` placed after the positional BAM arguments. bcftools stops option parsing at the first positional, so `-r` and the region became input filenames. +- `"$B"` quoted, collapsing three BAM paths into one argument. +- `2>/dev/null` on all of it, so neither error was visible. + +The region spot-checks were discarded entirely and every number in this note comes from complete whole-genome runs. + +Two process traps hit in the same hour, both already recorded elsewhere in this project: editing a running bash script (survivable only because `sed -i` renames rather than writing in place, so the running shell keeps the old inode), and `pgrep -f` / `pkill -f` patterns matching their own command line. The second killed the wrapper that was waiting to launch `-C 20`. diff --git a/.claude/development-notes/metadata-file.md b/.claude/development-notes/metadata-file.md index 5750dfd..1ffef79 100644 --- a/.claude/development-notes/metadata-file.md +++ b/.claude/development-notes/metadata-file.md @@ -2,24 +2,24 @@ **Written 2026-08-31, against the tree at `7d65893`.** Two more reserved prefixes have arrived since and the parser now documents seven kinds of column, not three; `diploidy` is `ploidy`; and "a design column is free to edit" has acquired a caveat the analysis layer created. All noted in place. -> **For the manual — all three are there now** (verified 2026-08-31), including the factor of two in the threshold and the freedom to edit design columns. Kept here as the statement of what the file is *for*: -> 1. **`RG_Sample` merges rows into one VCF column** — two lanes of a pool become one column whose depths are added. This changes what a number *means* and is the single most consequential thing in the file. -> 2. **`param_poolSize` sets a pool's detection limit**, `s = 1 / (2 * ploidy * poolSize)` — the smallest allele fraction that pool can produce. A reader interpreting a frequency table needs it. +> **For the manual -- all three are there now** (verified 2026-08-31), including the factor of two in the threshold and the freedom to edit design columns. Kept here as the statement of what the file is *for*: +> 1. **`RG_Sample` merges rows into one VCF column** -- two lanes of a pool become one column whose depths are added. This changes what a number *means* and is the single most consequential thing in the file. +> 2. **`param_poolSize` sets a pool's detection limit**, `s = 1 / (2 * ploidy * poolSize)` -- the smallest allele fraction that pool can produce. A reader interpreting a frequency table needs it. > 3. **Design columns never affect results**, so they can be added and edited freely. Worth stating positively; it is the reason the file exists. `metadata.csv`, the file the user authors to say what their samples are. Read by `bin/parse_metadata.py`, projected by `scripts/metadata.nf`, consumed by steps 0, 2, 4, 6 and 7. ## Why it replaced RGTags.csv -`RGTags.csv` was a SAM header fragment: its columns were raw `@RG` tags and nothing else fitted in it. Everything a pool-seq experiment actually needs recorded — which population a sample belongs to, which timepoint, which replicate, how many individuals went into the pool — had nowhere to live, so people encoded it in the `DS` field or in the sample name. E3a replaced it with a file the user writes for themselves. The analysis layer (E4) is the consumer the design columns exist for. +`RGTags.csv` was a SAM header fragment: its columns were raw `@RG` tags and nothing else fitted in it. Everything a pool-seq experiment actually needs recorded -- which population a sample belongs to, which timepoint, which replicate, how many individuals went into the pool -- had nowhere to live, so people encoded it in the `DS` field or in the sample name. E3a replaced it with a file the user writes for themselves. The analysis layer (E4) is the consumer the design columns exist for. ## One parser, one parsed structure `resolveParameters()` runs the parser once and attaches the rows to every run map as `run.metadata`. Nothing else parses the file, and that is the point of `scripts/metadata.nf` existing at all. -Before it, the file was parsed in **three** places: `head`/`awk`/`IFS` in step 4, a naive `split(',')` in step 6, and a third in the divergence analysis. Two of those already mis-read a quoted value containing a comma — and a description field is the likeliest place in this pipeline to find one. Step 4's version was worse than wrong-on-quotes: it projected *every* non-empty column into an `@RG` tag, which worked only while every column was a tag. A design column would have gone into the BAM header as an invented tag. +Before it, the file was parsed in **three** places: `head`/`awk`/`IFS` in step 4, a naive `split(',')` in step 6, and a third in the divergence analysis. Two of those already mis-read a quoted value containing a comma -- and a description field is the likeliest place in this pipeline to find one. Step 4's version was worse than wrong-on-quotes: it projected *every* non-empty column into an `@RG` tag, which worked only while every column was a tag. A design column would have gone into the BAM header as an invented tag. -It also let `RepairRGTagsLineEndings` be deleted outright. That stage existed only because the consumers read raw bytes; Python's `csv` handles CRLF natively, and nothing reads the file at task time any more. That closed a standing complaint — *step 0 rewrites a file the user wrote* — rather than documenting it. `03_pipeline` asserts its task count is 0 so nothing reintroduces it. +It also let `RepairRGTagsLineEndings` be deleted outright. That stage existed only because the consumers read raw bytes; Python's `csv` handles CRLF natively, and nothing reads the file at task time any more. That closed a standing complaint -- *step 0 rewrites a file the user wrote* -- rather than documenting it. `03_pipeline` asserts its task count is 0 so nothing reintroduces it. ## The three projections are one mechanism asked three questions @@ -33,11 +33,11 @@ They have to be, or the pipeline could share an artifact between two runs while A design column appears in none of them, which is what makes it free to add and edit. -**That is true of the pipeline and no longer true of the analysis layer.** `exp_`, `pt_` and `cov_` columns are what a module builds its published design from, and `metadataGuardLines()` still projects `RG_*` and `param_*` alone — so a design column edited after a run leaves the pipeline's results correct, exactly as this section says, and lets a module publish a design those results were never produced under. The freedom was the point of the file and is worth keeping; what is missing is that nothing records which version of a design column a result was analyzed under. See `change-guards.md`. +**That is true of the pipeline and no longer true of the analysis layer.** `exp_`, `pt_` and `cov_` columns are what a module builds its published design from, and `metadataGuardLines()` still projects `RG_*` and `param_*` alone -- so a design column edited after a run leaves the pipeline's results correct, exactly as this section says, and lets a module publish a design those results were never produced under. The freedom was the point of the file and is worth keeping; what is missing is that nothing records which version of a design column a result was analyzed under. See `change-guards.md`. **Sorted vs file order is the distinction that matters.** `metadataProjection()` sorts, because it answers "are two runs the same for this step". `metadataGuardLines()` does **not** sort, because the guard has to tell a value change from a permutation: tag values live in the BAMs and a reordering leaves them correct, while row order lives in the VCF and nothing else. Step 0 sorts both sides itself to draw that distinction. Adding a `.sort()` to the guard lines would silently collapse the two cases into one. -**Row order is a separate identity from row values.** bcftools names VCF sample columns in the order the BAMs are given; step 6 orders them by `metadataOrder()`, and the frequency tables inherit it. Two files that are permutations of each other therefore *share* step 4 and *diverge* at step 6 — which a single "the metadata file" identity token would get wrong in one direction whichever way it was written. +**Row order is a separate identity from row values.** bcftools names VCF sample columns in the order the BAMs are given; step 6 orders them by `metadataOrder()`, and the frequency tables inherit it. Two files that are permutations of each other therefore *share* step 4 and *diverge* at step 6 -- which a single "the metadata file" identity token would get wrong in one direction whichever way it was written. ## Decisions @@ -45,24 +45,24 @@ A design column appears in none of them, which is what makes it free to add and - **`RG_Sample` defaults to `SampleID`** when absent or blank, resolved *in the parser* so every consumer sees the effective row and nothing re-derives it. A second place to derive it is a second place for it to be derived differently. - **Three reserved prefixes** (Z, 2026-08-29): *"anything that touches the parameters should take a prefix `param_*` like `RG_*`, we control the list."* `SampleID`, `RG_*`, `param_*`, and everything else is the user's. E3a's bare `adapter1`/`adapter2` were renamed to `param_adapter1`/`param_adapter2` under that rule; nothing had shipped, so it was not a migration. - **Two more prefixes were reserved since, on the same rule.** `exp_` for the experimental design (F0b), then `pt_` for a phenotype and `cov_` for a covariate (F0f) — three rather than one because only `exp_` identifies a series, so a temperature recorded per timepoint under `exp_` would leave every series one point long. The parser's docstring opens **"SEVEN KINDS OF COLUMN"** now: `SampleID`, `RG_*`, `param_*`, `exp_*`, `pt_*`, `cov_*`, and the user's own. Each reserved prefix refuses its own unknown members, and the bare prefix with no name after it is refused by name. -- **Both closed lists are refusals, not filters.** A `param_` or `RG_` column the pipeline did not recognize would be a setting the user has written down, can see in their own file, and that was never applied. The parser also refuses a *bare* `poolSize`/`adapter1` — without that check it would be accepted as design metadata and silently ignored, which is the exact failure the prefixes exist to prevent. -- **The tables exist twice on purpose** — `RG_TAGS`/`PARAM_COLUMNS` in the parser, `rgTagMap()`/`paramColumns()` in `metadata.nf`. The parser must refuse an unknown column before anything runs; the Groovy side must render the tag and act on the parameter. Neither can do the other's job from where it sits. Two cases in `00_static.sh` check they agree, so a tag added to one and not the other fails a test instead of becoming a column that validates and then vanishes from the BAM. + **Two more prefixes were reserved since, on the same rule.** `exp_` for the experimental design (F0b), then `pt_` for a phenotype and `cov_` for a covariate (F0f) -- three rather than one because only `exp_` identifies a series, so a temperature recorded per timepoint under `exp_` would leave every series one point long. The parser's docstring opens **"SEVEN KINDS OF COLUMN"** now: `SampleID`, `RG_*`, `param_*`, `exp_*`, `pt_*`, `cov_*`, and the user's own. Each reserved prefix refuses its own unknown members, and the bare prefix with no name after it is refused by name. +- **Both closed lists are refusals, not filters.** A `param_` or `RG_` column the pipeline did not recognize would be a setting the user has written down, can see in their own file, and that was never applied. The parser also refuses a *bare* `poolSize`/`adapter1` -- without that check it would be accepted as design metadata and silently ignored, which is the exact failure the prefixes exist to prevent. +- **The tables exist twice on purpose** -- `RG_TAGS`/`PARAM_COLUMNS` in the parser, `rgTagMap()`/`paramColumns()` in `metadata.nf`. The parser must refuse an unknown column before anything runs; the Groovy side must render the tag and act on the parameter. Neither can do the other's job from where it sits. Two cases in `00_static.sh` check they agree, so a tag added to one and not the other fails a test instead of becoming a column that validates and then vanishes from the BAM. - **No generated intermediate.** Generating an RGTags.csv inside the pipeline was considered and rejected: `rgTagsOrder()` ran at DAG-build time, before any process, so a generated file would have broken first-run column ordering and then tripped the row-order change guard. -## Multi-lane pooling — tested, not reasoned about +## Multi-lane pooling -- tested, not reasoned about -Two FASTQ pairs whose rows share an `RG_Sample` produce two `@RG` IDs, two ready BAMs, and **one** VCF sample column carrying both lanes' depth. The frequency table follows the VCF, and the cohort assertion passes because it counts ready BAMs against read pairs, never VCF columns. Verified on the base fixture with `TestSample1`/`TestSample2` sharing `SM: Pool_A` — exit 0, five columns for six pairs, pooled frequencies differing from either lane's. +Two FASTQ pairs whose rows share an `RG_Sample` produce two `@RG` IDs, two ready BAMs, and **one** VCF sample column carrying both lanes' depth. The frequency table follows the VCF, and the cohort assertion passes because it counts ready BAMs against read pairs, never VCF columns. Verified on the base fixture with `TestSample1`/`TestSample2` sharing `SM: Pool_A` -- exit 0, five columns for six pairs, pooled frequencies differing from either lane's. -The shipped template had always done this, so pooling is documented usage rather than an edge case. **Therefore a per-sample override that changes a number is a property of the POOL, keyed by `RG_Sample`, not of the row** — and rows sharing a pool must agree on it or the file is refused. That rule was decided by the experiment run to decide it. +The shipped template had always done this, so pooling is documented usage rather than an edge case. **Therefore a per-sample override that changes a number is a property of the POOL, keyed by `RG_Sample`, not of the row** -- and rows sharing a pool must agree on it or the file is refused. That rule was decided by the experiment run to decide it. I had previously claimed multi-lane was unsupported. That was wrong: `rgTagsOrder()` orders BAMs by `SampleID` and never touches VCF columns, so more rows than columns is correct and nothing downstream miscounts. The two lists coincide in the fixture, which is why reading the code did not settle it and running it did. ## `param_poolSize` -A pool's size sets its detection limit: `s = 1 / (2 * ploidy * poolSize)` is the smallest allele fraction a pool that size can produce — one chromosome out of all of them — and the false-positive filter drops everything below it. It was one number for the whole VCF until E3b, which judged a pool of 10 at a pool of 100's resolution. +A pool's size sets its detection limit: `s = 1 / (2 * ploidy * poolSize)` is the smallest allele fraction a pool that size can produce -- one chromosome out of all of them -- and the false-positive filter drops everything below it. It was one number for the whole VCF until E3b, which judged a pool of 10 at a pool of 100's resolution. -`poolSizeArgument()` renders **sizes, not sensitivities**, because the filter derives `s` itself. That keeps the equation in one place on the per-pool path and puts the number the user actually wrote into the VCF header. Keyed by name rather than column position — see `false-positive-filter.md` for why that is a correctness property and not a convenience. +`poolSizeArgument()` renders **sizes, not sensitivities**, because the filter derives `s` itself. That keeps the equation in one place on the per-pool path and puts the number the user actually wrote into the VCF header. Keyed by name rather than column position -- see `false-positive-filter.md` for why that is a correctness property and not a convenience. A blank cell falls back to the global `poolSize`, so a project that sets none of them gets exactly the number it got before the column existed. @@ -78,6 +78,6 @@ Held here so the source can stay terse: ## `trim_galore.options` pinned, adapters overridden -A run may pin `trim_galore.options` outright rather than letting it be derived. That makes a per-sample adapter unactionable: the pinned string is used verbatim, so the row's adapters would be silently dropped. Step 0 refuses the combination rather than picking a winner — a run whose setting is quietly ignored is worse than one that will not start. +A run may pin `trim_galore.options` outright rather than letting it be derived. That makes a per-sample adapter unactionable: the pinned string is used verbatim, so the row's adapters would be silently dropped. Step 0 refuses the combination rather than picking a winner -- a run whose setting is quietly ignored is worse than one that will not start. `sampleTrimOptions()` builds through `resolve_parameters.nf`'s own `trimOptions()` rather than assembling the flags a second time, so a flag added to the derivation reaches the per-sample path too. A row that sets neither adapter gets the run's string unchanged, which is what keeps a project with no overrides byte-identical to one from before the column existed. diff --git a/.claude/development-notes/migration-hole.md b/.claude/development-notes/migration-hole.md new file mode 100644 index 0000000..539e12c --- /dev/null +++ b/.claude/development-notes/migration-hole.md @@ -0,0 +1,116 @@ +# The migration hole: a note in `config_migrate.sh`, and a default that was not one + +**Written 2026-09-30, against the tree at `8e8a078` plus uncommitted work.** The last of the eight items Z ordered on 2026-09-29. It ended as two small changes in two files, and took two wrong turns and two drafts to get there. Both are recorded, because both are reachable. + +## Part 1: the shape a new parameter's note has + +Z set it, 2026-09-30: **state the default behavior, and say what to write for the other one. Do not describe what the user had.** + + vcffilter.dropZeroDepth IS NEW, and your config now has its default. + + The default is true: a site where any pool has no reads at all is dropped. + To keep those sites, with the unread pool published as NA, set it to false. + +**Not describing the transition is what makes it true for every project rather than only some.** The first draft ran fifteen lines and narrated the change, which forced a second gate on `minDP 0`, because above that the zero term is already covered by the minimum and no site moves. A message that states only the default needs one gate: the parameter appears as `NEW` in the report. Read back from the report, as `OLD_MAXDEPTH` is, so it fires exactly when the migration classified it as added; a config that already sets it chose its own value and is told nothing. + +`variantCall.maxDepth`'s note is the other shape and stays as it is. That one exists because a value was deliberately **not** carried across, so the user's own number has to appear in it. + +## Part 2: the default was not a default + +**Measured through the real engine**, the ternary at `scripts/7_vcf2freq.nf:257` with `minDP 20`: + +| config | `run.vcffilter.dropZeroDepth` | expression built | +|---|---|---| +| key absent | `null` | `FMT/DP<20` | +| `= true`, the template default | `true` | `FMT/DP<20 \|\| FMT/DP==0` | + +So the shipped default was `true` and an absent key ran as `false`. Z: *"Then the default is not actually default, right? If unset it should fall back to default... nextflow.config is the right place to resolve defaults."* + +**Why `true` is the right default, which I had backwards.** Z: *"template default true matches v3.2.0 behavior. That's why I set the default to true."* At v3.2.0 the filter was `FMT/DP<${minDP}` alone, and at the shipped `minDP 20` that already excludes `DP 0`, so `true` reproduces v3.2.0 exactly. The two only differ at `minDP 0`, where v3.2.0 excluded nothing. I had described `false` as the v3.2.0-preserving value, which is true only at `minDP 0` and I stated it flatly. + +### The resolution, in `nextflow.config` below the include + + params.vcffilter = params.containsKey('vcffilter') && params.vcffilter != null ? params.vcffilter : [:] + params.vcffilter.dropZeroDepth = params.vcffilter.containsKey('dropZeroDepth') ? params.vcffilter.dropZeroDepth : true + +**Both lines are load-bearing and each was measured.** + +`containsKey` and not a plain assignment: that file is below `includeConfig`, and the file's own header says why -- *"Nextflow is later-wins, so the same assignment placed after parameters.config was read would beat whatever the project asked for, silently."* A plain `= true` there overrides every project on the machine. Proven: replacing the ternary with `params.vcffilter.dropZeroDepth = true` fails the second half of the new case. + +The map guard above it: **without it, a config carrying no `vcffilter` block at all fails to parse, and the error names the installation's `nextflow.config` rather than the user's file.** Measured directly. `params.cores` carries the same guard for the same reason. + +Resolved four ways, against the real repository file via `nextflow config -flat`: key absent gives `true`, explicit `false` stays `false`, explicit `true` stays `true`, no `vcffilter` block at all gives `true` and parses. + +## Coverage + +`01_migrate`, two cases, each biting on its own breakage: + +| broken | what fails | +|---|---| +| the note deleted | `a new parameter states its default`, all three assertions | +| the gate removed so it always fires | `the new parameter note is absent when it was set` | + +`05_guards`, one case, `an absent parameter resolves to its default`, in the shape of the `exec-defaults` case beside it -- own sandbox, `sandbox_config_flat`, assert on the flattened resolution. Both halves bite: + +| broken | what fails | +|---|---| +| the resolution removed | the absent-key half | +| `containsKey` dropped for a plain assignment | the project-wins half | + +**`nextflow.config` was not in `05_guards`' `# covers:` and now is.** Fourth instance of the selector gap in two days: the suite reads a file it never declared. + +## Part 3: the check that Part 2 made pointless, removed + +`bin/check_parameters.sh` carried a NOTE for `minDP 0` with `dropZeroDepth` off. **Its recorded justification is in `parameter-rules.md`, and Part 2 eliminated it**: *"That is the one combination that cannot tell it is in trouble, and it is precisely the migration hole's real exposure: someone copying an old `parameters.config` into a new project."* A copied old config now resolves to `true`. The stumble-into-it path is gone, and reaching the state takes two explicit non-default edits whose result labels itself `NA` in the published table. + +Z: *"I think this check is unnecessary. Unless we decide to report every single parameter and what they do in the logs. That's not what we're doing."* It also fails the file's own admission test -- it is not a setting that produces nothing, and `NA` is precisely what stopped it being silent. + +Removed: the rule, its two cases in `03_helpers`, and the manual's `#parameter-checks` row. Nothing else in any suite referenced it, checked by grep before touching it. The `#parameter-checks` paragraph about older configs was rewritten, which it needed regardless: it claimed an unmentioned parameter "resolves as unset, and for a true-or-false setting that is the same as `false`", which Part 2 made false. + +**This is the `how-to-know-a-change-works` rule running forwards rather than backwards.** The usual form is reverting a change whose reason was disproved. Here the reason was not disproved, it was removed by a better fix, and the check left behind would have read to a later session as a live hazard. + +## A third thing I got wrong, and it is the same shape as the first two + +I told Z the step 0 path had an unasserted consequence: that a project which ran before Part 2 and omitted the key would find its recorded baseline differs by a line. **Two pieces of wiring I had not read make that wrong.** + +- **The version guard runs first and exits.** `scripts/0_verify_environment.nf:1079` logs the mismatch, writes the report and `exit 0`, never reaching the parameter comparison at 1141. A project upgrading across releases cannot reach the difference at all. +- **A parameter a release introduces is already a classified case.** `bin/classify_manifest.sh` exists to tell it apart from a value the user changed, which is what `05_guards`' own header says the suite is for. + +What survives is coverage rather than hazard, and it is narrow: nothing asserts the manifest carries the key for a config that OMITS it. The existing flip case proves it carries the key for a config that SETS it. `analysisParams()` flattens the whole resolved map and excludes only a named list, which `vcffilter.dropZeroDepth` is not on, so it is carried. + +## The first wrong turn: the marker is not a version check + +`config_is_current()` at `lib/wrapper_lib.sh:459` is a grep for `storageDir`, shared by `require_migrated_config()` and `bin/check_project.sh`. Counting live assignments in `parameters.config.template` gives 101 at v3.0.0, v3.1.0, v3.1.1 and v3.2.0, and 102 at HEAD. From that I concluded the marker had been "correct for four releases by luck" and was a `gates-that-stopped-checking.md` case caught one release early. + +**Z: that is a false conclusion.** `dev/RELEASING.md` step 3 states the marker's job outright: *"It turns on one marker -- `storageDir` being assigned -- so a release that renames that root has to move the marker with it"*, held by `02_launcher`'s `a config from an older release is refused with the fix`. It answers **was this config carried across the root rename**. A parameter added within the 3.x line is the migration's business, and the migration is prepared every release at step 3. + +The count is a real measurement and still true. The inference was the error, and it is the one a key count invites. + +A second claim built on that reading was also wrong: I offered a user deleting a `fastqc.options` line as something the marker cannot see. It is template line 49, commented out, a knob the pipeline computes, and its deletion changes nothing. It is in the six-knob list CLAUDE.md spells out. + +## The second wrong turn: the wrong file + +With the marker ruled out, I read "the dropZeroDepth gap" as the reporting gap in `bin/check_parameters.sh`, which cannot tell an absent key from an explicit `false` because both take its `!= "true"` branch. That was built, with two cases, and reverted. **Z: *"Check parameters work as it should. #3 was about config migrate."*** The checker judges a resolved parameter set for a run that is about to happen; explaining what a release changed is the migration's job. + +Note that the second part of this note makes the checker's absent-key branch unreachable from a real run: `nextflow config -flat` now always carries the key. The `!= "true"` test stays correct and stays cheap, and the rule it guards is `false`, which a project can still choose. + +## Why two wrong turns on an easy step + +The failure was procedural rather than technical. **The item named its own file and I opened it last.** "#3 the migration hole" was treated as a problem to diagnose, so the search went to the two places a hole could be argued into existing, and each wrong turn produced real measurements that read as progress. Reading `config_migrate.sh` first would have shown the gap in a minute: the notes section is two entries long and neither is from this release. + +## The manual + +`## Install` is now `## Install and Update`, with `### Updating to a new release` added to it, carrying `migrate_config` at the front. The Upgrading page was already thorough; what it could not do is reach someone who never looked for it, which is the failure Z named. The new section points at Upgrading for the project-side rules rather than restating them. + +Nothing was added to the Upgrading page's per-release sections. The manual describes the current release's behavior; the transition from the previous one is what `migrate_config` prints. + +**A heading carrying an explicit `{ #anchor }` needs `nav:` in its page directive.** `render_nav()` takes the label from the raw heading text, so `## Install and Update { #install }` produced the mkdocs entry `Install and Update { #install }: getting-started/install.md`. The three module pages already declare `nav:` for the same reason, their headings being in backticks. `build_docs.py --check` cannot catch it: the braces are equally present in what it generates and in what it compares against, so the nav reads as current. The explicit anchor is what keeps the four `(#install)` links resolving after the rename; anchors went 373 to 374. + +## Two things about the harness, both of which cost a wrong reading here + +- **`--case` matches the underscored function name, not the displayed one.** `--case "zero depth"` selected nothing and printed `PASS 0 passed`; `--case zero_depth` selects the two. A filter that matches nothing reports success, which CLAUDE.md warns about and which I read past once here. +- **`build_docs.py --check` writes nothing.** If `mkdocs.yml` appears to change under it, `serve_docs.sh` is running in the background and regenerating it. + +## Still open + +`dev/RELEASING.md` step 3 tells you to check the migration report's categories. It does not tell you to ask whether a parameter this release **added** needs a note, nor whether its default is resolved for a config that does not set it. Both were stale here. Not built, not asked for. diff --git a/.claude/development-notes/module-dependencies.md b/.claude/development-notes/module-dependencies.md index 4ea3795..ce84193 100644 --- a/.claude/development-notes/module-dependencies.md +++ b/.claude/development-notes/module-dependencies.md @@ -1,35 +1,35 @@ # The four fields a manifest gained, and where each is enforced -**Written 2026-09-08, against the tree at `409aef7`.** E8's first half: the manifest schema. The conda half — what installing a module does to the shared environment — is not built yet and this note does not describe it. The manual is authoritative on what a user sees; `analysis/modules/README.md` is authoritative on what a module author writes. This is the record of why the shape is what it is. +**Written 2026-09-08, against the tree at `409aef7`.** E8's first half: the manifest schema. The conda half -- what installing a module does to the shared environment -- is not built yet and this note does not describe it. The manual is authoritative on what a user sees; `analysis/modules/README.md` is authoritative on what a module author writes. This is the record of why the shape is what it is. ## What forced it -A module could not declare a dependency at all. `install/environment-analysis.yml` is one hand-written conda spec for the whole analysis layer and a module has no file that feeds it, so the FST module — first in the F4 roster, built on `r-poolfstat` — was not installable at any price. 3.0.0 is the release that opens the store to third parties, who cannot wait for a PoolSeqFlow release to gain a package. +A module could not declare a dependency at all. `install/environment-analysis.yml` is one hand-written conda spec for the whole analysis layer and a module has no file that feeds it, so the FST module -- first in the F4 roster, built on `r-poolfstat` -- was not installable at any price. 3.0.0 is the release that opens the store to third parties, who cannot wait for a PoolSeqFlow release to gain a package. `license` had a second reason. It was already in `association` and `mds` and was doing nothing: `readManifest()` builds its return map from a literal list of keys and dropped every other one at parse time. Adding a field to a manifest was, until now, a way of writing a comment. ## The three shipped modules are GPL, and the argument that they were not was wrong -They carried `Apache-2.0` because that is what was written when the field did nothing. Z asked whether the 2026-09-03 decision had not been GPL. It had not — that decision was *per-module*, with the FST module going GPL because `poolfstat` is — but the question was right for a reason neither of us had checked. +They carried `Apache-2.0` because that is what was written when the field did nothing. Z asked whether the 2026-09-03 decision had not been GPL. It had not -- that decision was *per-module*, with the FST module going GPL because `poolfstat` is -- but the question was right for a reason neither of us had checked. Measured, `Rcpp` is `GPL (>= 2)`. Everything else the three load is permissive: `doFuture` Apache-2.0, `ggplot2` MIT, `data.table` MPL-2.0, `jsonlite` MIT. `optparse` is GPL-2+ and is in the environment, but no module loads it. I first argued the Apache label survived because the compiled path was optional. **Z corrected that and it is the whole point**: `usecpp: true` is the default in all three `main.nf`, `nocpp` is the opt-out, `r-rcpp` is pinned into `install/environment-analysis.yml`, and `mds.R:52` **stops** when Rcpp is absent rather than falling back. A dependency the default configuration hard-requires is not optional in any sense that a license question cares about. -What was left of the counter-argument — that we ship `.R` and `.cpp` as source and the user's machine does the linking, so no combined work is ever distributed by us — is an argument about *when* the obligation attaches, not about whether the dependency is real. Z's call, 2026-09-08: GPL-3.0-or-later on all three. +What was left of the counter-argument -- that we ship `.R` and `.cpp` as source and the user's machine does the linking, so no combined work is ever distributed by us -- is an argument about *when* the obligation attaches, not about whether the dependency is real. Z's call, 2026-09-08: GPL-3.0-or-later on all three. `verify` stays Apache-2.0 in `builtinModules()`. It belongs to the frame, runs no R, and installs nothing. -Consequence written down where a user meets it: `README.md` and the manual's License section now say the modules carry their own terms, because the release tarball ships an Apache-2.0 core beside GPL modules. That is ordinary aggregation — separate files, separate works, no linking — but a repository badged Apache-2.0 that ships GPL under it has to say so. +Consequence written down where a user meets it: `README.md` and the manual's License section now say the modules carry their own terms, because the release tarball ships an Apache-2.0 core beside GPL modules. That is ordinary aggregation -- separate files, separate works, no linking -- but a repository badged Apache-2.0 that ships GPL under it has to say so. ## Why validation splits across two functions -`readManifest()` is called by `moduleRoster()` for **every** directory in the store, and `moduleNames()` — which builds the "Available here:" list in a refusal — calls that. So anything thrown from `readManifest()` takes down every other module too: one bad manifest and you cannot run, or even list, the good ones. +`readManifest()` is called by `moduleRoster()` for **every** directory in the store, and `moduleNames()` -- which builds the "Available here:" list in a refusal -- calls that. So anything thrown from `readManifest()` takes down every other module too: one bad manifest and you cannot run, or even list, the good ones. That is the right behavior for a **broken** manifest and the wrong behavior for an **incompatible** one. A manifest missing `license` is an install that did not finish; a manifest asking for a newer frame is a module that is fine and simply does not belong here. So: -- **shape** — field present, `frame` matching `YYYYMMDD.NNN`, `environment` dotted-numeric, every `packages` entry pinned — stays in `readManifest()`, beside the existing `name`/`contract` checks it is indistinguishable from. -- **compatibility** — `frame` against `frameVersion()`, `environment` against the release — is `checkModuleCompatible()`, called from `analysisPlan()` beside `checkModuleNeeds` and `checkModuleOutputs`. It refuses one module and leaves the roster alone. +- **shape** -- field present, `frame` matching `YYYYMMDD.NNN`, `environment` dotted-numeric, every `packages` entry pinned -- stays in `readManifest()`, beside the existing `name`/`contract` checks it is indistinguishable from. +- **compatibility** -- `frame` against `frameVersion()`, `environment` against the release -- is `checkModuleCompatible()`, called from `analysisPlan()` beside `checkModuleNeeds` and `checkModuleOutputs`. It refuses one module and leaves the roster alone. ## The environment check only fires in one of the two halves, and that is enough @@ -37,7 +37,7 @@ That is the right behavior for a **broken** manifest and the wrong behavior for Both halves call `analysisPlan()`. The check skips a release it cannot read rather than guessing, which costs nothing: the verify half always runs first, always knows, and always refuses before the module is launched. The frame half of the check reads a file and works in both. -Measured alternative not taken: passing the release down through `frame.config` as a params key. It would reach the recorded manifest, and the identity check reads a new top-level key as a setting added since the results were produced — every analysis in the project would then refuse. That is the trap rule 11 of `analysis/modules/README.md` names, and it applies to the frame just as it does to a module. +Measured alternative not taken: passing the release down through `frame.config` as a params key. It would reach the recorded manifest, and the identity check reads a new top-level key as a setting added since the results were produced -- every analysis in the project would then refuse. That is the trap rule 11 of `analysis/modules/README.md` names, and it applies to the frame just as it does to a module. ## Why a package spec is `name=version` and nothing else @@ -54,9 +54,9 @@ Z, 2026-09-03: *"Set together to work together."* ## `environment` is checked for equality on a shipped module and as a minimum everywhere else -The field is a **minimum** — the oldest release whose analysis environment holds what the module needs — and a third-party module sets whatever is true of it. A module that ships **inside** a release is installed with that release's environment and no other, so for those the minimum is the release exactly, and `00_static` asserts equality against `nextflow.config`'s `manifest.version`. +The field is a **minimum** -- the oldest release whose analysis environment holds what the module needs -- and a third-party module sets whatever is true of it. A module that ships **inside** a release is installed with that release's environment and no other, so for those the minimum is the release exactly, and `00_static` asserts equality against `nextflow.config`'s `manifest.version`. -That is also the mechanism that makes E7b's version bump safe. All three shipped manifests say `2.2.0` today because that is what this tree is. When E7b bumps to 3.0.0 the static case fails by name and whoever bumps updates them. A script that rewrote the manifests along with the version would have hidden the one moment where a human should look at what a module claims about its environment — and `bump-version.sh` would then also owe each module a version bump of its own, since `check-analysis-versions.sh` watches those directories. +That is also the mechanism that makes E7b's version bump safe. All three shipped manifests say `2.2.0` today because that is what this tree is. When E7b bumps to 3.0.0 the static case fails by name and whoever bumps updates them. A script that rewrote the manifests along with the version would have hidden the one moment where a human should look at what a module claims about its environment -- and `bump-version.sh` would then also owe each module a version bump of its own, since `check-analysis-versions.sh` watches those directories. `frame` is a minimum in both cases: `00_static` only asserts it is not newer than `analysis/frame.version`. Frames bump on most days and a module's floor should not follow. @@ -66,46 +66,46 @@ That is also the mechanism that makes E7b's version bump safe. All three shipped The plan called it "precisely 'these packages install or the install fails, and nothing else moves'". **It is not.** `--freeze-installed` refuses to change a package the *solve reaches on its own*; a package **named on the command line** conda installs at the version asked for, downgrading what is there. -Measured, on the first run of `dev/scripts/check-module-packages.sh`: a fixture pinning `r-glue=1.8.0` against a baseline holding `r-glue=1.8.1` **downgraded it and reported the solve as a success**. That is exactly the failure the design exists to prevent — a module quietly moving a version every other module, and the release itself, computes against. +Measured, on the first run of `dev/scripts/check-module-packages.sh`: a fixture pinning `r-glue=1.8.0` against a baseline holding `r-glue=1.8.1` **downgraded it and reported the solve as a success**. That is exactly the failure the design exists to prevent -- a module quietly moving a version every other module, and the release itself, computes against. So `conda_install_packages` reads what the environment holds and refuses a disagreeing pin before conda is asked. The same pin at the same version is not a disagreement: two modules needing one package at one version is the reason the environment is shared at all. **This is the case for the release gate existing.** No amount of reading conda's documentation produced it; a stub could never produce it; and every launcher case passed with the hole wide open, because they assert on command lines and the command line was correct. The gate ran once and found it. -It also caught its own fixtures: `r-glue` was in the baseline, which made four of its checks meaningless — the install was a version change rather than an addition. It now asserts its fixtures are absent from the baseline before using them. +It also caught its own fixtures: `r-glue` was in the baseline, which made four of its checks meaningless -- the install was a version change rather than an addition. It now asserts its fixtures are absent from the baseline before using them. -**`conda remove` cascades, and the direction matters.** Measured against the real environment: `conda remove -n --dry-run --json r-pheatmap` names only `r-pheatmap`, so conda takes **dependents** and not orphaned dependencies. Every name a plan adds beyond the ones asked for is therefore something that depends on one of them and belongs to a module still installed — which is why the rule is "refuse on any collateral at all" rather than a subtraction. `--force-remove` is the wrong tool: it would leave a broken environment. +**`conda remove` cascades, and the direction matters.** Measured against the real environment: `conda remove -n --dry-run --json r-pheatmap` names only `r-pheatmap`, so conda takes **dependents** and not orphaned dependencies. Every name a plan adds beyond the ones asked for is therefore something that depends on one of them and belongs to a module still installed -- which is why the rule is "refuse on any collateral at all" rather than a subtraction. `--force-remove` is the wrong tool: it would leave a broken environment. -**Atomicity is ordering, not rollback.** `install_module` is download → checksum → unpack → validate files → parse and validate the specs → move into the store → conda install; a failed solve rolls back with `rm -rf "$dest"`, a local directory delete. A rollback `conda remove` would be a second mutation that can itself fail, and conda's own transaction is already atomic. +**Atomicity is ordering, not rollback.** `install_module` is download -> checksum -> unpack -> validate files -> parse and validate the specs -> move into the store -> conda install; a failed solve rolls back with `rm -rf "$dest"`, a local directory delete. A rollback `conda remove` would be a second mutation that can itself fail, and conda's own transaction is already atomic. ### The manifest is read twice, and that is a cost paid deliberately -The wrapper is shell reached without a JVM; `readManifest()` is Groovy reached without a shell. Neither can call the other, so `packages` has two readers and the pin rule has two regexes. The alternative was asking Nextflow for the list, which costs a ~22 s JVM start per install and does not work in `deploy_payload` at all — that runs while the analysis environment may not exist. +The wrapper is shell reached without a JVM; `readManifest()` is Groovy reached without a shell. Neither can call the other, so `packages` has two readers and the pin rule has two regexes. The alternative was asking Nextflow for the list, which costs a ~22 s JVM start per install and does not work in `deploy_payload` at all -- that runs while the analysis environment may not exist. What keeps them honest is `00_static`: it extracts both patterns from their own files and runs a shared table of seven specs past the shell one. Both patterns must be present and identical, so a change to one that is not made to the other fails by name. -`module_packages` in `lib/wrapper_lib.sh` is the **one** shell reader — `install_module`, `uninstall_module`, `deploy_payload`, `analysis install` and `export-environment.sh` all go through it. `export-environment.sh` sources `wrapper_lib.sh` rather than carrying its own copy, which is also how it reaches the installed store through `install_prefix`. +`module_packages` in `lib/wrapper_lib.sh` is the **one** shell reader -- `install_module`, `uninstall_module`, `deploy_payload`, `analysis install` and `export-environment.sh` all go through it. `export-environment.sh` sources `wrapper_lib.sh` rather than carrying its own copy, which is also how it reaches the installed store through `install_prefix`. ### `uninstall` degrades where `install` refuses -The plan said both should require the analysis environment. Building it showed that is wrong for `uninstall`: `analysis uninstall` removes the environment and keeps the store, so requiring it to remove a module leaves the user with a store they cannot tidy and no way out but reinstalling the environment. So `install` requires it — a module installed without its packages is broken — and `uninstall` removes the directory either way, saying the packages stayed because there is nothing to take them out of. `analysis install` then reconciles. +The plan said both should require the analysis environment. Building it showed that is wrong for `uninstall`: `analysis uninstall` removes the environment and keeps the store, so requiring it to remove a module leaves the user with a store they cannot tidy and no way out but reinstalling the environment. So `install` requires it -- a module installed without its packages is broken -- and `uninstall` removes the directory either way, saying the packages stayed because there is nothing to take them out of. `analysis install` then reconciles. ### What the stub can and cannot prove -`test/lib/sandbox.sh`'s fake conda logs every invocation, so `02_launcher` proves **which command lines are issued and in what order**: that an unpinned spec is refused before conda is reached, that a module declaring nothing issues no install at all, that a removal reads its plan first, that the store's packages come out before the payload wipe. It cannot prove a solve succeeds — a stub always says yes. +`test/lib/sandbox.sh`'s fake conda logs every invocation, so `02_launcher` proves **which command lines are issued and in what order**: that an unpinned spec is refused before conda is reached, that a module declaring nothing issues no install at all, that a removal reads its plan first, that the store's packages come out before the payload wipe. It cannot prove a solve succeeds -- a stub always says yes. -The stub gained a removal-plan answer for this, shaped like real conda's: one key per line, with a `dist_name` beside each `name` so the reader is shown not to pick that up instead. Writing it inline on one line was the first attempt and the sed missed it, which is the useful half of the story — the stub has to imitate the format, not just the content. +The stub gained a removal-plan answer for this, shaped like real conda's: one key per line, with a `dist_name` beside each `name` so the reader is shown not to pick that up instead. Writing it inline on one line was the first attempt and the sed missed it, which is the useful half of the story -- the stub has to imitate the format, not just the content. -`dev/scripts/check-module-packages.sh` is the other half, by hand before a release, against real conda. Z, 2026-09-03: *"there is no real union, we test it here."* Its fixtures are pins rather than module directories: the store side is what `02_launcher` covers, and what this asks is a solver question. F1, F2 and F3 declare no packages — `cmdscale`, `p.adjust`, `pt` and `eigen` are base R — so without fixtures the script would pass by having done nothing. +`dev/scripts/check-module-packages.sh` is the other half, by hand before a release, against real conda. Z, 2026-09-03: *"there is no real union, we test it here."* Its fixtures are pins rather than module directories: the store side is what `02_launcher` covers, and what this asks is a solver question. F1, F2 and F3 declare no packages -- `cmdscale`, `p.adjust`, `pt` and `eigen` are base R -- so without fixtures the script would pass by having done nothing. ### The sharpest trap, and it is a dev script `dev/scripts/export-environment.sh` regenerates `install/environment-analysis.yml` from a live environment. Run after a module install, it would fold that module's packages into the baseline **permanently and invisibly**: nothing would declare them, nothing could remove them, and every project would install them. It now refuses, naming the specs it found and which module declares each. -It looks only for a module's **own** specs. What conda pulled in beneath them is not distinguishable here from what the baseline needed anyway — which is exactly why the answer is to rebuild the environment rather than to subtract from it. +It looks only for a module's **own** specs. What conda pulled in beneath them is not distinguishable here from what the baseline needed anyway -- which is exactly why the answer is to rebuild the environment rather than to subtract from it. ## What the fixtures do about it -`test/lib/analysis.sh` gained `ANALYSIS_MANIFEST_FLOORS` — `"license": "Apache-2.0", "frame": "20260101.001", "environment": "0.0.0"` — interpolated into every fixture manifest. Those are the lowest values either field can express, so no fixture moves when the frame or the release does. `test_a_manifest_missing_a_field_refuses` deliberately does not use it and still asserts on `contract`, which is checked first. +`test/lib/analysis.sh` gained `ANALYSIS_MANIFEST_FLOORS` -- `"license": "Apache-2.0", "frame": "20260101.001", "environment": "0.0.0"` -- interpolated into every fixture manifest. Those are the lowest values either field can express, so no fixture moves when the frame or the release does. `test_a_manifest_missing_a_field_refuses` deliberately does not use it and still asserts on `contract`, which is checked first. One case installs two bad modules in sequence and has to delete the first before the second: `readManifest()` runs over the whole store, so the second run would refuse on the first module again and the assertion would pass while testing nothing. diff --git a/.claude/development-notes/module-optionality.md b/.claude/development-notes/module-optionality.md index 3eb1151..37fff7c 100644 --- a/.claude/development-notes/module-optionality.md +++ b/.claude/development-notes/module-optionality.md @@ -1,4 +1,4 @@ -# Modules are optional — the design change, and the audited blast radius +# Modules are optional -- the design change, and the audited blast radius **Written 2026-09-10 against the tree at `91e027f` plus three uncommitted edits.** v3.0.0 was released earlier the same day, *with* the modules inside it. Nothing here is a defect in the released version: v3.0.0 works exactly as shipped. This is about what the NEXT release has to be, and why. @@ -9,16 +9,16 @@ Z, 2026-09-10: *"The modules should not ship with the release. That's the whole Three parts follow from it: 1. **No module ships inside a release tarball.** The store starts empty; every module is installed from the catalogue. -2. **Each module declares its own conda pins** in its manifest, and `modules install` puts them into the shared analysis environment — the E8 machinery, which already exists. +2. **Each module declares its own conda pins** in its manifest, and `modules install` puts them into the shared analysis environment -- the E8 machinery, which already exists. 3. **The baseline analysis environment slims** to what the FRAME needs, and stops carrying module dependencies. ## What was actually wrong, stated precisely **The architecture was never wrong. The data was.** -The per-module conda machinery is built and was proven against real conda on 2026-09-10 by `dev/scripts/check-module-packages.sh` — every check `ok`: a fixture module's pins installed under `--freeze-installed`, R imported them, a second module sharing a pin moved nothing, uninstalling one left the other's packages alive, and the baseline never lost a package. `PoolSeqFlow:786-794` is the deferred install; `:1433` is the reconcile that `analysis install` performs over modules already in the store. Neither is missing anything. +The per-module conda machinery is built and was proven against real conda on 2026-09-10 by `dev/scripts/check-module-packages.sh` -- every check `ok`: a fixture module's pins installed under `--freeze-installed`, R imported them, a second module sharing a pin moved nothing, uninstalling one left the other's packages alive, and the baseline never lost a package. `PoolSeqFlow:786-794` is the deferred install; `:1433` is the reconcile that `analysis install` performs over modules already in the store. Neither is missing anything. -**The three shipped modules declare `packages: []`.** That was TRUE while they shipped inside the release, because the same release shipped a 191-package environment holding everything they need. It became FALSE the moment they were published independently — which happened on 2026-09-09/10. They were never really modules; they were release components wearing a module's shape, and they are the one case that never exercised the machinery built for them. +**The three shipped modules declare `packages: []`.** That was TRUE while they shipped inside the release, because the same release shipped a 191-package environment holding everything they need. It became FALSE the moment they were published independently -- which happened on 2026-09-09/10. They were never really modules; they were release components wearing a module's shape, and they are the one case that never exercised the machinery built for them. ## The measured package split @@ -26,49 +26,49 @@ Evidence: every `library()`/`requireNamespace()`/`pkg::` reference across `analy | | | |---|---| -| the frame uses | `jsonlite`, `knitr`, `rmarkdown` — plus `pandoc` and `typst` for the PDF report | +| the frame uses | `jsonlite`, `knitr`, `rmarkdown` -- plus `pandoc` and `typst` for the PDF report | | the modules use | `doFuture`, `ggplot2`, `Rcpp`, and `data.table` (basicstats only) | -| used by nothing at all | `r-optparse`, `r-pheatmap` — pinned since the first guess at the E4c roster | +| used by nothing at all | `r-optparse`, `r-pheatmap` -- pinned since the first guess at the E4c roster | -`foreach` and `future` arrive as `doFuture` dependencies. **The audit corrected two of my assumptions**: `mds` DOES need `ggplot2` unconditionally, and the compiled path needs the conda C++ toolchain, which `r-rcpp` does not pull — so the toolchain placement is part of the split, not incidental. +`foreach` and `future` arrive as `doFuture` dependencies. **The audit corrected two of my assumptions**: `mds` DOES need `ggplot2` unconditionally, and the compiled path needs the conda C++ toolchain, which `r-rcpp` does not pull -- so the toolchain placement is part of the split, not incidental. -## The audited blast radius — seven seams, 2026-09-10 +## The audited blast radius -- seven seams, 2026-09-10 -A seven-dimension audit with adversarial verification of every blocker and major finding. **Five verifier verdicts, zero refutations or corrections** — the findings below survived independent checking. +A seven-dimension audit with adversarial verification of every blocker and major finding. **Five verifier verdicts, zero refutations or corrections** -- the findings below survived independent checking. ### Blockers - **The clone-install route still ships all three modules.** `.gitattributes export-ignore` governs the TARBALL; `install` copies from a checkout tree where `analysis/modules/*` still exist. "The store starts empty" is false for every user following the manual's clone route. **This is the one I would have missed.** - **The three published catalogue rows become install-clean, fail-at-first-run.** Their `environment=3.0.0` passes the `version_at_most` check on any later release, their manifests declare no packages, so they install cleanly onto a slim baseline and then die missing `ggplot2`/`doFuture`/`Rcpp`. The `environment` field's semantics assume baselines only ever GAIN packages. -- **`00_static`'s `no package leaves a shipped environment file` cannot legitimately pass a slimmed baseline** — and it is the guard added on 2026-09-09 for exactly the typst class of bug. It compares against the last release tag, so it stays red for the whole dev cycle. `export-environment.sh --allow-removals` is the sanctioned escape for the export; the test needs its own answer. -- **`00_static` asserts every repo manifest's `environment` EQUALS the release version.** With no module shipping, that coupling is backwards — it forces a bump nobody's needs justify. +- **`00_static`'s `no package leaves a shipped environment file` cannot legitimately pass a slimmed baseline** -- and it is the guard added on 2026-09-09 for exactly the typst class of bug. It compares against the last release tag, so it stays red for the whole dev cycle. `export-environment.sh --allow-removals` is the sanctioned escape for the export; the test needs its own answer. +- **`00_static` asserts every repo manifest's `environment` EQUALS the release version.** With no module shipping, that coupling is backwards -- it forces a bump nobody's needs justify. - **The manual teaches the old design in at least three places**: "Four ship with the release", the `# Shipped Modules` section, and "No module shipped with this release names one \[a package\]". -- **`RELEASING.md` step 6**'s justification for automatic manifest rewriting — "travels in the same tarball as the analysis environment it names" — is exactly the premise being removed. +- **`RELEASING.md` step 6**'s justification for automatic manifest rewriting -- "travels in the same tarball as the analysis environment it names" -- is exactly the premise being removed. ### Major - **`bump-version.sh` now writes a wrong claim.** Automatically setting `environment := new release` is right for a shipped module and wrong for an independent one, whose minimum should move only when its needs do. Landed 2026-09-10 at Z's request; the request was correct under the old design. -- **No gate checks that a module's declared packages cover what its R actually loads.** Under-declaration is silent — and the three published tarballs declare NONE. This is the gate that would have caught tonight. +- **No gate checks that a module's declared packages cover what its R actually loads.** Under-declaration is silent -- and the three published tarballs declare NONE. This is the gate that would have caught tonight. - **The module suites SKIP rather than fail when `TEST_ANALYSIS_ENV` lacks a package**, so a slim environment silently retires the compiled and parallel coverage instead of reporting it. -- **`PoolSeqFlow analysis cite` cites zero R packages today** — a live bug, independent of any of this, in how the cite arm reads `analysis_r_packages`. +- **`PoolSeqFlow analysis cite` cites zero R packages today** -- a live bug, independent of any of this, in how the cite arm reads `analysis_r_packages`. - **`analysis check` would report "All checks passed" on an environment missing every module-declared package**, because it only verifies the baseline. -- **The slim baseline list must add `python3` and `rsync`** (the frame invokes both) and probably drop `samtools`/`bcftools`/`htslib`, which the audit found nothing in the frame invoking — verify before acting. +- **The slim baseline list must add `python3` and `rsync`** (the frame invokes both) and probably drop `samtools`/`bcftools`/`htslib`, which the audit found nothing in the frame invoking -- verify before acting. - **Module package arrays must be name-disjoint from the baseline's set**, or `export-environment.sh`'s refusal fires permanently on a name the baseline owns. - **Two tests fail against the uncommitted wrapper edit**: `02_launcher`'s modules-list case and `test_modules_list_reports_the_store` both plant a store module with no `.source`, which now prints `ships with this release`. -- **`test_an_unknown_module_refuses_before_any_task`** asserts the literal roster `Available here: association, basicstats, mds, verify`, read from the store at runtime — it only holds while the checkout's store has the three. -- **The store README** contradicts the new design in three places, including rule 6e: "Leave the field out when the release's own environment suffices — which is true of every module shipped here". -- **Offline/air-gapped installs regress**, which matters because this runs on HPC clusters: today a cluster user gets three working modules in the tarball; afterwards they need HTTPS to the catalogue AND to each tarball. A `file://` catalogue does not solve it — the rows carry `https` URLs independently of how the index was fetched. +- **`test_an_unknown_module_refuses_before_any_task`** asserts the literal roster `Available here: association, basicstats, mds, verify`, read from the store at runtime -- it only holds while the checkout's store has the three. +- **The store README** contradicts the new design in three places, including rule 6e: "Leave the field out when the release's own environment suffices -- which is true of every module shipped here". +- **Offline/air-gapped installs regress**, which matters because this runs on HPC clusters: today a cluster user gets three working modules in the tarball; afterwards they need HTTPS to the catalogue AND to each tarball. A `file://` catalogue does not solve it -- the rows carry `https` URLs independently of how the index was fetched. ## The uncommitted work, and what to do with it Three files, all on `dev`, none committed: -- **`.gitattributes`** — `analysis/modules/*/ export-ignore`, keeping `README.md`. **Correct and verified**: `git archive --worktree-attributes` leaves only `analysis/modules.nf` and the README. -- **`dev/scripts/verify-archive.sh`** — `excluded` widened to `analysis/modules/*/`, and the positive assertion strengthened to "no module directory in the archive AND the README is". **Correct**; `00_static` passes at 46 against the working tree. -- **`PoolSeqFlow`** — `module_was_installed()` plus `ships with this release` labels and the uninstall warning. **This encodes the ABANDONED middle design** and should probably be reverted: once nothing ships, everything in the store was installed, and the label distinction is dead weight. It also breaks two tests. `.source` remains a useful marker; the labels built on it do not. +- **`.gitattributes`** -- `analysis/modules/*/ export-ignore`, keeping `README.md`. **Correct and verified**: `git archive --worktree-attributes` leaves only `analysis/modules.nf` and the README. +- **`dev/scripts/verify-archive.sh`** -- `excluded` widened to `analysis/modules/*/`, and the positive assertion strengthened to "no module directory in the archive AND the README is". **Correct**; `00_static` passes at 46 against the working tree. +- **`PoolSeqFlow`** -- `module_was_installed()` plus `ships with this release` labels and the uninstall warning. **This encodes the ABANDONED middle design** and should probably be reverted: once nothing ships, everything in the store was installed, and the label distinction is dead weight. It also breaks two tests. `.source` remains a useful marker; the labels built on it do not. ## Sequencing, when this is picked up -The blockers interlock, so order matters. The clone-install path has to be settled before "the store starts empty" is true for anyone. The published catalogue rows need a decision — remove them, or republish at bumped versions declaring their packages — before the baseline slims, or a next release strands anyone who installed one. The two `00_static` gates need their own answers before the slim can be committed at all, or the suite is red for the whole cycle. +The blockers interlock, so order matters. The clone-install path has to be settled before "the store starts empty" is true for anyone. The published catalogue rows need a decision -- remove them, or republish at bumped versions declaring their packages -- before the baseline slims, or a next release strands anyone who installed one. The two `00_static` gates need their own answers before the slim can be committed at all, or the suite is red for the whole cycle. Estimated at roughly one working day, about half of it unattended suite and environment runs. diff --git a/.claude/development-notes/module-queue.md b/.claude/development-notes/module-queue.md index 9851093..e002852 100644 --- a/.claude/development-notes/module-queue.md +++ b/.claude/development-notes/module-queue.md @@ -1,18 +1,18 @@ -# The module queue — the plan from v3.1.1 onward +# The module queue -- the plan from v3.1.1 onward **Written 2026-09-10, against the tree at `8b4f0ca` "Version bump 3.1.1".** This is the plan as it was decided that day, not a document kept current. Z's cadence: **one module per week**. -The plan it replaced covered E1–E8, F0–F3, E5b and E7, and every line of it had landed. Nothing of that is repeated here; `git show` is the detail and the notes beside this one are the reasoning. +The plan it replaced covered E1-E8, F0-F3, E5b and E7, and every line of it had landed. Nothing of that is repeated here; `git show` is the detail and the notes beside this one are the reasoning. ## A week ends in a publish, not a release **This is the whole reason the module system exists and I argued past it twice before Z stopped me.** Z, 2026-09-10: *"I don't know why are you pushing so hard on this we made the modules this way so we don'r make new releases every time."* -The argument I built was half right and the conclusion was wrong, which is worth recording because the half that is right is a real constraint. `manualAnchors()` in `analysis/lib/nf/outputs.nf` resolves the manual as `${installDir()}/manual/PoolSeqFlow-manual.md` — the copy inside the **user's own installation**, which `.gitattributes` ships in the release tarball and which is frozen at whatever release they installed. The website is never consulted. So a module whose manual section exists only on `main` has no anchor a v3.1.1 user can resolve, and `checkModuleOutputs()` **refuses while the DAG is built** rather than merely rendering a dead link. +The argument I built was half right and the conclusion was wrong, which is worth recording because the half that is right is a real constraint. `manualAnchors()` in `analysis/lib/nf/outputs.nf` resolves the manual as `${installDir()}/manual/PoolSeqFlow-manual.md` -- the copy inside the **user's own installation**, which `.gitattributes` ships in the release tarball and which is frozen at whatever release they installed. The website is never consulted. So a module whose manual section exists only on `main` has no anchor a v3.1.1 user can resolve, and `checkModuleOutputs()` **refuses while the DAG is built** rather than merely rendering a dead link. -From that I concluded a module publish had to ride with a release. **It does not.** `checkModuleOutputs()` already accepts `url` in place of `anchor`, and the branch exists for exactly this case — its own error message says so: *"a url is for a module published separately, whose section is not in it."* Three lines below the check I was quoting. +From that I concluded a module publish had to ride with a release. **It does not.** `checkModuleOutputs()` already accepts `url` in place of `anchor`, and the branch exists for exactly this case -- its own error message says so: *"a url is for a module published separately, whose section is not in it."* Three lines below the check I was quoting. -**So the rule is: a module published between releases declares site URLs on its outputs.** `docs.yml` triggers on both `manual/**` and `modules/repo/**`, so the page, the tarball and the catalogue row deploy in one push, and any v3.1.x installation can install it the same afternoon. The three modules already published keep their anchors, because their sections shipped with the release that carries them. A release still happens on its own timetable — when the pipeline changes, or to fold accumulated module sections into the shipped manual — and nothing in a week waits for one. +**So the rule is: a module published between releases declares site URLs on its outputs.** `docs.yml` triggers on both `manual/**` and `modules/repo/**`, so the page, the tarball and the catalogue row deploy in one push, and any v3.1.x installation can install it the same afternoon. The three modules already published keep their anchors, because their sections shipped with the release that carries them. A release still happens on its own timetable -- when the pipeline changes, or to fold accumulated module sections into the shipped manual -- and nothing in a week waits for one. **The lesson, which is the reusable part:** I let a mechanism I had just finished reading drive a cadence decision, instead of asking what the mechanism had been built for. The escape hatch was in the same function. @@ -24,32 +24,32 @@ Dependency order, not priority order. Thirteen modules. **After a per-position depth track**: `callable` · `theta` · `sweep` · `scan`. -Z added positive selection as **two** modules rather than one: `selection` — per-allele *s* with an interval, from the trajectory under an N_e-aware model — early, and `scan` — a composite windowed genome scan — last. They answer different questions for different designs, and one of them needs a time series most projects do not have. Forcing them into one module would gate half of it behind that design. +Z added positive selection as **two** modules rather than one: `selection` -- per-allele *s* with an interval, from the trajectory under an N_e-aware model -- early, and `scan` -- a composite windowed genome scan -- last. They answer different questions for different designs, and one of them needs a time series most projects do not have. Forcing them into one module would gate half of it behind that design. **The depth track goes last deliberately.** It is a pipeline change and not a module: per-position depth from the **capped** BAMs step 6 actually piles up, `bgzip` + `tabix`, into `Analysis/Main`. Putting it at the end keeps the weekly cadence unbroken for nine weeks and lets the track land in a settled layer. The measurements that size it, and the two design errors not to repeat, are in `callable-sites.md` beside this note. -**`histograms` is not a substitute for the track**, and this is easy to get wrong because the class exists and looks like it would do. It is `samtools stats` COV rows: genome-wide, **per library** rather than per pool, taken from the **ready** BAM rather than the capped one the calls came from, and with no zero bin, so genome length is not recoverable from it. A denominator built on it describes a different read set than the numerator — the exact error the reverted `CallableSites` made. +**`histograms` is not a substitute for the track**, and this is easy to get wrong because the class exists and looks like it would do. It is `samtools stats` COV rows: genome-wide, **per library** rather than per pool, taken from the **ready** BAM rather than the capped one the calls came from, and with no zero bin, so genome length is not recoverable from it. A denominator built on it describes a different read set than the numerator -- the exact error the reverted `CallableSites` made. ## Two shape questions the roster raised, both left open on purpose -**`selection` and `drift` may collapse into one module.** An interval on *s* that excludes zero **is** the drift test, so the Wright–Fisher/N_e library belongs to whichever is built first and the second may turn out to be a table rather than a module. If `drift` survives it will be because it answers per-site *"is this more than drift"* genome-wide where `selection` answers per-allele *"how strong"*. Decide it while designing `selection`, not before. +**`selection` and `drift` may collapse into one module.** An interval on *s* that excludes zero **is** the drift test, so the Wright-Fisher/N_e library belongs to whichever is built first and the second may turn out to be a table rather than a module. If `drift` survives it will be because it answers per-site *"is this more than drift"* genome-wide where `selection` answers per-allele *"how strong"*. Decide it while designing `selection`, not before. -**`scan` reads another module's output, and the frame cannot express that.** `needs` names artifact classes produced by pipeline steps and `checkModuleNeeds()` refuses anything else; there is no class for *"what `fst` published into this results folder"*. Three shapes are possible and none is chosen: `scan` recomputes everything itself and depends on no module; the frame grows a class for a module's own published tables; or `scan` reads `target.results` directly and declares the dependency in `gates` with nothing verifying it. **Settle it before `theta` is built** — whichever it is decides whether the modules before it publish tables meant to be consumed or only read. +**`scan` reads another module's output, and the frame cannot express that.** `needs` names artifact classes produced by pipeline steps and `checkModuleNeeds()` refuses anything else; there is no class for *"what `fst` published into this results folder"*. Three shapes are possible and none is chosen: `scan` recomputes everything itself and depends on no module; the frame grows a class for a module's own published tables; or `scan` reads `target.results` directly and declares the dependency in `gates` with nothing verifying it. **Settle it before `theta` is built** -- whichever it is decides whether the modules before it publish tables meant to be consumed or only read. ## Week 1 is `fst`, and one thing about it is already settled -**Read the DEPTH table, not the VCF.** `vcf2pooldata` is poolfstat's documented entry point and it is the wrong one here. `needs: ["vcf"]` resolves through `artifactClasses()` to `Output/VCF/.vcf`, which is the **raw step-6 call set** — no false-positive filter, no depth or quality filter, and in the original reference encoding rather than major-allele normalized. Only the raw call set and its snpEff annotation survive a finished run; every VCF step 7 produces is consumed and deleted. So a module built on `vcf2pooldata` would compute FST over a different site set than `basicstats`, `association` and `mds`, silently and with nothing in the output saying so. +**Read the DEPTH table, not the VCF.** `vcf2pooldata` is poolfstat's documented entry point and it is the wrong one here. `needs: ["vcf"]` resolves through `artifactClasses()` to `Output/VCF/.vcf`, which is the **raw step-6 call set** -- no false-positive filter, no depth or quality filter, and in the original reference encoding rather than major-allele normalized. Only the raw call set and its snpEff annotation survive a finished run; every VCF step 7 produces is consumed and deleted. So a module built on `vcf2pooldata` would compute FST over a different site set than `basicstats`, `association` and `mds`, silently and with nothing in the output saying so. -Building `pooldata` from the depth table instead keeps the site set identical to every other module, and makes the one real difference — poolfstat is biallelic — a declared gate rather than an invisible one. **Unverified as of this note:** whether poolfstat exposes a usable matrix constructor taking `refallele.readcount`, `readcoverage`, `poolsizes` and `snp.info`. That is the first thing week 1 checks, because a no would force the design back onto the raw call set. +Building `pooldata` from the depth table instead keeps the site set identical to every other module, and makes the one real difference -- poolfstat is biallelic -- a declared gate rather than an invisible one. **Unverified as of this note:** whether poolfstat exposes a usable matrix constructor taking `refallele.readcount`, `readcoverage`, `poolsizes` and `snp.info`. That is the first thing week 1 checks, because a no would force the design back onto the raw call set. -`fst` is also the first module to declare `url` outputs, so it carries one-off work the rest of the queue inherits: `00_static: every declared manual anchor exists` checks `anchor` and goes quiet on a `url` module, and wants a companion asserting each declared `url` resolves to a heading in the manual here — which is what the site is built from. That is `gates-that-stopped-checking.md` arriving on schedule, and it was predicted rather than discovered, which is the first time that has happened. +`fst` is also the first module to declare `url` outputs, so it carries one-off work the rest of the queue inherits: `00_static: every declared manual anchor exists` checks `anchor` and goes quiet on a `url` module, and wants a companion asserting each declared `url` resolves to a heading in the manual here -- which is what the site is built from. That is `gates-that-stopped-checking.md` arriving on schedule, and it was predicted rather than discovered, which is the first time that has happened. ## Measured while planning: an abandoned constraint -The superseded plan carried *"The catalogue stays header-only — shipped modules live in the payload, and `analysis/modules-index.tsv` is for separately published ones. No dependency or license column is added to it in 3.0.0."* Z flagged it as possibly abandoned and asked for a measurement. It is abandoned three ways over: +The superseded plan carried *"The catalogue stays header-only -- shipped modules live in the payload, and `analysis/modules-index.tsv` is for separately published ones. No dependency or license column is added to it in 3.0.0."* Z flagged it as possibly abandoned and asked for a measurement. It is abandoned three ways over: - **`analysis/modules-index.tsv` does not exist.** No tracked file by that name; the catalogue is `modules/repo/index.tsv`, published at `/modules-repo/`. - **No module ships in the payload at all.** `modules/` is export-ignored, so `basicstats` is installed from the catalogue like any third-party module. -- **The catalogue did gain columns** — `kind`, `frame`, `environment`. +- **The catalogue did gain columns** -- `kind`, `frame`, `environment`. Only *no license column* survived, and it is now worth reopening rather than keeping: a user choosing what to install learns the terms only after the tarball is down and the manifest is read. Columns are matched by name, so adding one is additive and needs no `#!index-format` bump. diff --git a/.claude/development-notes/modules-and-libraries.md b/.claude/development-notes/modules-and-libraries.md index 8caddd9..cb8da58 100644 --- a/.claude/development-notes/modules-and-libraries.md +++ b/.claude/development-notes/modules-and-libraries.md @@ -1,4 +1,4 @@ -# Modules and libraries are optional, and installed — the rework +# Modules and libraries are optional, and installed -- the rework **Written 2026-09-10 against the tree at `6d38c88`.** Supersedes `module-optionality.md`, which recorded the audit that led here; that note's findings are all addressed below or listed as still open. v3.0.0 was released the same morning WITH the modules inside it, so everything here describes the release after it. @@ -15,13 +15,13 @@ Every symptom traced to that: the three modules shipped in the tarball because t | `modules//` | module sources, tracked, `export-ignore`d from releases | | `modules/lib//` | library sources, same | | `modules/repo/` | the published catalogue and tarballs | -| `analysis/` | the frame ALONE — `lib/nf/`, `lib/rmd/`, the entry pieces | +| `analysis/` | the frame ALONE -- `lib/nf/`, `lib/rmd/`, the entry pieces | | `analysis/modules/` | THE STORE. Gitignored, empty in a checkout, empty in a fresh install | | `analysis/modules/lib/` | the library store, under a name no module may take | -**The repo path and the install path deliberately differ.** A module's frame import is `'../../lib/nf/plan.nf'`, correct from the store and meaningless from `modules//`. I first "fixed" that by moving the store to `$INSTALL/modules/` so the two matched — **wrong**, because the store would then land on the sources in a checkout install. Z caught it. `00_static` lints modules in an assembled store layout instead. +**The repo path and the install path deliberately differ.** A module's frame import is `'../../lib/nf/plan.nf'`, correct from the store and meaningless from `modules//`. I first "fixed" that by moving the store to `$INSTALL/modules/` so the two matched -- **wrong**, because the store would then land on the sources in a checkout install. Z caught it. `00_static` lints modules in an assembled store layout instead. -**The published URL did NOT move** when `modules-repo/` became `modules/repo/`. `MODULE_INDEX_URL` compiles into v3.0.0 and asks for `/PoolSeqFlow/modules-repo/` for as long as that release exists. `build_docs.py` and `publish-module.sh` each carry two constants — source directory and published path — with a comment saying why. +**The published URL did NOT move** when `modules-repo/` became `modules/repo/`. `MODULE_INDEX_URL` compiles into v3.0.0 and asks for `/PoolSeqFlow/modules-repo/` for as long as that release exists. `build_docs.py` and `publish-module.sh` each carry two constants -- source directory and published path -- with a comment saying why. ## Libraries @@ -29,40 +29,40 @@ Eleven files in `analysis/lib/R` + three `.cpp` became **five libraries**, group `n_eff` (n_eff, pool_n_eff, harmonic_mean) · `chunk_ranges` · `allele_frequencies` (+cpp) · `site_diversity` (+cpp) · `nei_distance` (nei_distance, add_distance, mean_distance, +cpp) -**Measured, not assumed:** no library calls another. Every apparent dependency was a comment reference. Z's correction stands anyway — ***"They are standalone now. They can be used by other modules as we increase the size. Don't make assumptions."*** — so the manifest carries `libraries` regardless and resolution is transitive. +**Measured, not assumed:** no library calls another. Every apparent dependency was a comment reference. Z's correction stands anyway -- ***"They are standalone now. They can be used by other modules as we increase the size. Don't make assumptions."*** -- so the manifest carries `libraries` regardless and resolution is transitive. -**Two files had no consumer and moved to `test/tools/`**, on Z's rule *"If only test suite is using it it should be in test suite not here"*: `split_counts.R` and `pool_sensitivity.R`. Neither is dead — `split_counts` is the **oracle** the vectorized and compiled paths are checked against (mutating its split character fails 19 of 140 checks), and `pool_sensitivity` duplicated `poolSensitivity()` in `analysis/lib/nf/pools.nf`, which is the one modules actually read off `target.pools`. +**Two files had no consumer and moved to `test/tools/`**, on Z's rule *"If only test suite is using it it should be in test suite not here"*: `split_counts.R` and `pool_sensitivity.R`. Neither is dead -- `split_counts` is the **oracle** the vectorized and compiled paths are checked against (mutating its split character fails 19 of 140 checks), and `pool_sensitivity` duplicated `poolSensitivity()` in `analysis/lib/nf/pools.nf`, which is the one modules actually read off `target.pools`. -**`libraryFiles()` and `compiledFiles()` are gone from every `main.nf`.** The frame resolves both from the manifest — `moduleLibraryFiles(name)`, `moduleCompiledFiles(name)` — so the list exists once. `00_static:707` used to police the duplicate; there is no duplicate now. +**`libraryFiles()` and `compiledFiles()` are gone from every `main.nf`.** The frame resolves both from the manifest -- `moduleLibraryFiles(name)`, `moduleCompiledFiles(name)` -- so the list exists once. `00_static:707` used to police the duplicate; there is no duplicate now. ## Packages: declare everything Z: ***"Modules declare everything including doFuture, ggplot2 and data.table."*** And the rule that makes it safe, in Z's words: *"to uninstall take a diff between the uninstalled package and union of all installed packages + analysis base environment."* -**That was a live bug.** `uninstall_module` computed `going = mine − others` with **no baseline subtraction**, so a module declaring `r-ggplot2` would have stripped ggplot2 out of the shared environment on its way out, breaking the frame and every module beside it. Now `going = mine − (others + baseline)`, with `baseline_packages()` reading the shipped `environment-analysis.yml` — the shipped file, not the live environment, which has module additions merged in and cannot say which packages are the release's own. +**That was a live bug.** `uninstall_module` computed `going = mine - others` with **no baseline subtraction**, so a module declaring `r-ggplot2` would have stripped ggplot2 out of the shared environment on its way out, breaking the frame and every module beside it. Now `going = mine - (others + baseline)`, with `baseline_packages()` reading the shipped `environment-analysis.yml` -- the shipped file, not the live environment, which has module additions merged in and cannot say which packages are the release's own. -`doFuture`, `ggplot2`, `data.table` and `Rcpp` **stay in the baseline** — Z: *"Rcpp should remain in the frame. I settled this one before."* Modules declare them anyway. A library declares `r-rcpp` only if it ships a `.cpp`. +`doFuture`, `ggplot2`, `data.table` and `Rcpp` **stay in the baseline** -- Z: *"Rcpp should remain in the frame. I settled this one before."* Modules declare them anyway. A library declares `r-rcpp` only if it ships a `.cpp`. ## The catalogue Gained a **`kind` column** (`module` | `library`), additive and matched by name, so an older release ignores it and an empty value means `module`. `available` lists modules only; a library is never asked for by name. -**Every positional consumer had to move with it** — six `read` destructurings, an awk filter, and a `sort -k2,2Vr` that was then sorting by kind instead of version. Thirteen cases failed and pointed straight at them. **The columns are matched by name; the consumers destructure positionally. Adding a column is not free.** +**Every positional consumer had to move with it** -- six `read` destructurings, an awk filter, and a `sort -k2,2Vr` that was then sorting by kind instead of version. Thirteen cases failed and pointed straight at them. **The columns are matched by name; the consumers destructure positionally. Adding a column is not free.** ## Bugs made and caught while building this -- **`def LIBRARY_DIR = 'lib'` at the top level of a `.nf`** — the strict parser allows only declarations there. It is `libraryDirName()`. -- **`git log -1 --format=%ct `** returns nothing: a tree has no commit and no date. `--mtime="@"` made tar substitute a nonsense year. `publish-module.sh` now **refuses** rather than publishing something unreproducible — which is why publishing must follow a commit. +- **`def LIBRARY_DIR = 'lib'` at the top level of a `.nf`** -- the strict parser allows only declarations there. It is `libraryDirName()`. +- **`git log -1 --format=%ct `** returns nothing: a tree has no commit and no date. `--mtime="@"` made tar substitute a nonsense year. `publish-module.sh` now **refuses** rather than publishing something unreproducible -- which is why publishing must follow a commit. - **A `sed 1,/^-->$/d`** whose anchor was not at line start deleted the entire release-notes boilerplate. Caught by reading the output; the new case mutation-tests exactly that. - **`select-tests.py` listed files from the git index**, so a moved file still counted as a source. I masked it by staging before noticing. It now lists only what exists. -- **My own `every catalogue row` case went vacuous** for the positional reason above — and the `rows > 0` guard I had put in it made it **skip loudly** instead of passing. +- **My own `every catalogue row` case went vacuous** for the positional reason above -- and the `rows > 0` guard I had put in it made it **skip loudly** instead of passing. ## What is still open - **Nothing is published.** The three module tarballs carry `20260910.001` manifests with no `libraries` and no `packages`; the repo is at `.002`. The five libraries have never been published. `RELEASING.md` step 10 is the procedure. - **The JVM and pipeline tiers have never run against this layout.** `moduleLibraryFiles()` resolving a module's libraries at run time is new code exercised only statically. Z: *"we should not run the full suite until we are done with the work today."* - **The tarball extract/install/check step** (Z's step 3) is unbuilt. A spec exists and came back with four blockers including two vacuous assertions; it needs rewriting, not applying. -- **The docs pass is partial.** The manual's roster, modules section, package paragraph, store-wipe paragraph and library references are corrected; `modules/README.md` gained a libraries section and rule 6e was inverted; `RELEASING.md` gained step 10. **An exhaustive read of the manual was still running when this was written** — the directory-layout diagram and the Upgrading section are known to be untouched. +- **The docs pass is partial.** The manual's roster, modules section, package paragraph, store-wipe paragraph and library references are corrected; `modules/README.md` gained a libraries section and rule 6e was inverted; `RELEASING.md` gained step 10. **An exhaustive read of the manual was still running when this was written** -- the directory-layout diagram and the Upgrading section are known to be untouched. --- @@ -72,11 +72,11 @@ Written when the question "can we publish now?" was asked. The answer was no, an ## Publishing had to wait, and the frame version is why -**v3.0.0 compiled `MODULE_INDEX_COLUMNS="name version contract frame environment url sha256 summary"` — no `kind`.** It matches by name, drops what it does not know, and reads all eight rows as modules. Two gates then stand between a v3.0.0 user and a module built for the release after it: `contract`, which is `freq-1` on both sides and passes, and `frame`. +**v3.0.0 compiled `MODULE_INDEX_COLUMNS="name version contract frame environment url sha256 summary"` -- no `kind`.** It matches by name, drops what it does not know, and reads all eight rows as modules. Two gates then stand between a v3.0.0 user and a module built for the release after it: `contract`, which is `freq-1` on both sides and passes, and `frame`. -**`analysis/frame.version` was still `20260908.003`** — the exact value v3.0.0 shipped — after `analysis/lib/R/` and `analysis/lib/cpp/` were deleted out from under it. So the frame gate passed too, and publishing then would have offered every v3.0.0 user a `.002` module that installs cleanly, brings no library (v3.0.0 has no `install_library`), and dies at run time on a `moduleLibraryFiles()` its frame does not define. `install/environment-analysis.yml` is byte-identical to v3.0.0, so `environment: 3.0.0` is honest and cannot refuse anything: **the frame version was the only gate, and it was open.** +**`analysis/frame.version` was still `20260908.003`** -- the exact value v3.0.0 shipped -- after `analysis/lib/R/` and `analysis/lib/cpp/` were deleted out from under it. So the frame gate passed too, and publishing then would have offered every v3.0.0 user a `.002` module that installs cleanly, brings no library (v3.0.0 has no `install_library`), and dies at run time on a `moduleLibraryFiles()` its frame does not define. `install/environment-analysis.yml` is byte-identical to v3.0.0, so `environment: 3.0.0` is honest and cannot refuse anything: **the frame version was the only gate, and it was open.** -Now `20260910.001`, with all eight manifests declaring it — modules at `.003`, libraries at `.002`. `check-analysis-versions.sh` had been reporting `BEHIND` the whole time; nothing was reading it because it is a release gate rather than a suite case, which is correct and is also how it sat unnoticed for a day. +Now `20260910.001`, with all eight manifests declaring it -- modules at `.003`, libraries at `.002`. `check-analysis-versions.sh` had been reporting `BEHIND` the whole time; nothing was reading it because it is a release gate rather than a suite case, which is correct and is also how it sat unnoticed for a day. ## THE CLASS: a glob pointed at the install store passes over nothing @@ -84,14 +84,14 @@ Now `20260910.001`, with all eight manifests declaring it — modules at `.003`, | | | |---|---| -| `check-analysis-versions.sh` | the per-module version loop — **zero modules examined**; a release could ship any manifest unbumped | +| `check-analysis-versions.sh` | the per-module version loop -- **zero modules examined**; a release could ship any manifest unbumped | | `export-environment.sh` | half of the guard the plan calls *"the sharpest trap in the whole feature"* | | `check-module-packages.sh` | the only real conda solve of the modules' own pins, asking it of an empty set | -| `bib2citations.py` | `--check` said *"every citations.json matches (2 files)"* — three module bibliographies unread | -| `00_static` ×5 | the DOI check, three suite-list loops, the manifest-anchor glob | +| `bib2citations.py` | `--check` said *"every citations.json matches (2 files)"* -- three module bibliographies unread | +| `00_static` x5 | the DOI check, three suite-list loops, the manifest-anchor glob | | `07_analysis_frame` | asserted the three modules **ship**, which the rework inverted | | `08_analysis_rlib` | the no-package grep, which printed `grep: ... No such file` and passed | -| the three module suites | `cat analysis/lib/R/*.R` — **0 passed, 6 failed** in `basicstats` once actually run | +| the three module suites | `cat analysis/lib/R/*.R` -- **0 passed, 6 failed** in `basicstats` once actually run | **The suite agreed with the bug rather than catching it.** Both version fixtures in `00_static` planted their `demo` module *into the store*, so the gate found it there and the case went green over a loop that finds nothing in reality. A fixture that matches the defect is worse than no fixture: it is a standing assertion that the wrong thing is right. @@ -99,17 +99,17 @@ Guards added so it cannot repeat quietly: a `ghost` module planted in the store ## Two defects in shipped code, found while fixing the gates -**`module_packages()` and `module_libraries()` did not terminate their last line.** `store_packages` runs one per module and concatenates, so the last spec of one manifest and the first of the next arrive as **one token** — `r-shared=2.0r-shared=2.0`, `chunk_rangesn_eff`. `cut -d= -f1` then keeps the first name and the second is gone from the keep-list, so **`uninstall` removes a package a still-installed module declares.** Reproduced with three modules where the shared package sits at a list boundary: `r-fst` dropped while `gamma` still needed it. Fixed by ending both with `awk -F'"' 'NF > 1 { print $2 }'`, which terminates its records. +**`module_packages()` and `module_libraries()` did not terminate their last line.** `store_packages` runs one per module and concatenates, so the last spec of one manifest and the first of the next arrive as **one token** -- `r-shared=2.0r-shared=2.0`, `chunk_rangesn_eff`. `cut -d= -f1` then keeps the first name and the second is gone from the keep-list, so **`uninstall` removes a package a still-installed module declares.** Reproduced with three modules where the shared package sits at a list boundary: `r-fst` dropped while `gamma` still needed it. Fixed by ending both with `awk -F'"' 'NF > 1 { print $2 }'`, which terminates its records. It did not bite the three shipped modules, by luck: their library lists overlap enough that every glued name also appears cleanly somewhere else in the `sort -u`. Two modules with a boundary name in common and nothing else would lose it. -**`ANALYSIS_ENV_FILE` was defined in `PoolSeqFlow` and read in `lib/wrapper_lib.sh`.** So `baseline_packages()` returned **nothing** for any `dev/` script that sourced the library on its own — silently, through `2>/dev/null` on an empty path — and an empty baseline subtracts nothing. Now defaulted in `wrapper_lib.sh` beside the function that reads it. +**`ANALYSIS_ENV_FILE` was defined in `PoolSeqFlow` and read in `lib/wrapper_lib.sh`.** So `baseline_packages()` returned **nothing** for any `dev/` script that sourced the library on its own -- silently, through `2>/dev/null` on an empty path -- and an empty baseline subtracts nothing. Now defaulted in `wrapper_lib.sh` beside the function that reads it. `test_the_keep_list_survives_more_than_one_other_module` in `02_launcher` covers both; the existing case could not, because with a single other module there is no join to get wrong. ## Still open after this pass - **Nothing is published, and publishing waits for the release.** `RELEASING.md` step 10 already puts it after the version bump, and the frame version is why that ordering is load-bearing rather than tidy. -- **The JVM and pipeline tiers still have not run.** `--fast` is 313 passing, `--cost static` 264 — but every Nextflow half is skipped, and `moduleLibraryFiles()` resolving a module's libraries at run time has still never executed. +- **The JVM and pipeline tiers still have not run.** `--fast` is 313 passing, `--cost static` 264 -- but every Nextflow half is skipped, and `moduleLibraryFiles()` resolving a module's libraries at run time has still never executed. - **The tarball extract/install/check step** (Z's step 3) is unbuilt. - **The docs pass.** The four-way audit finished, 39 agents; the manual's own claims about the shared library are corrected in `analysis/references.bib` and regenerated, but the directory-layout diagram and the Upgrading section are untouched. diff --git a/.claude/development-notes/nei-distance.md b/.claude/development-notes/nei-distance.md index bdd0f59..b382988 100644 --- a/.claude/development-notes/nei-distance.md +++ b/.claude/development-notes/nei-distance.md @@ -4,7 +4,7 @@ ## The question Z asked -The roster's original formulation was `d(A,B) = Σ_sites Σ_alleles |f_A − f_B|`, which is Prevosti's distance. An earlier session measured that it makes depth the first MDS axis and proposed Nei's minimum distance instead. Z pushed back with a specific alternative, and it is a better one than the roster's: +The roster's original formulation was `d(A,B) = Σ_sites Σ_alleles |f_A - f_B|`, which is Prevosti's distance. An earlier session measured that it makes depth the first MDS axis and proposed Nei's minimum distance instead. Z pushed back with a specific alternative, and it is a better one than the roster's: > *"consider the distances within a site with larger effects and between sites the effect is additive but not A + B. My original idea was to calculate d(AB) = sum( | p(A) - p(B) |) for the site and assume that each site is orthogonal to each other and calculate the sqrt (sum (d(AB) ^2 )) not even average it over the sites."* @@ -23,58 +23,58 @@ So Z's proposal **is** Nei's uncorrected form, up to a global scale that `cmdsca ## What actually separates them, and it is only two things -**1. The correction, which is where the whole benefit is.** Six pools from ONE population, two at 30× and four at 400×, 20 000 sites: +**1. The correction, which is where the whole benefit is.** Six pools from ONE population, two at 30x and four at 400x, 20 000 sites: | | Prevosti | Z's form | Nei raw | Nei corrected | |---|---|---|---|---| -| 30–30 | 0.09660 | 35.21 | 0.015493 | −0.00030 | -| 30–400 | 0.07890 | 28.37 | 0.010058 | −0.00012 | -| 400–400 | 0.05301 | 19.13 | 0.004575 | +0.00002 | -| **shallow:deep** | **1.822** | **1.840** | **3.387** | — | +| 30-30 | 0.09660 | 35.21 | 0.015493 | -0.00030 | +| 30-400 | 0.07890 | 28.37 | 0.010058 | -0.00012 | +| 400-400 | 0.05301 | 19.13 | 0.004575 | +0.00002 | +| **shallow:deep** | **1.822** | **1.840** | **3.387** | -- | -**Going from mean-of-`d_s` to root-sum-of-`d_s²` moved the depth artefact from 1.822 to 1.840 — it does not help.** Both are linear in the noise scale, so the ratio survives. The correction is the only thing that removes it, and Z's form already does the one thing that makes correction possible: it squares, at the site level, which is the level the correction lives at. It is *Prevosti* that cannot be corrected, because it never squares. +**Going from mean-of-`d_s` to root-sum-of-`d_s²` moved the depth artefact from 1.822 to 1.840 -- it does not help.** Both are linear in the noise scale, so the ratio survives. The correction is the only thing that removes it, and Z's form already does the one thing that makes correction possible: it squares, at the site level, which is the level the correction lives at. It is *Prevosti* that cannot be corrected, because it never squares. -**2. Multiallelic sites.** `(Σ|Δ_j|)² = ΣΔ_j² + 2Σ_{j= 3 the correction is unavailable in principle and not merely in practice. ## What was measured about multiallelic sites, including two negative results -Z asked specifically about these. The correction is **k-agnostic** — `Σ_j Var(f̂_j) = h/n` whatever the arity, so the same expression is exact at every k. Residual after correction: 2.08% of the raw bias at k=2, 0.94% at k=3, 1.05% at k=4. **Nothing in the module branches on arity, and this is why.** +Z asked specifically about these. The correction is **k-agnostic** -- `Σ_j Var(f̂_j) = h/n` whatever the arity, so the same expression is exact at every k. Residual after correction: 2.08% of the raw bias at k=2, 0.94% at k=3, 1.05% at k=4. **Nothing in the module branches on arity, and this is why.** -Multiallelic sites take more than their share of the distance under both forms — at 5% of sites they carry 7.9% of Z's distance and 6.1% of Nei's; at 15%, 22.3% against 17.8%. Under the null that excess is entirely the higher `h` at those sites, which is exactly what the correction removes. +Multiallelic sites take more than their share of the distance under both forms -- at 5% of sites they carry 7.9% of Z's distance and 6.1% of Nei's; at 15%, 22.3% against 17.8%. Under the null that excess is entirely the higher `h` at those sites, which is exactly what the correction removes. **Two negative results worth keeping, because both were expected to go the other way:** - **The over-weighting does not compound the depth artefact.** 1.836 restricted to multiallelic sites against 1.840 over the whole corpus. -- **It does not move the ordination.** Three populations × two pools at Fst 0.03, Procrustes residual between the two forms: 0.00000 at uniform depth, 0.00213–0.00363 with uneven depth, across 0–35% multiallelic. Algebraically real, empirically immaterial once there is any structure to find. +- **It does not move the ordination.** Three populations x two pools at Fst 0.03, Procrustes residual between the two forms: 0.00000 at uniform depth, 0.00213-0.00363 with uneven depth, across 0-35% multiallelic. Algebraically real, empirically immaterial once there is any structure to find. -**And one that inverts a premise in the plan.** The plan attributes the negative-eigenvalue trap to L1 not being Euclidean-embeddable. Measured, **Z's form produced zero negative eigenvalue mass in every cell and Nei corrected produced up to 0.59%** — because Z's form is an L2 accumulation which at biallelic sites is squared Euclidean and therefore PSD, and it is subtracting a *different correction per pair* that perturbs the matrix off the cone. True of Prevosti, not of the form we adopted. Both trivial at these magnitudes; the reporting obligation stands but the reason for it is different from what was written down. +**And one that inverts a premise in the plan.** The plan attributes the negative-eigenvalue trap to L1 not being Euclidean-embeddable. Measured, **Z's form produced zero negative eigenvalue mass in every cell and Nei corrected produced up to 0.59%** -- because Z's form is an L2 accumulation which at biallelic sites is squared Euclidean and therefore PSD, and it is subtracting a *different correction per pair* that perturbs the matrix off the cone. True of Prevosti, not of the form we adopted. Both trivial at these magnitudes; the reporting obligation stands but the reason for it is different from what was written down. ## The constant, derived rather than ported The plan flagged this as the one unverified item, on F1's lesson about `n_eff`. It is now derived and unit-tested: -`D_m = (J_X + J_Y)/2 − J_XY` expands to `½ Σ(x−y)²` — verified at k = 2, 3, 4. The unbiased homozygosity estimator `(n·Σx² − 1)/(n − 1)` equals `Σx² − h/(n − 1)`, verified at three effective sizes. `J_XY` takes no correction because the two pools are sequenced independently. Both are cases in `test/tools/r_lib_tests.R` under `nei_distance`, with the arithmetic written out beside each. +`D_m = (J_X + J_Y)/2 - J_XY` expands to `½ Σ(x-y)²` -- verified at k = 2, 3, 4. The unbiased homozygosity estimator `(n·Σx² - 1)/(n - 1)` equals `Σx² - h/(n - 1)`, verified at three effective sizes. `J_XY` takes no correction because the two pools are sequenced independently. Both are cases in `test/tools/r_lib_tests.R` under `nei_distance`, with the arithmetic written out beside each. -**Why the code is written in the `J` form rather than as `½Σ(f_A − f_B)²`:** they are the same number, but the correction attaches to `J` and not to the difference, so the `J` form makes it obvious that there are two subtracted terms and not three. +**Why the code is written in the `J` form rather than as `½Σ(f_A - f_B)²`:** they are the same number, but the correction attaches to `J` and not to the difference, so the `J` form makes it obvious that there are two subtracted terms and not three. ## A residual the manual states rather than hides -The correction slightly **over**-corrects at shallow depth. Over 12 replicate simulations, mean residual −1.24e-4 at 30–30 against −0.09e-4 at 400–400, with 10 of 12 replicates putting the shallow pair *below* the deep pairs. Residual spread 1.15e-4 against a raw spread of 1.09e-2 — the artefact is down ~95× — but it runs the opposite way, so shallow pools come out marginally too **close** rather than too far. That is the conservative direction. It is the `n_eff` approximation showing through and is largest where `n_eff` is smallest. +The correction slightly **over**-corrects at shallow depth. Over 12 replicate simulations, mean residual -1.24e-4 at 30-30 against -0.09e-4 at 400-400, with 10 of 12 replicates putting the shallow pair *below* the deep pairs. Residual spread 1.15e-4 against a raw spread of 1.09e-2 -- the artefact is down ~95x -- but it runs the opposite way, so shallow pools come out marginally too **close** rather than too far. That is the conservative direction. It is the `n_eff` approximation showing through and is largest where `n_eff` is smallest. ## Rcpp, and the number that justified it -Z: *"for the distance calculations we should definitely use Rcpp."* Measured by `dev/scripts/bench-compiled-paths.R` at 3.2M sites, scaled to 100M, six pools: **710 s vectorized against 22 s compiled, 32×.** +Z: *"for the distance calculations we should definitely use Rcpp."* Measured by `dev/scripts/bench-compiled-paths.R` at 3.2M sites, scaled to 100M, six pools: **710 s vectorized against 22 s compiled, 32x.** -That ratio is not the parsers' ~10× and should not be averaged with it. The vectorized form makes one `rowsum` pass per *pair*, so what compiling removes is interpreted call overhead growing with the square of the pool count, where the parsers grow with the site count in memory traffic. The consequence for the manual: `mds` is the one module whose statistic costs about as much as reading the table for it — 675 s parse + 710 s distance vectorized, against 69 + 22 compiled. +That ratio is not the parsers' ~10x and should not be averaged with it. The vectorized form makes one `rowsum` pass per *pair*, so what compiling removes is interpreted call overhead growing with the square of the pool count, where the parsers grow with the site count in memory traffic. The consequence for the manual: `mds` is the one module whose statistic costs about as much as reading the table for it -- 675 s parse + 710 s distance vectorized, against 69 + 22 compiled. ## Where the code lives, and why the first answer was wrong @@ -88,12 +88,12 @@ So `nei_distance.R`, `add_distance.R` and `mean_distance.R` are library files (o ## Questions that were settled by precedent rather than re-argued -- **The `*` spanning deletion.** It reaches the SNP table and its frequency is depth-dependent in a way an ordinary allele's is not, and a site carrying it is multiallelic by construction — so it lands in exactly the bucket that gets extra weight. `basicstats` already ruled: it is one allele of the site like any other, and dropping it would renormalize the site to something no pool was sequenced at. `mds` carries the same gate rather than inventing a second answer. +- **The `*` spanning deletion.** It reaches the SNP table and its frequency is depth-dependent in a way an ordinary allele's is not, and a site carrying it is multiallelic by construction -- so it lands in exactly the bucket that gets extra weight. `basicstats` already ruled: it is one allele of the site like any other, and dropping it would renormalize the site to something no pool was sequenced at. `mds` carries the same gate rather than inventing a second answer. - **Pools, not units.** Rule 17c collapses to units because degrees of freedom come from units. An ordination is not a test and has no degrees of freedom, and two pools of one unit landing apart is the thing an ordination is read to see. `mds.tsv` carries the unit as a label so a reader can tell which points should have coincided. ## Two implementation traps, both found by building it - **`D_m` is a squared distance and `cmdscale` squares what it is given.** `cmdscale(D_m)` ordinates a quartic; `cmdscale(sqrt(D_m))` hits `NaN` on the negative entries the design deliberately does not floor. `ordinate()` double-centers directly, which is what `cmdscale` does internally, needs no square root, and is what lets the "not floored at zero" promise be kept. `test_the_coordinates_reproduce_the_distance_matrix` is the guard. -- **`n_eff` is exactly 1 at depth 1, whatever the pool holds** — `n·d/(n + d − 1)` with `d = 1` is `n/n`. So the correction divides by zero at any single-read site, and since observed `h` is 0 there it arrives as `0/0`. Made an explicit NA and a dropped site. The same expression is 1 for `n_chrom = 1` at *every* depth, which is a single haploid individual and is refused at load. +- **`n_eff` is exactly 1 at depth 1, whatever the pool holds** -- `n·d/(n + d - 1)` with `d = 1` is `n/n`. So the correction divides by zero at any single-read site, and since observed `h` is 0 there it arrives as `0/0`. Made an explicit NA and a dropped site. The same expression is 1 for `n_chrom = 1` at *every* depth, which is a single haploid individual and is refused at load. Related: `depth-cutoff.md` for how depth reaches a published number elsewhere, and `calibration.md` for the corpus this module's expectations were added to. diff --git a/.claude/development-notes/parameter-resolution.md b/.claude/development-notes/parameter-resolution.md index 3877507..272a713 100644 --- a/.claude/development-notes/parameter-resolution.md +++ b/.claude/development-notes/parameter-resolution.md @@ -1,10 +1,10 @@ # Parameter resolution -**Written 2026-08-31, against the tree at `7d65893`.** `resolve_parameters.nf` is unchanged — every function named below still exists and still does what is described. `diploidy` is `ploidy` since, corrected in place. +**Written 2026-08-31, against the tree at `7d65893`.** `resolve_parameters.nf` is unchanged -- every function named below still exists and still does what is described. `diploidy` is `ploidy` since, corrected in place. -`scripts/resolve_parameters.nf` — what the pipeline computes when you have not set it yourself, and how one run's parameters are built from a multi-run row. +`scripts/resolve_parameters.nf` -- what the pipeline computes when you have not set it yourself, and how one run's parameters are built from a multi-run row. -> **For the manual, not here.** The user-facing rule is worth stating plainly in the parameter reference: **a parameter you set is used exactly as written; one you leave commented out is computed; and a value you set feeds whatever is computed from it.** Pinning `cores.samtools` moves `cores.javaGc` with it, so pinning one thing never strands what depends on it. Also user-facing: any parameter may be varied per run, including the derived ones — but pinning a derived value costs the link back to what it was derived from, which step 0 reports. +> **For the manual, not here.** The user-facing rule is worth stating plainly in the parameter reference: **a parameter you set is used exactly as written; one you leave commented out is computed; and a value you set feeds whatever is computed from it.** Pinning `cores.samtools` moves `cores.javaGc` with it, so pinning one thing never strands what depends on it. Also user-facing: any parameter may be varied per run, including the derived ones -- but pinning a derived value costs the link back to what it was derived from, which step 0 reports. ## Why the computation moved out of parameters.config @@ -14,32 +14,32 @@ Measured against the real config, not assumed: | Override | What actually happened | |---|---| -| `bcftools.maxDepth` → 5000 | `bcftools.mpileupOptions` still emitted `-d 2000` (the scope is `variantCall` since `cc00833`; the behavior is unchanged) | -| `trim_galore.quality` → 15 | `trim_galore.options` still said `-q 25` | +| `bcftools.maxDepth` -> 5000 | `bcftools.mpileupOptions` still emitted `-d 2000` (the scope is `variantCall` since `cc00833`; the behavior is unchanged) | +| `trim_galore.quality` -> 15 | `trim_galore.options` still said `-q 25` | -No warning, exit 0 — the run reports the value you asked for and uses the old one. +No warning, exit 0 -- the run reports the value you asked for and uses the old one. -Overriding a **top-level** parameter has no such problem: `poolSize` really does re-derive `filterFalsePositives.sensitivity`, and `threads` really does re-derive the cores ladder. That is why parameters whose only inputs are top-level — the reference paths, `sensitivity`, `snpEff.db` — are still computed in the config file, where they read better. +Overriding a **top-level** parameter has no such problem: `poolSize` really does re-derive `filterFalsePositives.sensitivity`, and `threads` really does re-derive the cores ladder. That is why parameters whose only inputs are top-level -- the reference paths, `sensitivity`, `snpEff.db` -- are still computed in the config file, where they read better. ## Two Nextflow mechanics this depends on, both verified -- **Writing into an existing scope (`params.cores.trim = 4`) is visible everywhere**, because scope blocks are ordinary maps shared across modules. Creating a new **top-level** key from inside a module is **not** — it is visible only within that module, and processes see null. So every scope written here must already exist in the config file, which is why each carries its keys commented out rather than being absent. -- **Absence is the signal, not emptiness.** `containsKey` distinguishes "not set" from "deliberately set to nothing", and an empty string is a legitimate value — it is what `trim_galore.adapterOptions` holds whenever adapter autodetection is on. +- **Writing into an existing scope (`params.cores.trim = 4`) is visible everywhere**, because scope blocks are ordinary maps shared across modules. Creating a new **top-level** key from inside a module is **not** -- it is visible only within that module, and processes see null. So every scope written here must already exist in the config file, which is why each carries its keys commented out rather than being absent. +- **Absence is the signal, not emptiness.** `containsKey` distinguishes "not set" from "deliberately set to nothing", and an empty string is a legitimate value -- it is what `trim_galore.adapterOptions` holds whenever adapter autodetection is on. -`resolveParameters()` is called once, first thing in the entry workflow, so it runs before any process script is evaluated and before `analysisParams()` builds the change-guard manifest — which therefore records the values actually used, whether set or computed. +`resolveParameters()` is called once, first thing in the entry workflow, so it runs before any process script is evaluated and before `analysisParams()` builds the change-guard manifest -- which therefore records the values actually used, whether set or computed. **The analysis layer imports that same `analysisParams()`** rather than defining its own projection: `analysis/modules.nf` calls it and subtracts the module's own prefixes. So the pipeline's change guard and a module's identity check are the same list of settings asked at two moments, and cannot disagree about what a run was configured with. ## The duplication with parameters.config -`deriveRunPaths()` is **a second copy of logic that also exists in parameters.config**, and there is no way around it: config interpolation runs once at parse time against one set of values, and a run changing `referenceFile` needs `referencePath`, `reference` and `snpEff.db` to move with it. The config's copy cannot be dropped either — `nextflow config -flat` is how the wrapper learns the paths that `clean` and `reset` delete, and it only ever sees the config. +`deriveRunPaths()` is **a second copy of logic that also exists in parameters.config**, and there is no way around it: config interpolation runs once at parse time against one set of values, and a run changing `referenceFile` needs `referencePath`, `reference` and `snpEff.db` to move with it. The config's copy cannot be dropped either -- `nextflow config -flat` is how the wrapper learns the paths that `clean` and `reset` delete, and it only ever sees the config. The duplication is held together **by a test rather than by care**: for a single run the function must reproduce exactly what the config computed. Drift fails a test instead of silently sending one run's output somewhere else. ## Core counts - `coreLadder()` is the largest power of two at or below `threads`, capped at 8, which is where the published scaling for these tools flattens out. -- `trimCores()` costs Trim Galore on its **full footprint**, not its worker count: `--cores N` runs N+4 threads (N workers + 2 decompressors + 1 batcher + 1 writer) for any N ≥ 2, while `--cores 1` bypasses the pool and is genuinely single-threaded. So it picks the largest N whose N+4 still fits. +- `trimCores()` costs Trim Galore on its **full footprint**, not its worker count: `--cores N` runs N+4 threads (N workers + 2 decompressors + 1 batcher + 1 writer) for any N >= 2, while `--cores 1` bypasses the pool and is genuinely single-threaded. So it picks the largest N whose N+4 still fits. - `samtools -@` counts **additional** threads, so 0 means one core and 1 means two. `-XX:ParallelGCThreads` is a total, not an increment. ## Traps encoded here @@ -47,37 +47,37 @@ The duplication is held together **by a test rather than by care**: for a single - **`fill()` takes a computed value, not a closure.** The strict parser rejects invoking a closure-typed parameter. Harmless because every expression is pure and the calls are ordered so anything one reads is already filled. - **Plain recursive functions, not self-referencing closures.** `def walk; walk = { ... walk(...) }` fails with "`walk` is not defined". - **`deepCopy` has to be deep.** Scope blocks are shared maps: a shallow copy would leave every run writing into one `cores` and one `trim_galore`, so the last row parsed would silently decide the settings for all of them. -- **`setDotted` guards `parts[0..-2]`** — a top-level column like `poolSize` has no scopes to walk into, and `[0..-2]` on a one-element list is a *negative range* rather than an empty one. -- **Types are preserved from the config.** Everything arrives from CSV as a string, and a `poolSize` of `"50"` divides differently from `50` — `sensitivity` is `1.0 / (2 * ploidy * poolSize)`, so this is integer-vs-string, not cosmetic. When the key is being created there is nothing to copy a type from, so a digit string becomes an Integer — otherwise `task.cpus` gets a String and the process fails somewhere far from here. +- **`setDotted` guards `parts[0..-2]`** -- a top-level column like `poolSize` has no scopes to walk into, and `[0..-2]` on a one-element list is a *negative range* rather than an empty one. +- **Types are preserved from the config.** Everything arrives from CSV as a string, and a `poolSize` of `"50"` divides differently from `50` -- `sensitivity` is `1.0 / (2 * ploidy * poolSize)`, so this is integer-vs-string, not cosmetic. When the key is being created there is nothing to copy a type from, so a digit string becomes an Integer -- otherwise `task.cpus` gets a String and the process fails somewhere far from here. - **`-q` is mapping quality and `-Q` is base quality** in the mpileup options. They are not interchangeable and were once supplied to each other. ## The row is applied twice, on purpose -Between the two passes the derivations run. So a row setting an **input** to a derivation gets the derived value recomputed — setting `poolSize` moves `sensitivity` with it — and the second pass is what makes a row that sets a **derived** value directly win anyway. Both are required: any parameter may be varied, including the computed ones, because benchmarking needs it. +Between the two passes the derivations run. So a row setting an **input** to a derivation gets the derived value recomputed -- setting `poolSize` moves `sensitivity` with it -- and the second pass is what makes a row that sets a **derived** value directly win anyway. Both are required: any parameter may be varied, including the computed ones, because benchmarking needs it. -`runDefinitions()` **must** be called before `resolveParameters()`. Afterwards there is no way to tell a value the user pinned from one we filled in, and `fill` refuses to overwrite a key that is already there — so a run changing an input to a derivation would keep the base run's derived value and quietly use the old setting. The copy has to be taken while "absent" still means "not set". +`runDefinitions()` **must** be called before `resolveParameters()`. Afterwards there is no way to tell a value the user pinned from one we filled in, and `fill` refuses to overwrite a key that is already there -- so a run changing an input to a derivation would keep the base run's derived value and quietly use the old setting. The copy has to be taken while "absent" still means "not set". ## `knownParameterNames()` is not a whitelist Any parameter may be varied between runs; there is deliberately no list of permitted ones. This is the set of names that **are** parameters, so a column naming something else can be refused rather than quietly doing nothing. A run whose setting is silently ignored is worse than one that will not start. -Derived from the live `params` map rather than written out, so a parameter added to parameters.config is varyable the moment it exists. The derived names are unioned in because the template leaves them commented out — they do not exist in `params` until `resolveParameters()` fills them. +Derived from the live `params` map rather than written out, so a parameter added to parameters.config is varyable the moment it exists. The derived names are unioned in because the template leaves them commented out -- they do not exist in `params` until `resolveParameters()` fills them. ## Missing versus malformed `metadataRows()` splits the two deliberately: - **Missing is not an error here.** Absent is the ordinary state of a project someone has not finished setting up, and step 0 exists to say so in context, in a report kept beside the results. So it returns an empty list and lets the run reach the stage with the good message. Nothing computes in between: every step is gated on step 0. -- **Malformed throws.** `parse_metadata.py`'s message is already the best available — every problem at once, with line numbers — and throwing puts it in front of the user seconds after they start the run. The multi-run table gets the same treatment for the same reason; it runs while the DAG is being built, before step 0 executes, so the message has to stand on its own. +- **Malformed throws.** `parse_metadata.py`'s message is already the best available -- every problem at once, with line numbers -- and throwing puts it in front of the user seconds after they start the run. The multi-run table gets the same treatment for the same reason; it runs while the DAG is being built, before step 0 executes, so the message has to stand on its own. Both shell out rather than parsing in Groovy, because the Python parsers handle CSV quoting that `readPattern` needs (its default contains a comma) and are unit-tested at a millisecond a case. ## A run no longer gets its own storageDir (Z, 2026-08-27) -It used to default to `${storageDir}/${RunID}`, which gave every run a complete parallel tree — and once work is shared there is no such thing as one run's complete tree, so the parallel trees described something that had stopped being true. There is one storage root now, and `deriveRunPaths()` names the run *inside* `Output/` and `Logs/`. +It used to default to `${storageDir}/${RunID}`, which gave every run a complete parallel tree -- and once work is shared there is no such thing as one run's complete tree, so the parallel trees described something that had stopped being true. There is one storage root now, and `deriveRunPaths()` names the run *inside* `Output/` and `Logs/`. A `storageDir` column still works; it just means that run's results are somewhere else entirely. Two runs not sharing a storage root cannot share a results directory, so sharing between them is refused rather than resolved. -**`dir.utilized` keeps its per-run suffix, and that matters more than it looks.** It hangs off `mainDir`, and runs *share* `mainDir`. Without the suffix every run writes `Utilized/VCF/Test.vcf` to one path, and the second run's skip check finds the first run's file and symlinks to it — the whole VCF chain silently reused across runs that differ. +**`dir.utilized` keeps its per-run suffix, and that matters more than it looks.** It hangs off `mainDir`, and runs *share* `mainDir`. Without the suffix every run writes `Utilized/VCF/Test.vcf` to one path, and the second run's skip check finds the first run's file and symlinks to it -- the whole VCF chain silently reused across runs that differ. `dir.logs` is per run and not merely per project because log files are named for the step and the sample: three runs sharing one Logs tree would put three writers on one file, and one-writer-per-file is what lets tasks append without locking. diff --git a/.claude/development-notes/parameter-rules.md b/.claude/development-notes/parameter-rules.md new file mode 100644 index 0000000..3d3ae28 --- /dev/null +++ b/.claude/development-notes/parameter-rules.md @@ -0,0 +1,91 @@ +# Parameter validation, as built + +**Written 2026-09-29 and 30, against the tree at `f54a5ae` plus uncommitted work.** The build of what `parameter-validation.md` planned earlier the same day. That note is the design and the reasoning; this is what the code does and where it differs from the plan. + +## The shape the plan called for, and it survived + +`bin/check_parameters.sh` reads a resolved parameter set and writes **one tab-separated finding per rule that has something to say**: + + LEVEL label verdict detail explanation + +`FAIL`, `WARN` or `NOTE`. No color, no indentation, no wrapping: every caller formats it the way it formats everything else. Exit status is 1 when any finding is a `FAIL`, so a caller may use the status alone and read nothing. **Silence is the answer for a sound parameter set** -- a healthy project gets one `NOTHING TO FLAG` line rather than nine passes. + +**Two callers, one implementation**, which was the whole point: + +- `bin/check_project.sh` pipes `nextflow config -flat` into it and renders each level with its own `fail`/`warn`/`note`, folding the explanation under the row. +- `scripts/0_verify_environment.nf`'s `CheckRunParameters` pipes `current_params.txt` into it -- the same resolved set it already records for the change guard -- and logs findings through `log_message`, so they reach the archived report. A `FAIL` sets `STATUS=FAIL` and the run stops there. + +**The inline `scaleMapQ` block added to `check_project.sh` earlier the same day is gone**, and its seven cases went with it, as the plan insisted: two copies is how the two sides drift. + +## One thing the plan did not anticipate: two input shapes + +The callers hold the values differently and neither should have to reformat. `nextflow config -flat` writes `params. = `; step 0 already holds the same set as `=` from `analysisParams()`. The parser reads both. + +That also settles which values are available in step 0 without new plumbing. `analysisParams()` excludes paths, resources, `dir.`, `cores.`, `java.`, `software.` and `capBAM.histogramMax` -- and every key these rules read survives it, including `fastqc.memory`, which is not the top-level `memory` that `skipKey` names. + +## The rules, and that every threshold was measured + +| rule | level | where the number comes from | +|---|---|---| +| `scaleMapQ` below `varQualMin` | FAIL | measured on real pools at six values of `-q`: 12, 15, 20, 30, 40, 50. One below returned zero sites every time | +| `sampleThreshold` above 1 | FAIL | measured against the fixture VCF: 1.5 left 0 of 135 sites | +| `ploidy` or `poolSize` below 1 | FAIL | sensitivity is `1/(2*ploidy*poolSize)`, so below 1 it is infinite or negative | +| `fastqc.memory` carrying a unit | FAIL | the template already says FastQC rejects `2G` | +| `scaleMapQ` at or below 10 | WARN | 0, 5 and 10 are byte-identical to no adjustment | +| `scaleMapQ` under twice `varQualMin` | WARN | at `-C` equal to `-q`, 6% of the sites an unadjusted run called | +| `sampleThreshold` at or below 0 | WARN | the clause asks for nothing | +| `ploidy * poolSize` below 2 | WARN | see below | +| `minDP 0` with `dropZeroDepth` off | NOTE | the one combination that lets an unmeasured cell reach a published table | +| positive `variantCall.maxDepth` with `capBAM.maxDepth` at -1 | NOTE | two ceilings, and the smaller decides | + +**`ONE CHROMOSOME` was drafted as a FAIL and corrected to a WARN by measuring.** `n_eff(1, 50)` returns exactly 1, so the unbiased correction `n_eff/(n_eff - 1)` is `Inf` rather than an error: the **pipeline runs and publishes frequencies perfectly well**, and only an analysis degrades. Refusing a run at step 0 for that would be stricter than the thing being protected. The earlier draft's explanation also claimed the analysis layer "refuses such a pool", which it does not. + +## An absent key, and the one rule where absence is a value + +Z asked whether parameters with defaults were accounted for, and one rule was wrong because they were not. + +**`dropZeroDepth` absent is `false`, not unknown.** It is read in a Groovy ternary at `scripts/7_vcf2freq.nf`, so a config written before the parameter existed leaves it `null`, `null` is falsy, and the run builds the no-zero-term expression -- behaving exactly as `false` while nothing says so. The first draft tested `= "false"` and therefore emitted **no finding at all** for a config with `minDP = 0` and the key missing, measured. That is the one combination that cannot tell it is in trouble, and it is precisely the migration hole's real exposure: someone copying an old `parameters.config` into a new project. The test is `!= "true"` now. + +**Any later boolean read through a ternary inherits this**, which is why the comment says so rather than naming this parameter. + +**Every other rule declines to judge a key it does not have, and that is correct.** After resolution an absent key means the project's config never defined it, so there is nothing to compare; inventing a verdict about a parameter nobody set would be worse than silence. A parameter set holding one irrelevant line produces no crash, no spurious finding, and one honest `NOT CHECKED` for the pileup pair. + +Two cases cover this: an absent `dropZeroDepth` with `minDP 0` must be reported, and a near-empty set must produce nothing but that one note. The first fails when the rule is reverted to `= "false"`. + +`is_num` was also replaced while here. It had a nested parameter expansion meant to reject a lone `.` and was too clever to read; it is a plain character-class test now, with the arithmetic left to awk. + +## What is deliberately not in it + +**A setting that merely produces fewer sites.** `minDP 20` against `minDP 5` is a scientific choice and this has no opinion on it. The line is: does it make the run produce NOTHING, or silently change what a published number means? + +**Anything that needs runtime data.** `cutadapt.min_length` above the `-l` that `ClipReads` computes discards every read and exits 0 -- the template says so -- but the length it is compared against does not exist until the step runs, so it cannot be judged from a configuration. + +**Any repair.** Nothing here rewrites a value or picks one. It says what will happen and stops or does not. + +## Coverage, and the two places a case could be hollow + +- **12 rule cases in `03_helpers`**, static cost, calling the helper directly over the shipped defaults with one assignment replaced. The first of them asserts **silence on the defaults**, which is the case most likely to rot: a rule added with a wrong threshold makes every ordinary run noisy, and a user who sees a warning on a default stops reading warnings. +- **5 wiring cases in `02_launcher`**, against a stubbed `nextflow` so they need no JVM: that `check_project.sh` calls the helper, renders each level, folds the explanation, lets a `FAIL` decide its exit status, and reports the helper's own absence rather than passing over it. +- **3 step-0 cases in `05_guards`**: a FAIL stops the run with nothing published, a WARN does not, and the defaults raise no finding at all. Each uses a **fresh** sandbox rather than the baseline one, because changing a parameter in the baseline also trips the change guard and then one failure has two causes and no way to tell them apart. + +**One assertion had to be rewritten and the reason generalises.** It looked for `Raise scaleMapQ above varQualMin` in the step-0 report, and `fold -s` had landed between `Raise` and `scaleMapQ`. **No multi-word phrase is safe to assert against folded output.** It now asserts the deeper indent, which says a wrapped explanation was printed at all and holds however the prose changes. + +## What it did to the emptiness cases, the same day + +The guards in `empty-results.md` were built hours earlier, and this **pre-empted two of their three cases**: they reached the step 6 and step 7 guards by setting `scaleMapQ = 15` and `sampleThreshold = 1.5`, and both are now refused at step 0 before the pipeline starts. The earlier stop is the better behavior and the guards are untouched, but two cases were left asserting something that no longer happened. + +**The shape recurs and is worth naming: a new earlier check makes a later one unreachable by its original route, and the later one still needs a route.** + +**The cross-sample guard now has no case at all.** The only configuration that emptied that filter was `sampleThreshold` above 1. Measured while looking for another door: at `sampleThreshold 1.0` with `poolSize 2`, 39 of 135 sites still survive, and sensitivity is not the binding clause at fixture depth. No value step 0 permits empties it. The guard stays, as defense against DATA rather than configuration -- a cohort whose variants are shared by too few pools -- and `04_pipeline` carries a comment saying why a case cannot be written, so nobody writes one that cannot work. + +**The call-set guard was rerouted, after two dead ends worth not repeating:** + +- `scaleMapQ = 15` is the same failure by a shorter path, and step 0 refuses it. +- `baseQualMin = 60` **does nothing at all**. Measured on the fixture's own ready BAMs: 123 records at `-Q 30` and 123 at `-Q 60`, indels included. A synthetic BAM with uniform Q40 and one variant site had said otherwise, which is the second time in one session a synthetic fixture gave a confidently wrong answer about a bcftools flag. +- `varQualMin = 500` works. No read carries a mapping quality near it, so every read is skipped: 0 records against 123. + +**And the case is stronger for needing two settings.** `varQualMin = 500` alone trips the rule here, so `scaleMapQ` goes to 600 to clear it, which is permitted with a warning -- correctly, because that pair does emit records for ordinary data. So the case passes only when BOTH layers work: this check permissive enough to let it through, and the step 6 guard catching what a configuration cannot reveal. + +## The selector, again + +`dev/scripts/select-tests.py bin/check_parameters.sh` did not select `02_launcher`, because `check_project.sh` calls the helper by full path rather than by bare name and the derived graph cannot see that. Declared in that suite's `# covers:`. This is the third undeclared data dependency found in one day, after `parameters.config.template` and `metadata.csv.template`, and they are all the same shape: the graph reads imports, and a file named inside a string is invisible to it. diff --git a/.claude/development-notes/parameter-validation.md b/.claude/development-notes/parameter-validation.md new file mode 100644 index 0000000..7c008fa --- /dev/null +++ b/.claude/development-notes/parameter-validation.md @@ -0,0 +1,72 @@ +# Parameter validation: a plan, not a build + +**Written 2026-09-29, against the tree at `85eb380` plus uncommitted work. PLANNED, for before the next release cycle.** Z's call, after the `scaleMapQ` / `varQualMin` measurement showed that a setting can be individually reasonable and fatal in combination, and that the pipeline had nothing anywhere that would say so. + +## What is missing + +**`scripts/0_verify_environment.nf` validates no parameters at all.** It checks the environment, the tools, the files and the tables. It does not check a single value in `parameters.config`, and `vcffilter` and `mpileupOptions` appear nowhere in it. + +So every settings mistake is caught in one of three ways, all worse than step 0: + +- **By a runtime guard**, minutes or hours in. `capBAM.histogramMax` in step 5, and as of this cycle the three emptiness refusals in steps 6 and 7. +- **By `PoolSeqFlow check project`**, which is optional and which many users will never run. +- **Not at all.** Which was the case for everything until this cycle. + +## The rule for what belongs here + +**Validate what produces nothing, or what silently changes what a number means. Never validate what merely produces fewer sites.** + +`minDP = 20` against `minDP = 5` is a scientific choice and the pipeline has no business having an opinion. `scaleMapQ` below `varQualMin` produces an empty run, and `minDP = 0` with `dropZeroDepth = false` changes the published tables from "every cell measured" to "some cells are NA" - which every analysis module leans on. Those are not preferences. + +This is the existing house rule read strictly: *fail loudly or document; never automate away a decision*. Validation is the loud failure. It does not remove a knob and it never picks a value. + +## The catalogue, as far as this session established it + +Detectable from the configuration alone, so they belong in step 0: + +| setting | the wrong value | what happens now | +|---|---|---| +| `variantCall.scaleMapQ` vs `varQualMin` | below `varQualMin` | every read discarded, empty VCF, empty tables. **Measured at six thresholds** | +| `variantCall.scaleMapQ` | 1 to 10 | no adjustment at all, whatever was intended | +| `filterFalsePositives.sampleThreshold` | above 1 | every site removed. **Measured: 1.5 leaves zero of 135** | +| `filterFalsePositives.sampleThreshold` | 0 | the cross-sample filter is inert | +| `vcffilter.minDP` | 0 with `dropZeroDepth = false` | NA cells reach the published tables; the no-missing-cell guarantee the modules rest on is gone. Carried from the release triage as "`minDP > 0` is not validated" | +| `ploidy` | 0 or negative | `sensitivity` is infinite or negative and the false-positive filter is meaningless | +| `poolSize` | 1 | `basicstats` refuses a pool of one chromosome, four steps later | +| `variantCall.maxDepth` | a small positive number | mpileup's ceiling silently undercuts the per-sample cap step 5 measured | +| `capBAM.maxDepth` | a small positive number | caps nearly every read away, and it is not `-1` so nothing measures it | +| `fastqc.memory` | `'2G'` | FastQC rejects it. The template says "as a plain number" and that is the only defense | + +Needs runtime data, so it belongs in the step that computes it rather than in step 0: + +| setting | why step 0 cannot see it | +|---|---| +| `cutadapt.min_length` | cutadapt applies `-m` after the `-l` that `ClipReads` computes per sample, so whether it discards every read depends on a number that does not exist yet. **The template already documents this as "discards every read and exits 0"** | + +`capBAM.histogramMax` is already guarded at runtime in step 5 and wants nothing here. + +## The design point that matters: do not write the rules twice + +`bin/check_project.sh` has to make the same judgements step 0 makes, and a project told one thing before a run and another during it is worse than either alone. This project already has the pattern for that - `00_static` asserts *"the per sample parameter table is the same on both sides"* and *"both sides agree on what a package spec may be"*. + +**So the rules live in one place, and both callers read it.** The shape that works: a shell helper, because `check_project.sh` is bash and step 0 runs a shell script, and both can call the same thing where neither can call a Groovy function. + + bin/check_parameters.sh reads the flat config, prints a verdict per rule, exits non-zero on a failure + <- bin/check_project.sh in its configuration section + <- scripts/0_verify_environment.nf through log_message, so it lands in the archived report + +One implementation, one set of cases in `03_helpers` at static cost, and both callers exercised by the suites that already cover them. + +**This supersedes the ad-hoc check added to `check_project.sh` this cycle.** The `scaleMapQ` / `varQualMin` verdict is currently written directly into that script with seven cases in `02_launcher` behind it. When the helper is built, that block moves into it and the cases move with it - **do not leave two copies**, which is exactly how the two sides drift. + +## What it reads + +Both callers need the **composed** values, not the raw settings, for the same reason the `check_project.sh` block already does: a project that pins `variantCall.mpileupOptions` by hand makes `scaleMapQ` and `varQualMin` inert, and judging the settings it was not built from reports a problem that does not exist. `nextflow config -flat` is the source, `-C` and `-q` are read out of the composed string, and an option string carrying neither is reported as unjudged rather than guessed at. + +Step 0 already has the parsed configuration, so it needs no second parse. + +## Sizing + +Medium. The helper is the small part. What it carries: a rule per row of the catalogue above, each with a message naming the setting and what to do; two call sites; cases for the helper at static cost plus one through step 0; the move of the existing block out of `check_project.sh`; and a manual section - the troubleshooting page at `#empty-result-refusals` is where it belongs, since the refusals and the pre-run verdicts are two halves of one story. + +**It is not a release blocker and it is not a module.** It goes in before the release cycle opens, because a release is when the manual is swept and these messages are user-facing text. diff --git a/.claude/development-notes/promotion.md b/.claude/development-notes/promotion.md index 78d37c8..9759948 100644 --- a/.claude/development-notes/promotion.md +++ b/.claude/development-notes/promotion.md @@ -1,6 +1,6 @@ -# Promotion — moving artifacts from the working volume to permanent storage +# Promotion -- moving artifacts from the working volume to permanent storage -**Written 2026-08-31 at 15:18, against the tree at `7d65893`.** Everything about promotion itself still holds — five attachment points, five rows in the table, the same gates. One thing below describes `atomic_mv.sh` as it was for another seven hours: the staging fix landed the same evening and the reasoning is corrected in place. +**Written 2026-08-31 at 15:18, against the tree at `7d65893`.** Everything about promotion itself still holds -- five attachment points, five rows in the table, the same gates. One thing below describes `atomic_mv.sh` as it was for another seven hours: the staging fix landed the same evening and the reasoning is corrected in place. `scripts/9_completion.nf`. @@ -8,55 +8,55 @@ ## Why it is one file -*"Was this file really moved, and is the source gone"* is the question that decides whether a result exists, so every answer to it should be readable in one place. The alternative — a move at the end of each step's own module — spreads the same three lines across eight files and makes a missing one invisible. +*"Was this file really moved, and is the source gone"* is the question that decides whether a result exists, so every answer to it should be readable in one place. The alternative -- a move at the end of each step's own module -- spreads the same three lines across eight files and makes a missing one invisible. ## Why it wires at several points and not one terminal node -A single node at the end of the DAG would hold roughly 2.5–3× the raw data on the fast volume until the run finished, which is the opposite of the point. Trimmed reads can go as soon as alignment succeeds; alignments as soon as cleaning succeeds. The fast volume is released as the run proceeds. +A single node at the end of the DAG would hold roughly 2.5-3x the raw data on the fast volume until the run finished, which is the opposite of the point. Trimmed reads can go as soon as alignment succeeds; alignments as soon as cleaning succeeds. The fast volume is released as the run proceeds. ## Three constraints, all established by probing -1. **Promotion cannot be driven off a staged input.** Six process inputs in this pipeline are pure ordering barriers rather than real reads — the process names an absolute `params.*` path in its script and never touches the staged copy. For those, "the consumer has the file" is a scheduling fact, not a data-flow one. So promotion is triggered by a completion **signal**, never by receiving the artifact. -2. **It is the LAST consumer that matters, not "the" consumer.** `Test.vcf` has two — step 7 always, step 8 whenever `annotate` is true — with no ordering between them. So a trigger can be parameter-dependent, and anything assuming one consumer will delete a file another step is about to read. +1. **Promotion cannot be driven off a staged input.** Six process inputs in this pipeline are pure ordering barriers rather than real reads -- the process names an absolute `params.*` path in its script and never touches the staged copy. For those, "the consumer has the file" is a scheduling fact, not a data-flow one. So promotion is triggered by a completion **signal**, never by receiving the artifact. +2. **It is the LAST consumer that matters, not "the" consumer.** `Test.vcf` has two -- step 7 always, step 8 whenever `annotate` is true -- with no ordering between them. So a trigger can be parameter-dependent, and anything assuming one consumer will delete a file another step is about to read. 3. **Cardinality is fragile.** This module attaches as an **additional consumer** of an existing channel, alongside the real consumer and never in front of it. Nothing upstream changes shape, so there is nothing to flip. That mattered most while every singleton rode an implicit value channel, where inserting an operator would have turned it into a queue channel and silently reduced N tasks to 1 with the run still reporting success. -## Why the artifact→gate table is written out +## Why the artifact->gate table is written out Nextflow's own dependency graph cannot answer this, and all three limits were found by probing: - it **asserts dependencies that are not reads** (the six ordering barriers above); -- it **misses reads that are never declared** — `VerifyAll` and `CheckMetadataFile` take `val` for files and then read them by absolute path into other tasks' work directories; +- it **misses reads that are never declared** -- `VerifyAll` and `CheckMetadataFile` take `val` for files and then read them by absolute path into other tasks' work directories; - it **describes one session**, while an artifact's lifetime here spans runs. This pipeline resumes by looking at the filesystem, `-resume` is unused, and `cleanup = true` removes the work directories behind it. -The table grows **one row per stage**, each landing with the change that makes it true. A stage with no row yet returns null and is recorded without moving anything. A stage **not in the table at all is an error**, not a no-op — a mistyped name would otherwise promote nothing, silently, on every run from then on. +The table grows **one row per stage**, each landing with the change that makes it true. A stage with no row yet returns null and is recorded without moving anything. A stage **not in the table at all is an error**, not a no-op -- a mistyped name would otherwise promote nothing, silently, on every run from then on. `subpath` is relative to both roots: `Utilized/` and `Output/` differ only in their root, which is what makes promotion a move between two spellings of one path. ## What enters Utilized, and what does not -**Something enters `Utilized` exactly when it will be read again.** Artifacts with no consumer at all — unpaired reads, trim reports, the FastQC htmls, both step 5 reports — are never "an output that becomes an input", so they are written straight to `storageDir`. +**Something enters `Utilized` exactly when it will be read again.** Artifacts with no consumer at all -- unpaired reads, trim reports, the FastQC htmls, both step 5 reports -- are never "an output that becomes an input", so they are written straight to `storageDir`. Row-by-row reasoning: - **Trimmed reads** are step 3's only input and nothing after step 3 reads them, so a sample's alignment succeeding releases that sample's reads. The `*_val_*` reads in the same directory are deliberately absent: `ClipReads` deletes them outright, so they never become a result. -- **The FastQC zips** are working data by the same rule — `ClipReads` unzips them to work out its clipping bounds — but their gate is `ClipReads` finishing rather than alignment, which is why they are a row of their own. The htmls are not here: nothing reads those. +- **The FastQC zips** are working data by the same rule -- `ClipReads` unzips them to work out its clipping bounds -- but their gate is `ClipReads` finishing rather than alignment, which is why they are a row of their own. The htmls are not here: nothing reads those. - **Aligned BAMs** live in a flat directory rather than per-sample, so the key selects the *file* instead of the folder. That is why `subpath` and `patterns` are both resolved against the key. -- **Ready BAMs** are the first artifact with two consumers — step 5 for reports, step 6 for calling — so the gate is both, assembled at the call site. The index travels with its BAM: nothing reads the index by name, both step-5 processes just expect it beside the file, so promoting one without the other would leave a BAM that looks complete and is not. +- **Ready BAMs** are the first artifact with two consumers -- step 5 for reports, step 6 for calling -- so the gate is both, assembled at the call site. The index travels with its BAM: nothing reads the index by name, both step-5 processes just expect it beside the file, so promoting one without the other would leave a BAM that looks complete and is not. - **The called VCF** also has two consumers, the second conditional. This is the one place where "the step that consumes it" is not a single step. ## One task, one destination -**One promotion task per (producing variant, stage, key), not one per member.** The five attachment points hang off channels carrying variants, so a shared artifact would otherwise get N promotion tasks over one set of files — and that is where `atomic_mv.sh`'s missing lock actually bites: two concurrent callers stage through the same `${DEST}.part`, and one's EXIT trap deletes the other's staged copy between its two `mv` calls. +**One promotion task per (producing variant, stage, key), not one per member.** The five attachment points hang off channels carrying variants, so a shared artifact would otherwise get N promotion tasks over one set of files -- and that is where `atomic_mv.sh`'s missing lock actually bites: two concurrent callers stage through the same `${DEST}.part`, and one's EXIT trap deletes the other's staged copy between its two `mv` calls. **That mechanism was replaced seven hours after this was written**, in `f4f1104`: each caller stages through its own `mktemp -d` and `${DEST}.part` no longer exists anywhere. The rule above is unchanged; what changed is how much it is worth. N promotion tasks over one set of files is duplicated work now, not a race that can lose the artifact. `concurrency.md` carries the measurements on both sides of that fix. -**One destination**, which is a property of the results layout rather than of this file. A variant writes to the single directory named for whoever owns it — `All_Runs`, `Shared_` or one run — and every member reads it there, so there is nothing to copy into a second place. This briefly carried a *list* of destinations; the one-results-tree layout made it dead and it went. +**One destination**, which is a property of the results layout rather than of this file. A variant writes to the single directory named for whoever owns it -- `All_Runs`, `Shared_` or one run -- and every member reads it there, so there is nothing to copy into a second place. This briefly carried a *list* of destinations; the one-results-tree layout made it dead and it went. ## Details that prevent a wrong-sample promotion - **The key is derived from the completion signal, not from pairing two channels.** Two channels would be matched by arrival order, and a promotion aimed at the wrong sample would delete a file another task still needs. -- **`stage` stays a separate input** because it is a literal at the call site and therefore a value channel that broadcasts. The run travels *with* the key: they are one fact — who released what — and separating them would match them by arrival order too. -- **The source must be GONE, not merely copied.** `find_artifact.sh` reports the first root that has an artifact and the working volume is searched second, so a copy left behind is harmless now and wrong the moment the promoted one is edited, replaced or reset — it would go on satisfying every later skip check. +- **`stage` stays a separate input** because it is a literal at the call site and therefore a value channel that broadcasts. The run travels *with* the key: they are one fact -- who released what -- and separating them would match them by arrival order too. +- **The source must be GONE, not merely copied.** `find_artifact.sh` reports the first root that has an artifact and the working volume is searched second, so a copy left behind is harmless now and wrong the moment the promoted one is edited, replaced or reset -- it would go on satisfying every later skip check. - **"Already promoted" is the ordinary case; "in neither root" is not.** The gate reported success, so the artifact has to exist somewhere, and the likeliest cause is a wrong subpath in the table. - **Patterns are quoted** so the loops iterate over the patterns themselves rather than whatever they happen to match in the task directory. @@ -64,18 +64,18 @@ Row-by-row reasoning: This note used to say they were still open: written by their own process to `run.dir.output.*`, which for a shared variant would be the **lead member's** storage and no one else's, so they needed the destination handling promoted artifacts get. -**The layout removed the problem instead.** `variants.nf` gives every variant its own `dir.outputs` — `${variant.storageDir}/Output${owner}`, where the owner is `All_Runs`, `Shared_` or one run — so a consumer-less output written to `run.dir.output.*` lands in the directory that variant owns, whoever its members are. There is no lead member's `Output/` any more, so there is nothing to distribute. Verified in `scripts/5_reports.nf`, whose three reports are exactly this class. +**The layout removed the problem instead.** `variants.nf` gives every variant its own `dir.outputs` -- `${variant.storageDir}/Output${owner}`, where the owner is `All_Runs`, `Shared_` or one run -- so a consumer-less output written to `run.dir.output.*` lands in the directory that variant owns, whoever its members are. There is no lead member's `Output/` any more, so there is nothing to distribute. Verified in `scripts/5_reports.nf`, whose three reports are exactly this class. ## How the wrapper finds a project's results directories `clean` and `reset` have to name every directory a run wrote, and under a run table those are `Output/`, `Output/All_Runs` and `Output/Shared_`. -**`nextflow config -flat` is the wrapper's only path oracle, and it reports the BASE configuration** — so it cannot name `Output/`, which the resolver builds per run. +**`nextflow config -flat` is the wrapper's only path oracle, and it reports the BASE configuration** -- so it cannot name `Output/`, which the resolver builds per run. -**So the wrapper asks the disk, and specifically the COPY of the multi-run table that step 0 keeps beside the results.** That copy is written by the pipeline, at a path the wrapper already knows, and it describes the runs that actually produced what is there. Reading the user's own table instead would be a third copy of the rule that decides where results go — the two that already exist are only kept in step by a test — and it would answer wrongly for a project whose table has since been edited or deleted. +**So the wrapper asks the disk, and specifically the COPY of the multi-run table that step 0 keeps beside the results.** That copy is written by the pipeline, at a path the wrapper already knows, and it describes the runs that actually produced what is there. Reading the user's own table instead would be a third copy of the rule that decides where results go -- the two that already exist are only kept in step by a test -- and it would answer wrongly for a project whose table has since been edited or deleted. **Shared directories are found by their `members.txt` rather than by the table**, because their names are assigned by the divergence analysis and are not in it. -**Both helpers end in an explicit `return 0`.** The script runs under `set -e`, a function's exit status is its last command's, and the last command is a test that is FALSE in the ordinary single-run case — so without it `ROOTS=$(run_storage_roots …)` aborted `reset` silently, before it printed anything at all. +**Both helpers end in an explicit `return 0`.** The script runs under `set -e`, a function's exit status is its last command's, and the last command is a test that is FALSE in the ordinary single-run case -- so without it `ROOTS=$(run_storage_roots ...)` aborted `reset` silently, before it printed anything at all. `Utilized_` sits beside `Utilized` under mainDir, so the glob prefix is ours and cannot pick up anything of the user's. diff --git a/.claude/development-notes/sequencer-suffix.md b/.claude/development-notes/sequencer-suffix.md new file mode 100644 index 0000000..a677c35 --- /dev/null +++ b/.claude/development-notes/sequencer-suffix.md @@ -0,0 +1,45 @@ +# The sequencer suffix, and why nothing strips it + +**Written 2026-09-29, against the tree at `6746280` plus uncommitted work.** Raised by Z from a collaborator's run: *"If sample name is A08781 but then they were submitted as A08781_S58 the metadata check cannot distinguish."* Shelved at the time with a manual section; picked up here and resolved as documentation, by Z's decision. + +## What the problem actually was + +`readPairChannel` uses `fromFilePairs`, so the sample identifier is **derived from the file name** -- whatever is left after `readPattern` is stripped. Illumina's `bcl2fastq` and BCL Convert append `_S`, the sample's row number in the sample sheet, so `A08781` is delivered as `A08781_S58_R1.fq.gz` and the identifier is `A08781_S58`. Nothing in the pipeline chose that and nothing removes it. + +**The join failing is not the defect. The defect was that the failure is reported as two unrelated problems**, in two directions checked asymmetrically: + +| direction | verdict | message | +|---|---|---| +| reads with no row | hard FAIL | `Sample 'A08781_S58' has reads but no row in metadata.csv` | +| row with no reads | NOTE only | `NOTE: metadata.csv has a row for 'A08781', which has no reads in Data/` | + +A user who wrote the short name gets both and nothing connects them. At sixty samples that is a hundred and twenty lines, none of which names the cause. Step 0 has both halves of the answer in hand and never puts them together -- which is exactly what "cannot distinguish" meant. + +**And there is a second half that is worse, because it does not fail.** `RG_Sample` defaults to `SampleID` (`parse_metadata.py`, `pool = row.get("RG_Sample") or row.get(SAMPLE_ID, "")`). Fix the join by writing `SampleID = A08781_S58`, leave `RG_Sample` blank, and the run **succeeds** with the sample sheet's row number in every VCF column, every frequency table header and every analysis result. Worse, a sample split across lanes stays two separate pools rather than one, so each carries half its depth and the false-positive filter sees twice as many pools as exist. + +## Why stripping the suffix is the wrong fix + +The obvious fix -- strip a trailing `_S` from `SampleID` -- is actively wrong, and the reason is the mechanism that makes lanes work. A sample split across two lanes arrives as `A08781_S58_L001` and `A08781_S58_L002`: two rows, one `RG_Sample`, merged into one pool. Stripping would give both rows the same identifier, and a repeated `SampleID` is refused, so one of the pairs would be lost. + +The suffix is load-bearing in `SampleID` and meaningless in `RG_Sample`. That asymmetry is the whole design and it is correct. + +## What was decided, and what was not + +**Z, 2026-09-29: documentation, not code.** *"First we tell explicitly to the user to use the full name for SampleID... RG_Sample will need to be entered too explicitly. And we will ask the user to put what they want the pipeline to output as sample names in the outputs."* Then, on the defaulting: *"If we haven't changed automatic filling of RG_Sample, it's fine. We can add a note... It's good documentation practice."* + +So the defaulting is unchanged, and is now stated plainly rather than presented as a fine choice: + +- `SampleID` is defined in both places the manual defines it as **the whole name before the read tag, as delivered**, naming `_S` and `_L00`. +- The read-group section carries a two-row table separating the questions the two columns answer -- `SampleID`: which files is this row, from your sequencer; `RG_Sample`: what name do I want in my results, from you. +- A new subsection gives **both failure modes with the real messages quoted verbatim**, checked against `0_verify_environment.nf`: the two unconnected lines when `SampleID` is short, and the `METADATA CHECK: ... is one column, pooling ...` line that catches a blank `RG_Sample` before any compute is spent. +- `metadata.csv.template`'s eight example rows became `Sample1T1_S1_L001 -> Sample1T1`. Same eight rows and four pools, verified through the parser -- but the example now demonstrates the two columns doing different jobs, where before they looked redundant. + +**NOT built, and not asked for:** step 0 connecting the two messages when an unmatched read id equals an unmatched row id plus a recognizable suffix, and a warning when `RG_Sample` was defaulted. Both are still the right shape if anyone wants them. + +There is a practical obstacle to the second that is worth recording: **the parser fills `RG_Sample` before anything downstream sees it**, and emits a plain list of rows, so nothing after the parse can tell a written value from a defaulted one. A runtime warning has to come from `parse_metadata.py` itself, which has a notes channel -- but those notes reach `PoolSeqFlow check project` and NOT a run's step 0, because `resolve_parameters.nf` keeps a parser's stderr only when it fails. + +## One thing measured, because it decides whether the advice is safe + +**Filling `RG_Sample` with the value it was already defaulting to does not trip the metadata change guard.** The parser fills a blank cell, so the guard's projection -- `metadataGuardLines()`, which records each RG column of the resolved row -- is byte-identical whether the column is absent or written out with the same value. Verified by parsing both shapes. + +That is what makes "always fill it in" safe to tell every existing project: no guard trip, no results invalidated. A project that *should* have been pooling and was not will trip when it fixes the column, which is correct -- those results really were produced under the wrong pooling. The manual says so, so nobody hesitates to add the column. diff --git a/.claude/development-notes/shell-and-nextflow-gotchas.md b/.claude/development-notes/shell-and-nextflow-gotchas.md index c5dc180..5be5355 100644 --- a/.claude/development-notes/shell-and-nextflow-gotchas.md +++ b/.claude/development-notes/shell-and-nextflow-gotchas.md @@ -1,6 +1,6 @@ # Shell and Nextflow traps -Every entry here produced a **confident wrong answer** at least once before it was caught — a test that passed against broken code, a survey that missed a file, a run that failed three lines from its cause. None of them is exotic; that is the point. Check this list before believing a result. +Every entry here produced a **confident wrong answer** at least once before it was caught -- a test that passed against broken code, a survey that missed a file, a run that failed three lines from its cause. None of them is exotic; that is the point. Check this list before believing a result. The companion notes are `dag-wiring.md` for channel shape and `parameter-resolution.md` for how a run's parameters are built. The largest single source of silent wrong values in this project is config interpolation and params mutation, which has its own note. @@ -13,96 +13,96 @@ The companion notes are `dag-wiring.md` for channel shape and `parameter-resolut **The agent's Bash tool runs zsh; the pipeline runs bash.** Three differences have bitten: - **`${var%$pat}` does not glob-expand a pattern held in a variable in zsh.** A sample-ID-stripping test reported "nothing is stripped" while the code was correct. Run anything the pipeline will run with `bash -c`. -- **zsh does not word-split unquoted parameters.** `$2` holding `-c file` arrives as one argument and the tool reports `Unknown option`. The same rule silently breaks a loop: `FILES="a b c"; for f in $FILES` iterates **once**, with the whole string as the filename. It cost a bulk rename on 2026-08-30 — `cp` and `sed` both errored on one absurd path, only the command *after* the loop took effect, and the tree was left half renamed. +- **zsh does not word-split unquoted parameters.** `$2` holding `-c file` arrives as one argument and the tool reports `Unknown option`. The same rule silently breaks a loop: `FILES="a b c"; for f in $FILES` iterates **once**, with the whole string as the filename. It cost a bulk rename on 2026-08-30 -- `cp` and `sed` both errored on one absurd path, only the command *after* the loop took effect, and the tree was left half renamed. - **`grep -rn --include='*.nf'` needs the glob quoted**, or zsh expands it against the cwd and the whole command dies with `no matches found`. --- ## Bash under `set -e` and `set -u` -**An unmatched glob is passed to the loop body as the pattern itself.** `for f in *_unpaired_*` with nothing matching iterates once with `f` set to the literal `*_unpaired_*`. Handing that to a helper that refuses a missing source kills the task under `set -eo pipefail`. Reproduced 2026-09-02: `atomic_mv: source not found: *_unpaired_*`, and the loop never reached its end. Seven loops in `scripts/2_trim_reads.nf` were exposed this way, the live one being Trim Galore's unpaired reads — written only when a mate is discarded, so a run where every pair survived trimming died on the pattern. Guard with `if [ -e "$f" ]; then … fi`. `[ -e "$f" ] && cmd` is also safe (the `&&` exemption holds, tested), but the `if` needs no reader to know that. +**An unmatched glob is passed to the loop body as the pattern itself.** `for f in *_unpaired_*` with nothing matching iterates once with `f` set to the literal `*_unpaired_*`. Handing that to a helper that refuses a missing source kills the task under `set -eo pipefail`. Reproduced 2026-09-02: `atomic_mv: source not found: *_unpaired_*`, and the loop never reached its end. Seven loops in `scripts/2_trim_reads.nf` were exposed this way, the live one being Trim Galore's unpaired reads -- written only when a mate is discarded, so a run where every pair survived trimming died on the pattern. Guard with `if [ -e "$f" ]; then ... fi`. `[ -e "$f" ] && cmd` is also safe (the `&&` exemption holds, tested), but the `if` needs no reader to know that. -**`set -e` is suppressed inside a function invoked as an `&&` operand.** `( f && touch x ) || exit 1` means nothing inside `f` aborts on failure — which is why copy loops need explicit `|| return 1` rather than relying on `set -e`. +**`set -e` is suppressed inside a function invoked as an `&&` operand.** `( f && touch x ) || exit 1` means nothing inside `f` aborts on failure -- which is why copy loops need explicit `|| return 1` rather than relying on `set -e`. -**The mirror image is worse because it is silent: under `set -e`, `X=$(f)` aborts the script when `f`'s LAST command returns non-zero.** A function ending in a `for` loop whose body is `[ -f … ] && printf …` returns 1 whenever nothing matched — the ordinary case — so `./PoolSeqFlow reset` exited 1 having printed nothing at all, before its own first `echo`. **Empty output plus status 1 is the signature.** End such helpers with an explicit `return 0`, and write loop bodies as `if …; then …; fi`: a loop's exit status is its last body command's, so an empty list leaves the loop returning 1 too — which in a deletion loop would abort partway through. Found 2026-08-26. +**The mirror image is worse because it is silent: under `set -e`, `X=$(f)` aborts the script when `f`'s LAST command returns non-zero.** A function ending in a `for` loop whose body is `[ -f ... ] && printf ...` returns 1 whenever nothing matched -- the ordinary case -- so `./PoolSeqFlow reset` exited 1 having printed nothing at all, before its own first `echo`. **Empty output plus status 1 is the signature.** End such helpers with an explicit `return 0`, and write loop bodies as `if ...; then ...; fi`: a loop's exit status is its last body command's, so an empty list leaves the loop returning 1 too -- which in a deletion loop would abort partway through. Found 2026-08-26. **`A || B && continue` is not "skip when either holds".** Bash parses it `(A || B) && continue`, so when neither holds the compound's status is non-zero and `set -e` ends the script rather than falling through. Write `if A || B; then continue; fi`. -**A function called inside `$(…)` runs in a subshell, so anything it assigns is lost.** `out=$(run_launcher …)` meant the status variable never reached the caller. Assign in the caller (`OUT=$(cd x && cmd)` then `STATUS=$?`), and feed stdin with a herestring (`f <<< "y"`) — a pipeline is also a subshell. The same rule is why a loud `exit 1` inside a helper that is always called in a command substitution would silently do nothing. +**A function called inside `$(...)` runs in a subshell, so anything it assigns is lost.** `out=$(run_launcher ...)` meant the status variable never reached the caller. Assign in the caller (`OUT=$(cd x && cmd)` then `STATUS=$?`), and feed stdin with a herestring (`f <<< "y"`) -- a pipeline is also a subshell. The same rule is why a loud `exit 1` inside a helper that is always called in a command substitution would silently do nothing. **`local a="$1" b=".../$a"` dies under `set -u`.** Referencing a name in the same `local` that declares it is unbound. Split the declaration. In a test helper this failed silently mid-case and presented as three unrelated assertion failures. -**`grep -c` exits 1 when it counts none**, and a Nextflow process script runs under `bash -ue`, so `N=$(grep -c PATTERN file)` kills the task when the answer is zero. Use `|| true`. The older code sidesteps this with `grep … | wc -l`, whose status is `wc`'s. +**`grep -c` exits 1 when it counts none**, and a Nextflow process script runs under `bash -ue`, so `N=$(grep -c PATTERN file)` kills the task when the answer is zero. Use `|| true`. The older code sidesteps this with `grep ... | wc -l`, whose status is `wc`'s. -**An optional positional parameter needs `${3:-}`, not `$3`.** Widening a test helper from two arguments to three, 2026-09-07: every existing two-argument call died on `analysis.sh: line 266: $3: unbound variable`, and the suite stopped mid-file rather than failing a case — so the log showed a suite that simply ended. `local design="${3:-}"` is the whole fix. The same shape bit an argument that was *always* passed but sometimes empty, which `set -u` does not mind and `[ -n "$x" ] && printf …` does: a failing test is the last command of an AND-list, so under `set -e` an empty body ends the case. Write it as `if …; then …; fi`. +**An optional positional parameter needs `${3:-}`, not `$3`.** Widening a test helper from two arguments to three, 2026-09-07: every existing two-argument call died on `analysis.sh: line 266: $3: unbound variable`, and the suite stopped mid-file rather than failing a case -- so the log showed a suite that simply ended. `local design="${3:-}"` is the whole fix. The same shape bit an argument that was *always* passed but sometimes empty, which `set -u` does not mind and `[ -n "$x" ] && printf ...` does: a failing test is the last command of an AND-list, so under `set -e` an empty body ends the case. Write it as `if ...; then ...; fi`. --- ## awk and sed -**An apostrophe inside an embedded awk program breaks the shell parse.** `bin/filterFalsePositives.sh` and `bin/config_migrate.sh` hold their awk program in a single-quoted shell string, so any `'` inside it — **including one in an awk `#` comment** — closes the string and the file stops parsing. Writing "the VCF's own header" into a comment broke it during the 2026-08-30 comment pass; so did "the block's keys". The code uses the `'"'"'` dance where an apostrophe is unavoidable, and `config_migrate.sh` says "a value is yours to keep" rather than "the user's" for exactly this reason. **`bash -n` catches it instantly — run it over every shell file after any edit, however cosmetic.** +**An apostrophe inside an embedded awk program breaks the shell parse.** `bin/filterFalsePositives.sh` and `bin/config_migrate.sh` hold their awk program in a single-quoted shell string, so any `'` inside it -- **including one in an awk `#` comment** -- closes the string and the file stops parsing. Writing "the VCF's own header" into a comment broke it during the 2026-08-30 comment pass; so did "the block's keys". The code uses the `'"'"'` dance where an apostrophe is unavoidable, and `config_migrate.sh` says "a value is yours to keep" rather than "the user's" for exactly this reason. **`bash -n` catches it instantly -- run it over every shell file after any edit, however cosmetic.** -**`awk -v x=…` applies escape-sequence processing.** A commit subject containing `\t` becomes a tab; `\n` injects a newline. Pass arbitrary text through `ENVIRON` instead (`X="$X" awk '… ENVIRON["X"] …'`). The same applies to *patterns*: `-v pat='^## \['` dies with `fatal: invalid regexp: unbalanced [`, which is why the two insertion blocks in `dev/scripts/bump-version.sh` were deliberately not factored into a shared helper. `-v` is fine for values you control. +**`awk -v x=...` applies escape-sequence processing.** A commit subject containing `\t` becomes a tab; `\n` injects a newline. Pass arbitrary text through `ENVIRON` instead (`X="$X" awk '... ENVIRON["X"] ...'`). The same applies to *patterns*: `-v pat='^## \['` dies with `fatal: invalid regexp: unbalanced [`, which is why the two insertion blocks in `dev/scripts/bump-version.sh` were deliberately not factored into a shared helper. `-v` is fine for values you control. -**`\t` in a sed PATTERN is a GNU extension, and BSD sed reads it as a literal `t` without complaining.** Caught before shipping in E3b's pool-size guard, where a condition decided whether a metadata edit touched only the pool sizes. On macOS it would never hold, and a size edit would have been reported as a read-group change — telling that user, and only that user, to delete every BAM. Replaced with awk. `sed -i` without a suffix is also GNU-only but fails **loudly** there, which is why the codebase gets away with it; `\t` fails silently. **Prefer awk for anything tab-delimited.** +**`\t` in a sed PATTERN is a GNU extension, and BSD sed reads it as a literal `t` without complaining.** Caught before shipping in E3b's pool-size guard, where a condition decided whether a metadata edit touched only the pool sizes. On macOS it would never hold, and a size edit would have been reported as a read-group change -- telling that user, and only that user, to delete every BAM. Replaced with awk. `sed -i` without a suffix is also GNU-only but fails **loudly** there, which is why the codebase gets away with it; `\t` fails silently. **Prefer awk for anything tab-delimited.** -**A line-level `grep -v` filter hides a line that holds BOTH patterns.** Surveying the `samtools`/`bcftools` scope rename with `grep … | grep -v software` reported 12 references in 7 files and missed `scripts/6_variant_call.nf` entirely, because one line carries `run.software.bcftools` and `run.bcftools.mpileupOptions` together. The bad count reached a plan Z used to weigh two designs. **Verify a filtered survey by searching for what should REMAIN afterwards**, not by trusting the filtered list. +**A line-level `grep -v` filter hides a line that holds BOTH patterns.** Surveying the `samtools`/`bcftools` scope rename with `grep ... | grep -v software` reported 12 references in 7 files and missed `scripts/6_variant_call.nf` entirely, because one line carries `run.software.bcftools` and `run.bcftools.mpileupOptions` together. The bad count reached a plan Z used to weigh two designs. **Verify a filtered survey by searching for what should REMAIN afterwards**, not by trusting the filtered list. --- ## Git and time -**`%ct` is whole seconds.** Two commits landing in the same second compare equal, so a check of the form "were the sources committed after the version" silently passes on a real drift. Found 2026-09-02 while re-verifying `dev/scripts/check-analysis-versions.sh` — a path that should have reported BEHIND reported PASS. That check compares **days** now, taken from the version string itself, and no timestamps are involved. +**`%ct` is whole seconds.** Two commits landing in the same second compare equal, so a check of the form "were the sources committed after the version" silently passes on a real drift. Found 2026-09-02 while re-verifying `dev/scripts/check-analysis-versions.sh` -- a path that should have reported BEHIND reported PASS. That check compares **days** now, taken from the version string itself, and no timestamps are involved. **`git check-attr export-ignore` does not report a file inside an export-ignored directory.** A pattern like `dev/` removes the directory from `git archive` but `check-attr` on a file within it answers `unspecified`. Do not use it as an oracle for "will this ship". -**A checker must not read its exclusion list from the file that causes the bug.** `dev/scripts/verify-archive.sh` was briefly rewritten to read its `export-ignore` rules out of `.gitattributes` at the same ref — which made it blind to exactly the accident it exists for, since an export-ignore added by mistake takes the file out of the *check* as well as out of the archive. Measured in a throwaway clone: `bin/find_artifact.sh export-ignore` passed the gate green. The list is hand-kept in the script now, so an export-ignore fails the gate until it is named there too. **Write the negative test; a checker that only ever says "fine" is worse than none.** +**A checker must not read its exclusion list from the file that causes the bug.** `dev/scripts/verify-archive.sh` was briefly rewritten to read its `export-ignore` rules out of `.gitattributes` at the same ref -- which made it blind to exactly the accident it exists for, since an export-ignore added by mistake takes the file out of the *check* as well as out of the archive. Measured in a throwaway clone: `bin/find_artifact.sh export-ignore` passed the gate green. The list is hand-kept in the script now, so an export-ignore fails the gate until it is named there too. **Write the negative test; a checker that only ever says "fine" is worse than none.** --- ## The Nextflow strict parser (26.04) -**Top-level statements are a hard error** — "Statements cannot be mixed with script declarations". A script-level constant (`def DEFAULTS = [runs: 'all']`) is therefore impossible; a hardcoded default has to be returned by a function. Multiple declarations on one line (`def a = 1, b = 2`) do not parse either. +**Top-level statements are a hard error** -- "Statements cannot be mixed with script declarations". A script-level constant (`def DEFAULTS = [runs: 'all']`) is therefore impossible; a hardcoded default has to be returned by a function. Multiple declarations on one line (`def a = 1, b = 2`) do not parse either. -**`-entry` does not exist.** `nextflow run analysis.nf -entry Complete` fails with *"The `-entry` option is not supported with the strict parser"*, and **`nextflow lint` does not catch it** — a script with an anonymous workflow plus named ones lints clean and fails only when run. Cost half a suite run and seven red cases before the error was read rather than guessed at. The answer is the pattern the codebase already uses: **one entry script per command** — `poolseqflow.nf`, `dryrun.nf`, `analysis.nf`, `analysis/complete.nf`, and every module's `main.nf`. +**`-entry` does not exist.** `nextflow run analysis.nf -entry Complete` fails with *"The `-entry` option is not supported with the strict parser"*, and **`nextflow lint` does not catch it** -- a script with an anonymous workflow plus named ones lints clean and fails only when run. Cost half a suite run and seven red cases before the error was read rather than guessed at. The answer is the pattern the codebase already uses: **one entry script per command** -- `poolseqflow.nf`, `dryrun.nf`, `analysis.nf`, `analysis/complete.nf`, and every module's `main.nf`. -**`switch` is rejected.** `switch (x) { case 'a': … }` fails with `Unexpected input: '\n'` at the `case` line, and the knock-on is misleading: the file fails to parse, so every workflow it exports is reported "is not defined" at each call site in another file. Use a map literal plus `containsKey` — that is why `promotionRow()` in `scripts/9_completion.nf` is shaped the way it is. +**`switch` is rejected.** `switch (x) { case 'a': ... }` fails with `Unexpected input: '\n'` at the `case` line, and the knock-on is misleading: the file fails to parse, so every workflow it exports is reported "is not defined" at each call site in another file. Use a map literal plus `containsKey` -- that is why `promotionRow()` in `scripts/9_completion.nf` is shaped the way it is. -**Both `for` forms are rejected.** `for (int k = 2; k <= n; k++)` fails with `Unexpected input: 'k'`; `for (part in list)` fails with `` `part` is not defined `` at the first *use* rather than at the loop. Use a range (`(2..n).each { k -> … }`) or `.each`. Reassigning a captured local **is** allowed — `dotted.tokenize('.').each { part -> cur = cur[part] }` is committed and works — it is only the loop syntax that is refused. +**Both `for` forms are rejected.** `for (int k = 2; k <= n; k++)` fails with `Unexpected input: 'k'`; `for (part in list)` fails with `` `part` is not defined `` at the first *use* rather than at the loop. Use a range (`(2..n).each { k -> ... }`) or `.each`. Reassigning a captured local **is** allowed -- `dotted.tokenize('.').each { part -> cur = cur[part] }` is committed and works -- it is only the loop syntax that is refused. -**A local closure called by name is rejected**: `def refuse = { … }; refuse('x')` gives `refuse is not defined`. Use a plain `def` function. +**A local closure called by name is rejected**: `def refuse = { ... }; refuse('x')` gives `refuse is not defined`. Use a plain `def` function. -**An interpolated slashy regex is rejected** — `return ~/${pattern}/` gives `Unexpected input: '/'`, and the knock-on is the misleading one again: the file stops parsing, so every workflow it exports is reported "is not defined" at each call site in three other files, and `analysis.nf` looks like the broken one. Use `java.util.regex.Pattern.compile("${pattern}".toString())` and match with `matcher.matcher(value).matches()` rather than `==~`. Found 2026-09-05 building `missingValueMatcher()` in `analysis/lib/nf/design.nf`; a plain `~/literal/` with no interpolation is fine. +**An interpolated slashy regex is rejected** -- `return ~/${pattern}/` gives `Unexpected input: '/'`, and the knock-on is the misleading one again: the file stops parsing, so every workflow it exports is reported "is not defined" at each call site in three other files, and `analysis.nf` looks like the broken one. Use `java.util.regex.Pattern.compile("${pattern}".toString())` and match with `matcher.matcher(value).matches()` rather than `==~`. Found 2026-09-05 building `missingValueMatcher()` in `analysis/lib/nf/design.nf`; a plain `~/literal/` with no interpolation is fine. -**An implicit closure parameter is deprecated** — `collect { "'${it}'" }` warns; write `collect { p -> … }`. `00_static` requires zero warnings. +**An implicit closure parameter is deprecated** -- `collect { "'${it}'" }` warns; write `collect { p -> ... }`. `00_static` requires zero warnings. -**A named `emit` is rejected when a workflow has only one** — "Emit name should be omitted when there is only one emit". +**A named `emit` is rejected when a workflow has only one** -- "Emit name should be omitted when there is only one emit". **`take:` / `main:` / `emit:` must be on their own lines.** A one-liner gives `Invalid workflow definition -- check for missing or out-of-order section labels`. -**`nextflow.config` rejects `if` statements** — "If statements cannot be mixed with config statements". There is no way to validate a parameter in the config file; validation belongs in `bin/config_migrate.sh`. Use a ternary where a conditional is unavoidable. +**`nextflow.config` rejects `if` statements** -- "If statements cannot be mixed with config statements". There is no way to validate a parameter in the config file; validation belongs in `bin/config_migrate.sh`. Use a ternary where a conditional is unavoidable. -**`workflow.onComplete` in `nextflow.config` is deprecated.** The working form is a top-level `def` plus one registration line *inside* the entry workflow — which is why `assembleCombinedLog()` in `poolseqflow.nf` looks the way it does. +**`workflow.onComplete` in `nextflow.config` is deprecated.** The working form is a top-level `def` plus one registration line *inside* the entry workflow -- which is why `assembleCombinedLog()` in `poolseqflow.nf` looks the way it does. **`.execute()` and `groovy.json.JsonSlurper` DO work**, at DAG-build time. That is what lets `runDefinitions()` shell out to a tested Python parser instead of reimplementing CSV quoting in Groovy. --- -**An EMPTY config block reaches `params` as nothing at all.** Measured 2026-09-07: `phenotypes { pt_wingspan { }; pt_other { levels = ['a','b'] } }` arrives as `[pt_other:[levels:[a, b]]]` — `pt_wingspan` is *absent*, not present and empty. So a checker that iterates a declaration scope cannot refuse an empty declaration: it never sees one, and it cannot tell that block from a column nobody wrote about. This bites any open scope declared per column — `analysis.metadata.phenotypes` and `analysis.metadata.covariates` both. **The defense is to report what is UNdeclared**, which covers the empty block for free; a test case asserting that an empty block refuses will fail, and correctly. +**An EMPTY config block reaches `params` as nothing at all.** Measured 2026-09-07: `phenotypes { pt_wingspan { }; pt_other { levels = ['a','b'] } }` arrives as `[pt_other:[levels:[a, b]]]` -- `pt_wingspan` is *absent*, not present and empty. So a checker that iterates a declaration scope cannot refuse an empty declaration: it never sees one, and it cannot tell that block from a column nobody wrote about. This bites any open scope declared per column -- `analysis.metadata.phenotypes` and `analysis.metadata.covariates` both. **The defense is to report what is UNdeclared**, which covers the empty block for free; a test case asserting that an empty block refuses will fail, and correctly. --- ## Interpolation, and one character that shifts everything -**A bare `$?` in a Nextflow script string SHIFTS every interpolation after it by one slot.** It does not error. A `` `[ $? -eq 0 ]` `` written inside a **shell comment** in a script block made `${workflow.runName}` print the session id, `${workflow.sessionId}` print `1`, `${task.attempt}` print a directory path, and `${dir_log}` come out as the run name — so the task died on `No such file or directory` for a log path, three lines from the cause and looking nothing like it. Being inside a `#` comment does not help: the substitution happens in Groovy, long before bash sees the line. **Write `\$?` even in comments.** Found 2026-08-25. +**A bare `$?` in a Nextflow script string SHIFTS every interpolation after it by one slot.** It does not error. A `` `[ $? -eq 0 ]` `` written inside a **shell comment** in a script block made `${workflow.runName}` print the session id, `${workflow.sessionId}` print `1`, `${task.attempt}` print a directory path, and `${dir_log}` come out as the run name -- so the task died on `No such file or directory` for a log path, three lines from the cause and looking nothing like it. Being inside a `#` comment does not help: the substitution happens in Groovy, long before bash sees the line. **Write `\$?` even in comments.** Found 2026-08-25. -**A heredoc terminator cannot be indented.** A block interpolated into an already-indented script takes its indentation with it, so `<<'X'` … `X` never terminates and the whole task script breaks. Render such blocks with `printf` and escaped quotes instead. Found 2026-09-01 building the citations shell. +**A heredoc terminator cannot be indented.** A block interpolated into an already-indented script takes its indentation with it, so `<<'X'` ... `X` never terminates and the whole task script breaks. Render such blocks with `printf` and escaped quotes instead. Found 2026-09-01 building the citations shell. -**`for x in ; do … done` is a bash SYNTAX error, not an empty loop.** Groovy-interpolating a list that is normally empty into a shell `for` breaks every task of that process. Emit the calls themselves — `link_into "a"` per element, nothing when the list is empty. +**`for x in ; do ... done` is a bash SYNTAX error, not an empty loop.** Groovy-interpolating a list that is normally empty into a shell `for` breaks every task of that process. Emit the calls themselves -- `link_into "a"` per element, nothing when the list is empty. -**`join('\\t')` in Groovy source is what samtools wants.** A single-quoted `'\\t'` in a `.nf` file is backslash-then-t, two characters, which is the literal `\t` an `@RG` line expects — not a tab. Getting this "right" with a real tab breaks it. +**`join('\\t')` in Groovy source is what samtools wants.** A single-quoted `'\\t'` in a `.nf` file is backslash-then-t, two characters, which is the literal `\t` an `@RG` line expects -- not a tab. Getting this "right" with a real tab breaks it. -**A backslash-u-0000 escape typed into a source file may be written as a real NUL byte**, which turns the file `binary` to git (`Bin 12716 -> 17407 bytes` in `git diff --stat` instead of a line count) and makes the change unreviewable. Groovy accepts it silently and the pipeline runs. **Check `git diff --stat` for `Bin` before handing a change over** — or `file` it, which says `data` instead of `Unicode text`. +**A backslash-u-0000 escape typed into a source file may be written as a real NUL byte**, which turns the file `binary` to git (`Bin 12716 -> 17407 bytes` in `git diff --stat` instead of a line count) and makes the change unreviewable. Groovy accepts it silently and the pipeline runs. **Check `git diff --stat` for `Bin` before handing a change over** -- or `file` it, which says `data` instead of `Unicode text`. That escape is written out in words above for the same reason: the first draft of this very entry put the escape in a code span, and it was written into this file as a real NUL. `file` caught it. **The trap survives being documented; spell it, do not quote it.** @@ -110,13 +110,13 @@ That escape is written out in words above for the same reason: the first draft o ## Channels -**`combine` FLATTENS.** `a.collect().combine(b.collect())` emits ONE list holding everything from both, not a tuple of two lists. A closure declaring two parameters looks right and fails at runtime with ``Invalid method invocation `call` with arguments: […] (java.util.LinkedList)``. +**`combine` FLATTENS.** `a.collect().combine(b.collect())` emits ONE list holding everything from both, not a tuple of two lists. A closure declaring two parameters looks right and fails at runtime with ``Invalid method invocation `call` with arguments: [...] (java.util.LinkedList)``. -**The rule behind it: `combine` treats a channel ITEM that is a List as a tuple and spreads it.** `queue.combine(valueChannelHoldingAListOfN)` gives an (N+1)-element tuple, not a pair — and every downstream closure is then called with the wrong arity, far from the cause. A list *inside* a tuple slot is safe (`[key, [f1..f5]]` → `[key, x, [f1..f5]]`, verified); it is a bare list **as** the item that spreads. `count()` is a scalar and cannot spread, which is the fix when the channel is only an ordering gate. +**The rule behind it: `combine` treats a channel ITEM that is a List as a tuple and spreads it.** `queue.combine(valueChannelHoldingAListOfN)` gives an (N+1)-element tuple, not a pair -- and every downstream closure is then called with the wrong arity, far from the cause. A list *inside* a tuple slot is safe (`[key, [f1..f5]]` -> `[key, x, [f1..f5]]`, verified); it is a bare list **as** the item that spreads. `count()` is a scalar and cannot spread, which is the fix when the channel is only an ordering gate. -**`combine(by: 0)` is the explicit form of what an implicit value channel did for free** — the cartesian product *within* a key, which is exactly "broadcast this per-run singleton across that run's samples". `join(by: 0)` is wrong there: it matches one-to-one, so every sample after the first silently gets nothing. +**`combine(by: 0)` is the explicit form of what an implicit value channel did for free** -- the cartesian product *within* a key, which is exactly "broadcast this per-run singleton across that run's samples". `join(by: 0)` is wrong there: it matches one-to-one, so every sample after the first silently gets nothing. -**Verified safe by probe for run-keyed channels:** a Map as element 0 works as a key for `combine(by: 0)`, `join(by: 0)`, `join(by: [0,1])` and `groupTuple(by: 0)`; a process can take `tuple val(someMap), …` and read it in `tag { }` / `cpus { }` closures and in `output: path("${someMap.field}")`. Re-probe rather than assume if the Nextflow version moves. +**Verified safe by probe for run-keyed channels:** a Map as element 0 works as a key for `combine(by: 0)`, `join(by: 0)`, `join(by: [0,1])` and `groupTuple(by: 0)`; a process can take `tuple val(someMap), ...` and read it in `tag { }` / `cpus { }` closures and in `output: path("${someMap.field}")`. Re-probe rather than assume if the Nextflow version moves. **Almost every value channel in this pipeline is IMPLICIT.** There are three explicit `channel.value` calls; everything else comes from Nextflow's rule that a process emits a value channel when all its inputs were value channels. So the cardinality risk is worse than "do not put an operator in a value channel": **nothing in the source marks which channels those are.** `test_each_step_runs_once_per_sample` is the only statement of the property in the repository, which makes those task-count assertions load-bearing. @@ -130,15 +130,15 @@ That escape is written out in words above for the same reason: the first draft o **`trace`, `report`, `timeline` and `dag` CANNOT be set from a `.nf`.** Assigned on the session config inside `workflow{}` they write zero files; the identical settings in a config file write both. The observers are constructed during session init, strictly before the entry script's body runs. These four are what stop an analysis run overwriting `Output/Reports/PoolSeqFlow_pipeline_*`, so the frame config cannot be hardcoded away. -**Config-to-config MERGES a nested scope.** Frame declares `analysis { runs; folderName; installDir }`, project sets only `runs` → all three keys survive. The replace-wholesale gotcha is about a **script**-declared map being replaced by a config leaf, which is a different case — do not generalise it to config files. Where a default lives in an accessor instead, use `containsKey`, never `?:`: measured, `runs = []` must stay `[]` rather than falling back to `'all'`. +**Config-to-config MERGES a nested scope.** Frame declares `analysis { runs; folderName; installDir }`, project sets only `runs` -> all three keys survive. The replace-wholesale gotcha is about a **script**-declared map being replaced by a config leaf, which is a different case -- do not generalise it to config files. Where a default lives in an accessor instead, use `containsKey`, never `?:`: measured, `runs = []` must stay `[]` rather than falling back to `'all'`. -**A later `-c` file wins over an earlier one, including whole scopes.** Measured with a failing task: frame sets `errorStrategy='finish'` → exit 1; a project `analysis.config` passed as a second `-c` sets `'ignore'` → exit 0. This is why the analysis layer removes no knob. +**A later `-c` file wins over an earlier one, including whole scopes.** Measured with a failing task: frame sets `errorStrategy='finish'` -> exit 1; a project `analysis.config` passed as a second `-c` sets `'ignore'` -> exit 0. This is why the analysis layer removes no knob. -**Later-wins is also how `nextflow.config` used to remove seven of them.** Its `includeConfig "${launchDir}/parameters.config"` sat above its own assignments, so a project setting `cleanup=false`, `conda.enabled=false`, `errorStrategy='ignore'` or `maxRetries=99` resolved to `true/true/'finish'/3` — silently, while the config advertised the alternatives in a comment beside each one. **Fixed 2026-09-02 (E6g):** everything reading no parameter moved above the include, so a project's value wins. `workDir` and `env.PATH` stay below and stay the installation's — both read a parameter and so cannot move, and both are structural rather than tuning. The `process` block is deliberately two blocks now; the merge rule above is what makes that safe. +**Later-wins is also how `nextflow.config` used to remove seven of them.** Its `includeConfig "${launchDir}/parameters.config"` sat above its own assignments, so a project setting `cleanup=false`, `conda.enabled=false`, `errorStrategy='ignore'` or `maxRetries=99` resolved to `true/true/'finish'/3` -- silently, while the config advertised the alternatives in a comment beside each one. **Fixed 2026-09-02 (E6g):** everything reading no parameter moved above the include, so a project's value wins. `workDir` and `env.PATH` stay below and stay the installation's -- both read a parameter and so cannot move, and both are structural rather than tuning. The `process` block is deliberately two blocks now; the merge rule above is what makes that safe. -**`params.dir.bin = …` from a library function at DAG-build time sticks**, and both a later Groovy read and a process script block see the new value. **`env.PATH` does not** — `env` is config-only and config is parsed first, so it has to compute the installation itself with `System.getenv('POOLSEQFLOW_HOME')`. +**`params.dir.bin = ...` from a library function at DAG-build time sticks**, and both a later Groovy read and a process script block see the new value. **`env.PATH` does not** -- `env` is config-only and config is parsed first, so it has to compute the installation itself with `System.getenv('POOLSEQFLOW_HOME')`. -**`System.getenv('X')` works inside a `-c` config file** — the one way a config learns something Nextflow does not compute. The wrapper exports `POOLSEQFLOW_HOME` before running anything. +**`System.getenv('X')` works inside a `-c` config file** -- the one way a config learns something Nextflow does not compute. The wrapper exports `POOLSEQFLOW_HOME` before running anything. --- @@ -146,13 +146,13 @@ That escape is written out in words above for the same reason: the first draft o **`moduleDir` is the file's own directory; `projectDir` is the entry script's.** Tested on 26.04.6: a function defined in `analysis/lib/nf/paths.nf` and *called from* a module's `main.nf` still sees `moduleDir = /analysis/lib/nf`. That is how a library locates the installation without being told which script is the entry point. -**`projectDir` follows the ENTRY SCRIPT, and `env{}` interpolates once.** Running a script from a subdirectory makes `params.dir.bin` (`${projectDir}/bin`) point there. Overriding the param is not enough — `env { PATH = "${params.dir.bin}:$PATH" }` was already interpolated against the old value, so the helpers are simply not found. Set both. +**`projectDir` follows the ENTRY SCRIPT, and `env{}` interpolates once.** Running a script from a subdirectory makes `params.dir.bin` (`${projectDir}/bin`) point there. Overriding the param is not enough -- `env { PATH = "${params.dir.bin}:$PATH" }` was already interpolated against the old value, so the helpers are simply not found. Set both. **An entry script outside the installation loads NONE of the installation's `nextflow.config`.** Nextflow reads `/nextflow.config` and `/nextflow.config` only, and for a module both are somewhere else. Measured, in this order: `params.dir.bin` became `//bin` so `parse_metadata.py` was not found; then `params.cores` was null and `fill()` died at `resolve_parameters.nf:8` with `Cannot invoke method containsKey() on null object`. Also silently absent: `conda.enabled`, the `process` block with `resourceLimits`, `env.PATH`, and `enabled = true` on all four observers. **Anything the pipeline gets from `nextflow.config`, a second entry point must be given again.** -**`workflow.manifest.version` is null for an entry script not beside `nextflow.config`.** The repo root gets the real version; a subdirectory gets null. Anything writing a release into a provenance record from a module has nothing to write — which is why the analysis frame carries its version in `analysis/frame.version` instead. +**`workflow.manifest.version` is null for an entry script not beside `nextflow.config`.** The repo root gets the real version; a subdirectory gets null. Anything writing a release into a provenance record from a module has nothing to write -- which is why the analysis frame carries its version in `analysis/frame.version` instead. -**Nextflow `include` is static** (26.04.6, 2026-08-31). An interpolated path is rejected at lint AND at run time (`Unexpected input: '"'`). An include of an ABSENT file is a hard compile error (`Invalid include source`), so "ship every include, install some" is impossible. An absolute or relative single-quoted literal works. The entry script on the command line is the one path Nextflow lets you choose at runtime — which is why the analysis layer runs a module's own `main.nf` rather than including it. +**Nextflow `include` is static** (26.04.6, 2026-08-31). An interpolated path is rejected at lint AND at run time (`Unexpected input: '"'`). An include of an ABSENT file is a hard compile error (`Invalid include source`), so "ship every include, install some" is impossible. An absolute or relative single-quoted literal works. The entry script on the command line is the one path Nextflow lets you choose at runtime -- which is why the analysis layer runs a module's own `main.nf` rather than including it. **An unused `include` still couples you.** Deleting a workflow from a library broke a module that named it in its include list but never called it. Import only what you call. @@ -165,35 +165,41 @@ That escape is written out in words above for the same reason: the first draft o - **A `def` local of the workflow body is NOT visible to the handler.** The closure resolves the name against the script binding when it eventually runs, finds nothing, and dies on `Cannot get property 'x' on null object`. Declare the value **without** `def` so it lands in the binding, or compute it from `params`/`workflow` inside the handler. - **A handler that throws replaces the real diagnosis.** Nextflow prints ``Failed to invoke `workflow.onComplete` event handler`` *instead of* the error that actually stopped the run, so a fault in logging code hides a genuine configuration error completely. It cost a multi-run dictionary conflict its entire careful message. Make the handler call exactly one guarded function and nothing else: anything in the **argument list** is evaluated outside that function's own try/catch, which is where the throw was. -**Catching an exception thrown by a module function inside a workflow body wraps it in `InvocationTargetException`, whose own `message` is null.** The real message is on `.cause`. Nextflow's top-level handler unwraps it correctly, so the user sees the right text — which means a test written around `try/catch` asserts on the harness rather than on what anyone will see. Test such a throw through the real entry point. +**Catching an exception thrown by a module function inside a workflow body wraps it in `InvocationTargetException`, whose own `message` is null.** The real message is on `.cause`. Nextflow's top-level handler unwraps it correctly, so the user sees the right text -- which means a test written around `try/catch` asserts on the harness rather than on what anyone will see. Test such a throw through the real entry point. -**That unwrapping does NOT happen inside an OPERATOR closure** — `subscribe`, `map`, `flatMap`. There the user sees `ERROR ~ Unexpected error [InvocationTargetException]` and a line number, and nothing else. Verified 2026-08-27. **Print the diagnosis to stderr first; throw only to make the run fail.** The exit status is still 1, so the failure is never lost — only the explanation. +**That unwrapping does NOT happen inside an OPERATOR closure** -- `subscribe`, `map`, `flatMap`. There the user sees `ERROR ~ Unexpected error [InvocationTargetException]` and a line number, and nothing else. Verified 2026-08-27. **Print the diagnosis to stderr first; throw only to make the run fail.** The exit status is still 1, so the failure is never lost -- only the explanation. **Groovy `list[0..-2]` on a ONE-element list is a negative range, not an empty one.** It throws `fromIndex = -1`, which Nextflow reports as a bare `ERROR ~ fromIndex = -1` with a line number and nothing else. Guard with `if (parts.size() > 1)`. -**`.unique()` on a Map's `values()` throws `UnsupportedOperationException`** — `values()` is an unmodifiable view and `unique()` sorts in place. Use `map.values().toList().unique()`. Cost one pipeline run, because the throw surfaced as the same contentless "Unexpected error" as every other DAG-build-time exception. +**`.unique()` on a Map's `values()` throws `UnsupportedOperationException`** -- `values()` is an unmodifiable view and `unique()` sorts in place. Use `map.values().toList().unique()`. Cost one pipeline run, because the throw surfaced as the same contentless "Unexpected error" as every other DAG-build-time exception. + +**A process script that exits early loses its own log, and the terminal hides it.** Every step here ends by appending `.command.log` to `Logs/`, at the bottom of the script. An `exit 1` above that -- a refusal, a guard -- never reaches it, so `Logs//` holds nothing for the one run that most needed explaining. Nextflow still prints the text to the terminal and `.nextflow.log`, which is what makes it look fine and is no use to anyone reading `Logs/` on a cluster afterwards. Verified 2026-09-29 by asserting on the log file: with the copy skipped the message is absent, with it present it is there. `0_verify_environment.nf` has had the fix since it was written -- an `archive_logs` function called on both paths, with a comment saying why. **Steps 2, 5 and 9 still have the hole for their own refusals.** + +**`$$` in a process script is eaten by Groovy and leaves the command without its argument.** A triple-quoted string interpolates `$`, so `kill -TERM $$` reaches `.command.sh` as `kill -TERM` -- a bash usage error, exit 2, which under `set -e` looks exactly like the tool failing. Write `\$\$`. Found 2026-09-29 while testing what survives a signal: two rows of a results table said a design survived SIGKILL when no signal had been sent at all. **Read `.command.sh` in the work directory before believing any result that depends on what a process script actually ran.** + +**`.command.log` is the COMBINED stream, so `>&2` is not what keeps a message out of it.** Measured the same day, by restoring the redirect and re-running: a guard's text reaches `Logs/` identically whether it went to stdout or stderr. The choice between them is about which section Nextflow files it under in the terminal -- "Command output:" or "Command error:" -- and nothing else. --- ## What the tools actually do -**A tool given an ABSOLUTE output path will not create the directory for you.** snpEff exits 255 with "Error creating summary: …/Reports/snpeff_summary.html (No such file or directory)". It had worked for years only because some earlier step in the same run made that directory on the way past — an accident that held until a work item could be the first thing writing into its own output directory. **"Another step will have made it" is not a guarantee the DAG expresses anywhere.** +**A tool given an ABSOLUTE output path will not create the directory for you.** snpEff exits 255 with "Error creating summary: .../Reports/snpeff_summary.html (No such file or directory)". It had worked for years only because some earlier step in the same run made that directory on the way past -- an accident that held until a work item could be the first thing writing into its own output directory. **"Another step will have made it" is not a guarantee the DAG expresses anywhere.** **snpEff's `-stats` writes TWO files.** It derives the gene table's name from the summary's, so `-stats snpeff_summary.html` also writes `snpeff_summary.genes.txt` beside it. Measured against a real run 2026-09-02; the plan and the manual had both recorded one file. -**The annotated VCF is not byte-reproducible across two runs of the same annotation.** `##SnpEffCmd` records the mktemp name of the normalized input and `##bcftools_normCommand` records a wall-clock `Date=`. Both are the tools' own provenance and neither is a defect — but any "the VCF is unchanged" assertion has to exclude those two lines. +**The annotated VCF is not byte-reproducible across two runs of the same annotation.** `##SnpEffCmd` records the mktemp name of the normalized input and `##bcftools_normCommand` records a wall-clock `Date=`. Both are the tools' own provenance and neither is a defect -- but any "the VCF is unchanged" assertion has to exclude those two lines. -**`samtools stats -c min,max,step` emits COV rows only for the depths that OCCUR.** Measured on a hand-built BAM: `-c 1,20,1` over depths 1 and 2 gives two rows, not twenty, and no open-bin row at all when nothing exceeds the ceiling. This is what makes `capBAM.histogramMax` unable to change a result — every value a run completes at yields the identical histogram — and it is why that parameter is excluded from the recorded manifest. +**`samtools stats -c min,max,step` emits COV rows only for the depths that OCCUR.** Measured on a hand-built BAM: `-c 1,20,1` over depths 1 and 2 gives two rows, not twenty, and no open-bin row at all when nothing exceeds the ceiling. This is what makes `capBAM.histogramMax` unable to change a result -- every value a run completes at yields the identical histogram -- and it is why that parameter is excluded from the recorded manifest. **vcftools echoes its own parsed parameters, not your command line.** `--minQ 1000` is logged as `--minQ 1e+03`. An assertion on the literal value fails while the pipeline does exactly the right thing. **`paste -sd', '` cycles the delimiter, it does not use the string.** `-d` takes a delimiter LIST and rotates through its characters, so a two-item list joins with `,` and a three-item one gives `a,b c`. For "join with comma-space" use `paste -sd, file | sed 's/,/, /g'`. Caught in review, where the two-member case looked correct. -**`cp -r src dest/` merges when `dest/src` already exists** — it does not nest into `dest/src/src`. Verified, because the nesting variant would have been silent. +**`cp -r src dest/` merges when `dest/src` already exists** -- it does not nest into `dest/src/src`. Verified, because the nesting variant would have been silent. -**`realpath -m`** resolves symlinks, `..` and trailing slashes without requiring the path to exist. That is what makes the `mainDir` ≠ `storageDir` check honest: a string comparison passes on `/x` vs `/x/` vs `/x/y/..` vs a symlink, and each is the same directory. +**`realpath -m`** resolves symlinks, `..` and trailing slashes without requiring the path to exist. That is what makes the `mainDir` != `storageDir` check honest: a string comparison passes on `/x` vs `/x/` vs `/x/y/..` vs a symlink, and each is the same directory. -**`cleanup = true` is session-scoped.** It removes only its own task leaves and never the workDir root, so two concurrent runs sharing a workDir do **not** delete each other's task directories — six concurrent runs from one checkout all exited 0, `.nextflow/cache` is partitioned by session UUID, and `.nextflow/history` survived an 8-way simultaneous append. The opposite was previously asserted here as a sharp hazard; it was tested and is false. +**`cleanup = true` is session-scoped.** It removes only its own task leaves and never the workDir root, so two concurrent runs sharing a workDir do **not** delete each other's task directories -- six concurrent runs from one checkout all exited 0, `.nextflow/cache` is partitioned by session UUID, and `.nextflow/history` survived an 8-way simultaneous append. The opposite was previously asserted here as a sharp hazard; it was tested and is false. **`.nextflow.log` is a single 10-slot rotation per launch directory**, so the tenth run started during a long run unlinks that run's live log while it is still writing. `NXF_CACHE_DIR` + `NXF_LOG_FILE` isolate completely and leave zero `.nextflow*` residue. `NXF_WORK` does **not** override a `workDir =` set in config; CLI `-work-dir` does. @@ -201,24 +207,24 @@ That escape is written out in words above for the same reason: the first draft o ## The test harness -**A process's own `echo` never reaches the suite's `run.out`.** It goes to that task's `.command.log`, and from there to `Logs///*_nextflow.log`; `run.out` holds only what Nextflow itself prints. A test asserting on a process message must read the per-process log — asserting against `run.out` fails even when the pipeline did exactly the right thing, which presents as a passing mtime check beside a failing message check. Those logs accumulate across runs with a per-run header, so searching the whole file is usually right. +**A process's own `echo` never reaches the suite's `run.out`.** It goes to that task's `.command.log`, and from there to `Logs///*_nextflow.log`; `run.out` holds only what Nextflow itself prints. A test asserting on a process message must read the per-process log -- asserting against `run.out` fails even when the pipeline did exactly the right thing, which presents as a passing mtime check beside a failing message check. Those logs accumulate across runs with a per-run header, so searching the whole file is usually right. -**A sandbox snapshots the installation when it is CREATED.** Re-running an old sandbox after editing `PoolSeqFlow`, `scripts/` or `poolseqflow.nf` tests the code as it was then. It cost a false "the reset bug does not reproduce" — the reused sandbox predated the edit by an hour. Rebuild it, or `cp` the edited file into `$sb/install/` first. +**A sandbox snapshots the installation when it is CREATED.** Re-running an old sandbox after editing `PoolSeqFlow`, `scripts/` or `poolseqflow.nf` tests the code as it was then. It cost a false "the reset bug does not reproduce" -- the reused sandbox predated the edit by an hour. Rebuild it, or `cp` the edited file into `$sb/install/` first. **A kept sandbox is shared across cases.** A report read out of `$TEST_TMPDIR/guards` after a full run belongs to whichever case ran last, not the one being investigated. Reproduce against a fresh sandbox. **Substring assertions and versioned names do not mix.** `assert_not_contains "conda activate PoolSeqFlow"` fires against the *correct* string `conda activate PoolSeqFlow-2.2.0`. Compare whole lines when the wrong value is a prefix of the right one. Caught twice this way. -**`ls | sort` is locale-dependent** — the default collation puts `_` before `.`, so `Test_annotated.vcf` sorts ahead of `Test.vcf`. Use `LC_ALL=C sort` when asserting on a file listing, or the assertion depends on the machine rather than on the pipeline. +**`ls | sort` is locale-dependent** -- the default collation puts `_` before `.`, so `Test_annotated.vcf` sorts ahead of `Test.vcf`. Use `LC_ALL=C sort` when asserting on a file listing, or the assertion depends on the machine rather than on the pipeline. **A test that strips comments must actually strip them.** An assertion that a source file does not mention `workflow.manifest.version` fails against a file that mentions it *in a comment explaining why it is not used*. Filter with `grep -vE '^\s*//'` first. -**Never pipe a suite run through `tail`.** The runner ends a failing run with `FAIL n passed, m failed`, a blank line, `Failed:`, then one ` / ` line per failure. Cut the head off that and what remains is a list of case names indented two spaces — which is the shape of the *passing* output, and `tail`'s own exit status is 0 whatever the suite did. On 2026-09-05 a six-suite run came back as twelve such lines and read as green; twelve cases had failed. Redirect the whole run to a file and read the summary line, or grep for `^FAIL`. The two output shapes are only distinguishable by the ` / ` separator. +**Never pipe a suite run through `tail`.** The runner ends a failing run with `FAIL n passed, m failed`, a blank line, `Failed:`, then one ` / ` line per failure. Cut the head off that and what remains is a list of case names indented two spaces -- which is the shape of the *passing* output, and `tail`'s own exit status is 0 whatever the suite did. On 2026-09-05 a six-suite run came back as twelve such lines and read as green; twelve cases had failed. Redirect the whole run to a file and read the summary line, or grep for `^FAIL`. The two output shapes are only distinguishable by the ` / ` separator. -**`--case` matches the function name, not the name the runner prints.** The display name has underscores turned into spaces, so `--case "missing encoding"` matches nothing while `--case missing_encoding` matches — and a filter that matches nothing reports `PASS 0 passed`, which reads exactly like a suite where everything succeeded. Two things follow: use the underscored form, and treat a `0 passed` line as a filter that missed rather than as a green run. It also means a case whose name shares no word with its siblings cannot be reached by the group filter that finds the rest of them — 2026-09-05, two new phenotype cases were silently not run because neither name contained `phenotype`. +**`--case` matches the function name, not the name the runner prints.** The display name has underscores turned into spaces, so `--case "missing encoding"` matches nothing while `--case missing_encoding` matches -- and a filter that matches nothing reports `PASS 0 passed`, which reads exactly like a suite where everything succeeded. Two things follow: use the underscored form, and treat a `0 passed` line as a filter that missed rather than as a green run. It also means a case whose name shares no word with its siblings cannot be reached by the group filter that finds the rest of them -- 2026-09-05, two new phenotype cases were silently not run because neither name contained `phenotype`. --- ## One rule that is not about syntax -**Anything put in a run map is a candidate for the parameter manifest, and a List lands whole.** `analysisParams()` flattens the map through `flattenParams()`, which recurses into Maps and takes anything else verbatim — so `run.metadata` (a List of Maps) would become one enormous `metadata=[{…}, …]` line, and every edit to a design column would fail the change guard as a changed *parameter*. It is excluded by name in `skipKey`, with the reason written there. **The general rule: when attaching derived data to a run map, check `skipKey` first — the manifest is the default destination, not an opt-in one.** +**Anything put in a run map is a candidate for the parameter manifest, and a List lands whole.** `analysisParams()` flattens the map through `flattenParams()`, which recurses into Maps and takes anything else verbatim -- so `run.metadata` (a List of Maps) would become one enormous `metadata=[{...}, ...]` line, and every edit to a design column would fail the change guard as a changed *parameter*. It is excluded by name in `skipKey`, with the reason written there. **The general rule: when attaching derived data to a run map, check `skipKey` first -- the manifest is the default destination, not an opt-in one.** diff --git a/.claude/development-notes/step-0-verification.md b/.claude/development-notes/step-0-verification.md index 47b9ce1..8dfa418 100644 --- a/.claude/development-notes/step-0-verification.md +++ b/.claude/development-notes/step-0-verification.md @@ -1,10 +1,10 @@ -# Step 0 — verification +# Step 0 -- verification **Written 2026-08-31, against the tree at `7d65893`.** The machinery is unchanged: the keyed checks, the logging rule, `VerifyAll`'s single joined tuple and the three broadcast stages beside it are all still as described. `analysisParams()`'s exclusion list has grown, and the one addition that needed a reason is noted below. `scripts/0_verify_environment.nf`. Everything the pipeline checks before it spends compute, and the machinery that decides how many times each check runs. -> **For the manual, not here.** The stage list and what each stage reports is user-facing — a person reading `0_verify_environment.txt` needs it. So is the rule that **every step is gated on step 0**: nothing computes while a check is failing. +> **For the manual, not here.** The stage list and what each stage reports is user-facing -- a person reading `0_verify_environment.txt` needs it. So is the rule that **every step is gated on step 0**: nothing computes while a check is failing. ## A check runs once per distinct value of what it reads @@ -14,14 +14,14 @@ Z, 2026-08-27: *"the pipeline first needs to parse out the multi-run csv, and de So `checkParameterMap()` names what each check actually reads, a check runs once per distinct key, and its verdict is handed back to every run that shares that value. -**It carries the same risk as `stepParameterMap()`, and worse.** Name too few parameters and one run's verdict is used for a run whose value differs — which is more serious here than there, because catching exactly that is what the check is for. A static case re-extracts each process body and fails if it reads anything its entry does not declare. +**It carries the same risk as `stepParameterMap()`, and worse.** Name too few parameters and one run's verdict is used for a run whose value differs -- which is more serious here than there, because catching exactly that is what the check is for. A static case re-extracts each process body and fails if it reads anything its entry does not declare. The names are dotted paths into a **run's own** parameters, read with `dig()`. Never `params`, which is the base configuration rather than what any particular run is using. Entries worth their reasoning: - **`CheckReference` and `CheckGFF` are one check per file**, deliberately not step 1's dictionary key, which is a (reference, GFF) *pair*. What this stage asks is whether one file is on disk. -- **`SkipGFFCheck` reads nothing** — its report is two fixed lines — so one task serves every run that does not annotate. The empty list is a statement, not an omission. +- **`SkipGFFCheck` reads nothing** -- its report is two fixed lines -- so one task serves every run that does not annotate. The empty list is a statement, not an omission. - **`CheckData` names `dataSource` as well as `dir.data`**, because the report prints it, and a name that reaches the report is a name that decides the report. - **`CheckDirectories` names only two of the four roots.** The installation and the launch directory are properties of the invocation and cannot differ between runs. @@ -29,48 +29,48 @@ The **storage root is part of every check key**, for the same reason it is part ## Where a step-0 stage logs -Not in a run's own Logs tree. A check keyed to what it validates can answer for two runs out of three, which belongs to neither of their trees — and step 0 runs before any run has results for a log to sit beside. So every stage logs at invocation level. `All_Runs` means "the invocation" in this one place rather than "shared by every run"; the file name says which runs each task answered for. `VerifyAll` is the exception and stays in the run's own tree, because it really is per run. +Not in a run's own Logs tree. A check keyed to what it validates can answer for two runs out of three, which belongs to neither of their trees -- and step 0 runs before any run has results for a log to sit beside. So every stage logs at invocation level. `All_Runs` means "the invocation" in this one place rather than "shared by every run"; the file name says which runs each task answered for. `VerifyAll` is the exception and stays in the run's own tree, because it really is per run. **One directory per workflow** (Z, 2026-08-28), not one per process. Every log file already carries the step, the stage and the runs it answered for, so a per-process directory repeated what the name said at the cost of a level of nesting in a tree the user is expected to read. -**One writer per file.** Tasks append without locking, which is safe only while no two share a file — and a keyed check runs N times into one directory. Naming the file after the runs it answered for makes collision impossible. The old RGTags repair stage had this wrong since E1t: two runs naming two different tables gave two tasks appending to one file. +**One writer per file.** Tasks append without locking, which is safe only while no two share a file -- and a keyed check runs N times into one directory. Naming the file after the runs it answered for makes collision impossible. The old RGTags repair stage had this wrong since E1t: two runs naming two different tables gave two tasks appending to one file. ## Assembling the report `reportPerRun()` uses `combine(by: 0)` on the key rather than a join. What is matched is the same string computed by one function on both sides, and every run reaches exactly one task of every check that applies to it. A join would drop an unmatched key silently. -**`VerifyAll` takes ONE tuple, joined on the run.** These used to be nine separate `val` inputs, which Nextflow matches **positionally** — item k of each channel paired with item k of the others. Safe while every stage emitted exactly one report; a silent mismatch the moment there are N, with nothing making run B's reference check line up with run B's trim check. The report would have described a run that never existed. +**`VerifyAll` takes ONE tuple, joined on the run.** These used to be nine separate `val` inputs, which Nextflow matches **positionally** -- item k of each channel paired with item k of the others. Safe while every stage emitted exactly one report; a silent mismatch the moment there are N, with nothing making run B's reference check line up with run B's trim check. The report would have described a run that never existed. -The three invocation-level stages — `CheckInstalledSoftware`, `CheckRunParameters` and, since E1u, `CheckMultiRun` — stay separate on purpose: they ride **value channels**, which broadcast to every task, which is what is wanted for a check that ran once for the whole invocation. +The three invocation-level stages -- `CheckInstalledSoftware`, `CheckRunParameters` and, since E1u, `CheckMultiRun` -- stay separate on purpose: they ride **value channels**, which broadcast to every task, which is what is wanted for a check that ran once for the whole invocation. -## `analysisParams()` — what invalidates a result +## `analysisParams()` -- what invalidates a result An **exclusion list on purpose**: a parameter added in a later release is treated as analysis-affecting until someone decides otherwise, which fails safe. -It takes the run's own effective parameters rather than the global `params`. Called once per run against `params`, it would write N identical manifests describing the base config — so every run would record settings it did not use, and the guard would never fire. +It takes the run's own effective parameters rather than the global `params`. Called once per run against `params`, it would write N identical manifests describing the base config -- so every run would record settings it did not use, and the guard would never fire. Exclusions with a reason: - **`dataSource` is deliberately NOT excluded.** It names the subdirectory the reads come from, so two datasets under one `storageDir` are two different analyses. While it *was* excluded, both passed the check and the second run reused the first dataset's trimmed reads, because step 2 keys its skip test on the sample id alone. Nothing recorded which data produced a set of outputs. - **`runId`** names where results go, not what they are. It is also the one key that does not exist in parameters.config at all, so leaving it in would add a line to every manifest and fail the change check on every project upgrading into 3.0. -- **`dryRun`/`dryRunDir`** describe the invocation, not the project. Leaving them in would make every dry run report the parameters as changed — the one thing a preview must not do. +- **`dryRun`/`dryRunDir`** describe the invocation, not the project. Leaving them in would make every dry run report the parameters as changed -- the one thing a preview must not do. - **`metadata`** is the parsed file carried in every run map. The metadata guard compares its own projection; flattening it here would put the whole file in the manifest, so adding a design column would fail the *parameter* check. -**The list has grown since, and almost all of it needs no reasoning.** What was added is the paths and resources Z ruled out from the start — `mainDir`, `storageDir`, `threads`, `memory`, the resolved `*Path` and `reference`/`gff`/`reads` names, and the `dir.`, `cores.`, `java.` and `software.` prefixes — none of which can change a number. One addition is a genuine setting and carries its reason in the source: **`capBAM.histogramMax`** bounds how deep step 5 *looks* while measuring a sample's depth, not what it decides, so moving it cannot move a cap and cannot invalidate a result. +**The list has grown since, and almost all of it needs no reasoning.** What was added is the paths and resources Z ruled out from the start -- `mainDir`, `storageDir`, `threads`, `memory`, the resolved `*Path` and `reference`/`gff`/`reads` names, and the `dir.`, `cores.`, `java.` and `software.` prefixes -- none of which can change a number. One addition is a genuine setting and carries its reason in the source: **`capBAM.histogramMax`** bounds how deep step 5 *looks* while measuring a sample's depth, not what it decides, so moving it cannot move a cap and cannot invalidate a result. -**The stored copies are refreshed only on a clean pass.** `.poolseqflow_params`, `.parameters.config` and `.multirun.csv` are rewritten under `[ "$STATUS" = "PASS" ]` and nowhere else. Refreshing them on a failure would overwrite the baseline with the very values that failed against it, and the run after that would compare current against current and pass — the guard fires exactly once and then never again. The same applies to `record_baseline()` in the metadata check, which is why that function is the only writer of `.poolseqflow_metadata`. +**The stored copies are refreshed only on a clean pass.** `.poolseqflow_params`, `.parameters.config` and `.multirun.csv` are rewritten under `[ "$STATUS" = "PASS" ]` and nowhere else. Refreshing them on a failure would overwrite the baseline with the very values that failed against it, and the run after that would compare current against current and pass -- the guard fires exactly once and then never again. The same applies to `record_baseline()` in the metadata check, which is why that function is the only writer of `.poolseqflow_metadata`. ## Traps encoded here -- **`findNameExpr()` cannot use a bracket class.** `{1,2}` → `[1,2]` happens to work only because each alternative is one character. `{R1,R2}` → `[R1,R2]` is a class matching a single character out of `R`, `1`, `,` or `2`, so it matches no real FASTQ — the check finds nothing and **passes vacuously** while the run has no data. Expanding the group into one `-name` per alternative is exact for any length. +- **`findNameExpr()` cannot use a bracket class.** `{1,2}` -> `[1,2]` happens to work only because each alternative is one character. `{R1,R2}` -> `[R1,R2]` is a class matching a single character out of `R`, `1`, `,` or `2`, so it matches no real FASTQ -- the check finds nothing and **passes vacuously** while the run has no data. Expanding the group into one `-name` per alternative is exact for any length. - **The sample-ID split takes its mate token from `readPattern`**, never assuming `_R1`/`_R2`. Step 2 keys every sample off `Channel.fromFilePairs`, which derives the prefix from the glob and accepts any `{1,2}` scheme, so this check has to agree with it or it rejects valid layouts. - **The mate token must be separated from the sample name.** Without a separator the split is guesswork: `Sample11`/`Sample12` read equally as one sample's two mates or as two samples. - **`CheckInstalledSoftware` is handed the union of every run's software settings**, not `params.software`. A table may name a different binary for one run, and a check reading only the base config would pass while that run's tool was missing. -- **`drop_size()` is awk and must stay awk.** It strips one tab-separated field from a guard line, which sed would do in one expression — except that `\t` in a sed *pattern* is a GNU extension. BSD sed reads it as a literal `t`, so the expression silently matches nothing and every project on macOS reports its pool sizes as changed on the first run after an edit elsewhere. +- **`drop_size()` is awk and must stay awk.** It strips one tab-separated field from a guard line, which sed would do in one expression -- except that `\t` in a sed *pattern* is a GNU extension. BSD sed reads it as a literal `t`, so the expression silently matches nothing and every project on macOS reports its pool sizes as changed on the first run after an edit elsewhere. - **`any_exists()` tests one candidate at a time on purpose.** The obvious `ls a b >/dev/null` is wrong in the same direction as the sed trap: `ls` exits non-zero when *either* operand is missing, so a project that has BAMs but no VCF would be read as having neither, and the guard would re-record the baseline instead of comparing against it. ## Stage 9, the multi-run table -Parsing and syntax live in `bin/parse_multirun.py`, for a correctness reason first: the values are parameter values, and `readPattern` — exactly the sort of thing people vary — defaults to `*_R{1,2}.fq.gz`, which contains a comma. Splitting on commas would cut it in half and report a row with the wrong field count, as a completely different mistake. Second reason: a script is unit-testable in milliseconds, while a check written here costs a JVM start. +Parsing and syntax live in `bin/parse_multirun.py`, for a correctness reason first: the values are parameter values, and `readPattern` -- exactly the sort of thing people vary -- defaults to `*_R{1,2}.fq.gz`, which contains a comma. Splitting on commas would cut it in half and report a row with the wrong field count, as a completely different mistake. Second reason: a script is unit-testable in milliseconds, while a check written here costs a JVM start. -**From E1u this is not the first gate.** The runs have to exist before the DAG can be built, so `resolve_parameters.nf` validates the table earlier still and an unusable one stops the invocation before any task is submitted — including this one. What is left here is the part worth having in the durable record: what the table expanded to, and what it detached from its derivation. The FAIL branches are a backstop, and because the stage can be run on its own. +**From E1u this is not the first gate.** The runs have to exist before the DAG can be built, so `resolve_parameters.nf` validates the table earlier still and an unusable one stops the invocation before any task is submitted -- including this one. What is left here is the part worth having in the durable record: what the table expanded to, and what it detached from its derivation. The FAIL branches are a backstop, and because the stage can be run on its own. diff --git a/.claude/development-notes/unmeasured-cells.md b/.claude/development-notes/unmeasured-cells.md new file mode 100644 index 0000000..20ebc56 --- /dev/null +++ b/.claude/development-notes/unmeasured-cells.md @@ -0,0 +1,95 @@ +# A cell nobody measured, and how it is published + +**Written 2026-09-29, against the tree at `eedf598` plus uncommitted work.** Raised by Z the moment `vcffilter.dropZeroDepth` landed: if the pipeline can now be told to keep a site some pool has no reads at, what does the frequency table say about that pool, and does the analysis layer need to change. + +The first half is fixed here. The second half is measured and **one decision is left open**, named at the bottom. + +## What a zero-depth cell actually looks like + +Measured rather than assumed, because the fixture VCF this project unit-tests against has no such record and its README says so: *"no sample has `DP=0` -- a division-by-zero case has to be constructed, not found here."* + +Constructed one: a 120 bp reference, twenty reads for `SampleA` over 11-50 carrying an alternate at position 30, twenty reads for `SampleB` over 71-110 and none anywhere near it. Through `bcftools mpileup` with the release's own option string, then `bcftools call -m -A -v`: + +``` +chrT 30 A C GT:PL:DP:SP:AD 1/1:214,60,0:20:0:0,20 ./.:0,0,0:0:0:0,0 +``` + +**A sample with no reads gets `AD` as `0,0`, not as a missing value.** So the depth table cell reads `0,0` -- real zeros -- and nothing about it is missing in the VCF sense. That matters because `allele_frequencies()` and `site_diversity()` both carry a `flat[flat == "."] <- NA` line written for bcftools' missing value; that line is not what handles this case, `total <= 0` is. + +## What the frequency table said before + +`bin/depth2freq.awk` divided each count by the cell's total and had `(total > 0) ? counts[j] / total : 0`. Run over the record above: + +``` +CHROM POS REF ALLELE TOTAL_AD SampleA SampleB +chrT 30 A A 0 0 0 +chrT 30 A C 1 1 0 +``` + +**`SampleA`'s zero and `SampleB`'s zero are the same character and mean opposite things.** SampleA was read twenty times and carries no `A`; SampleB was not read at all. A reader has no way to tell them apart, and the one tell -- that SampleB's column sums to 0 across the site where every other column sums to 1 -- is not something any reader is going to compute. + +This was unreachable before `dropZeroDepth`: `vcffilter.minDP` removes a site where any sample falls short, and the smallest minDP that means anything is 1. The branch existed and could not fire. `dropZeroDepth = false` with `minDP = 0` is what makes it reachable, which is two deliberate edits from any default. + +## The change + +`freqs[j] = (total > 0) ? counts[j] / total : "NA"`. Every allele of that site in that pool reads `NA`; every other pool at the same site is untouched, and the same pool at every other site is untouched. `TOTAL_AD` is converted by the same loop and takes the same rule. + +`NA` rather than an empty field or a `.`: the table is read by R more than by anything else, and `read.table` takes `NA` natively. `pandas.read_table` gives `NaN` for it too. + +**The depth table is not changed.** `0,0` there is unambiguous already -- zero reads supporting each allele, with no competing reading -- and it is what every module computes from. + +`delete parsed_vals` went in at the same time. It had been sitting in the release triage as a one-liner to apply whenever something next touched this file, and this is that. `parsed_vals` is global, the write loop runs over the cell's counts and the print loop over the ALT column's alleles, so a cell holding fewer counts than its row declares alleles printed whatever an earlier row left at that index. Measured with the clear removed: a row at `chr1:200` published `0.25` for its third allele, carried from `chr1:100`. It cannot reach this from a run -- `AD` is `Number=R` and `MajorAlleleToRef.py` stops on a ragged cell first -- so the case in `03_helpers` is written against the converter's contract rather than against what feeds it. + +## The analysis layer: per site it was already right + +Nothing in the frame or in any module reads the frequency table's numbers. `basicstats` declares `frequencies` in `needs`, which requires the file to exist, and its `main.nf` passes only depth tables to R. Every module computes from the depth table, which is the standing rule and is what made this a publishing defect rather than a numerical one. + +And the per-site arithmetic already treats an unmeasured cell as unmeasured, in four places, each with the behavior stated in its own comment: + +| | | +|---|---| +| `allele_frequencies()` | `depth` 0, `freq` NA on every allele of that site | +| `site_diversity()` | `depth` 0, `h` NA | +| `n_eff()` | NA at depth 0, so the site carries no weight | +| `mean_distance()` | every pair divided by its own site count | + +Run, not read: `allele_frequencies(list(A = c("0,20"), B = c("0,0")))` returns depth `20, 0` and freq `0, 1` against `NA, NA`. + +`association` is safe by the same route -- the weight is `n_eff`, NA becomes 0, and a weight of 0 removes the pool from that site's fit. `mds` is safe and already says so in its gates: *"A pool with no reads at a site drops it for that pool's pairs and leaves the others intact."* + +## The analysis layer: per pool it is not + +`basicstats` aggregates each pool's per-site depths into summary columns, and those aggregates run over **every called site**, including the ones that pool never saw. One pool, four sites, `nChrom` 200, with and without a single zero-depth site among them: + +| | `depth_mean` | `depth_harmonic` | `n_eff_harmonic` | `sites` | `pi_per_called_site` | +|---|---|---|---|---|---| +| four sites read | 100.00 | 100.00 | 66.89 | 4 | 0.43653 | +| one of them unmeasured | **75.00** | **0.00** | **NA** | 4 | 0.41284 | + +Three different behaviors in one row: + +- **`pi_per_called_site` is correct.** `h_sum` sums with `na.rm = TRUE` and the denominator is `sum(!is.na(corrected))`, so the unmeasured site leaves both. This is the column the manual already described as possibly smaller than `sites`. +- **`depth_harmonic` collapses to 0 and takes `n_eff_harmonic` to NA with it.** `harmonic_mean()` returns 0 if any value is 0, deliberately and with a comment saying so: *"a position carrying no reads carries no information."* Over one position that is right. Over a genome it says the pool has no information anywhere, out of one missed site in a million. It is at least **loud**: a depth_harmonic of exactly 0 beside a mean in the hundreds is not a number anyone reads past. +- **`depth_mean` is quietly wrong.** 75 where the pool was read at 100. It is a plausible number, it carries no marker, and `depth_median` behaves the same way. This is the one that would be believed. + +`sites` in both tables stays the full count, so it no longer equals the number of sites any of the depth columns were taken over. + +The step 5 depth histogram path is unaffected: `harmonic_mean(hist$depth, hist$positions)` reads the COV rows of `samtools stats`, which have no zero bin. + +## Decided 2026-09-29, Z: aggregate over the sites the pool was read at + +`read_at <- !is.na(stats$depth) & stats$depth > 0`, and `depth_mean`, `depth_median` and `depth_harmonic` are taken over that subset in both `depth.tsv` and `diversity.tsv`. A new `unmeasured` column in each publishes what was left out, so the denominator is readable rather than assumed. `sites` keeps its meaning and stays the full count. + +`neff.tsv`'s called-site rows moved with it: `positions` is now `sites - unmeasured`, which is what that column has always meant -- how many positions the figure beside it was measured over -- and it would otherwise have named a count the depth was not taken over. + +An all-unmeasured pool gives NA on all three rather than a mix. `mean(numeric(0))` is `NaN` while `median(numeric(0))` is `NA`, so the empty case is branched explicitly instead of being left to R's two answers. + +`basicstats` goes to `20260929.001` and gains a gate stating the rule. + +**What the case that guards it had to work around:** `awk -v n="NA" 'BEGIN { exit !(n > 0) }'` succeeds. `-v` assigns `NA` as a string, `"NA" > 0` compares it with `"0"` character by character, and `N` sorts above `0`. The first version of the `n_eff_harmonic` assertion passed on exactly the NA it existed to catch. It now tests the string first and forces the number with `n + 0`. + +Reverting `basicstats.R` to `HEAD` fails all eight assertions of that case, including `depth_mean` at 60 where it should be 80 and `depth_harmonic` at 0. + +## Still true afterwards + +The dead zone and the destructive band of `scaleMapQ` are untouched by any of this, and **nothing in step 7 refuses an empty VCF or an empty frequency table**. `scaleMapQ = 15` still gives an empty run that reports success. See `mapq-downgrade.md`; the emptiness guard is still unbuilt. diff --git a/.claude/development-notes/variant-model.md b/.claude/development-notes/variant-model.md index 3fd10ed..0e775bb 100644 --- a/.claude/development-notes/variant-model.md +++ b/.claude/development-notes/variant-model.md @@ -1,86 +1,86 @@ -# The variant model — how work is shared between runs +# The variant model -- how work is shared between runs -**Written 2026-08-31, against the tree at `7d65893`.** The model is unchanged and every function named here still exists. Two entries in `stepParameterMap()` have moved since — step 5 gained one, and step 7's is spelled differently — and one forward-looking clause about `members.txt` did not come true. All three are corrected in place. +**Written 2026-08-31, against the tree at `7d65893`.** The model is unchanged and every function named here still exists. Two entries in `stepParameterMap()` have moved since -- step 5 gained one, and step 7's is spelled differently -- and one forward-looking clause about `members.txt` did not come true. All three are corrected in place. -`scripts/variants.nf`. A multi-run table usually describes runs differing in a few late parameters — one reference at three filtering settings — and running each from the reads costs hours to produce byte-identical intermediates. This file works out, once and before anything runs, which runs agree up to which step. +`scripts/variants.nf`. A multi-run table usually describes runs differing in a few late parameters -- one reference at three filtering settings -- and running each from the reads costs hours to produce byte-identical intermediates. This file works out, once and before anything runs, which runs agree up to which step. > **For the manual, not here.** One thing in this file is user-facing science: **what `Output/All_Runs/`, `Shared_/` and `/` mean**, and that only divergence gets a name. A person reading a results tree needs it. `members.txt` inside a `Shared_` directory is the other half. Everything below is development history. ## The unit is a variant, not a run -A variant is a parameter set that one or more runs share at a given step — a node of the divergence tree. It carries exactly what a process needs, so a process still takes `tuple val(x), ...` and reads `x.dir.*` exactly as it did when `x` was a run; only what `x` *means* changes. A run is then a path from the root of the tree to a leaf, and runs appear only at the **edges**: step 0 validates each run's own configuration, and publishing sends a variant's outputs to each of its member runs. +A variant is a parameter set that one or more runs share at a given step -- a node of the divergence tree. It carries exactly what a process needs, so a process still takes `tuple val(x), ...` and reads `x.dir.*` exactly as it did when `x` was a run; only what `x` *means* changes. A run is then a path from the root of the tree to a leaf, and runs appear only at the **edges**: step 0 validates each run's own configuration, and publishing sends a variant's outputs to each of its member runs. **Z reframed this twice while it was being planned, and both corrections shrank it.** ### It is an analysis, not a deduplication (Z, 2026-08-26) -The obvious alternative is to hash each step's inputs and group by the digest. It produces the same partition, but it makes the digest do two jobs — identity *and* location — and the second is where the cost sits: hashes in paths, a content hash that races step 0's own CRLF repair, GString-vs-String key mismatches, and a "leader" that moves when the CSV is reordered. Comparing parameter values for **equality** needs none of it. There is no hash anywhere in the file. +The obvious alternative is to hash each step's inputs and group by the digest. It produces the same partition, but it makes the digest do two jobs -- identity *and* location -- and the second is where the cost sits: hashes in paths, a content hash that races step 0's own CRLF repair, GString-vs-String key mismatches, and a "leader" that moves when the CSV is reordered. Comparing parameter values for **equality** needs none of it. There is no hash anywhere in the file. ### Steps EXPAND, they never fan back -Every fan-back the expansions replaced was a `join`, and a join drops an unmatched key *silently*. The only cardinality check that existed sat inside a `.map{}` reached only by keys that had already survived one — so a run lost during fan-back produced no VCF, no tables, no promotion, and reported SUCCESS. +Every fan-back the expansions replaced was a `join`, and a join drops an unmatched key *silently*. The only cardinality check that existed sat inside a `.map{}` reached only by keys that had already survived one -- so a run lost during fan-back produced no VCF, no tables, no promotion, and reported SUCCESS. An expansion enumerates its children from the plan, so the arity is decided before anything runs and nothing can be dropped. `assertEveryRunProduced()` asserts it anyway rather than trusting it, because the failure it guards against is silent and the check costs nothing. ## Sharing was built switched off, and the switch is still there -`sharingEnabled()` returned false for two whole stages on purpose: the rewiring was built and proven inert first — old and new code over one fixture, nothing moved — so that when task counts finally did change, the change was unambiguously that one line and not the refactor underneath it. The suite's three-run table fell from 233 tasks to 112 when it was turned on. +`sharingEnabled()` returned false for two whole stages on purpose: the rewiring was built and proven inert first -- old and new code over one fixture, nothing moved -- so that when task counts finally did change, the change was unambiguously that one line and not the refactor underneath it. The suite's three-run table fell from 233 tasks to 112 when it was turned on. -**It is now `return true` and nothing else,** so `variantKey`'s `if (!sharingEnabled())` branch — which prefixes the run id and forces every run into a singleton variant — is unreachable. Whether that is a kill switch worth keeping or dead code worth deleting has not been decided. It is cheap to keep and it is the one lever that isolates "the partition is wrong" from "the wiring is wrong", which is the failure this file exists to guard against. +**It is now `return true` and nothing else,** so `variantKey`'s `if (!sharingEnabled())` branch -- which prefixes the run id and forces every run into a singleton variant -- is unreachable. Whether that is a kill switch worth keeping or dead code worth deleting has not been decided. It is cheap to keep and it is the one lever that isolates "the partition is wrong" from "the wiring is wrong", which is the failure this file exists to guard against. ## `stepParameterMap()` is the whole correctness risk -Name too few parameters and two runs share an artifact one of them did not ask for — reads trimmed to someone else's settings, with nothing downstream able to tell. +Name too few parameters and two runs share an artifact one of them did not ask for -- reads trimmed to someone else's settings, with nothing downstream able to tell. It is **authored rather than derived**, because it has to be reviewable, and a static case re-extracts it from the source and fails if any step reads a parameter its entry does not declare. Authoring alone drifts; extraction alone is fragile, since a regex cannot see an indirect read. Both. `artifact` refines the partition permanently: once what a step passes on differs, everything after it differs. `publish` affects only what that step writes for you to look at, so it splits that step's own side outputs and the branch rejoins immediately. -It deliberately disagrees with `analysisParams()` in one place: `dir.subpath.*` **is** included here, because it is half of an artifact's identity even though it cannot invalidate a result. The two lists answer different questions — "what invalidates a result" versus "what makes two results the same" — so they are allowed to disagree, but only on purpose. +It deliberately disagrees with `analysisParams()` in one place: `dir.subpath.*` **is** included here, because it is half of an artifact's identity even though it cannot invalidate a result. The two lists answer different questions -- "what invalidates a result" versus "what makes two results the same" -- so they are allowed to disagree, but only on purpose. Entries worth their reasoning: -- **Step 2's `fastqc.options` is `publish`, not `artifact`.** It is FastQC over the *clipped* reads, run after cutadapt, for a report nothing consumes. Trim Galore's own FastQC — whose zips `ClipReads` really does read — takes `--fastqc_args` instead. +- **Step 2's `fastqc.options` is `publish`, not `artifact`.** It is FastQC over the *clipped* reads, run after cutadapt, for a report nothing consumes. Trim Galore's own FastQC -- whose zips `ClipReads` really does read -- takes `--fastqc_args` instead. - **`threads` is deliberately NOT an identity** anywhere. It is a resource, it changes no byte of output, and including it would redo an entire analysis because someone asked for more cores. -- **Step 5 read no analysis parameter at all** when this was written, so it could never be a branch point. **E2 inverted that.** Step 5 now decides each sample's depth cap, so it declares `capBAM.maxDepth` as `artifact` — the cap refines what the step passes on — and `capBAM.histogramMax` as `publish`, which changes nothing it writes but makes runs sharing the step agree on it, since a ceiling too low for one sample fails all of them. Note the same parameter is *excluded* from `analysisParams()`, which is the disagreement two paragraphs up being used on purpose. +- **Step 5 read no analysis parameter at all** when this was written, so it could never be a branch point. **E2 inverted that.** Step 5 now decides each sample's depth cap, so it declares `capBAM.maxDepth` as `artifact` -- the cap refines what the step passes on -- and `capBAM.histogramMax` as `publish`, which changes nothing it writes but makes runs sharing the step agree on it, since a ceiling too low for one sample fails all of them. Note the same parameter is *excluded* from `analysisParams()`, which is the disagreement two paragraphs up being used on purpose. - **Step 7 declares `poolSize` and `ploidy` directly**, not only through the derived sensitivity: with per-pool sizes the filter computes each pool's threshold from them, so a run changing ploidy alone would otherwise be told it could reuse the filtered VCF. `diploidy` was the name when this was written. `poolSize` stands beside it because a sensitivity pinned by hand would otherwise let two runs of different pool sizes share one results directory. -- **Step 8 declares `annotate`** even though `8_annotate_variants.nf` never reads it. The reader is `variants.nf` itself — `variant.executes = (step != 8) || (variant.annotate as boolean)` — and `poolseqflow.nf` then filters on `.executes`. It belongs to step 8's identity because it decides whether the step runs *at all*. +- **Step 8 declares `annotate`** even though `8_annotate_variants.nf` never reads it. The reader is `variants.nf` itself -- `variant.executes = (step != 8) || (variant.annotate as boolean)` -- and `poolseqflow.nf` then filters on `.executes`. It belongs to step 8's identity because it decides whether the step runs *at all*. -`stepFolders()` is deliberately **wider** than `stepParameterMap()`: that map names the folders holding the artifact a step passes on, while this also names side outputs nothing consumes — the unpaired reads, the trim reports, step 5's coverage. Someone opening `Shared_1` sees all of them. Step 5 declares no output at all (both reports are written by absolute path), so it would be invisible here if this were derived rather than authored. +`stepFolders()` is deliberately **wider** than `stepParameterMap()`: that map names the folders holding the artifact a step passes on, while this also names side outputs nothing consumes -- the unpaired reads, the trim reports, step 5's coverage. Someone opening `Shared_1` sees all of them. Step 5 declares no output at all (both reports are written by absolute path), so it would be invisible here if this were derived rather than authored. ## Step 1 is not part of the analysis -Its artifacts are keyed by reference **name** and live under `mainDir`, so several genomes coexist and a later project reuses them — they are not keyed by what produced them. That is why `dictionaryKey` groups on output paths and `dictionarySettings` **throws** when two runs on one key disagree, rather than quietly splitting them. `dictionaryRuns` deliberately **merges** runs that disagree about `annotate`, because one database serves both; splitting that group would have two of them build concurrently into one directory. +Its artifacts are keyed by reference **name** and live under `mainDir`, so several genomes coexist and a later project reuses them -- they are not keyed by what produced them. That is why `dictionaryKey` groups on output paths and `dictionarySettings` **throws** when two runs on one key disagree, rather than quietly splitting them. `dictionaryRuns` deliberately **merges** runs that disagree about `annotate`, because one database serves both; splitting that group would have two of them build concurrently into one directory. -So what the chain needs from step 1 is not an input digest but the identity of the artifact it will find there — split in two, because one identity over-splits. `referenceIdentity()` is what steps 3 and 6 read; `snpEffIdentity()` is what step 8 reads. Two runs differing only in `gffFile` must not redo alignment and calling. +So what the chain needs from step 1 is not an input digest but the identity of the artifact it will find there -- split in two, because one identity over-splits. `referenceIdentity()` is what steps 3 and 6 read; `snpEffIdentity()` is what step 8 reads. Two runs differing only in `gffFile` must not redo alignment and calling. ## `stepDependencies()` is not "the step before" -The pipeline branches after calling. Step 6 reads the ready BAMs from step 4, not step 5's reports; step 8 reads the called VCF from step 6, exactly as step 7 does, so a run differing only in a step-7 filter still shares annotation. Walking `2..k` linearly gets both wrong — which is how it was written first, and what dumping the partition for a real table exposed. +The pipeline branches after calling. Step 6 reads the ready BAMs from step 4, not step 5's reports; step 8 reads the called VCF from step 6, exactly as step 7 does, so a run differing only in a step-7 filter still shares annotation. Walking `2..k` linearly gets both wrong -- which is how it was written first, and what dumping the partition for a real table exposed. One dependency per step is enforced with a throw. A step reading two independent artifacts would have to JOIN them, and a join dropping a key silently is what this design exists to avoid. ## Keys, groups and destinations -- **The storage root is part of every key.** Two runs whose `storageDir` differ have no directory in common to put a shared result in, so they must not group — prefixing the key makes that automatic rather than a refusal. Verified: a table where one run points elsewhere leaves it alone in its own tree at every step while the others share. -- **The lead member is the lowest RunID**, not the first table row, so reordering the CSV cannot change which run's map a shared variant carries — which would orphan the previous invocation's artifacts under a different root and recompute everything. +- **The storage root is part of every key.** Two runs whose `storageDir` differ have no directory in common to put a shared result in, so they must not group -- prefixing the key makes that automatic rather than a refusal. Verified: a table where one run points elsewhere leaves it alone in its own tree at every step while the others share. +- **The lead member is the lowest RunID**, not the first table row, so reordering the CSV cannot change which run's map a shared variant carries -- which would orphan the previous invocation's artifacts under a different root and recompute everything. - **One results tree** (Z, 2026-08-27). A run no longer gets its own storageDir. The group number is per distinct *member set*, not per step: groups nest or are disjoint, so one member set is one group however many steps it owns, and `Shared_1` holds everything that group produced. -- **A single run is none of these** — nothing to be distinguished from — so its tree stays where it always was, and `Utilized/` stays unsuffixed. No synthetic key when there is only one run. +- **A single run is none of these** -- nothing to be distinguished from -- so its tree stays where it always was, and `Utilized/` stays unsuffixed. No synthetic key when there is only one run. -**Where a skip check looks**: permanent storage first, so a promoted artifact outranks a residue copy; then the variant's own working root; then its ancestors', because a step reading a *shared* artifact reads it from the root of the variant that produced it. Two runs sharing step 2 and diverging at step 3 is the whole case — the trimmed reads are under the step-2 variant's root, which neither step-3 variant would otherwise search. +**Where a skip check looks**: permanent storage first, so a promoted artifact outranks a residue copy; then the variant's own working root; then its ancestors', because a step reading a *shared* artifact reads it from the root of the variant that produced it. Two runs sharing step 2 and diverging at step 3 is the whole case -- the trimmed reads are under the step-2 variant's root, which neither step-3 variant would otherwise search. ## Promotion gates The gate is keyed by the **producing** step's branch. Once a producer is shared its consumers may not be, so releasing on the first consumer to finish would delete a file another still needs. -The count is arithmetic, not a runtime reference count: the structure is a tree, so a consuming step's variants partition their parent's members and how many there are is known before anything runs. `groupKey` carries that number so each group is released as soon as *its* consumers are in — waiting for the channel to close would hold every artifact on the working volume until the slowest run finished, which is the opposite of what promotion is for. +The count is arithmetic, not a runtime reference count: the structure is a tree, so a consuming step's variants partition their parent's members and how many there are is known before anything runs. `groupKey` carries that number so each group is released as soon as *its* consumers are in -- waiting for the channel to close would hold every artifact on the working volume until the slowest run finished, which is the opposite of what promotion is for. ## Decisions - **A publish-only disagreement inside a group is refused** (Z, 2026-08-27). Publish parameters must not split the artifact partition, but they cannot split the step's own outputs either: two step-2 variants feeding one step-3 variant is a MERGE, and the expansion model exists so that cannot happen. Same shape as `dictionarySettings()` throwing. - **`members.txt` is a record, not a guard.** What stops a table edit from mixing two groupings in one directory is the stored copy of the table itself. This exists because Z asked for the grouping to be recoverable from the results, and because the analysis layer will want it. - **The analysis layer did not want it.** It recomputes `variantPlan()` rather than reading what the pipeline left on disk — the same reasoning as the first half of this entry, one level up: a record can be edited after the fact, and a recomputation cannot disagree with the run it describes. `members.txt` stays what its first sentence says it is, and `dry-run.md` carries the same corrected prediction from the other end. + **The analysis layer did not want it.** It recomputes `variantPlan()` rather than reading what the pipeline left on disk -- the same reasoning as the first half of this entry, one level up: a record can be edited after the fact, and a recomputation cannot disagree with the run it describes. `members.txt` stays what its first sentence says it is, and `dry-run.md` carries the same corrected prediction from the other end. - **The partition is reported before compute is spent.** A wrong parameter map is the failure this design risks and it is silent, so step 0 states the partition where it can be disagreed with rather than leaving it inferred from which directories happen to exist. ## Traps encoded in this file @@ -89,4 +89,4 @@ The count is arithmetic, not a runtime reference count: the structure is a tree, - **`runToken()` exists because a null key matches nothing, silently.** A single run has no RunID at all, so it travels as the literal `-`. - **Recursion, not a self-referencing closure.** The strict parser rejects `def f; f = { ... f(...) }`. It also rejects both `for` forms, which is why `dig()` uses `.each` with a reassigned capture. - **`values()` is an unmodifiable view** and `unique()` sorts in place, so `publishConflicts()` calls `toList()` first or it throws `UnsupportedOperationException`. -- **An exception inside an operator closure reaches Nextflow wrapped in an `InvocationTargetException` whose own message is null** — the user sees "Unexpected error" and a line number. Verified, not assumed. That is why `assertEveryRunProduced()` prints its diagnosis to stderr first and throws only to fail the run. +- **An exception inside an operator closure reaches Nextflow wrapped in an `InvocationTargetException` whose own message is null** -- the user sees "Unexpected error" and a line number. Verified, not assumed. That is why `assertEveryRunProduced()` prints its diagnosis to stderr first and throws only to fail the run. diff --git a/.claude/development-notes/wrapper.md b/.claude/development-notes/wrapper.md index bfba3a4..aa685e0 100644 --- a/.claude/development-notes/wrapper.md +++ b/.claude/development-notes/wrapper.md @@ -1,22 +1,22 @@ # The `PoolSeqFlow` wrapper -**Written 2026-08-31, against the tree at `7d65893`.** Everything here still holds — the stamp, the lazy `require_install`, the `env_exists` name-column comparison, both deletion fixes, and what `init` refuses to write. Only the arity amendment below has been overtaken, and it is corrected in place. +**Written 2026-08-31, against the tree at `7d65893`.** Everything here still holds -- the stamp, the lazy `require_install`, the `env_exists` name-column comparison, both deletion fixes, and what `init` refuses to write. Only the arity amendment below has been overtaken, and it is corrected in place. -The shell script in front of everything. The root model it implements is in `four-roots.md`; this file is about the script itself — the traps it is shaped around, and the things it used to do wrongly. +The shell script in front of everything. The root model it implements is in `four-roots.md`; this file is about the script itself -- the traps it is shaped around, and the things it used to do wrongly. -**Arity, amended 2026-08-31.** It took exactly one word for its whole life, and that contract was load-bearing enough that the analysis layer was given a wrapper of its own to avoid touching it. Z reversed that and merged the two: **every subcommand still takes no arguments except `analysis`, which takes exactly one** — an analysis command, or the name of a module to run. `analysis-wrapper.md` covers that half and what the merge removed. +**Arity, amended 2026-08-31.** It took exactly one word for its whole life, and that contract was load-bearing enough that the analysis layer was given a wrapper of its own to avoid touching it. Z reversed that and merged the two: **every subcommand still takes no arguments except `analysis`, which takes exactly one** -- an analysis command, or the name of a module to run. `analysis-wrapper.md` covers that half and what the merge removed. **It widened again, first the same evening and then further.** A module run is `analysis [nocpp]`, and the store is `analysis modules install []`, so `analysis` no longer takes exactly one word. What survived is the *shape* rather than the count: the top-level subcommands still take nothing, and everything variable hangs off `analysis`. `usage_analysis` in the script is the current statement of it, and `four-roots.md` records the same widening from the root model's side. ## Why the installed copy is stamped rather than self-locating -`POOLSEQFLOW_INSTALLED_HOME` is written into the deployed copy by `install`, with a `sed` that then verifies its own substitution took. The alternative — having the wrapper resolve its own path at run time — fails on exactly the case that matters: an installed wrapper is reached through a symlink on `$PREFIX/bin`, and **resolving a symlink back to its target is the single most system-dependent step in the whole script**. `readlink -f` is GNU; BSD `readlink` has no `-f` at all. So the source tree's copy falls back to `readlink -f "$0"` (which is fine — a clone is not reached through a symlink) and the installed copy never needs to. +`POOLSEQFLOW_INSTALLED_HOME` is written into the deployed copy by `install`, with a `sed` that then verifies its own substitution took. The alternative -- having the wrapper resolve its own path at run time -- fails on exactly the case that matters: an installed wrapper is reached through a symlink on `$PREFIX/bin`, and **resolving a symlink back to its target is the single most system-dependent step in the whole script**. `readlink -f` is GNU; BSD `readlink` has no `-f` at all. So the source tree's copy falls back to `readlink -f "$0"` (which is fine -- a clone is not reached through a symlink) and the installed copy never needs to. The stamp is a **default, not an override**. `POOLSEQFLOW_HOME` still wins, which is what lets a developer point an installed wrapper at a working checkout without reinstalling. ## Why `require_install` is called per subcommand instead of at startup -`version`, `cite`, `list`, `uninstall` and `uninstall_all` are answerable from the wrapper alone. Refusing them because `poolseqflow.nf` is missing would take away exactly the commands someone reaches for **when an installation is broken** — which is the moment they are most needed. So the check is lazy and each subcommand opts in. +`version`, `cite`, `list`, `uninstall` and `uninstall_all` are answerable from the wrapper alone. Refusing them because `poolseqflow.nf` is missing would take away exactly the commands someone reaches for **when an installation is broken** -- which is the moment they are most needed. So the check is lazy and each subcommand opts in. ## Why `require_project_config` exists at all @@ -24,7 +24,7 @@ Nextflow does detect the missing include, but what it prints is `ERROR ~ Unable ## `env_exists` compares the name column only, and this is not hypothetical -`conda env list` prints ` [*] ` per row, so grepping the whole line for a name also matches any **other** environment whose path happens to contain it. The repository directory is itself called `PoolSeqFlow`, so an environment created underneath it matches. Hence `awk '{print $1}'` first, then `grep -qxF` — whole (`-x`) and literal (`-F`), so a name containing `.` or `+` still works. +`conda env list` prints ` [*] ` per row, so grepping the whole line for a name also matches any **other** environment whose path happens to contain it. The repository directory is itself called `PoolSeqFlow`, so an environment created underneath it matches. Hence `awk '{print $1}'` first, then `grep -qxF` -- whole (`-x`) and literal (`-F`), so a name containing `.` or `+` still works. ## What a second environment actually costs @@ -42,32 +42,32 @@ So a second version costs roughly **300 MB even where every package matches**, p ## Uninstalling the copy you are running from is safe -On Linux the shell holds the script open, and unlinking an open file keeps its contents readable until the last close — so `remove_payload` deleting its own payload mid-execution works. **Verified rather than assumed**: there is a self-uninstall case in the launcher suite. +On Linux the shell holds the script open, and unlinking an open file keeps its contents readable until the last close -- so `remove_payload` deleting its own payload mid-execution works. **Verified rather than assumed**: there is a self-uninstall case in the launcher suite. The plain `PoolSeqFlow` symlink is re-pointed at the newest remaining version, or removed once none remain. A dangling symlink on PATH is worse than an absent command: it fails far less clearly. ## `nf_config_value` is the only path oracle, and both roots resolve through it -Resolved through Nextflow rather than by parsing `parameters.config`, because values are interpolated — `workDir` is built from `mainDir` — and text matching gets them wrong. The installation is named explicitly on the command line because that is where `nextflow.config` lives, while the current directory supplies `parameters.config`. **Verified that both resolve correctly from a project directory**: `${launchDir}` entries come from the project, `${projectDir}` entries from the installation. What it cannot do is name per-run directories; see `promotion.md`. +Resolved through Nextflow rather than by parsing `parameters.config`, because values are interpolated -- `workDir` is built from `mainDir` -- and text matching gets them wrong. The installation is named explicitly on the command line because that is where `nextflow.config` lives, while the current directory supplies `parameters.config`. **Verified that both resolve correctly from a project directory**: `${launchDir}` entries come from the project, `${projectDir}` entries from the installation. What it cannot do is name per-run directories; see `promotion.md`. ## Two deletion bugs the current shape is the fix for -**`clean` deleted Nextflow's history silently.** `rm -rf .nextflow*` used to sit *below* the if/else that reports on `workDir` rather than inside it. A project whose `workDir` could not be resolved was therefore told only that the work directory was being left in place — while `.nextflow*` was deleted anyway, unmentioned. The deletion was right; the silence was not. Everything that will be removed is now listed before anything is, including that branch. +**`clean` deleted Nextflow's history silently.** `rm -rf .nextflow*` used to sit *below* the if/else that reports on `workDir` rather than inside it. A project whose `workDir` could not be resolved was therefore told only that the work directory was being left in place -- while `.nextflow*` was deleted anyway, unmentioned. The deletion was right; the silence was not. Everything that will be removed is now listed before anything is, including that branch. -**`reset` had an `rm -rf` on a user glob.** `$STORAGE/PoolSeqFlow_*` was removed here too, until it was traced: the `PoolSeqFlow_pipeline_{dag,trace,timeline,report}` files have gone to `${params.dir.output.reports}` in every release since 1.0.0, never to `storageDir` itself. The glob therefore **never matched anything this pipeline created**. All it could ever have done is delete a user's own file or directory that happened to start with `PoolSeqFlow_` — undisclosed, and with `rm -rf`. Dropped. +**`reset` had an `rm -rf` on a user glob.** `$STORAGE/PoolSeqFlow_*` was removed here too, until it was traced: the `PoolSeqFlow_pipeline_{dag,trace,timeline,report}` files have gone to `${params.dir.output.reports}` in every release since 1.0.0, never to `storageDir` itself. The glob therefore **never matched anything this pipeline created**. All it could ever have done is delete a user's own file or directory that happened to start with `PoolSeqFlow_` -- undisclosed, and with `rm -rf`. Dropped. The surviving legacy removals are deliberate and harmless: `$STORAGE/Reports` only exists for projects created before 2.1.0, `$STORAGE/Reference` only for those created before 3.0.0, and the guard files at the storage root only for those created before 3.0. Leaving one of the last would make the next run fail its checks for no reason, or attribute fresh results to an earlier release. ## Why the dry-run preview is the one directory that gets vetted -`dryrun` rebuilds the preview from scratch and `dryclean` removes it, both with `rm -rf` on a path taken from the configuration — so both call `dryrun_is_ours` first. It is the only one of the three roots that sits **in the project** rather than in a storage root, which is precisely where somebody is most likely to have put something of their own. Finding anything that is not an empty directory, a `README.txt` or a `members.txt` is reported and stops the command; the answer is never to delete it anyway. +`dryrun` rebuilds the preview from scratch and `dryclean` removes it, both with `rm -rf` on a path taken from the configuration -- so both call `dryrun_is_ours` first. It is the only one of the three roots that sits **in the project** rather than in a storage root, which is precisely where somebody is most likely to have put something of their own. Finding anything that is not an empty directory, a `README.txt` or a `members.txt` is reported and stops the command; the answer is never to delete it anyway. ## What `init` will not write for you -`parameters.config` is copied from the template because it is a settings file edited in place, so the template is the right starting point. `metadata.csv` and the run table are **not** written, only their `.example` copies: both are tables whose entire content is the experiment — which pairs are one pool, which parameters differ between runs — so a copied one would describe somebody else's. See `metadata-file.md`. +`parameters.config` is copied from the template because it is a settings file edited in place, so the template is the right starting point. `metadata.csv` and the run table are **not** written, only their `.example` copies: both are tables whose entire content is the experiment -- which pairs are one pool, which parameters differ between runs -- so a copied one would describe somebody else's. See `metadata-file.md`. `init_multi` flips `multiRun` to `true` only in a config it created this run. Editing one the user has already filled in would be reaching into their settings behind their back, so if the config was already there it says so and asks them to set it themselves. ## `uninstall` asks, and refuses to guess -This wrapper's own version is only the newest **by default**, so silently removing "this version" would sometimes delete the copy most projects are using. One installed version needs no question; several do. With nothing attached to prompt — a script, CI, a hook — it refuses and names the exact command that removes a specific version (`PoolSeqFlow- uninstall`), because guessing here deletes somebody's installation. +This wrapper's own version is only the newest **by default**, so silently removing "this version" would sometimes delete the copy most projects are using. One installed version needs no question; several do. With nothing attached to prompt -- a script, CI, a hook -- it refuses and names the exact command that removes a specific version (`PoolSeqFlow- uninstall`), because guessing here deletes somebody's installation. diff --git a/.gitignore b/.gitignore index b8d9282..36e580e 100644 --- a/.gitignore +++ b/.gitignore @@ -59,6 +59,13 @@ plugins-prod docs/ site/ +# The mkdocs virtual environment, which dev/scripts/serve_docs.sh looks for. It belongs under +# dev/: `manual` is a payload item, so the installer's `cp -r` would carry one placed there into +# every installation made from a checkout. manual/.venv/ is listed only so that a copy made out of +# habit cannot be committed. +dev/manual/.venv/ +manual/.venv/ + __pycache__/ *.pyc diff --git a/CHANGELOG.md b/CHANGELOG.md index 3aa05f9..b2c67c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to PoolSeqFlow will be documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -**The public API is what a result depends on**: the published table formats, `parameters.config`, `metadata.csv`, and what an analysis folder contains. The command line is documented, not frozen — a verb may gain a word or a listing may get shorter without that being a breaking change, because nothing already computed becomes unreproducible. A version moves when something changes, never on a schedule. +**The public API is what a result depends on**: the published table formats, `parameters.config`, `metadata.csv`, and what an analysis folder contains. The command line is documented, not frozen -- a verb may gain a word or a listing may get shorter without that being a breaking change, because nothing already computed becomes unreproducible. A version moves when something changes, never on a schedule. In practice a version number says what upgrading will cost you: a third number is an inconvenience fixed, a second means the tool works mostly fine but there is a caveat worth reading and an incompatibility with its immediate fix, and a first is a different experience. @@ -18,14 +18,14 @@ In practice a version number says what upgrading will cost you: a third number i ### Added -- **Reads may sit in subfolders of `Data/`.** However your facility handed them over is how you can leave them. A sample is named by its **file**, never by its folder — `Sample1_R1.fq.gz` is sample `Sample1` wherever it lies — so `metadata.csv` does not change, `readPattern` does not change, and the pipeline reads no meaning into the folders you use. A pair whose two mates are in *different* folders is still that pair. What does not work is the same file name in two places: that is one sample twice over, each copy would overwrite the other's results, and the run stops and names both directories. +- **Reads may sit in subfolders of `Data/`.** However your facility handed them over is how you can leave them. A sample is named by its **file**, never by its folder -- `Sample1_R1.fq.gz` is sample `Sample1` wherever it lies -- so `metadata.csv` does not change, `readPattern` does not change, and the pipeline reads no meaning into the folders you use. A pair whose two mates are in *different* folders is still that pair. What does not work is the same file name in two places: that is one sample twice over, each copy would overwrite the other's results, and the run stops and names both directories. - **Hidden folders are skipped.** Anything beginning with a dot is not searched. `.snapshot` is the one that matters: NetApp exposes it read-only inside every directory on a great deal of shared storage, and it holds a copy of every file per snapshot, so searching it would find every sample many times over. If one of them does hold reads, step 0 names it, because a folder full of reads ignored in silence is how you lose samples without being told. -- **Tab completion**, installed with the pipeline. It completes every command, the two targets `check` takes, the analysis commands, and the modules **you actually have installed** — so `PoolSeqFlow analysis ` lists what is in your store rather than a fixed list. bash finds it in your next shell with nothing to do; zsh needs two lines, which the installer prints. `PoolSeqFlow analysis modules install ` offers nothing on purpose: those names come from the catalogue over the network, and a keystroke should not make a network request. +- **Tab completion**, installed with the pipeline. It completes every command, the two targets `check` takes, the analysis commands, and the modules **you actually have installed** -- so `PoolSeqFlow analysis ` lists what is in your store rather than a fixed list. There is one for bash and a native one for zsh, which also shows what each command is for. bash finds its own in your next shell with nothing to do; zsh needs one line adding a directory to `$fpath`, which the installer prints. `PoolSeqFlow analysis modules install ` offers nothing on purpose: those names come from the catalogue over the network, and a keystroke should not make a network request. ### Changed -- **A sample missing one of its mates is refused.** `readPattern` takes the two together, and a lone file was simply not a pair — so it was left out of the run, silently, with nothing to say a sample had gone. The only check against it asked whether the total number of FASTQ files was even, which two samples each missing a mate satisfy between them. Every sample is now required to have exactly one of each mate, and the run names the ones that do not. This is the change most likely to stop a project that used to start: it is telling you about data that was already incomplete. -- **macOS is no longer listed as a supported platform.** No released version has installed on one. It ran in early development, when the environment was a plain list of names and versions. Conda solved that per platform, picking builds for whatever machine it is on. Once the environment was frozen by export it stopped being portable, because an export names the exact builds one machine received: `install/environment.yml` has pinned `linux-64` since the repository's first commit, so conda cannot solve it on a Mac and the install fails before anything else is reached. The Requirements table had said "Linux or macOS" since before 3.1.0 and that was never true of a release anyone could download. The manual now says Linux, and explains what supporting macOS would take — three sets of pinned files instead of one, since Apple Silicon and Intel are separate platforms again. It is planned, not prioritized. +- **A sample missing one of its mates is refused.** `readPattern` takes the two together, and a lone file was simply not a pair -- so it was left out of the run, silently, with nothing to say a sample had gone. The only check against it asked whether the total number of FASTQ files was even, which two samples each missing a mate satisfy between them. Every sample is now required to have exactly one of each mate, and the run names the ones that do not. This is the change most likely to stop a project that used to start: it is telling you about data that was already incomplete. +- **macOS is no longer listed as a supported platform.** No released version has installed on one. It ran in early development, when the environment was a plain list of names and versions. Conda solved that per platform, picking builds for whatever machine it is on. Once the environment was frozen by export it stopped being portable, because an export names the exact builds one machine received: `install/environment.yml` has pinned `linux-64` since the repository's first commit, so conda cannot solve it on a Mac and the install fails before anything else is reached. The Requirements table had said "Linux or macOS" since before 3.1.0 and that was never true of a release anyone could download. The manual now says Linux, and explains what supporting macOS would take -- three sets of pinned files instead of one, since Apple Silicon and Intel are separate platforms again. It is planned, not prioritized. ### Removed @@ -52,10 +52,10 @@ In practice a version number says what upgrading will cost you: a third number i ### Fixed -- **`analysis install` failed on any machine with glibc older than 2.39**, which is most clusters. `install/environment-analysis.yml` pinned `sysroot_linux-64=2.39`, and that is a demand on your machine rather than something conda installs — `__glibc` is conda's name for the glibc you already have. Nothing asked for it: the solver took the newest the exporting machine allowed. Both environments now install on anything from glibc 2.28 up. The pipeline layer was never affected, so if `PoolSeqFlow install` worked while `analysis install` did not, this was why. -- **`rehash: command not found` on every conda call**, where something in your environment exports `ZSH_VERSION`. conda runs `\rehash` — zsh's name for `hash -r` — whenever that is set, including inside the bash wrapper, where the command does not exist. Nothing was failing underneath it. The wrapper now defines `rehash`. +- **`analysis install` failed on any machine with glibc older than 2.39**, which is most clusters. `install/environment-analysis.yml` pinned `sysroot_linux-64=2.39`, and that is a demand on your machine rather than something conda installs -- `__glibc` is conda's name for the glibc you already have. Nothing asked for it: the solver took the newest the exporting machine allowed. Both environments now install on anything from glibc 2.28 up. The pipeline layer was never affected, so if `PoolSeqFlow install` worked while `analysis install` did not, this was why. +- **`rehash: command not found` on every conda call**, where something in your environment exports `ZSH_VERSION`. conda runs `\rehash` -- zsh's name for `hash -r` -- whenever that is set, including inside the bash wrapper, where the command does not exist. Nothing was failing underneath it. The wrapper now defines `rehash`. - **`~/.local/bin is NOT on your PATH` when it is.** The check compared `$PATH` as text, so a trailing slash, a doubled slash or a symlinked home each made it answer no. It resolves directories now. Worth fixing for what follows it: the message tells you to edit a shell profile that was correct. -- **`uninstall` reported `Directory not empty` and stopped there**, on network storage, where removing a file that is still open leaves a `.nfs*` placeholder — and the process holding them open is `PoolSeqFlow` itself. It now finishes the uninstall and tells you the placeholders go when it exits. Previously it left `~/.local/bin` pointing at the release you were removing. +- **`uninstall` reported `Directory not empty` and stopped there**, on network storage, where removing a file that is still open leaves a `.nfs*` placeholder -- and the process holding them open is `PoolSeqFlow` itself. It now finishes the uninstall and tells you the placeholders go when it exits. Previously it left `~/.local/bin` pointing at the release you were removing. ### Changed @@ -88,13 +88,13 @@ In practice a version number says what upgrading will cost you: a third number i ### Fixed -- **`analysis modules available` listed every published version of every module.** The catalogue holds one row per version — that is what `install` resolves against — so a listing of rows was each module's history rather than something to act on, and it grew with every publish. It now shows one line per module: the newest version *your* release can run, which is exactly what `install ` would take. Both commands ask the same question through the same code, so they cannot tell you different things. Older versions stay published and stay installable by naming one. -- **Installing a module asked conda to install packages that were already there.** A module declares everything it needs whether or not the release's own environment already carries it, and most of what a manifest names normally is already there — so installing one meant a network round trip and a solve to be told nothing had to happen. Only what is actually missing is installed now, and a module needing nothing new says so instead of listing seven packages it is not going to touch. What a manifest declares is unchanged, and uninstalling still reasons over the whole list. -- **The published modules page offered libraries as though you could install them.** [Published modules](https://ozankiratli.github.io/PoolSeqFlow/modules-repo/) rendered every catalogue row, so the five libraries appeared beside the modules — and a library is never installed by name. It now lists modules only, newest version each, matching what `available` shows, and names the libraries as reference. The page corrects itself on the next site deploy rather than on upgrade. +- **`analysis modules available` listed every published version of every module.** The catalogue holds one row per version -- that is what `install` resolves against -- so a listing of rows was each module's history rather than something to act on, and it grew with every publish. It now shows one line per module: the newest version *your* release can run, which is exactly what `install ` would take. Both commands ask the same question through the same code, so they cannot tell you different things. Older versions stay published and stay installable by naming one. +- **Installing a module asked conda to install packages that were already there.** A module declares everything it needs whether or not the release's own environment already carries it, and most of what a manifest names normally is already there -- so installing one meant a network round trip and a solve to be told nothing had to happen. Only what is actually missing is installed now, and a module needing nothing new says so instead of listing seven packages it is not going to touch. What a manifest declares is unchanged, and uninstalling still reasons over the whole list. +- **The published modules page offered libraries as though you could install them.** [Published modules](https://ozankiratli.github.io/PoolSeqFlow/modules-repo/) rendered every catalogue row, so the five libraries appeared beside the modules -- and a library is never installed by name. It now lists modules only, newest version each, matching what `available` shows, and names the libraries as reference. The page corrects itself on the next site deploy rather than on upgrade. ### Changed -- **The CHANGELOG says what the public API is.** Semantic Versioning requires a project to declare one and this had not, which left every release number a judgment call. It is what a result depends on — the published table formats, `parameters.config`, `metadata.csv`, and what an analysis folder contains. The command line is documented, not frozen. +- **The CHANGELOG says what the public API is.** Semantic Versioning requires a project to declare one and this had not, which left every release number a judgment call. It is what a result depends on -- the published table formats, `parameters.config`, `metadata.csv`, and what an analysis folder contains. The command line is documented, not frozen. ### Commits @@ -108,7 +108,7 @@ In practice a version number says what upgrading will cost you: a third number i ## [3.1.0] - 2026-09-10 -**This version makes the module system do what it was built for.** 3.0.0 introduced modules that are published, versioned and installed on their own timetable — and then shipped three of them inside the release, which is the one thing that design was meant to avoid. A module in the payload is a module that moves when the pipeline moves. Now nothing ships: `analysis/modules/` is a store that arrives empty and holds what you put in it. +**This version makes the module system do what it was built for.** 3.0.0 introduced modules that are published, versioned and installed on their own timetable -- and then shipped three of them inside the release, which is the one thing that design was meant to avoid. A module in the payload is a module that moves when the pipeline moves. Now nothing ships: `analysis/modules/` is a store that arrives empty and holds what you put in it. **Upgrading leaves you with an empty module store, and that is the whole of the upgrade.** Ask the release you are leaving what it has, then install each into the new one. Nothing else moves: your projects, your configuration and every analysis already published are untouched, and the old release keeps its own modules and still runs. @@ -117,23 +117,23 @@ PoolSeqFlow-3.0.0 analysis modules list # what the old release has PoolSeqFlow analysis modules install mds # and again, into the new one ``` -**`PoolSeqFlow check` now takes a word.** There are two questions — is this installation sound, and is this project sound — and one command answering both meant answering neither well. A bare `check` is refused rather than guessing which you meant, because whichever it picked would leave the other unchecked while reporting success. +**`PoolSeqFlow check` now takes a word.** There are two questions -- is this installation sound, and is this project sound -- and one command answering both meant answering neither well. A bare `check` is refused rather than guessing which you meant, because whichever it picked would leave the other unchecked while reporting success. ### Changed -- **No module ships inside a release, and no library either.** `analysis/modules/` is the install store: gitignored, absent from the tarball, and empty until you install something. `PoolSeqFlow analysis modules install ` puts one there. This is what lets a module be fixed, improved or published without waiting for a pipeline release — and the cost is that a new installation starts with nothing in it and you choose what goes back. -- **A module arrives with the libraries it declares.** The shared arithmetic more than one module wants — effective pool size, gene diversity, per-site allele frequencies, Nei's distance, the chunking — is now five libraries, each published and versioned like a module and installed into `analysis/modules/lib/`. You never ask for one by name: it arrives with whatever needs it, and leaves when nothing installed still declares it. A published result still carries the library code folded into the script that produced it, so a result explains itself whatever the store holds later. -- **`check` is two commands.** `check install` verifies the installation — every tool the release is built to run, and every helper in `bin/`. `check project` verifies a project — that `parameters.config` is current and parses, that `metadata.csv` and the run table parse, and that every command resolves *as that project configures them*, so a tool repointed at a system binary is checked the way the run will call it. Run `check install` from anywhere, including before you have a project; run `check project` from your project directory. -- **`check install` asks the release's own environment rather than your `PATH`.** Every tool it looks for is pinned in `install/environment.yml`, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in — at some other version, on your machine only. That is now reported as `OUTSIDE THE ENVIRONMENT` and fails the check. It is worth catching because it is quiet: the pipeline runs, the results look fine, and nothing reproduces anywhere else. +- **No module ships inside a release, and no library either.** `analysis/modules/` is the install store: gitignored, absent from the tarball, and empty until you install something. `PoolSeqFlow analysis modules install ` puts one there. This is what lets a module be fixed, improved or published without waiting for a pipeline release -- and the cost is that a new installation starts with nothing in it and you choose what goes back. +- **A module arrives with the libraries it declares.** The shared arithmetic more than one module wants -- effective pool size, gene diversity, per-site allele frequencies, Nei's distance, the chunking -- is now five libraries, each published and versioned like a module and installed into `analysis/modules/lib/`. You never ask for one by name: it arrives with whatever needs it, and leaves when nothing installed still declares it. A published result still carries the library code folded into the script that produced it, so a result explains itself whatever the store holds later. +- **`check` is two commands.** `check install` verifies the installation -- every tool the release is built to run, and every helper in `bin/`. `check project` verifies a project -- that `parameters.config` is current and parses, that `metadata.csv` and the run table parse, and that every command resolves *as that project configures them*, so a tool repointed at a system binary is checked the way the run will call it. Run `check install` from anywhere, including before you have a project; run `check project` from your project directory. +- **`check install` asks the release's own environment rather than your `PATH`.** Every tool it looks for is pinned in `install/environment.yml`, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in -- at some other version, on your machine only. That is now reported as `OUTSIDE THE ENVIRONMENT` and fails the check. It is worth catching because it is quiet: the pipeline runs, the results look fine, and nothing reproduces anywhere else. - **The installation directories say what they are for.** `bin/` holds everything that is run rather than sourced, the check scripts included; `lib/` holds what is sourced; `install/` holds the two pinned environment files and nothing else; and `citations/` is new, holding the pipeline's own `references.bib` and the `citations.json` generated from it. Nothing you set moves, and no project is affected. ### Added -- **`PoolSeqFlow check project`** — the configuration and the commands a project names, checked without spending a run. It reads `parameters.config` through Nextflow itself and the two tables through the same parsers step 0 uses, so what it tells you is what a run would tell you. +- **`PoolSeqFlow check project`** -- the configuration and the commands a project names, checked without spending a run. It reads `parameters.config` through Nextflow itself and the two tables through the same parsers step 0 uses, so what it tells you is what a run would tell you. ### Fixed -- **Uninstalling a module could take a package another module still needs.** The keep-list is built by reading one manifest per installed module, and the reader did not terminate its last line — so with several modules in the store the last package of one and the first of the next arrived joined, and a name at that boundary dropped out of the list of things to keep. The same defect applied to libraries. The shipped modules declared no packages, so this could only be reached by a module published against 3.0.0 that declared its own. +- **Uninstalling a module could take a package another module still needs.** The keep-list is built by reading one manifest per installed module, and the reader did not terminate its last line -- so with several modules in the store the last package of one and the first of the next arrived joined, and a name at that boundary dropped out of the list of things to keep. The same defect applied to libraries. The shipped modules declared no packages, so this could only be reached by a module published against 3.0.0 that declared its own. - **Uninstalling a module could take a package the release itself is built on.** A module declares what it needs whether or not the baseline already carries it, so `r-ggplot2` appears in a manifest and in `install/environment-analysis.yml` both. The removal now subtracts the baseline, and nothing the release provides can leave with a module. ### Commits @@ -155,36 +155,36 @@ PoolSeqFlow analysis modules install mds # and again, into the new one **This version is about accessibility.** I tried to do as much engineering as possible using the most common tools and knowledge to make sure that the pipeline can create reproducible results for the users. The outputs now contain, not only the parameter set used in each analysis, there is a list of citations for all the tools used for each portion of the analysis. The pipeline refuses to run when parameter combination is changed mid-run, this is because, one cannot say which one is used for certain analysis if they change it mid-run. This was a reproducibility choice. However, if the user wants to compare multiple parameter combinations, multi-run feature is added. The pipeline handles it in the most efficient way, by finding where the divergent parameter applies and creates separate workflows for each parameter combination. Analysis layer is built to accommodate different ploidies and multiallelic sites. I also improved the manual/website which now has all explanation and history about the tool. -**Upgrading is not automatic and it is not optional reading.** Your project now has two directories instead of one, `RGTags.csv` is replaced by a file that does not convert from it, and the depth ceiling is measured per sample rather than fixed. Run `./PoolSeqFlow migrate_config` before anything else — it carries your settings across, prints the `mv` commands for the files that have to move, and explains each change in place. A configuration from 2.2.0 is now **refused** rather than half-read, so there is no way to discover this partway through a run. +**Upgrading is not automatic and it is not optional reading.** Your project now has two directories instead of one, `RGTags.csv` is replaced by a file that does not convert from it, and the depth ceiling is measured per sample rather than fixed. Run `./PoolSeqFlow migrate_config` before anything else -- it carries your settings across, prints the `mv` commands for the files that have to move, and explains each change in place. A configuration from 2.2.0 is now **refused** rather than half-read, so there is no way to discover this partway through a run. **The pipeline also grew a second half.** `./PoolSeqFlow analysis` runs statistical modules over a finished run's frequency tables: three ship in this release, more install from a repository without waiting for a PoolSeqFlow release, and every published result carries the script that produced it, the assumptions it was computed under, and a PDF report. It runs in its own conda environment and does not touch the pipeline's. ### Changed -- **A project now has two roots and they must be different paths.** `mainDir` holds your reads, reference, `parameters.config` and `metadata.csv`, and is where you launch; `storageDir` holds finished results. Before 3.0 there was one directory doing both, called `projectDir`. `migrate_config` renames the parameter, reports it under `Renamed this release`, and prints the moves for the files that were on the old root — it never moves anything itself. -- **`RGTags.csv` is replaced by `metadata.csv`, and it is not a rename.** The old file carried SAM read-group tags and nothing else. The new one describes the experiment: it names each sample, decides which rows merge into one pool through `RG_Sample`, and carries per-sample pool sizes and adapters. It also has somewhere to put the experiment itself — `exp_` for what you set, `pt_` for what you measured as a response, `cov_` for what you measured alongside — which is what the analysis layer reads. **Nothing converts the old file, and the run stops at step 0 until the new one exists.** Start from `metadata.csv.template`, which documents every column. +- **A project now has two roots and they must be different paths.** `mainDir` holds your reads, reference, `parameters.config` and `metadata.csv`, and is where you launch; `storageDir` holds finished results. Before 3.0 there was one directory doing both, called `projectDir`. `migrate_config` renames the parameter, reports it under `Renamed this release`, and prints the moves for the files that were on the old root -- it never moves anything itself. +- **`RGTags.csv` is replaced by `metadata.csv`, and it is not a rename.** The old file carried SAM read-group tags and nothing else. The new one describes the experiment: it names each sample, decides which rows merge into one pool through `RG_Sample`, and carries per-sample pool sizes and adapters. It also has somewhere to put the experiment itself -- `exp_` for what you set, `pt_` for what you measured as a response, `cov_` for what you measured alongside -- which is what the analysis layer reads. **Nothing converts the old file, and the run stops at step 0 until the new one exists.** Start from `metadata.csv.template`, which documents every column. - **The depth ceiling is measured per sample instead of fixed at 2000.** Step 5 reads each sample's own depth histogram and step 6 caps that sample's BAM before calling, so a shallow library is no longer judged at a deep one's ceiling. `capBAM.maxDepth = -1` is that measurement; `variantCall.maxDepth` becomes a second ceiling on top of it and ships as `0`, which mpileup reads as no limit. Your old `2000` is **not** carried across, and `migrate_config` reports it under `Format changed this release` with the reason. **To reproduce 2.2.0 results exactly: `variantCall.maxDepth = 2000` and `capBAM.maxDepth = 0`.** - **Pool size, ploidy and detection sensitivity can vary per sample**, set in `metadata.csv` through `param_poolSize`. One number for a whole run judged a pool of 10 at a pool of 500's resolution. Rows sharing an `RG_Sample` must agree, and a blank cell counts as a different answer rather than as agreement. -- **Parameters renamed for what they do rather than which tool runs them.** `samtools.*` is `cleanBAM.*`, `bcftools.*` is `variantCall.*`, `diploidy` is `ploidy` — the pipeline was never limited to diploids and the name said otherwise. `migrate_config` carries every value across. +- **Parameters renamed for what they do rather than which tool runs them.** `samtools.*` is `cleanBAM.*`, `bcftools.*` is `variantCall.*`, `diploidy` is `ploidy` -- the pipeline was never limited to diploids and the name said otherwise. `migrate_config` carries every value across. - **A configuration from an older release is refused rather than partly read.** Nextflow reads `parameters.config` as given, so an absent parameter used to interpolate into a path as the literal string `null` and the run started anyway. `run`, `resume`, `dryrun`, `reset`, `analysis complete` and running a module now stop and name `migrate_config`. `clean`, `dryclean` and `migrate_config` itself are unaffected. -- **The `cores` block and the tool `options` strings are computed for you and ship commented out.** They are not gone: `migrate_config` reports them under `Still yours to set`, and uncommenting a line takes one back. Coming from 2.2.0 that is thirteen parameters — the eight `cores` values and the five `options` strings your file already had. +- **The `cores` block and the tool `options` strings are computed for you and ship commented out.** They are not gone: `migrate_config` reports them under `Still yours to set`, and uncommenting a line takes one back. Coming from 2.2.0 that is thirteen parameters -- the eight `cores` values and the five `options` strings your file already had. ### Added - **The analysis layer.** `./PoolSeqFlow analysis ` runs a module over a finished run's frequency tables, in a conda environment of its own that `./PoolSeqFlow analysis install` creates. Three modules ship: **`basicstats`** (site counts, depth, effective pool size and gene diversity per pool), **`association`** (each allele's frequency regressed on a phenotype measured per pool, with a permutation *p*), and **`mds`** (the pools placed by Nei's minimum distance, corrected for sampling, on a classical MDS). Each publishes the script that produced its numbers, the shared library folded in, a `references.bib` for the methods it used, and a PDF report of the whole folder. -- **Every module states what it cannot answer.** A module's manifest carries its assumptions and its limits as text, and the run prints them beside the result — the permutation floor a small design cannot go below, why an MDS distance can be negative and that this is correct, that a capped BAM is biased toward reads mapping earliest. A result that is model-based says so where it is read. +- **Every module states what it cannot answer.** A module's manifest carries its assumptions and its limits as text, and the run prints them beside the result -- the permutation floor a small design cannot go below, why an MDS distance can be negative and that this is correct, that a capped BAM is biased toward reads mapping earliest. A result that is model-based says so where it is read. - **A module store and a repository.** `./PoolSeqFlow analysis modules {list|available|install|uninstall}` installs a module published separately from the pipeline, with its conda packages, checked against a checksum and against what this release can run. Installing one that needs a GPL package tells you so: the pipeline stays Apache-2.0 and each module carries its own license. -- **Every run writes its own citations.** A pipeline run leaves `citations.txt` in `Output/`, naming each tool it actually invoked with the version that tool reported — probed at run time, so a tool repointed at a system binary is recorded as what ran rather than as what shipped. A published analysis carries `CITATIONS.md` and `references.bib` beside its results, covering the methods each module used as well as the software. What to cite stops being something you reconstruct months later. +- **Every run writes its own citations.** A pipeline run leaves `citations.txt` in `Output/`, naming each tool it actually invoked with the version that tool reported -- probed at run time, so a tool repointed at a system binary is recorded as what ran rather than as what shipped. A published analysis carries `CITATIONS.md` and `references.bib` beside its results, covering the methods each module used as well as the software. What to cite stops being something you reconstruct months later. - **Multi-run projects.** One data source, several parameter sets, described in `runs.csv` with `multiRun = true`. Runs sharing an input share the work rather than repeating it, and step 0 prints what is shared before any compute is spent. -- **`./PoolSeqFlow dryrun` and `dryclean`** — check the configuration and preview what a run would do, writing nothing into either root, then remove the preview. -- **`./PoolSeqFlow init`** — populate an empty directory with the template configuration and metadata, ready to edit. +- **`./PoolSeqFlow dryrun` and `dryclean`** -- check the configuration and preview what a run would do, writing nothing into either root, then remove the preview. +- **`./PoolSeqFlow init`** -- populate an empty directory with the template configuration and metadata, ready to edit. - **Several versions install side by side.** Each release has its own environment and payload, so an in-flight project can finish on the release it started on. `./PoolSeqFlow list` shows what is installed and `uninstall` asks which. - **A test suite**, nineteen suites split by what they cost, so a change runs only the cases it can reach: `test/run_tests.sh --changed` picks them from the file you edited. A module ships its own cases inside its own directory. - **Both conda environments are pinned to exact builds.** `install/environment-analysis.yml` was a hand-written specification and is now an export of an environment the full suite passed against, as `install/environment.yml` already was. ### Fixed -- **FastQC was given up to eight threads and needed two.** Its `-t` counts *files* processed simultaneously, not threads per file, and step 2 hands it one pair. Measured on a pair of 2M-read files, `-t 2` is 1.93× faster than `-t 1` while `-t 4`, `-t 6` and `-t 8` are no faster at all — and every thread past the second costs roughly 250 MB of resident memory, twice per sample. On a memory-constrained machine that is the difference between a run finishing and being killed. The documented ladder always said two; the code had drifted. +- **FastQC was given up to eight threads and needed two.** Its `-t` counts *files* processed simultaneously, not threads per file, and step 2 hands it one pair. Measured on a pair of 2M-read files, `-t 2` is 1.93x faster than `-t 1` while `-t 4`, `-t 6` and `-t 8` are no faster at all -- and every thread past the second costs roughly 250 MB of resident memory, twice per sample. On a memory-constrained machine that is the difference between a run finishing and being killed. The documented ladder always said two; the code had drifted. - **`DepthProfile` declared one of the three files it publishes**, so the depth histogram and the depth report were outside Nextflow's tracking and outside the skip that avoids rebuilding them. - **Two data-loss defects in `atomic_mv.sh`**, both found by reading rather than by a failure: moving a directory onto an existing one could lose a file that only the destination had. - **`snpEff`'s configuration file is settable through `parameters.config`** rather than fixed, and multiple annotation databases are supported. @@ -317,26 +317,26 @@ PoolSeqFlow analysis modules install mds # and again, into the new one ## [2.2.0] - 2026-08-16 -**This release changes results.** `vcffilter.minDP` previously had no effect on the output at all; it now removes sites. Read the first entry under Changed before upgrading a project that has outputs you intend to keep — and expect step 0 to stop your next run, because the analysis parameters have changed. That is the guardrail working; the report names the folders to delete. +**This release changes results.** `vcffilter.minDP` previously had no effect on the output at all; it now removes sites. Read the first entry under Changed before upgrading a project that has outputs you intend to keep -- and expect step 0 to stop your next run, because the analysis parameters have changed. That is the guardrail working; the report names the folders to delete. Alongside that: a documentation site, an installation check that fails an install rather than letting a half-built environment through, and a release process that publishes a verified download so nobody has to clone the repository to use the pipeline. ### Changed -- **`vcffilter.minDP` now filters, where before it did nothing.** The depth filter was `vcftools --minDP`, which expresses a failed genotype-level test by rewriting `FORMAT/GT` and nothing else — it never touches `AD` or `DP`, and it never removes a site. Because step 7's major-allele normalization sets every `GT` to `./.` before that filter runs, and because frequency conversion reads `AD` rather than `GT`, the setting had no path to the output: running the old command with `--minDP 20` and with `--minDP 50` produced byte-identical frequency tables. It is now `bcftools view -e "FMT/DP, built with MkDocs Material and published from `main` by GitHub Actions. It goes well beyond the README: when Pool-seq fits and when it does not, why the pipeline replaces Nextflow's `-resume` and what that costs, the full filter chain from alignment flags to frequency conversion with what each stage removes, and how to read the frequency tables. Broken internal links fail the build. -- **`./PoolSeqFlow check`** — verifies an installation and reports what it finds. Every command the pipeline invokes, with the version each reports; every helper in `bin/`, present *and* executable, since they are called by bare name off `nextflow.config`'s `PATH` and a lost executable bit fails mid-run; and that `parameters.config` parses. With a config present the tool list is read from `params.software` through `nextflow config`, so a command repointed at a system binary is checked as configured rather than as shipped. It also runs at the end of `install` and **fails the install** if anything is missing — an environment that was created but is short a tool would otherwise surface partway through step 4, hours in. -- **`./PoolSeqFlow cite`** — prints the citation for the copy you have, with its version filled in, and explains which DOI to use. -- **A release workflow.** Tagging `v*` publishes a curated tarball: the pipeline only, in a versioned directory, built with `git archive` so the executable bit on `./PoolSeqFlow` and `bin/*` comes from the git index rather than the runner's umask. What ships is decided by `export-ignore` in `.gitattributes`, and the workflow asserts both directions — required files present, repository furniture absent — along with the executable bits, shell syntax and the version the extracted wrapper reports. It refuses to publish unless the tag, both version strings in `./PoolSeqFlow` and a changelog section all agree. `SHA256SUMS` is attached, and `PoolSeqFlow.tar.gz` carries a stable name for scripted installs. -- **`config_migrate.sh` handles renamed parameters.** A rename was previously two unrelated events — one `DROPPED`, one `NEW` — and your tuned value silently reverted to the template default. Renames now carry the value across and report it as `Renamed this release`. If a rename also changes what the parameter *means*, adding it to `reformatted()` makes the template value win while still surfacing the change. +- **`./PoolSeqFlow check`** -- verifies an installation and reports what it finds. Every command the pipeline invokes, with the version each reports; every helper in `bin/`, present *and* executable, since they are called by bare name off `nextflow.config`'s `PATH` and a lost executable bit fails mid-run; and that `parameters.config` parses. With a config present the tool list is read from `params.software` through `nextflow config`, so a command repointed at a system binary is checked as configured rather than as shipped. It also runs at the end of `install` and **fails the install** if anything is missing -- an environment that was created but is short a tool would otherwise surface partway through step 4, hours in. +- **`./PoolSeqFlow cite`** -- prints the citation for the copy you have, with its version filled in, and explains which DOI to use. +- **A release workflow.** Tagging `v*` publishes a curated tarball: the pipeline only, in a versioned directory, built with `git archive` so the executable bit on `./PoolSeqFlow` and `bin/*` comes from the git index rather than the runner's umask. What ships is decided by `export-ignore` in `.gitattributes`, and the workflow asserts both directions -- required files present, repository furniture absent -- along with the executable bits, shell syntax and the version the extracted wrapper reports. It refuses to publish unless the tag, both version strings in `./PoolSeqFlow` and a changelog section all agree. `SHA256SUMS` is attached, and `PoolSeqFlow.tar.gz` carries a stable name for scripted installs. +- **`config_migrate.sh` handles renamed parameters.** A rename was previously two unrelated events -- one `DROPPED`, one `NEW` -- and your tuned value silently reverted to the template default. Renames now carry the value across and report it as `Renamed this release`. If a rename also changes what the parameter *means*, adding it to `reformatted()` makes the template value win while still surfacing the change. ### Fixed @@ -359,29 +359,29 @@ Alongside that: a documentation site, an installation check that fails an instal ## [2.1.1] - 2026-08-15 -A stability release. Nothing new to configure and no change to how a run is invoked — this closes the gaps where a result could be quietly wrong or quietly irreproducible. An existing `parameters.config` needs no changes. +A stability release. Nothing new to configure and no change to how a run is invoked -- this closes the gaps where a result could be quietly wrong or quietly irreproducible. An existing `parameters.config` needs no changes. ### Added - **Step 0 refuses to run when the analysis parameters changed** since the existing outputs were produced. Completed steps are skipped by looking for output files, not by checking what produced them, so a changed `poolSize` or filter threshold would otherwise leave one output folder holding results from two different settings. The values behind a set of outputs are recorded in `.poolseqflow_params` and mirrored to a read-only `Output/run_parameters.txt`. Path, resource and software parameters are excluded; anything added in a later release counts as analysis-affecting until decided otherwise. - **Step 0 refuses to run when `RGTags.csv` changed** after the file was consumed. The tags are written into the BAMs at step 4 and the row order is fixed into the VCF at step 6, and neither is re-derived once its output exists. The report separates a changed tag value (invalidates `Ready/`, `VCF/`, `Frequencies/`) from a reordering (invalidates `VCF/`, `Frequencies/` only) and names the folders to delete; deleting them is what clears the check. Projects whose outputs predate this release adopt their current file as the baseline, with a note to confirm it against the BAM headers. - **Sample columns follow `RGTags.csv` row order**, so results come out arranged the way the samples were laid out rather than however they sort as strings. Where several rows share an `SM`, the merged column takes the position of the first of them. -- **Duplicate `ID` detection.** A row is looked up by `ID` and only the first match is read, so a repeated `ID` silently discarded the later rows and gave that sample the wrong tags — producing a perfectly valid BAM that nothing downstream could flag. +- **Duplicate `ID` detection.** A row is looked up by `ID` and only the first match is read, so a repeated `ID` silently discarded the later rows and gave that sample the wrong tags -- producing a perfectly valid BAM that nothing downstream could flag. - **CRLF repair for `RGTags.csv`.** A file saved from Excel on Windows carries a stray carriage return into the last tag of every row; it previously failed with `Invalid tag 'PU'`, which names nothing useful. Step 0 now rewrites the file with Unix line endings, preserving permissions and ownership, and reports `RGTAGS LINE ENDING CHECK: FIXED`. -- **`bin/atomic_mv.sh`** — moves that cross a filesystem boundary now stage through a `.part` file and rename into place. +- **`bin/atomic_mv.sh`** -- moves that cross a filesystem boundary now stage through a `.part` file and rename into place. ### Changed -- **`workDir` is now under `mainDir`.** It was a relative path, so the scratch/permanent split the pipeline documents was not actually in effect — work directories landed wherever the pipeline was launched from. +- **`workDir` is now under `mainDir`.** It was a relative path, so the scratch/permanent split the pipeline documents was not actually in effect -- work directories landed wherever the pipeline was launched from. - **Variant calling receives its BAMs in a defined order.** `collect()` emitted them in task-completion order, so the sample column order of the VCF varied between runs on identical input; three consecutive runs gave three different orders. The sort keys on the sample id, because the file paths begin with Nextflow's work-directory hash and sorting those is no better than chance. -- **All 22 cross-filesystem moves are atomic.** A plain `mv` across filesystems is a copy followed by an unlink, so a job killed mid-move left a truncated file under its final name — which the existence-based skip logic then accepted as a completed step. +- **All 22 cross-filesystem moves are atomic.** A plain `mv` across filesystems is a copy followed by an unlink, so a job killed mid-move left a truncated file under its final name -- which the existence-based skip logic then accepted as a completed step. - **`reset` is behind a typed `DELETE_MY_ANALYSIS` confirmation** and also clears `.poolseqflow_params` and `.poolseqflow_rgtags`, which would otherwise fail the next run's checks against outputs that no longer exist. - **`clean` and `reset` resolve paths through `nextflow config`** rather than parsing `parameters.config` as text. Values are interpolated, so text matching returned the wrong path. - `RGTags.csv.template` now shows the replicate and `SM`-merge pattern, with `DS` carrying a per-replicate descriptor instead of repeating the sample name. ### Fixed -- The sensitivity formula in `bin/filterFalsePositives.sh -h` was missing a factor of two. It read `s = 1 / ([DIPLOIDY] / [POOLSIZE per SAMPLE])` and should read `s = 1 / 2 * ([DIPLOIDY] / [POOLSIZE per SAMPLE])`. Help text only — anyone who ran the script by hand and followed it would have passed the wrong `-s`. +- The sensitivity formula in `bin/filterFalsePositives.sh -h` was missing a factor of two. It read `s = 1 / ([DIPLOIDY] / [POOLSIZE per SAMPLE])` and should read `s = 1 / 2 * ([DIPLOIDY] / [POOLSIZE per SAMPLE])`. Help text only -- anyone who ran the script by hand and followed it would have passed the wrong `-s`. ### Commits @@ -401,7 +401,7 @@ Resource allocation is now declared to Nextflow rather than only passed to the t ### Added -- **`./PoolSeqFlow migrate_config`** — rebuilds `parameters.config` from the current template, backs the original up, carries across every setting whose parameter still exists, and reports what it kept, what is new, what the pipeline now computes for itself, and what it dropped. It refuses to carry a value the template derives, so it cannot reintroduce a stale `snpEff.db` or a hand-set thread count. The report is a starting point: a parameter whose behavior changed while its value still looks ordinary will be carried across, so compare against the template afterwards. +- **`./PoolSeqFlow migrate_config`** -- rebuilds `parameters.config` from the current template, backs the original up, carries across every setting whose parameter still exists, and reports what it kept, what is new, what the pipeline now computes for itself, and what it dropped. It refuses to carry a value the template derives, so it cannot reintroduce a stale `snpEff.db` or a hand-set thread count. The report is a starting point: a parameter whose behavior changed while its value still looks ordinary will be carried across, so compare against the template afterwards. - **Every process declares `cpus`**, so Nextflow schedules against real requirements instead of assuming one core per task. Previously three `Align` tasks each using ~2.2 cores ran concurrently on an 8-core machine with `cpus=1` recorded for each. - `params.memory`, feeding `resourceLimits` alongside `params.threads`, so one place sizes a run. @@ -416,7 +416,7 @@ Resource allocation is now declared to Nextflow rather than only passed to the t ### Fixed -- **`parameters.config.template` was missing parameters the pipeline requires** — `annotate`, `snpEff.runOptions`, `rgTagsPath`, `diploidy` — and carried a different `vcftools.minDP` and different report directory names. A configuration built from it failed step 0 with `RGTAGS VERIFICATION: STATUS=FAIL`. The template is now generated from the reference configuration and resolves identically to it. +- **`parameters.config.template` was missing parameters the pipeline requires** -- `annotate`, `snpEff.runOptions`, `rgTagsPath`, `diploidy` -- and carried a different `vcftools.minDP` and different report directory names. A configuration built from it failed step 0 with `RGTAGS VERIFICATION: STATUS=FAIL`. The template is now generated from the reference configuration and resolves identically to it. - **Step 7 created the wrong output directory.** `SortRefAltByFrequency` ran `mkdir -p` on the frequencies folder and then moved into the VCF folder, which only worked because step 6 had created it first. - `parameters.config` contained the `dir { }` block and the reference path assignments twice, byte-identical. @@ -438,7 +438,7 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba - **Requires Nextflow 26** (`26.04.6`). The `cleanup { }` block in `nextflow.config` was invalid and is rejected by the stricter config parser; the pipeline could not start on 26 before this release. - **Requires Trim Galore 2.x** (`2.3.0`). The bundled FastQC engine and `--basename` naming are both assumed. -- **`parameters.config` is no longer tracked in git.** Copy `parameters.config.template` and re-apply your settings — see *Upgrading from an earlier release* in the README. Carrying an older file over causes a later step to fail with a bare `null: command not found`. +- **`parameters.config` is no longer tracked in git.** Copy `parameters.config.template` and re-apply your settings -- see *Upgrading from an earlier release* in the README. Carrying an older file over causes a later step to fail with a bare `null: command not found`. - **Trimmed read filenames changed** from `_R1_val_1.fq.gz` to `_val_1.fq.gz`. Trimming is redone once on the first run after upgrading. - **SnpEff database name is derived from the GFF filename** instead of being set by hand, so an existing database directory is not found and is rebuilt. - **`params.fastqc.memory` is now a plain number of megabytes** (`2048`). The previous `"2G"` was rejected by FastQC, which silently fell back to its 512 MB default. @@ -447,7 +447,7 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba ### Added - Automatic core allocation: a `params.cores` block derives every tool's thread count from `params.threads`. Trim Galore is costed on its true footprint (`--cores N` runs N+4 threads), and `threads = 1` forces everything single-core. -- `params.trim_galore.autodetect` — when `true`, no adapter is passed and Trim Galore detects it; when `false`, both adapter sequences are required. +- `params.trim_galore.autodetect` -- when `true`, no adapter is passed and Trim Galore detects it; when `false`, both adapter sequences are required. - Step 0 now validates trimming parameters, failing early if auto-detection is off and the adapters are missing or are not DNA sequences. - `unzip` added to `environment.yml` and to `params.software`, so step 0 verifies it. It was always required by the clipping step but never declared. - Trim Galore 2.x `*_trimming_report.json` files are kept alongside the `.txt` reports. @@ -461,7 +461,7 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba - **Alignment and coverage reports paired BAMs with indexes by position** rather than by sample; the two channels are now joined on `pair_id`. - **`SkipGFFCheck` was unparseable** because of a duplicated `script:` label, so `annotate = false` could not run at all on Nextflow 26. - **Resuming a completed run failed at the frequency step**, which linked a bare filename and created a self-referential symlink. -- **`parameters.config.template` did not parse on Nextflow 26** — it used `${mainDir}` instead of `${params.mainDir}` in nine places. +- **`parameters.config.template` did not parse on Nextflow 26** -- it used `${mainDir}` instead of `${params.mainDir}` in nine places. - README documented parameters that do not exist (`refGenome`, `refGFF`, `ploidy`) and placed the data directory under the wrong root. --- @@ -473,7 +473,7 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba --- -## [1.0.0] - 2026-03-26 — Initial Public Release +## [1.0.0] - 2026-03-26 -- Initial Public Release [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19245612.svg)](https://doi.org/10.5281/zenodo.19245612) @@ -483,35 +483,35 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba - End-to-end Pool-seq analysis workflow (`poolseqflow.nf`) with 9 modular steps - Wrapper script (`PoolSeqFlow`) exposing `install`, `run`, `resume`, `clean`, and `reset` subcommands -**Step 0 — Environment verification** +**Step 0 -- Environment verification** - Pre-run checks for all required input files, folder structure, RGTags CSV format, and software dependencies - Generates `Reports/0_verify_environment.txt` -**Step 1 — Reference indexing** +**Step 1 -- Reference indexing** - Builds BWA, SAMtools (`.fai`), and SnpEff indices from a gzipped reference FASTA and GFF -**Step 2 — Quality control and trimming** +**Step 2 -- Quality control and trimming** - FastQC assessment of raw reads - Adapter trimming via Trim Galore with user-specified adapter sequences - Automated per-cycle base-composition analysis of FastQC reports -- Intelligent hard-clipping via Cutadapt driven by A/T and G/C imbalance thresholds — no manual parameter tuning required +- Intelligent hard-clipping via Cutadapt driven by A/T and G/C imbalance thresholds -- no manual parameter tuning required -**Step 3 — Alignment** +**Step 3 -- Alignment** - Paired-end alignment to the reference genome using BWA-MEM -**Step 4 — BAM post-processing** -- Full SAMtools-based cleanup: name-sort → fixmate → coord-sort → markdup → addreplacerg → filter → index +**Step 4 -- BAM post-processing** +- Full SAMtools-based cleanup: name-sort -> fixmate -> coord-sort -> markdup -> addreplacerg -> filter -> index - Configurable alignment filter flags (`samFlags.filter`, `samFlags.required`) -**Step 5 — Alignment reporting** +**Step 5 -- Alignment reporting** - Per-sample alignment statistics via `bamtools stats` - Coverage summaries via `samtools coverage` -**Step 6 — Variant calling** +**Step 6 -- Variant calling** - Multi-sample SNP and indel calling with BCFtools mpileup + call in multiallelic mode - Outputs VCFs with per-sample `AD` and `DP` FORMAT fields -**Step 7 — VCF to allele frequency tables** +**Step 7 -- VCF to allele frequency tables** - Major-allele normalization: VCF re-encoded so the major allele is always REF - Multiallelic site support throughout variant calling and frequency conversion - Ploidy- and pool-size-aware minimum frequency filter: $f_{\min} = 1 / (2 \times ploidy \times poolSize)$ @@ -519,12 +519,12 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba - SNP / INDEL split - Export to tab-separated allele frequency tables -**Step 8 — Variant annotation (optional)** +**Step 8 -- Variant annotation (optional)** - SnpEff-based functional annotation, toggled via `params.annotate` **Resume logic** - Custom filesystem-based resume strategy using symbolic links between `mainDir` (working directory) and `projectDir` (permanent storage) -- Completed steps are skipped based on presence of permanent output files — resilient to job timeouts, reboots, and `work/` directory cleanups +- Completed steps are skipped based on presence of permanent output files -- resilient to job timeouts, reboots, and `work/` directory cleanups - Supports HPC environments where compute nodes and storage are on separate filesystems **Configuration** diff --git a/CLAUDE.md b/CLAUDE.md index 94892ee..68238c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,7 @@ The first clause earns its place. The second is a decision and rides in on its b ### Why this matters more than tidiness -**A design decision left in a comment reads to a future session as a current constraint, and it will be argued from against what you are actually being asked for.** Z, 2026-08-30: *"In multiple occasions, the previous design decisions have skewed your interpretation of my asks, we had to spend way too much time on simple tasks."* Stale reasoning in the source is not clutter — it is misinformation with authority. A decision we changed five times is also confusing to a reviewer and must not end up in the source or in the manual. +**A design decision left in a comment reads to a future session as a current constraint, and it will be argued from against what you are actually being asked for.** Z, 2026-08-30: *"In multiple occasions, the previous design decisions have skewed your interpretation of my asks, we had to spend way too much time on simple tasks."* Stale reasoning in the source is not clutter -- it is misinformation with authority. A decision we changed five times is also confusing to a reviewer and must not end up in the source or in the manual. ### The third category, which a binary test will wrongly delete @@ -22,10 +22,10 @@ Not every "why" is a decision. The cut is: **does the code become inexplicable w | | | |---|---| -| **Inexplicable — keep** | `// The trailing return 0 is load-bearing under set -e.` Without it the line reads as dead code and someone deletes it. | -| **Unjustified — cut** | `// Removed rather than kept as PoolSeqFlow-, so the environment ships built fresh...` A choice among alternatives that all work. | +| **Inexplicable -- keep** | `// The trailing return 0 is load-bearing under set -e.` Without it the line reads as dead code and someone deletes it. | +| **Unjustified -- cut** | `// Removed rather than kept as PoolSeqFlow-, so the environment ships built fresh...` A choice among alternatives that all work. | -A constraint that makes an otherwise-pointless line explicable is *what the code does*. A constraint that defends a choice is a decision. Losing this distinction reintroduces bugs — `_sort_fp_dq.vcf` sat in `Output/VCF` on every run since 1.0 because the reason for a deletion's placement was never written down. +A constraint that makes an otherwise-pointless line explicable is *what the code does*. A constraint that defends a choice is a decision. Losing this distinction reintroduces bugs -- `_sort_fp_dq.vcf` sat in `Output/VCF` on every run since 1.0 because the reason for a deletion's placement was never written down. ### The check @@ -35,7 +35,7 @@ Grep before calling a file done. `dev/scripts/comment-audit.sh` does it: on purpose | deliberately | rather than | because | would otherwise | so that | which is why ``` -Each hit is a candidate, not a verdict — some are legitimate. This is a review aid and never a test case: a hard failure would train the next session to reword around the words instead of deleting the decision. +Each hit is a candidate, not a verdict -- some are legitimate. This is a review aid and never a test case: a hard failure would train the next session to reword around the words instead of deleting the decision. ### Three destinations @@ -45,11 +45,11 @@ Each hit is a candidate, not a verdict — some are legitimate. This is a review | `manual/PoolSeqFlow-manual.md` | anything that changes what a result **means**, or that a person interpreting output needs. The manual is authoritative; nothing in it is repeated in the source. | | `.claude/development-notes/` | the record of how it got here. Design churn, alternatives dropped, measurements, who decided what and when. | -**No source file references the notes.** A note is dated and is not updated to follow the code, so a comment pointing at one imports a description of the code as it used to be — the exact failure the rule above exists to prevent. **No source file references the test suite** — the dependency runs tests → scripts, never back. A test may say what it is testing; a script may not say it is tested. +**No source file references the notes.** A note is dated and is not updated to follow the code, so a comment pointing at one imports a description of the code as it used to be -- the exact failure the rule above exists to prevent. **No source file references the test suite** -- the dependency runs tests -> scripts, never back. A test may say what it is testing; a script may not say it is tested. ### Mid-feature is not the deadline, but the feature has one -**A comment carrying more than the code below it needs is tolerable while a feature is being built, and is not tolerable when it ships.** Z, 2026-08-31: *"During development of a feature it is fine, but we should not forget about cleaning them at the end of the development."* So a working comment may explain more than it should for as long as the stage is open — and **every stage ends with a pass over the comments it added**, trimmed to what the code needs. Put that pass in the stage's own checklist when the stage starts, or it is the thing that gets skipped. +**A comment carrying more than the code below it needs is tolerable while a feature is being built, and is not tolerable when it ships.** Z, 2026-08-31: *"During development of a feature it is fine, but we should not forget about cleaning them at the end of the development."* So a working comment may explain more than it should for as long as the stage is open -- and **every stage ends with a pass over the comments it added**, trimmed to what the code needs. Put that pass in the stage's own checklist when the stage starts, or it is the thing that gets skipped. ### Not source, and out of scope @@ -57,11 +57,11 @@ Each hit is a candidate, not a verdict — some are legitimate. This is a review ### `dev/` and `test/` follow the opposite rule: keep everything but abandoned ideas -**They carry a different role in the project.** Nothing in them ships, nothing in them computes a published number, and neither is read by someone trying to understand what the pipeline does — they are read by someone trying to understand *whether it is right*. There the reasoning IS the content: a measurement with no account of what it measured is a number nobody can act on, and a case with no record of the bug it guards is a case the next session deletes as arbitrary. +**They carry a different role in the project.** Nothing in them ships, nothing in them computes a published number, and neither is read by someone trying to understand what the pipeline does -- they are read by someone trying to understand *whether it is right*. There the reasoning IS the content: a measurement with no account of what it measured is a number nobody can act on, and a case with no record of the bug it guards is a case the next session deletes as arbitrary. So the source filter is **inverted** here. Keep the why. Keep the measurements, the alternatives weighed, the constraint that made a harness awkward, the reason an assertion is phrased the way it is. A `dev/` script that explains its own methodology at length is doing its job. -**The one thing to cut is an abandoned idea** — an approach tried and dropped, described as though it were still how things work. That is the failure the source rule exists to prevent and it is just as damaging here: it reads to a later session as a current constraint. The test is not "is this a decision" but **"does this describe something that is still true?"** A superseded approach whose *finding* still governs the current design is not abandoned — keep it, and say which part is live. +**The one thing to cut is an abandoned idea** -- an approach tried and dropped, described as though it were still how things work. That is the failure the source rule exists to prevent and it is just as damaging here: it reads to a later session as a current constraint. The test is not "is this a decision" but **"does this describe something that is still true?"** A superseded approach whose *finding* still governs the current design is not abandoned -- keep it, and say which part is live. Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different role in the project."* @@ -69,28 +69,37 @@ Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different ## Other standing rules +- **`dev/RELEASING.md` is a living procedure, not a record.** Correct it when a release teaches you something; it carries no date and describes no particular version. It holds only what a person does: a command, what it must show, and what to do when it does not. Everything else belongs in `.claude/development-notes/` or in the script's own header, and a step must never point into a dated note -- that is how it came to promise an audit that did not exist. +- **Several release gates live in the suite rather than in that document**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`), the analysis versions (`check-analysis-versions.sh`) and the version-consistency case all run inside `00_static`. A green suite is how they report. Do not add a step for one. +- **The manual keeps real typographic characters. Everything else in the repository is ASCII.** `manual/PoolSeqFlow-manual.md` is rendered to a website and read as prose, so an arrow is `→`, a reverse arrow `←`, a product `×`, an ellipsis `…`, a negative `−`, a comparison `≥`. **Never ASCII-ify the manual**, and never let a sweep do it: a pass asked to remove em dashes has twice taken 53 arrows, 29 products and 16 ellipses with them. Two carve-outs inside the manual, and only two. **Em dashes are out on style** -- rewrite the sentence, do not substitute `--`. And **a command someone copies and runs stays ASCII**: that is the test, not whether something sits in a fenced block, because the manual fences its diagrams and formulas too. `distance = (J_A + J_B) / 2 − J_AB` and the labels inside a ```mermaid block keep their glyphs. Every other file, including the templates, the notes and this one, is ASCII as the global rule says. + +- **A ```mermaid diagram exists on the site only.** `mkdocs.yml` carries the custom fence for it and material loads the renderer from a CDN, so the diagram needs a network to draw and the shipped `manual/PoolSeqFlow-manual.md` shows its `graph TD` source. Use it for what ASCII cannot draw. Anything a reader must have in the tarball stays as text, or as an `.svg` under `manual/assets/` like the five depth-histogram figures. +- **The manual names what happens, never the step number.** Step numbers are the maintainer's bookkeeping and a reader has no model of them: write "the BAM cleanup", "the pileup filter", "variant calling". There is exactly one place a number is the reader's own: the log directory a run writes, `Logs/4_clean/`, so a number may appear when naming that path, and as a label in the Steps reference where the step itself is the subject. Never in a heading, and never as the subject of a sentence a reader is meant to act on. +- **A manual heading is a name, not a claim.** `Configuration is a file, never a flag` makes an anchor that reads as a slogan and cannot be linked to soberly, and a heading carrying a step number bakes bookkeeping into a URL. Name the thing; put the claim in the prose under it. Renaming one means fixing every link to it in the same edit, which `build_docs.py --check` and `00_static` both catch. - **Never `git push`.** Z publishes; nothing else does, for any reason. -- **Never `git commit` unless asked**, and a commit instruction covers only the work that existed when it was given. Do the work, leave it uncommitted, say what changed in prose, stop. The uncommitted tree is the review surface — never build a diff artifact or a summary page as a substitute. +- **Never `git commit` unless asked**, and a commit instruction covers only the work that existed when it was given. Do the work, leave it uncommitted, say what changed in prose, stop. The uncommitted tree is the review surface -- never build a diff artifact or a summary page as a substitute. - **Use the `Edit` tool for file changes, never a script that rewrites a file.** Z reviews side by side in the IDE diff view as it lands. Announced mechanical renames via `sed` are the one exception. - **No hard wrapping in markdown.** One paragraph is one line, in every `.md`. -- **American English, everywhere.** Prose, comments, identifiers, user-facing output. The repository was swept on 2026-09-08 and `dev/scripts/americanize.py` is what finds a regression; run it before a release. Matching the surrounding file is the usual reason a Britishism appears, so it spreads from whatever it lands next to — which is why the sweep is a tool and not a habit. Two things are NOT errors and the script says so: `catalogue` is current American English and is this project's name for a specific thing, and `analyses` is the plural of `analysis` far more often than it is the British verb. +- **American English, everywhere.** Prose, comments, identifiers, user-facing output. The repository was swept on 2026-09-08 and `dev/scripts/americanize.py` is what finds a regression; run it before a release. Matching the surrounding file is the usual reason a Britishism appears, so it spreads from whatever it lands next to -- which is why the sweep is a tool and not a habit. Two things are NOT errors and the script says so: `catalogue` is current American English and is this project's name for a specific thing, and `analyses` is the plural of `analysis` far more often than it is the British verb. - **Fail loudly or document; never automate away a decision.** Compute the default, never remove the knob. -- **Never point anything at `Project/`** — real data, hours of compute, tens of GB. The test sandbox's `guard_path` refuses anything outside `TEST_TMPDIR` and anything inside the repository. +- **Never point anything at `Project/`** -- real data, hours of compute, tens of GB. The test sandbox's `guard_path` refuses anything outside `TEST_TMPDIR` and anything inside the repository. - **No legacy fallbacks in `.nf` files.** A dropped parameter is handled by `bin/config_migrate.sh`. ## Verifying a comment pass -Not one line of code may change. Per file, diff the non-comment lines against `HEAD`. For Python, a docstring is not a `#` comment and the line count misleads — prove it with an AST comparison that strips docstrings. +Not one line of code may change. Per file, diff the non-comment lines against `HEAD`. For Python, a docstring is not a `#` comment and the line count misleads -- prove it with an AST comparison that strips docstrings. -Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts — files linted, cases passed — against the run before it rather than only the exit status: a filter that matches nothing also reports success. Neither number is written down here, because both move with every file added. +Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts -- files linted, cases passed -- against the run before it rather than only the exit status: a filter that matches nothing also reports success. Neither number is written down here, because both move with every file added. -**Lint the tree without `modules/`**, and expect zero errors and zero warnings: +**Lint through the suite, at zero errors and zero warnings.** Seven seconds: ``` -nextflow lint analysis analysis.nf dryrun.nf poolseqflow.nf scripts +bash test/run_tests.sh --suite 00_static --case lint ``` -`nextflow lint .` **cannot pass and is not the command.** A module's `main.nf` imports the frame as `'../../lib/nf/plan.nf'` — correct from `analysis/modules//`, where it is installed, and unresolvable from `modules//`, where it is written. The path is right and the tree is wrong for it, so linting from the repository root reports one `Invalid include source` per import on every module. **`00_static` is what lints them**: it assembles a store layout in a sandbox and lints that, which is the only place those imports resolve. +**Do not write the `nextflow lint ...` command out by hand.** It needs `nextflow`, which lives in the release's conda environment and nowhere a plain shell will find it -- the suite locates the environment itself. And a hand-written path list drifts: the copy that used to be here had lost `lib`, `bin`, `install` and `nextflow.config`, so it linted a subset of what the suite does while reading as the authority. + +The two cases behind that filter are `nextflow lint is clean`, which lints the tree without `modules/`, and `every module lints in the store layout`. **`nextflow lint .` cannot pass and is not the command.** A module's `main.nf` imports the frame as `'../../lib/nf/plan.nf'` -- correct from `analysis/modules//`, where it is installed, and unresolvable from `modules//`, where it is written. The path is right and the tree is wrong for it, so linting from the repository root reports one `Invalid include source` per import on every module. The second case is what lints them, by assembling a store layout in a sandbox, which is the only place those imports resolve. ## What to run while building @@ -100,29 +109,29 @@ nextflow lint analysis analysis.nf dryrun.nf poolseqflow.nf scripts bash test/run_tests.sh --suite 07_analysis --case citation ``` -`--case` matches any part of a case name and filters at dispatch, so the suite's fixtures are still built by the cases that define them. Three cases in under a minute against ten for the suite they live in — which is the difference between checking a change and avoiding checking it. +`--case` matches any part of a case name and filters at dispatch, so the suite's fixtures are still built by the cases that define them. Three cases in under a minute against ten for the suite they live in -- which is the difference between checking a change and avoiding checking it. -**The full suite belongs to a release**, not to a step. `--fast` is the cheap sweep when a change reaches beyond its own step — the wrapper, packaging, a shared library — and a whole `--suite ` is for when the step *is* that suite. +**The full suite belongs to a release**, not to a step. `--fast` is the cheap sweep when a change reaches beyond its own step -- the wrapper, packaging, a shared library -- and a whole `--suite ` is for when the step *is* that suite. | you changed | run | |---|---| -| `bin/` | `03_helpers` — except the three `check_*.sh`, which are `02_launcher` | +| `bin/` | `03_helpers` -- except the three `check_*.sh`, which are `02_launcher` | | `PoolSeqFlow`, install/uninstall, the check scripts | `02_launcher` | | `bin/config_migrate.sh`, the templates | `01_migrate` | | step 0, parameter resolution, the change guards | `05_guards` | | wiring, channels, promotion, a step's script | `04_pipeline` | | `dryrun.nf`, `dryrun`/`dryclean` | `06_dryrun` | | version strings, packaging, syntax | `00_static` | -| a module library under `modules/lib/` | `analysis_rlib` — no JVM, 3 seconds | +| a module library under `modules/lib/` | `analysis_rlib` -- no JVM, 3 seconds | | `analysis/lib/nf/`, the frame | the analysis seam you touched: `analysis_frame`, `analysis_plan`, `analysis_verify`, `analysis_design`, `analysis_time`, `analysis_series`, `analysis_modules`, `analysis_results` | | a module | `--suite `; its cases travel with it under `modules//test/` | `--fast` runs everything that does not start a JVM; what it skips is `04_pipeline`, `05_guards`, and the pipeline halves of `06_dryrun` and the analysis suites. -**`bash test/run_tests.sh --changed` picks the suites for you**, from what each suite declares it runs expanded through the include graph. `dev/scripts/select-tests.py ` shows the reasoning without running anything. It errs wide — a change to `test/lib/` or to the selector selects everything — so a narrow answer is trustworthy and a wide one is only expensive. +**`bash test/run_tests.sh --changed` picks the suites for you**, from what each suite declares it runs expanded through the include graph. `dev/scripts/select-tests.py ` shows the reasoning without running anything. It errs wide -- a change to `test/lib/` or to the selector selects everything -- so a narrow answer is trustworthy and a wide one is only expensive. -**Every suite declares what it may cost** — `static`, `jvm` or `pipeline` — in a `# cost:` line in its own header. `--cost static` is the set that completes with nothing installed: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose. +**Every suite declares what it may cost** -- `static`, `jvm` or `pipeline` -- in a `# cost:` line in its own header. `--cost static` is the set that completes with nothing installed: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose. **`--suite` and `--case` accumulate and match by name, not by number.** `--suite analysis_time --suite analysis_series` runs both, and `--suite analysis` runs all nine. Every run prints the scope it selected, so a narrowed run cannot be mistaken for a full one; renumbering a suite therefore costs nothing, because nothing addresses one by its number. -A template's comments are checked differently again: the **commented-out parameter assignments are knobs, not commentary**, and `grep -vE '^\s*//'` hides them from the settings diff. Count them separately. `parameters.config.template` has 14 commented lines that look like assignments, and they are two different things: **6 are knobs** you can uncomment as they stand — `fastqc.options`, `trim_galore.adapterOptions`, `trim_galore.options`, `cutadapt.options`, `bwa.options`, `variantCall.mpileupOptions` — while the **other 8 are the `cores` block**, where the right-hand side describes how each value is derived rather than being a value. Do not cut either, and do not count them as one thing. +A template's comments are checked differently again: the **commented-out parameter assignments are knobs, not commentary**, and `grep -vE '^\s*//'` hides them from the settings diff. Count them separately. `parameters.config.template` has 14 commented lines that look like assignments, and they are two different things: **6 are knobs** you can uncomment as they stand -- `fastqc.options`, `trim_galore.adapterOptions`, `trim_galore.options`, `cutadapt.options`, `bwa.options`, `variantCall.mpileupOptions` -- while the **other 8 are the `cores` block**, where the right-hand side describes how each value is derived rather than being a value. Do not cut either, and do not count them as one thing. diff --git a/PoolSeqFlow b/PoolSeqFlow index a27e10a..32286b4 100755 --- a/PoolSeqFlow +++ b/PoolSeqFlow @@ -5,10 +5,10 @@ # email : ozankiratli@protonmail.com # # Wrapper script for PoolSeqFlow -# Usage: PoolSeqFlow {install|init|init_multi|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall|uninstall_all} +# Usage: PoolSeqFlow {install|init [multi]|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall [all]} # install - creates this release's conda environment and installs the pipeline # init - populates the current directory as a project -# init_multi - the same, plus multiRun on +# init multi - the same, plus multiRun on # check - verifies the installation # run - runs the pipeline; also resumes one # dryrun - creates the directory tree a run would write, empty @@ -30,7 +30,7 @@ # list - lists what is installed on this machine # uninstall - removes one installed version, asking which when several exist, and # always confirming before anything goes -# uninstall_all - removes every PoolSeqFlow environment and installation +# uninstall all - removes every PoolSeqFlow environment and installation # # Each top-level subcommand takes no arguments. `analysis` takes one word of its own; a module # name takes an optional `nocpp` after it, and `analysis modules` takes a verb plus whatever @@ -46,7 +46,7 @@ eval "$(conda shell.bash hook)" VERSION="3.2.0" usage() { - echo "Usage: $0 {install|init|init_multi|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall|uninstall_all}" + echo "Usage: $0 {install|init [multi]|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall [all]}" exit 1 } @@ -416,38 +416,94 @@ completion_dir() { # into a directory that is gone. A failure here does not fail the install: completion is a # convenience and the tool works without it. install_completion() { - local src="$1/lib/poolseqflow-completion.bash" dir - [ -f "$src" ] || return 0 - dir=$(completion_dir) - mkdir -p "$dir" 2>/dev/null || { echo "Could not create $dir; tab completion not installed."; return 0; } - if ! cp "$src" "$dir/PoolSeqFlow" 2>/dev/null; then - echo "Could not write $dir/PoolSeqFlow; tab completion not installed." + local bash_src="$1/lib/poolseqflow-completion.bash" zsh_src="$1/lib/_PoolSeqFlow" + local bash_dir zsh_dir wrote="" + bash_dir=$(completion_dir) + zsh_dir=$(zsh_completion_dir) + + if [ -f "$bash_src" ] && mkdir -p "$bash_dir" 2>/dev/null && + cp "$bash_src" "$bash_dir/PoolSeqFlow" 2>/dev/null; then + wrote="$wrote $bash_dir/PoolSeqFlow"$'\n' + fi + if [ -f "$zsh_src" ] && mkdir -p "$zsh_dir" 2>/dev/null && + cp "$zsh_src" "$zsh_dir/_PoolSeqFlow" 2>/dev/null; then + wrote="$wrote $zsh_dir/_PoolSeqFlow"$'\n' + fi + if [ -z "$wrote" ]; then + echo "Could not write the tab completions; the tool works without them." return 0 fi + echo "Tab completion installed:" - echo " $dir/PoolSeqFlow" + printf '%s' "$wrote" echo "" - echo " bash picks it up in your next shell." - echo " zsh does not read that directory. Add this to ~/.zshrc:" - echo "" - echo " autoload -U +X bashcompinit && bashcompinit" - echo " . $dir/PoolSeqFlow" + completion_advice "$bash_dir/PoolSeqFlow" "$zsh_dir" echo "" } +# Where a user's own zsh completions live. Not on zsh's default fpath - every entry there needs +# root - so the user adds it, which is the line completion_advice prints. +zsh_completion_dir() { + printf '%s/zsh/site-functions' "${XDG_DATA_HOME:-$HOME/.local/share}" +} + +# Where bash-completion itself lives, or empty. Without it nothing reads the directory above, +# so bash needs the file sourced by hand like zsh does. +bash_completion_present() { + local p + for p in /usr/share/bash-completion/bash_completion \ + /etc/bash_completion \ + /usr/local/share/bash-completion/bash_completion; do + [ -r "$p" ] && { printf '%s' "$p"; return 0; } + done + return 1 +} + +# WHAT TO DO NEXT, FOR THE SHELL YOU ACTUALLY USE. Printing both cases means the half that +# applies to you arrives buried in the half that does not, and the half that gets skipped is +# the one with work in it. $SHELL is the login shell, which is what your interactive sessions +# will be, whatever this install happens to be running under. +completion_advice() { + local file="$1" zdir="$2" + case "${SHELL##*/}" in + zsh) + echo " Your shell is zsh. Add this to ~/.zshrc, BEFORE compinit or oh-my-zsh:" + echo "" + echo " fpath=($zdir \$fpath)" + ;; + bash) + if bash_completion_present > /dev/null; then + echo " bash-completion is installed." + else + echo " bash-completion is not installed on your system." + echo " Either install bash-completion or add this line to ~/.bashrc:" + echo "" + echo " . $file" + fi + ;; + *) + echo " bash reads its directory when bash-completion is installed." + echo " zsh needs its own on \$fpath, in ~/.zshrc before compinit:" + echo "" + echo " fpath=($zdir \$fpath)" + ;; + esac +} + # Remove the completion, but only when no other installed version still provides one. remove_completion() { local prefix file prefix=$(install_prefix) - file="$(completion_dir)/PoolSeqFlow" - [ -f "$file" ] || return 0 - # Another version left installed keeps it: the file is the same for every version, and the - # command it completes still exists. + # Another version left installed keeps them: the files are the same for every version, and + # the command they complete still exists. if ls -d "$prefix"/opt/PoolSeqFlow-*/ > /dev/null 2>&1; then return 0 fi - rm -f "$file" 2>/dev/null || true - echo "Removed $file" + for file in "$(completion_dir)/PoolSeqFlow" "$(zsh_completion_dir)/_PoolSeqFlow"; do + [ -f "$file" ] || continue + rm -f "$file" 2>/dev/null || true + echo "Removed $file" + done } # Every PoolSeqFlow environment except this version's own and the legacy unversioned one. @@ -1089,7 +1145,7 @@ report_other_envs() { echo "None of them is used by this copy. Keep the ones whose results you still need to" echo "reproduce; remove the rest:" printf ' %-32s %s\n' "conda env remove -n " "one environment" - printf ' %-32s %s\n' "./$(basename "$0") uninstall_all" "every PoolSeqFlow environment" + printf ' %-32s %s\n' "./$(basename "$0") uninstall all" "every PoolSeqFlow environment" echo "" } @@ -1205,14 +1261,24 @@ fi COMMAND=$1 -# Two subcommands carry a word of their own. `analysis` takes an analysis command or the module +# Renamed to two words. Named here rather than as a case arm below, so the arm list and the usage +# line stay the same set - which is what `usage and implementation agree` compares. +if [ "$COMMAND" = "init_multi" ] || [ "$COMMAND" = "uninstall_all" ]; then + echo "ERROR: '$COMMAND' is now two words: '${COMMAND/_/ }'." >&2 + exit 1 +fi + +# Four subcommands carry a word of their own. `analysis` takes an analysis command or the module # to run, and how many words follow that is each arm's own business, enforced there. `check` # takes exactly one, saying which check; an absent or unknown one is reported by its own arm, -# which can name the two by name, so anything at all is passed through from here. Every other -# subcommand takes none. +# which can name the two by name, so anything at all is passed through from here. `init` and +# `uninstall` each take one optional word, and only one value is a word at all, so a wrong one is +# a usage error here. Every other subcommand takes none. ANALYSIS_COMMAND="" ANALYSIS_ARGS=() CHECK_TARGET="" +INIT_MODE="" +UNINSTALL_SCOPE="" if [ "$COMMAND" = "analysis" ]; then [ "$#" -ge 2 ] || usage_analysis ANALYSIS_COMMAND=$2 @@ -1221,12 +1287,20 @@ if [ "$COMMAND" = "analysis" ]; then elif [ "$COMMAND" = "check" ]; then [ "$#" -le 2 ] || usage CHECK_TARGET="${2:-}" +elif [ "$COMMAND" = "init" ]; then + [ "$#" -le 2 ] || usage + INIT_MODE="${2:-}" + [ -z "$INIT_MODE" ] || [ "$INIT_MODE" = "multi" ] || usage +elif [ "$COMMAND" = "uninstall" ]; then + [ "$#" -le 2 ] || usage + UNINSTALL_SCOPE="${2:-}" + [ -z "$UNINSTALL_SCOPE" ] || [ "$UNINSTALL_SCOPE" = "all" ] || usage elif [ "$#" -ne 1 ]; then usage fi # What the user typed, for the messages that tell them to type it again somewhere else. -INVOCATION="$COMMAND${ANALYSIS_COMMAND:+ $ANALYSIS_COMMAND}${CHECK_TARGET:+ $CHECK_TARGET}" +INVOCATION="$COMMAND${ANALYSIS_COMMAND:+ $ANALYSIS_COMMAND}${CHECK_TARGET:+ $CHECK_TARGET}${INIT_MODE:+ $INIT_MODE}${UNINSTALL_SCOPE:+ $UNINSTALL_SCOPE}" # One environment per release, named from VERSION above so a bump carries it. ENV_NAME="PoolSeqFlow-${VERSION}" @@ -1329,6 +1403,65 @@ populate_project() { echo " into one pool." } +# Every PoolSeqFlow environment and installation on the machine. A function rather than an arm +# of its own: `uninstall all` is one command with a word after it, and the arm list has to stay +# the same set as the usage line. +uninstall_every_version() { + INSTALLED=$(poolseqflow_envs) + if [ -z "$INSTALLED" ]; then + echo "No PoolSeqFlow conda environments are installed. Nothing to do." + exit 0 + fi + PREFIX=$(install_prefix) + PAYLOADS=$(installed_payload_versions "$PREFIX") + echo "" + echo "This removes every PoolSeqFlow conda environment on this machine:" + echo "$INSTALLED" | sed 's/^/ /' + if [ -n "$PAYLOADS" ]; then + echo "" + echo "and every installed pipeline under $PREFIX/opt:" + for v in $PAYLOADS; do echo " v$v"; done + echo "" + echo "including the symlinks in $PREFIX/bin." + fi + echo "" + echo "Only the software goes. Your project directories - parameters.config, Data/," + echo "Reference/ - and everything in storageDir are untouched: none of them live" + echo "inside an installation." + echo "Reinstall this version afterwards with:" + echo " ./$(basename "$0") install" + echo "" + echo "Removing an environment means results produced with that version can no" + echo "longer be reproduced without reinstalling it." + echo "" + printf 'Remove them? [y/N]: ' + if ! read -r CONFIRM; then + echo "" + echo "Aborted - no confirmation received. Nothing was removed." + exit 1 + fi + case "$CONFIRM" in + y|Y|yes|YES|Yes) ;; + *) echo "Aborted - nothing was removed."; exit 1 ;; + esac + # An active environment has to be stepped out of first; there may be none. + conda deactivate 2>/dev/null || true + # A plain `for`, not `... | while read`: a piped loop is a subshell, where `set -e` + # would not stop the script. Environment names cannot contain whitespace. + for env in $INSTALLED; do + echo "Removing $env..." + conda env remove -n "$env" -y + done + # remove_payload re-points the plain wrapper each time and removes it with the last + # version, so nothing is left on PATH pointing at a directory that has gone. + for v in $PAYLOADS; do + remove_payload "$v" || true + done + remove_completion + echo "" + echo "All PoolSeqFlow environments and installations removed." +} + case $COMMAND in install) require_install @@ -1353,12 +1486,11 @@ case $COMMAND in init) require_install - populate_project single - ;; - - init_multi) - require_install - populate_project multi + if [ "$INIT_MODE" = "multi" ]; then + populate_project multi + else + populate_project single + fi ;; check) @@ -1965,7 +2097,7 @@ EOF echo "" fi echo "Remove one environment: conda env remove -n " - echo "Remove all of them: ./$(basename "$0") uninstall_all" + echo "Remove all of them: ./$(basename "$0") uninstall all" ;; version) @@ -1977,6 +2109,11 @@ EOF poolseqflow_citation "$VERSION" ;; uninstall) + # `uninstall all` takes the machine; `uninstall` alone asks which installation. + if [ "$UNINSTALL_SCOPE" = "all" ]; then + uninstall_every_version + exit 0 + fi PREFIX=$(install_prefix) # Decided first, and may be an installation other than this wrapper's. Canceling, # or having no way to ask, stops here. @@ -2071,61 +2208,6 @@ EOF fi ;; - uninstall_all) - INSTALLED=$(poolseqflow_envs) - if [ -z "$INSTALLED" ]; then - echo "No PoolSeqFlow conda environments are installed. Nothing to do." - exit 0 - fi - PREFIX=$(install_prefix) - PAYLOADS=$(installed_payload_versions "$PREFIX") - echo "" - echo "This removes every PoolSeqFlow conda environment on this machine:" - echo "$INSTALLED" | sed 's/^/ /' - if [ -n "$PAYLOADS" ]; then - echo "" - echo "and every installed pipeline under $PREFIX/opt:" - for v in $PAYLOADS; do echo " v$v"; done - echo "" - echo "including the symlinks in $PREFIX/bin." - fi - echo "" - echo "Only the software goes. Your project directories - parameters.config, Data/," - echo "Reference/ - and everything in storageDir are untouched: none of them live" - echo "inside an installation." - echo "Reinstall this version afterwards with:" - echo " ./$(basename "$0") install" - echo "" - echo "Removing an environment means results produced with that version can no" - echo "longer be reproduced without reinstalling it." - echo "" - printf 'Remove them? [y/N]: ' - if ! read -r CONFIRM; then - echo "" - echo "Aborted - no confirmation received. Nothing was removed." - exit 1 - fi - case "$CONFIRM" in - y|Y|yes|YES|Yes) ;; - *) echo "Aborted - nothing was removed."; exit 1 ;; - esac - # An active environment has to be stepped out of first; there may be none. - conda deactivate 2>/dev/null || true - # A plain `for`, not `... | while read`: a piped loop is a subshell, where `set -e` - # would not stop the script. Environment names cannot contain whitespace. - for env in $INSTALLED; do - echo "Removing $env..." - conda env remove -n "$env" -y - done - # remove_payload re-points the plain wrapper each time and removes it with the last - # version, so nothing is left on PATH pointing at a directory that has gone. - for v in $PAYLOADS; do - remove_payload "$v" || true - done - remove_completion - echo "" - echo "All PoolSeqFlow environments and installations removed." - ;; *) usage ;; diff --git a/README.md b/README.md index 591bead..8b256bb 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19245611.svg)](https://doi.org/10.5281/zenodo.19245611) -### 📖 [Read the documentation →](https://ozankiratli.github.io/PoolSeqFlow/) +### 📖 [Read the documentation ->](https://ozankiratli.github.io/PoolSeqFlow/) > **Platform note:** PoolSeqFlow is developed and tested on **Linux**. macOS and Windows are not supported: the shipped conda environments are pinned to `linux-64` builds, and the resume logic relies on symbolic links and Unix-style paths that are not compatible with native Windows filesystems. @@ -20,7 +20,7 @@ PoolSeqFlow takes raw FASTQ files and a reference genome and gives back allele frequency tables. It automates quality control, adapter trimming with composition-aware clipping, alignment, BAM post-processing, variant calling and VCF-to-frequency conversion, with optional annotation. -Pool-seq sequences many individuals together, so the unit of analysis is not a genotype but a **frequency** — and that difference runs through every stage. Multiallelic sites are preserved rather than collapsed, VCFs are re-encoded so the most-read allele is the reference, and the minimum credible frequency is derived from your pool size and ploidy instead of a fixed cutoff. See [When to use PoolSeqFlow](https://ozankiratli.github.io/PoolSeqFlow/concepts/) for what the design assumes about your data. +Pool-seq sequences many individuals together, so the unit of analysis is not a genotype but a **frequency** -- and that difference runs through every stage. Multiallelic sites are preserved rather than collapsed, VCFs are re-encoded so the most-read allele is the reference, and the minimum credible frequency is derived from your pool size and ploidy instead of a fixed cutoff. See [When to use PoolSeqFlow](https://ozankiratli.github.io/PoolSeqFlow/concepts/) for what the design assumes about your data. ``` Raw FASTQ reads @@ -32,13 +32,13 @@ Raw FASTQ reads [Step 1] Build reference dictionaries (BWA, SAMtools, SnpEff) │ ▼ -[Step 2] QC & trimming (FastQC → Trim Galore → composition-aware clipping) +[Step 2] QC & trimming (FastQC -> Trim Galore -> composition-aware clipping) │ ▼ [Step 3] Alignment (BWA-MEM) │ ▼ -[Step 4] BAM cleanup (name-sort → fixmate → coord-sort → markdup → addRG → filter → index) +[Step 4] BAM cleanup (name-sort -> fixmate -> coord-sort -> markdup -> addRG -> filter -> index) │ ▼ [Step 5] Alignment, coverage & depth reports; each sample's depth ceiling @@ -48,7 +48,7 @@ Raw FASTQ reads │ ├────────────────────────────────────────────┐ ▼ ▼ -[Step 7] VCF → allele frequency tables [Step 8] Annotation (SnpEff, optional) +[Step 7] VCF -> allele frequency tables [Step 8] Annotation (SnpEff, optional) ``` --- @@ -72,11 +72,11 @@ PoolSeqFlow init # then edit parameters.config: mainDir, storageDir, readPattern, # referenceFile, poolSize, ploidy -# 4. Run — this is also the resume command +# 4. Run -- this is also the resume command PoolSeqFlow run ``` -Your project directory needs a `Data/` folder of FASTQs, a reference genome (gzipped or not), and a `metadata.csv`. `init` writes `metadata.csv.example` for you to start from — the file is not only metadata, it decides which FASTQ pairs count as one pool, the order your result columns come out in, and each pool's detection limit. +Your project directory needs a `Data/` folder of FASTQs, a reference genome (gzipped or not), and a `metadata.csv`. `init` writes `metadata.csv.example` for you to start from -- the file is not only metadata, it decides which FASTQ pairs count as one pool, the order your result columns come out in, and each pool's detection limit. Full walkthrough: [Install](https://ozankiratli.github.io/PoolSeqFlow/getting-started/install/) and [Quick Start](https://ozankiratli.github.io/PoolSeqFlow/getting-started/quick-start/). @@ -90,27 +90,27 @@ Full walkthrough: [Install](https://ozankiratli.github.io/PoolSeqFlow/getting-st |---|---| | `PoolSeqFlow install` | Create the conda environment, install the pipeline, then verify both | | `PoolSeqFlow init` | Populate the current directory as a project | -| `PoolSeqFlow init_multi` | The same, for a project running several parameter sets over one set of reads | -| `PoolSeqFlow check install` | Verify an installation — the tools and helpers it is built to run | -| `PoolSeqFlow check project` | Verify a project — its configuration, and the commands it names | -| `PoolSeqFlow run` | Start — or resume — the pipeline | +| `PoolSeqFlow init multi` | The same, for a project running several parameter sets over one set of reads | +| `PoolSeqFlow check install` | Verify an installation -- the tools and helpers it is built to run | +| `PoolSeqFlow check project` | Verify a project -- its configuration, and the commands it names | +| `PoolSeqFlow run` | Start -- or resume -- the pipeline | | `PoolSeqFlow dryrun` | Create the directory tree a run would write, empty, before any compute is spent | | `PoolSeqFlow dryclean` | Remove that preview | | `PoolSeqFlow migrate_config` | Carry an older `parameters.config` onto the current template | | `PoolSeqFlow clean` | Remove Nextflow work directories | | `PoolSeqFlow reset` | Remove all progress and start fresh (typed confirmation required) | -| `PoolSeqFlow analysis ` | The analysis layer — see below | +| `PoolSeqFlow analysis ` | The analysis layer -- see below | | `PoolSeqFlow version` | Print the installed version | | `PoolSeqFlow cite` | Print how to cite this copy, and which DOI to use | | `PoolSeqFlow list` | List the pipelines and conda environments installed on this machine | -| `PoolSeqFlow uninstall` | Remove one installed version — environment and pipeline together, after confirmation | -| `PoolSeqFlow uninstall_all` | Remove every PoolSeqFlow environment and installation, after confirmation | +| `PoolSeqFlow uninstall` | Remove one installed version -- environment and pipeline together, after confirmation | +| `PoolSeqFlow uninstall all` | Remove every PoolSeqFlow environment and installation, after confirmation | -**`./PoolSeqFlow install` is the only command you type with a `./`.** Until it has run there is no `PoolSeqFlow` on your `PATH`, so that first one is called from inside the folder you extracted. It installs the command, and everything after it — including `init`, which you run from your own project directory — is called by name from wherever you are. +**`./PoolSeqFlow install` is the only command you type with a `./`.** Until it has run there is no `PoolSeqFlow` on your `PATH`, so that first one is called from inside the folder you extracted. It installs the command, and everything after it -- including `init`, which you run from your own project directory -- is called by name from wherever you are. `analysis` takes `install`, `check`, `modules`, `version`, `cite`, `uninstall`, or the name of a module to run. `analysis modules available|install|list|uninstall` manages the modules themselves, which are published separately from the pipeline. The analysis layer ships with the pipeline and is enabled separately with `PoolSeqFlow analysis install`, which creates the conda environment that carries R. -There is no `-resume` flag. Every step checks whether its outputs already exist in permanent storage and skips itself if they do, so `run` both starts and resumes — and that survives job timeouts, reboots and `work/` cleanups. [Why →](https://ozankiratli.github.io/PoolSeqFlow/pipeline/resume/) +There is no `-resume` flag. Every step checks whether its outputs already exist in permanent storage and skips itself if they do, so `run` both starts and resumes -- and that survives job timeouts, reboots and `work/` cleanups. [Why ->](https://ozankiratli.github.io/PoolSeqFlow/pipeline/resume/) --- @@ -124,8 +124,8 @@ There is no `-resume` flag. Every step checks whether its outputs already exist | [Interpreting results](https://ozankiratli.github.io/PoolSeqFlow/concepts/interpreting-results/) | The frequency table format, and the mistakes that are easy to make reading it | | [Configuration](https://ozankiratli.github.io/PoolSeqFlow/configuration/) | Every parameter, sorted by whether it changes your results | | [Metadata](https://ozankiratli.github.io/PoolSeqFlow/configuration/metadata/) | `metadata.csv`, and why `RG_Sample` decides what counts as a pool | -| [Pipeline steps](https://ozankiratli.github.io/PoolSeqFlow/pipeline/steps/) | Steps 0–8 in detail | -| [Upgrading](https://ozankiratli.github.io/PoolSeqFlow/getting-started/upgrading/) | Your `parameters.config` is never touched by an update — read this first | +| [Pipeline steps](https://ozankiratli.github.io/PoolSeqFlow/pipeline/steps/) | Steps 0-8 in detail | +| [Upgrading](https://ozankiratli.github.io/PoolSeqFlow/getting-started/upgrading/) | Your `parameters.config` is never touched by an update -- read this first | | [Troubleshooting](https://ozankiratli.github.io/PoolSeqFlow/reference/troubleshooting/) | Errors by symptom | --- @@ -138,15 +138,15 @@ Your installed copy prints its own citation, with the version filled in: PoolSeqFlow cite ``` -**Cite the version you actually ran, not the newest one.** Zenodo issues a separate DOI for every release, and results depend on which release produced them — filters, defaults and parameter names have all changed between versions. Step 0 records the release that produced a project's results in `.poolseqflow_version`, named again in the header of the readable `Output/run_parameters.txt`, and refuses to run under a different one — so a project belongs to one release and there is never a question of which to cite. +**Cite the version you actually ran, not the newest one.** Zenodo issues a separate DOI for every release, and results depend on which release produced them -- filters, defaults and parameter names have all changed between versions. Step 0 records the release that produced a project's results in `.poolseqflow_version`, named again in the header of the readable `Output/run_parameters.txt`, and refuses to run under a different one -- so a project belongs to one release and there is never a question of which to cite. -[10.5281/zenodo.19245611](https://doi.org/10.5281/zenodo.19245611) is the **all-versions** DOI: it always resolves to the newest release. Use it to refer to the software in general, and a version DOI when reporting results. [Details →](https://ozankiratli.github.io/PoolSeqFlow/reference/citation/) +[10.5281/zenodo.19245611](https://doi.org/10.5281/zenodo.19245611) is the **all-versions** DOI: it always resolves to the newest release. Use it to refer to the software in general, and a version DOI when reporting results. [Details ->](https://ozankiratli.github.io/PoolSeqFlow/reference/citation/) --- ## License -Apache 2.0 — see [LICENSE](LICENSE). The tools PoolSeqFlow invokes carry their own licenses. +Apache 2.0 -- see [LICENSE](LICENSE). The tools PoolSeqFlow invokes carry their own licenses. **Analysis modules under `analysis/modules/` carry their own**, declared in each `manifest.json` and printed in the report of every analysis. `basicstats`, `association` and `mds` are GPL-3.0-or-later; `verify` belongs to the frame and is Apache 2.0. diff --git a/analysis/analysis.config.template b/analysis/analysis.config.template index 48a2892..5eb9a06 100644 --- a/analysis/analysis.config.template +++ b/analysis/analysis.config.template @@ -1,4 +1,4 @@ -// PoolSeqFlow analysis layer — project settings. The manual explains every setting in this file: +// PoolSeqFlow analysis layer -- project settings. The manual explains every setting in this file: // https://ozankiratli.github.io/PoolSeqFlow/ // // OPTIONAL. Without it every module uses the values below, which are what PoolSeqFlow falls @@ -58,7 +58,7 @@ params { folderName = '' // HOW YOUR METADATA FILE IS READ. Everything in here describes metadata.csv rather than - // this invocation, and none of it changes what the pipeline computed — exp_ and pt_ + // this invocation, and none of it changes what the pipeline computed -- exp_ and pt_ // columns are recorded and never acted on, so results you already have stay valid. // // A key this scope does not have is refused by name. That is the point of the scope: a @@ -73,17 +73,17 @@ params { // ['NA', 'N/A', 'na', '-'] several // ['na_*'] a wildcard: * is any run, ? is one character // - // Matched WHOLE and CASE SENSITIVELY, so 'NA' does not catch 'na' — list both if + // Matched WHOLE and CASE SENSITIVELY, so 'NA' does not catch 'na' -- list both if // your file has both. That is deliberate: a level genuinely called `na` has to stay // a level, and no setting can tell the two apart for you. // // Applies to exp_ and pt_ columns, which are the only ones the analysis layer reads. // An entry that would match everything is refused rather than obeyed, and the - // verification report names every cell this turned into a blank — read that line, + // verification report names every cell this turned into a blank -- read that line, // because a pattern wider than you meant removes values and leaves no error behind. missingValueEncoding = [] - // THE TIME AXIS. Set this whenever your metadata has an exp_time column — a project + // THE TIME AXIS. Set this whenever your metadata has an exp_time column -- a project // that has one and does not say how to read it is refused, because time is the one // variable whose ORDER changes what a result means and it is never guessed. // @@ -98,7 +98,7 @@ params { // // is meaningful. Needs `unit` below. // // 'datetime' the values are dates. Needs `format`, and `locale` if the // // month names are not English. Measured in days. - // // 'categorical' the values are labels in an order you give. Rank only — + // // 'categorical' the values are labels in an order you give. Rank only -- // // no rate can be fitted, and no unit may be set. // kind = 'numerical' // @@ -109,11 +109,11 @@ params { // // step evenly spaced in something unnamed // unit = 'generation' // - // // datetime only. java.time patterns — yyyy or uuuu year, MM month digits, + // // datetime only. java.time patterns -- yyyy or uuuu year, MM month digits, // // MMM short name, MMMM full name, dd day, HH 24-hour, hh with a for 12-hour. // // 07/03/2024 is a valid date under dd/MM/yyyy AND MM/dd/yyyy, four months // // apart, so this is asked for rather than guessed. The verification report - // // prints the dates it read, as ISO, in the order it will use them — read it. + // // prints the dates it read, as ISO, in the order it will use them -- read it. // format = 'dd/MM/yyyy' // // // datetime only. The language month and weekday names are written in. @@ -126,7 +126,7 @@ params { // order = ['pre', 'post'] // } - // PHENOTYPES — what was MEASURED on each pool, for a module to test against. + // PHENOTYPES -- what was MEASURED on each pool, for a module to test against. // // One block per pt_ column, and declare as many as you record. Those are the columns // checked for agreeing across the rows of one pool, which is what stops a pool @@ -135,7 +135,7 @@ params { // // DECLARING ONE DOES NOT CHOOSE IT. Which phenotype an analysis tests against is the // module's own setting, so a project with three of them runs three analyses from one - // declaration block — set folderName so each lands somewhere of its own. + // declaration block -- set folderName so each lands somewhere of its own. // // A pt_ column you do not declare is still recorded, checked and reported; it simply // carries no scale, so no module can test against it. The verification report names @@ -144,20 +144,20 @@ params { // // phenotypes { // pt_wingspan { - // // WHAT KIND OF SCALE IT IS. Never guessed — 0 and 1 read as numbers as + // // WHAT KIND OF SCALE IT IS. Never guessed -- 0 and 1 read as numbers as // // readily as they encode two groups, and no file says which of 'case' // // and 'control' is the affected one. // // // // 'quantitative' a measurement. Every value must parse as a number. // // Counts and percentages are this too. - // // 'binary' PRESENCE and ABSENCE — affected or not, resistant or + // // 'binary' PRESENCE and ABSENCE -- affected or not, resistant or // // not. Exactly two levels, as [absent, present]. // // 'ordinal' groups whose ORDER means something and whose spacing // // does not: low, medium, high. // // 'nominal' groups with no order: spotted, striped, curly. // // - // // Two groups that are not an absence and a presence — coastal and inland, - // // two host plants — are 'nominal', not 'binary'. + // // Two groups that are not an absence and a presence -- coastal and inland, + // // two host plants -- are 'nominal', not 'binary'. // kind = 'quantitative' // } // @@ -167,7 +167,7 @@ params { // // REQUIRED for binary, ordinal and nominal; refused for quantitative. // // // // For 'binary' the order is [absent, present] and it decides the SIGN of - // // every effect reported against it — write it the wrong way round and + // // every effect reported against it -- write it the wrong way round and // // every result reverses, with nothing in the data to tell you. // // For 'ordinal' the order IS the scale. // // For 'nominal' the order only sets which group others are reported @@ -180,7 +180,7 @@ params { // } // } - // COVARIATES — what was MEASURED alongside, and is neither set nor the response. + // COVARIATES -- what was MEASURED alongside, and is neither set nor the response. // A cage temperature, an altitude, a collection site, a technician when every // library of a pool shares one. // @@ -191,15 +191,15 @@ params { // // Declaring one is optional. Undeclared, it is recorded, checked and reported like // any other pool-level column. Declared, it also gets a scale, so a module can - // compute with it — the kinds are the phenotype's and mean the same things. + // compute with it -- the kinds are the phenotype's and mean the same things. // // WHAT IT HOLDS IS SET HERE; WHAT IT DOES IS SET IN design.covariates above. A // covariate left out of the design is still resolved, still reported and still - // published — it is simply not a term in anybody's model. + // published -- it is simply not a term in anybody's model. // // EVEN ON THE RECORD ONLY IT EARNS ITS PLACE. With six or twenty pools there are // few degrees of freedom to spend, and a value in the report lets a reader see that - // the high-phenotype pools were also the warm ones — which is the confounding + // the high-phenotype pools were also the warm ones -- which is the confounding // nobody would otherwise suspect. // // covariates { @@ -212,14 +212,14 @@ params { // } // } // - // What differs BETWEEN two rows of one pool — the lane, the run, the technician who - // handled one library — is NOT a covariate and cannot be made one. The reads are + // What differs BETWEEN two rows of one pool -- the lane, the run, the technician who + // handled one library -- is NOT a covariate and cannot be made one. The reads are // merged, so no analysis can attribute one to the row it came from. Record it in an // unprefixed column; it is kept and never interpreted. } - // WHAT YOUR EXPERIMENT WAS — which pools are independent of each other. + // WHAT YOUR EXPERIMENT WAS -- which pools are independent of each other. // // Its own scope beside `metadata`, not inside it: that one says how the file is READ, // this one says what the file DESCRIBES. This is where degrees of freedom come from, @@ -233,12 +233,12 @@ params { // design { // // The variables that identify one thing you set up. Left empty it is every exp_ // // variable except time. SET IT if one of your exp_ columns is recorded ON each - // // pool rather than saying what the pool is — a temperature, a census — because + // // pool rather than saying what the pool is -- a temperature, a census -- because // // that differs pool to pool and would split every unit into singletons. // by = ['exp_population', 'exp_replicate'] // // // Which of those index REPEATS rather than naming a condition. Both take any - // // number of columns — two lanes crossed with three sequencing runs is + // // number of columns -- two lanes crossed with three sequencing runs is // // technicalRep = ['exp_lane', 'exp_seqrun'], six technical replicates each. // // // // The two are not interchangeable. Biological replicates are independent repeats @@ -250,13 +250,13 @@ params { // // column out of it and each lane counts as an independent replicate, which // // roughly halves every standard error in the project. // // - // // A key column named in neither is a CONDITION — what is being compared. The + // // A key column named in neither is a CONDITION -- what is being compared. The // // verification report prints every key column under its role, which is the only // // place a column in the wrong place can be seen. // biologicalRep = ['exp_replicate'] // technicalRep = ['exp_lane'] // - // // Which of your cov_ columns are PART OF THE DESIGN — the ones a module may + // // Which of your cov_ columns are PART OF THE DESIGN -- the ones a module may // // adjust for, rather than only report. Left empty it is every cov_ column that // // has a declared scale under metadata.covariates; naming any restricts it. // // @@ -269,7 +269,7 @@ params { // covariates = ['cov_temperature'] // // // WHAT A TIME AXIS MAKES OF ALL THAT. Only a time course can leave a trajectory - // // with a gap in it, so this is the one thing the series scope holds — what a + // // with a gap in it, so this is the one thing the series scope holds -- what a // // series IS comes from the keys above. // series { // // What to do when a series does not cover every timepoint. diff --git a/bin/cap_depth.awk b/bin/cap_depth.awk index 4f1e8e2..592d8d8 100755 --- a/bin/cap_depth.awk +++ b/bin/cap_depth.awk @@ -11,7 +11,10 @@ # A read is kept only if every position it covers is still below the cap. Reads are dropped # individually, so a pair may lose one mate. # -# THE INPUT MUST BE COORDINATE-SORTED. +# THE INPUT MUST BE COORDINATE-SORTED. Every read already kept therefore starts at or before +# this one, so it covers a position only if its end reaches that far: the depth over this +# read's span is non-increasing and its first position carries the maximum. Coverage is held +# as a difference array and read off a running total at that one position. function reflen(cigar, i, c, num, len) { # Only M, D, N, = and X consume the reference; I, S, H and P do not. @@ -34,28 +37,30 @@ BEGIN { } chrom = "" low = 0 + cur = 0 } /^@/ { print; next } { # A new reference sequence shares no positions with the last one. - if ($3 != chrom) { delete depth; chrom = $3; low = $4 + 0 } + if ($3 != chrom) { delete diff; chrom = $3; low = $4 + 0; cur = 0 } pos = $4 + 0 span = reflen($6) # Unmapped, or a CIGAR that consumes no reference: nothing to count, and nothing to cap. if (pos == 0 || span <= 0) { kept++; print; next } - # Trimmed from behind: the stream is sorted, so nothing can add to a position it has passed. - # This is what keeps the array to one read's span instead of a chromosome's. - while (low < pos) { delete depth[low]; low++ } + # Carry the running depth forward to this read's first position, consuming each entry as + # it is passed. `in` rather than a plain read: referencing diff[low] would create it, and + # most positions hold nothing. + while (low < pos) { low++; if (low in diff) { cur += diff[low]; delete diff[low] } } - end = pos + span - 1 - for (p = pos; p <= end; p++) { - if (depth[p] + 0 >= cap) { dropped++; next } - } - for (p = pos; p <= end; p++) depth[p]++ + # cur is the depth at pos, which is the largest over this read's span. + if (cur >= cap) { dropped++; next } + + cur++ # this read covers pos + diff[pos + span]-- # and stops one past its last position kept++ print } diff --git a/bin/check_parameters.sh b/bin/check_parameters.sh new file mode 100755 index 0000000..1d7f08e --- /dev/null +++ b/bin/check_parameters.sh @@ -0,0 +1,149 @@ +#!/bin/bash +# +# Judge a resolved parameter set, one finding per line. +# +# nextflow config -flat | check_parameters.sh +# +# Reads flattened parameter assignments on stdin, in either of the two forms its callers hold +# them in, and writes one tab-separated finding per rule that has something to say: +# +# LEVEL label verdict detail explanation +# +# LEVEL is FAIL, WARN or NOTE. Every caller formats those itself; nothing here prints color or +# indentation. Exit status is 1 when any finding is a FAIL, 0 otherwise, so a caller may use the +# status alone and read nothing. +# +# Silence is the answer for a parameter set with nothing wrong with it. +# +# WHAT BELONGS HERE: a setting that makes the run produce NOTHING, or that silently changes what +# a published number means. What does not: a setting that merely produces fewer sites. minDP 20 +# against minDP 5 is a scientific choice and this has no opinion on it. +# +# VALUES ARE READ FROM THE COMPOSED OPTION STRINGS WHERE THERE IS ONE. A project may pin +# variantCall.mpileupOptions by hand, which makes scaleMapQ and varQualMin inert, so judging the +# settings it was not built from would report a problem that does not exist. An option string +# carrying neither flag is reported as unjudged rather than guessed at. + +set -uo pipefail + +FAILED=0 + +# One finding. Tabs are the separator, so no field may contain one. +say() { # level label verdict detail explanation + printf '%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "$4" "${5:-}" + [ "$1" = "FAIL" ] && FAILED=1 + return 0 +} + +# Every assignment, flattened, with surrounding quotes removed. TWO FORMS ARE READ, because the +# two callers have the values in different shapes and neither should have to reformat: `nextflow +# config -flat` writes `params. = `, and step 0 already holds the same set as +# `=` from analysisParams(). +declare -A P=() +while IFS= read -r line; do + case $line in + params.*' = '*) key=${line%% = *}; key=${key#params.}; value=${line#* = } ;; + *=*) key=${line%%=*}; value=${line#*=} ;; + *) continue ;; + esac + value=${value#\'}; value=${value%\'} + value=${value#\"}; value=${value%\"} + P["$key"]=$value +done + +get() { printf '%s' "${P[$1]:-}"; } + +# True when the value is a whole number, so a rule can decline to judge a blank or an expression +# it cannot evaluate rather than comparing a string to a number. +is_int() { case ${1:-} in ''|*[!0-9-]*) return 1 ;; *) return 0 ;; esac; } +# True for a decimal, which sampleThreshold is. Deliberately plain: it rejects an empty value and +# anything carrying a character that is not a digit, a dot or a minus, and leaves the arithmetic to +# awk. An earlier version tried to reject a lone "." as well and was too clever to read. +is_num() { case ${1:-} in ''|*[!0-9.-]*) return 1 ;; *) return 0 ;; esac; } + +# ------------------------------------------------------------- the pileup pair -- +# +# -C caps every read's adjusted mapping quality near its own value and -q then rejects anything +# below the minimum, so a -C under -q leaves bcftools call nothing and it writes a header with no +# records. Measured on real pools at six values of -q (12, 15, 20, 30, 40, 50): -C one below -q +# returned zero sites every time, -C equal to it returned sites every time. +mpileup=$(get variantCall.mpileupOptions) +scale=$(printf '%s\n' "$mpileup" | grep -o -- '-C [0-9][0-9]*' | head -1 | awk '{print $2}') +minq=$(printf '%s\n' "$mpileup" | grep -o -- '-q [0-9][0-9]*' | head -1 | awk '{print $2}') + +if ! is_int "$scale" || ! is_int "$minq"; then + say NOTE "variantCall.scaleMapQ" "NOT CHECKED" "no -C or -q in mpileupOptions" \ + "The option string was pinned by hand and carries no mapping-quality pair to judge." +elif [ "$scale" -eq 0 ]; then + say NOTE "variantCall.scaleMapQ" "OFF" "no mapping-quality adjustment" "" +elif [ "$scale" -le 10 ]; then + say WARN "variantCall.scaleMapQ" "INERT AT $scale" "10 and below changes nothing" \ + "bcftools applies no adjustment at all below 11, so this run is the same as scaleMapQ 0. Write 0 if that is what you meant." +elif [ "$scale" -lt "$minq" ]; then + say FAIL "variantCall.scaleMapQ" "DISCARDS EVERY READ" "$scale is below varQualMin $minq" \ + "-C caps every read's mapping quality near $scale and -q then rejects anything under $minq, so the pileup reaches bcftools call empty and the run produces no variants at all. Raise scaleMapQ above varQualMin, or lower varQualMin below scaleMapQ." +elif [ "$scale" -lt $(( minq * 2 )) ]; then + say WARN "variantCall.scaleMapQ" "SEVERE AT $scale" "close to varQualMin $minq" \ + "It emits records, but only the best-placed reads clear -q. Measured on real pools: scaleMapQ equal to varQualMin kept 6% of the sites an unadjusted run called. Recovery is gradual and the manual has the curve." +fi + +# ------------------------------------------------------- the cross-sample filter -- +# +# sampleThreshold is the fraction of samples an allele must appear in. Above 1 that is more +# samples than exist, so every site goes: measured at 1.5 against the fixture VCF, 0 of 135 +# survived. +threshold=$(get filterFalsePositives.sampleThreshold) +if is_num "$threshold"; then + if awk -v t="$threshold" 'BEGIN { exit !(t > 1) }'; then + say FAIL "filterFalsePositives.sampleThreshold" "REMOVES EVERY SITE" "$threshold is above 1" \ + "It is a fraction of your samples, so above 1 it asks for more samples than the run has and no site can satisfy it. Measured: 1.5 left 0 of 135 sites." + elif awk -v t="$threshold" 'BEGIN { exit !(t <= 0) }'; then + say WARN "filterFalsePositives.sampleThreshold" "INERT AT $threshold" "nothing is required" \ + "At or below 0 the cross-sample requirement asks for nothing, so the filter removes no allele on that clause." + fi +fi + +# ------------------------------------------------------------------- the pool -- +# +# n_chrom is ploidy * poolSize and is what every effective size is computed from. Below 1 the +# detection limit is infinite or negative, which is broken. At exactly 1 the pipeline still runs +# and only diversity degrades, so that case is a warning and not a refusal. +ploidy=$(get ploidy) +poolsize=$(get poolSize) +if is_int "$ploidy" && [ "$ploidy" -lt 1 ]; then + say FAIL "ploidy" "NOT A PLOIDY" "$ploidy" \ + "Every detection limit and every effective size is computed from ploidy times poolSize, so below 1 they are meaningless rather than merely small." +fi +if is_int "$poolsize" && [ "$poolsize" -lt 1 ]; then + say FAIL "poolSize" "NOT A POOL" "$poolsize" \ + "Every detection limit and every effective size is computed from ploidy times poolSize, so below 1 they are meaningless rather than merely small." +fi +if is_int "$ploidy" && is_int "$poolsize" && [ "$ploidy" -ge 1 ] && [ "$poolsize" -ge 1 ] \ + && [ $(( ploidy * poolsize )) -lt 2 ]; then + say WARN "poolSize" "ONE CHROMOSOME" "ploidy $ploidy times poolSize $poolsize" \ + "The pipeline runs and publishes frequencies, but every effective sample size is 1, so the unbiased diversity correction n_eff/(n_eff - 1) is infinite and any diversity an analysis computes over this pool is meaningless. A warning rather than a refusal because the pipeline itself is unaffected." +fi + +# ---------------------------------------------------------- the two ceilings -- +# +# capBAM.maxDepth -1 asks step 5 to measure a ceiling per sample from its own histogram. A +# positive variantCall.maxDepth then applies one flat number to every sample at pileup time, on +# top of that, so the smaller of the two wins and the measured ceilings stop deciding anything. +capmax=$(get capBAM.maxDepth) +vcmax=$(get variantCall.maxDepth) +if is_int "$capmax" && is_int "$vcmax" && [ "$capmax" -eq -1 ] && [ "$vcmax" -gt 0 ]; then + say NOTE "variantCall.maxDepth" "OVERRIDES THE MEASURED CEILINGS" "$vcmax, with capBAM.maxDepth -1" \ + "Step 5 measures a ceiling for each sample and mpileup then caps every sample at $vcmax as well, so whichever is smaller decides. Set it to 0 to let the per-sample ceilings stand." +fi + +# --------------------------------------------------------------- plain numbers -- +# +# FastQC takes megabytes as a bare integer and rejects a size with a unit on it, which the +# template says and is easy to undo. +fqmem=$(get fastqc.memory) +if [ -n "$fqmem" ] && ! is_int "$fqmem"; then + say FAIL "fastqc.memory" "NOT A PLAIN NUMBER" "$fqmem" \ + "FastQC takes megabytes as a bare integer and refuses a value carrying a unit, so write 2048 rather than 2G." +fi + +exit "$FAILED" diff --git a/bin/check_project.sh b/bin/check_project.sh index 26ad6b5..7908230 100755 --- a/bin/check_project.sh +++ b/bin/check_project.sh @@ -77,6 +77,27 @@ check_tool() { fi } +# The byte-order mark a file begins with: utf-8, utf-16le, utf-16be, or nothing at all. +# +# Read as bytes through `od`, because a mark is invisible everywhere it matters. Nextflow's own +# refusal prints `Unexpected character: ''` with the character rendered as nothing, and no +# editor shows one, so a file that is wrong looks identical to a file that is right. +# +# Measured against this release, 2026-09-26: a UTF-8 mark makes Nextflow refuse the config it +# leads, whether that file is nextflow.config or an included parameters.config, while both CSV +# parsers read past one. The same three bytes are therefore fatal in one file and harmless in +# another, and each caller below says which it is looking at. CRLF is not checked: a config +# with CRLF line endings parses, measured the same day. +byte_order_mark() { + local first + first=$(od -An -tx1 -N3 "$1" 2>/dev/null | tr -d ' \n') + case $first in + efbbbf*) printf 'utf-8' ;; + fffe*) printf 'utf-16le' ;; + feff*) printf 'utf-16be' ;; + esac +} + echo "PoolSeqFlow project check" echo "=========================" echo " $PROJECT_DIR" @@ -107,6 +128,22 @@ else "$DIM" "$RESET" "$stale" fi +# Before the parse below, because the parse is what a mark breaks. Nextflow names the file and +# the column and then prints the offending character as nothing, so the error it produces on its +# own is a file the user can see is correct and a cause they cannot see at all. +checked=$((checked + 1)) +config_mark=$(byte_order_mark "$CONFIG") +if [ -z "$config_mark" ]; then + pass "parameters.config" "NO BYTE-ORDER MARK" +else + fail "parameters.config" "STARTS WITH A BYTE-ORDER MARK" "run: dos2unix $CONFIG" + printf ' %sA %s mark sits before the first setting, and Nextflow refuses the\n' \ + "$DIM" "$config_mark" + printf ' file over a character nothing displays. Editors on Windows add one\n' + printf ' when they save as UTF-8; dos2unix removes it, and converts UTF-16\n' + printf ' to UTF-8 as well.%s\n' "$RESET" +fi + # Nextflow's own parse, from the project against the installation, exactly as a run would. # Everything below reads settings out of this, so a failure here is reported and the rest is # skipped by name rather than passing over an empty answer. @@ -163,6 +200,11 @@ check_table() { fi if out=$(python3 "$INSTALL_DIR/bin/$parser" "$path" 2>&1 >/dev/null); then pass "$label" "PARSES" + # A parser that exits 0 may still have written notes, and this discarded them. The only + # note either parser had before 2026-09-26 - a pool given two different exp_, pt_ or + # cov_ values - could therefore be reached by no user through any path: resolve_ + # parameters.nf keeps a parser's stderr only when it fails, and so did this. + [ -n "$out" ] && printf '%s\n' "$out" | sed "s/^/ ${DIM}/;s/$/${RESET}/" else fail "$label" "FAILED TO PARSE" printf '%s\n' "$out" | sed 's/^/ /' @@ -185,6 +227,43 @@ else fi fi +# THE PARAMETER RULES LIVE IN bin/check_parameters.sh, which step 0 runs as well. A project told +# one thing here and another during the run would be worse than either alone, so there is one +# implementation and two callers. +# +# It reads the COMPOSED config, so a project that pinned an option string by hand is judged on +# what will actually run. Silence means nothing to flag. +if [ "$PARSED" -eq 1 ]; then + checked=$((checked + 1)) + if [ ! -x "$INSTALL_DIR/bin/check_parameters.sh" ]; then + warn "parameters" "NOT CHECKED" "bin/check_parameters.sh is missing" + else + findings=$(cd "$PROJECT_DIR" && nextflow config -flat "$INSTALL_DIR" 2>/dev/null \ + | "$INSTALL_DIR/bin/check_parameters.sh") || true + if [ -z "$findings" ]; then + pass "parameters" "NOTHING TO FLAG" + else + printf '%s\n' "$findings" | while IFS=$'\t' read -r level label headline detail why; do + [ -n "$level" ] || continue + case $level in + FAIL) fail "$label" "$headline" "$detail" ;; + WARN) warn "$label" "$headline" "$detail" ;; + *) note "$label" "$headline${detail:+: $detail}" ;; + esac + # Folded here rather than in the helper: the helper emits one line per finding so + # a caller can read it with `read`, and step 0 wraps to its own width. + [ -n "$why" ] && printf '%s' "$why" | fold -s -w 72 \ + | sed "s|^| ${DIM}|;s|$|${RESET}|" + done + # The pipeline above runs in a subshell, so `missing` did not survive it. Ask the + # helper again for the status alone, which is what decides this script's own. + if printf '%s\n' "$findings" | grep -q '^FAIL'; then + missing=$((missing + 1)) + fi + fi + fi +fi + echo # ----------------------------------------------------------------- 2. tools -- diff --git a/bin/config_migrate.sh b/bin/config_migrate.sh index 80f5c86..b3447e5 100755 --- a/bin/config_migrate.sh +++ b/bin/config_migrate.sh @@ -356,4 +356,16 @@ if [ -n "$RGFILE" ]; then echo fi +# Read back from the report, so this fires exactly when the migration added the parameter: a +# config that already sets it has chosen its own value and is told nothing. +NEW_DROPZERO=$(awk -F'\t' '$1 == "NEW" && $2 == "vcffilter.dropZeroDepth" { print $3 }' "$REPORT") +if [ -n "$NEW_DROPZERO" ]; then + note_heading + echo " vcffilter.dropZeroDepth IS NEW, and your config now has its default." + echo + echo " The default is $NEW_DROPZERO: a site where any pool has no reads at all is dropped." + echo " To keep those sites, with the unread pool published as NA, set it to false." + echo +fi + echo "Review $OUT before running the pipeline." diff --git a/bin/depth2freq.awk b/bin/depth2freq.awk index cb28b94..a10f969 100755 --- a/bin/depth2freq.awk +++ b/bin/depth2freq.awk @@ -12,8 +12,8 @@ # EVERY COLUMN FROM 5 ON IS CONVERTED THE SAME WAY, TOTAL_AD included. # # Frequencies print through awk's default CONVFMT, so each is a six-significant-digit rendering -# of a ratio the input holds exactly, and a sample with no reads at a site prints 0 rather than -# a blank. Anything computing from these numbers reads the DEPTH table instead. +# of a ratio the input holds exactly. A sample with no reads at a site prints NA on every allele +# of that site. Anything computing from these numbers reads the DEPTH table instead. BEGIN { FS = OFS = "\t"; @@ -27,6 +27,9 @@ NR == 1 { chrom = $1; pos = $2; ref = $3; + # Cleared per row: a cell holding fewer counts than ALT declares alleles would otherwise + # print the entry an earlier row left at that index. + delete parsed_vals; # Process each sample column (from column 5 onwards) for (i = 5; i <= NF; i++) { @@ -39,9 +42,9 @@ NR == 1 { total += counts[j]; } - # Calculate frequencies + # Calculate frequencies, or NA on every allele where the sample has no reads at all. for (j = 1; j <= length(counts); j++) { - freqs[j] = (total > 0) ? counts[j] / total : 0; + freqs[j] = (total > 0) ? counts[j] / total : "NA"; } for (j in freqs) { diff --git a/bin/parse_metadata.py b/bin/parse_metadata.py index 61e67df..0954b5f 100755 --- a/bin/parse_metadata.py +++ b/bin/parse_metadata.py @@ -90,9 +90,13 @@ def rows_of(path): Comments are whole lines starting with `#`, tested before parsing so that a `#` inside a quoted value is left alone. CRLF is tolerated; the file is never rewritten. + + `utf-8-sig` consumes a leading UTF-8 byte-order mark, which Excel writes when it saves as + "CSV UTF-8". Plain `utf-8` leaves the mark on the front of the first header name, where no + editor shows it and no message printed from here can name it. """ out = [] - with open(path, newline="", encoding="utf-8") as handle: + with open(path, newline="", encoding="utf-8-sig") as handle: for lineno, raw in enumerate(handle, start=1): stripped = raw.strip("\r\n").strip() if not stripped or stripped.startswith("#"): @@ -102,6 +106,20 @@ def rows_of(path): return out +def leading_mark(path): + """The name of the byte-order mark the file begins with, or None. + + Read as raw bytes, because rows_of() decodes with `utf-8-sig` and consumes a UTF-8 mark: by + the time anything else here sees a column name, the evidence is gone. Only the UTF-8 mark is + looked for - a UTF-16 file never reaches this, having failed to decode. + """ + try: + with open(path, "rb") as handle: + return "UTF-8" if handle.read(3) == b"\xef\xbb\xbf" else None + except OSError: + return None + + def check(path): errors = [] # Reported and not refused: the caller exits 0 with these on stderr. @@ -115,6 +133,12 @@ def check(path): return None, [f"{path}: {exc.strerror}"], [] except csv.Error as exc: return None, [f"{path}: could not be read as CSV: {exc}"], [] + except UnicodeDecodeError: + return None, [ + f"{path}: is not UTF-8 text, so none of it could be read. Excel's " + f"'Unicode Text' and a PowerShell redirect both write UTF-16. Save it as " + f"'CSV UTF-8' and try again." + ], [] if not rows: return None, [f"{path}: is empty (only blank lines and comments)"], [] @@ -178,7 +202,9 @@ def check(path): if SAMPLE_ID not in header: errors.append( f"line {header_line}: no '{SAMPLE_ID}' column. Every row needs one: it is what " - f"joins the row to a pair of FASTQ files, and it becomes the read group's ID." + f"joins the row to a pair of FASTQ files, and it becomes the read group's ID. " + f"The names read from that line were: " + f"{', '.join(repr(column) for column in header)}" ) if not body: @@ -339,6 +365,17 @@ def main(argv): print(f" {message}", file=sys.stderr) return 1 + # A note about the FILE rather than the table, so it is added here rather than in check(): + # this one is true of the bytes on disk and says nothing about any row. + mark = leading_mark(argv[1]) + if mark: + warnings.append( + f"it begins with a {mark} byte-order mark, which was read past. Nothing about this " + f"file is wrong, but the editor that added one adds it to every file it saves, and " + f"in parameters.config it stops the run: check that file too, and dos2unix removes " + f"the mark." + ) + # stderr and exit 0: the caller reads the records off stdout either way, and step 0 reports # these in the context of the project rather than of the file. if warnings: diff --git a/bin/parse_multirun.py b/bin/parse_multirun.py index 68e5026..b146e60 100755 --- a/bin/parse_multirun.py +++ b/bin/parse_multirun.py @@ -32,9 +32,13 @@ def rows_of(path): Comments are whole lines starting with `#`, tested before parsing so that a `#` inside a quoted value is left alone. + + `utf-8-sig` consumes a leading UTF-8 byte-order mark, which Excel writes when it saves as + "CSV UTF-8". Plain `utf-8` leaves the mark on the front of the first header name, where no + editor shows it and no message printed from here can name it. """ out = [] - with open(path, newline="", encoding="utf-8") as handle: + with open(path, newline="", encoding="utf-8-sig") as handle: for lineno, raw in enumerate(handle, start=1): # CRLF is tolerated; the file the user wrote is never rewritten. stripped = raw.strip("\r\n").strip() @@ -45,6 +49,20 @@ def rows_of(path): return out +def leading_mark(path): + """The name of the byte-order mark the file begins with, or None. + + Read as raw bytes, because rows_of() decodes with `utf-8-sig` and consumes a UTF-8 mark: by + the time anything else here sees a column name, the evidence is gone. Only the UTF-8 mark is + looked for - a UTF-16 file never reaches this, having failed to decode. + """ + try: + with open(path, "rb") as handle: + return "UTF-8" if handle.read(3) == b"\xef\xbb\xbf" else None + except OSError: + return None + + def check(path): errors = [] @@ -56,6 +74,12 @@ def check(path): return None, [f"{path}: {exc.strerror}"] except csv.Error as exc: return None, [f"{path}: could not be read as CSV: {exc}"] + except UnicodeDecodeError: + return None, [ + f"{path}: is not UTF-8 text, so none of it could be read. Excel's " + f"'Unicode Text' and a PowerShell redirect both write UTF-16. Save it as " + f"'CSV UTF-8' and try again." + ] if not rows: return None, [f"{path}: is empty (only blank lines and comments)"] @@ -88,7 +112,9 @@ def check(path): if RUN_ID not in header: errors.append( f"line {header_line}: no '{RUN_ID}' column. Every run needs a name: it is what " - f"separates the runs' outputs from each other." + f"separates the runs' outputs from each other. " + f"The names read from that line were: " + f"{', '.join(repr(column) for column in header)}" ) if not body: @@ -163,6 +189,20 @@ def main(argv): print(f" {message}", file=sys.stderr) return 1 + # stderr and exit 0, as parse_metadata.py does: the caller reads the table off stdout either + # way. A note about the FILE rather than the rows, which is why it is not one of check()'s + # errors - nothing here is wrong. + mark = leading_mark(argv[1]) + if mark: + print(f"{argv[1]}: usable, with notes.", file=sys.stderr) + print( + f" it begins with a {mark} byte-order mark, which was read past. Nothing about " + f"this file is wrong, but the editor that added one adds it to every file it " + f"saves, and in parameters.config it stops the run: check that file too, and " + f"dos2unix removes the mark.", + file=sys.stderr, + ) + json.dump(runs, sys.stdout) sys.stdout.write("\n") return 0 diff --git a/bin/write_citations.py b/bin/write_citations.py index 37ea0e4..a3dd45d 100755 --- a/bin/write_citations.py +++ b/bin/write_citations.py @@ -44,7 +44,7 @@ def prose(entry: dict, key: str, reported: dict[str, str]) -> str: year = entry.get("year", "") tail = ", ".join(p for p in (who, where, year) if p) - line = f"- **{head}** — {tail}" if tail else f"- **{head}**" + line = f"- **{head}** -- {tail}" if tail else f"- **{head}**" if entry.get("doi"): line += f" \n " elif entry.get("url"): diff --git a/dev/RELEASING.md b/dev/RELEASING.md index 597a263..d4f8429 100644 --- a/dev/RELEASING.md +++ b/dev/RELEASING.md @@ -1,18 +1,12 @@ # Releasing PoolSeqFlow -The order to do a release in, what each step must show, and what bites. - -**This is a living procedure, not a record.** Correct it when a release teaches you something; it is not dated and it does not describe a particular version. `.claude/development-notes/` is where the dated records go. - -Everything happens on `dev` until step 4. Steps 4 to 10 happen on `main`. Step 11 brings `main` back. - -**Two things are not written down here and are checked by the suite instead**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`), the analysis versions (`check-analysis-versions.sh`) and the version-consistency case all run inside `00_static`. If the suite is green they passed. What follows is only the work the suite cannot do for you. +Everything happens on `dev` until step 6. Steps 6 to 11 happen on `main`. Step 12 brings `main` back. --- ## 1. Read the manual, top to bottom -In document order, not by concept. **Verify every claim against the code before trusting it** — this has caught something on nearly every page, and the existing text is not evidence. +In document order, not by concept. Verify every claim against the code -- the existing text is not evidence, and this has caught something on nearly every page. Live preview while you work: @@ -20,138 +14,159 @@ Live preview while you work: dev/scripts/serve_docs.sh # http://127.0.0.1:8055/PoolSeqFlow/ , regenerates on save ``` -What this pass is looking for: +``` +dev/scripts/check-manual-parameters.sh +``` + +Writes two scratch files into `.tmp/release-review/` and audits the parameters: + + - `commits.md` -- every commit since the last tag, in the form `bump-version.sh` will prepend to the CHANGELOG + - `manual.diff` -- the manual as published with that tag against the manual now, which is what has never been read in a release pass + - `parameters.txt` -- every settable key, whether the manual names it, and which file declares it -- **A number that moves with every commit.** Case counts, suite counts, file counts, running totals. The fix is to remove the number, not correct it. A count tied to a tag or to a closed list is fine. -- **A warning about a bug that is fixed.** Nothing prompts you to delete one when the bug goes. -- **Developer shorthand in user-facing prose.** "The pipeline will refuse", not "step 0 refuses". -- **A capability written as the only path.** "Each pool *can be set to* get its own", not "each pool gets its own". +Every settable parameter must be named, with what it is, what it does, and how to set it. Silence on stdout means every key is named, which is not the same as documented: `parameters.txt` is the list to read the manual against. -**Every settable parameter must be named in the manual**, and for each one: what it is, what it does, how to set it. Re-run the audit after any template change — the extraction and the grep are in `.claude/development-notes/` under the manual pass. A key that lives in `nextflow.config` rather than in the template is invisible to a template-driven audit, so extract from both. +Remove `.tmp/release-review/` when the read is done. -Version-dependent prose is read here in the *old* version's state, because the bump is step 6. Re-read just those passages after step 6 rather than moving this step. +Version-dependent prose is stale here: the bump is step 7. Note those passages and re-read them after it. ## 2. Update, prove and freeze both conda environments -### Update and freeze +### Update, prove and freeze ``` dev/scripts/prep-version.sh ``` -It clones **both** environments — `PoolSeqFlow-` and `PoolSeqFlow--analysis` — into scratch copies, runs `conda update --all` in each so the tools move as one consistent set, runs the **full suite** against both at once, and exports them to `install/environment.yml` and `install/environment-analysis.yml` **only if that passes**. Nothing is frozen against an environment the suite did not go green on — that refusal is the point of the script. + - solves **both shipped files** (`install/environment.yml` and `install/environment-analysis.yml`) into scratch environments, which is the baseline a user installs from rather than whatever is installed here + - runs `conda update --all` in each + - **moves any module pin the update left behind, and bumps that module with it**, before the suite, so the suite runs on what will ship + - runs the **full suite**, and if it passes, + - reads what each environment requires of its host and exports both files + - then proves the files it wrote: `check-exported-floor.sh` solves each from nothing and reads its floor, and `check-module-packages.sh` puts the module pins through a baseline built from the new file + +The scratch environments are removed on every exit. `--no-cleanup` keeps them, to reproduce a failure against. The conda package cache is left alone. -Three things it refuses before it starts, so none of them costs you the run: a scratch environment left over from an earlier attempt; an analysis environment that is not there (build one with `./PoolSeqFlow analysis install`, or name another with `--from-analysis `); and an analysis environment carrying an installed module's packages, which the export at the end would refuse to freeze. +Then read `dev/logs/prep--/`, including the per-environment table of what moved. -**Read the Nextflow line if one appears.** Both environments carry Nextflow and `install/environment-analysis.yml` says the analysis one carries the pipeline environment's version. Two independent solves can land either side of a release, so the script reports a divergence and leaves the decision to you. +**A moved pin is a module change.** There is one version a pin may name -- whatever the shared analysis environment holds -- so when the update moves a package, `prep-version.sh` rewrites the pin in every manifest declaring it and bumps that module's version. Read those edits in the diff with the environment files: they are uncommitted like everything else here, and every module whose version moved is republished at step 11. A package version that changes what a module computes is the thing to look for. -Read `dev/logs/prep--/` afterwards, including the table per environment of which packages moved. +**Raising the host floor takes three things together**: a line in the manual's Requirements, a move of `HOST_GLIBC_FLOOR` in `export-environment.sh` (`2.28`), and a CHANGELOG entry naming the machines that lose support. The floor is never raised to make a solve pass. -### Prove the module packages still solve against what was just frozen +**Skippable when nothing has drifted**, which is an hour. Export both to a scratch path and diff -- the second argument is what keeps this off the shipped files: ``` -dev/scripts/check-module-packages.sh +dev/scripts/export-environment.sh PoolSeqFlow- /tmp/a.yml +dev/scripts/export-environment.sh PoolSeqFlow--analysis /tmp/b.yml +diff /tmp/a.yml install/environment.yml && diff /tmp/b.yml install/environment-analysis.yml ``` -**Real conda, real network, minutes.** It is deliberately not in the suite, which is exactly why it gets skipped — it is on this checklist or it is lost. - -It builds the baseline, installs what the modules published from this repository declare, and checks a module cannot move a version another module or the release itself is running on. It found on its first run that `--freeze-installed` does **less** than its name suggests: it refuses to change a package the solve reaches on its own, but a package named on the command line it installs at the version asked for, downgrading what is there. +Identical both ways and the freeze still holds. Run the two proofs anyway: they answer a question about the files, which no diff does: -Every check must say `ok`. A failure here is a compatibility problem between this release and a module, and it is settled by publishing, not on a user's machine. +``` +dev/scripts/check-exported-floor.sh +dev/scripts/check-module-packages.sh +``` -### Prove the frozen environment installs on an older host than this one +### Read both diffs ``` -dev/scripts/check-exported-floor.sh +git diff install/environment.yml install/environment-analysis.yml ``` -It solves each exported file into a scratch environment of its own, reads what that really requires of its host, and discards it. Minutes, and a real network solve. +Classify before reading line by line: -**It replaces `./PoolSeqFlow analysis install` + `check-host-floor.sh`, which could not answer this and was unsafe besides.** The bump is step 6, so here the wrapper still declares the *old* version. `analysis install` therefore looks for `PoolSeqFlow--analysis`, finds it, prints `already exists` and creates nothing — and `check-host-floor.sh` defaults to that same old name, so the step read out the floor of the release being replaced while looking like a check on the new one. And removing the old environment first, which is the obvious way to make the install actually run, is worse: it builds the **new** file under the **old** version's name, leaving the release you have not replaced yet pointing at an environment that is no longer its own. + - a version that moved + - a build string that moved on its own (a conda-forge rebuild, usually `libgcc`) + - a package added + - a package **removed** -- stop on these -**This is the step v3.1.1 did not have, and a cluster found what it missed.** The analysis environment carries a compiler, because every module builds its hot path on the user's machine, and a compiler is built against a particular glibc. `conda update --all` takes the newest build of everything *this* machine can install, so `sysroot_linux-64` climbs to the maintainer's own glibc unless something says otherwise — v3.1.1 shipped `sysroot_linux-64=2.39`, which declares `__glibc >=2.39`, and no host below that could solve it. It installed perfectly here and nowhere older. +The first pinned export of `environment-analysis.yml` shows a large *added* count that is not new software: a hand-written spec names what you ask for, an export names what you get. -`prep-version.sh` now pins the floor into the scratch environment before it updates, and `export-environment.sh` refuses to write a file that breaks it, so this should pass without incident. Run it anyway: those two guard the package whose *version* is the glibc it targets, and **every other package carries its constraint in conda metadata instead**. `libsanitizer=16.2.0` requires `__glibc >=2.17` and nothing in the string `16.2.0` says so. This reads `conda-meta/*.json` in the installed environment, which is where the real constraints are, and reports the highest lower bound anything actually imposes. +## 3. Prepare the configuration migration -It needs an environment that really exists, which is why the script builds one rather than asking. A dry-run solve cannot substitute: measured 2026-09-21, `conda env create --dry-run --json` returns no `depends` key at all — 0 of 190 records carried one. +Double check that `bin/config_migrate.sh` is ready for this release. List what this release did to the parameter set, then for each one: -**And it asks a different question from the one `prep-version.sh` already answered at `[4/5]`.** That checks the floor of the **cloned and updated** scratch environments, before the export, which is what catches an update raising the floor while the solve is still in hand. This checks the file that was then **written**, solved from nothing the way a user's install resolves it. A clone carries whatever the source environment held; a file names constraints and lets the solver choose again. + - **renamed** -- a line in `renamed()`, or it reads as one DROPPED plus one NEW and the user silently gets the template default + - **meaning or format changed** -- a line in `reformatted()`, so the template value wins + - **added, and its absence would change behavior** -- resolve its default in `nextflow.config`, below the include and guarded by `containsKey` + - **added** -- a note under "Read these before your next run", giving the default and what to write for the other value + - **its default moved** -- nothing to do here, and that is the point: an existing config keeps its own value, so an upgrading project goes on behaving as it did while a new project behaves differently from the same reads. `config_migrate.sh` must not change an analysis setting, so the CHANGELOG is the only place a user can learn it -If it refuses, the floor is a release decision and not a solve artifact. Raising it drops machines, so it takes a line in the manual's Requirements, a move of `HOST_GLIBC_FLOOR` in `export-environment.sh`, and a CHANGELOG entry saying which machines just lost support. +If this release renames `storageDir`, move the marker in `config_is_current()` with it. -### Run `00_static` again, against the files that were just written +Then read the report a migrating user would see: ``` -bash test/run_tests.sh --suite 00_static +dev/scripts/check-migration.sh ``` -**`prep-version.sh` runs the suite at `[3/5]` and exports at `[4/5]`, so the files it writes are never seen by the suite that approved them.** The cases that read them — no `name:` or `prefix:` key, no absolute home path, and no package leaving a shipped file — only bite on a run after the export. +**Read `Kept your value` as carefully as the rest.** It is where a moved default shows up, written as `template -> yours`, and it is the one block that looks like nothing happened. -### Read both diffs +Also run the language sweep before the merge: ``` -git diff install/environment.yml install/environment-analysis.yml +dev/scripts/americanize.py # report; --fix rewrites the safe ones ``` -A pin that moved is a change to what every result was produced under. Classify before reading line by line: a version that moved, a build string that moved on its own (a conda-forge rebuild, usually because `libgcc` moved), a package added, a package **removed**. Removals are the ones to stop on, and the first pinned export of `environment-analysis.yml` will show a large *added* count that is not new software — the hand-written spec named what you ask for and an export names what you get, so the transitive dependencies become explicit all at once. +`catalogue` and `analyses` are **not** errors and the script says so. Read every risky hit by hand. -To export one environment by hand, or to ask whether an export would be accepted without writing anything: +## 4. Run the analysis version gate ``` -dev/scripts/export-environment.sh PoolSeqFlow--analysis -dev/scripts/export-environment.sh --check PoolSeqFlow--analysis +dev/scripts/check-analysis-versions.sh --release ``` -**It refuses to export an environment carrying an analysis module's packages.** Exporting one in that state folds them into the baseline every project installs, where nothing declares them and nothing can remove them — permanently and silently. Uninstall the modules first, or rebuild the environment from the shipped file and export that. - -## 3. Prepare the configuration migration +**Must pass.** Commit anything outstanding under `analysis/` first, and not on a shallow clone: `--release` treats a question it could not answer as a failure rather than a skip. -`bin/config_migrate.sh` carries a user's `parameters.config` onto the new template. Two halves: +Bump whatever it names behind: -- **The migration itself.** Every dropped parameter is handled here; there are no legacy fallbacks in `.nf` files. Check the migration report's categories are right — a knob that is merely commented out in the new template is *not* a parameter that is gone, and the report must not say it is. -- **The user who never runs it.** `require_migrated_config` in the wrapper refuses a config from an older release and names `migrate_config`, so this path ends somewhere useful rather than in a step interpolating an absent parameter into a path. **It turns on one marker — `storageDir` being assigned — so a release that renames that root has to move the marker with it**, and `02_launcher`'s `a config from an older release is refused with the fix` is what says so. It guards what runs the pipeline, runs a module, or acts on either's outputs; never `migrate_config`, `clean` or `dryclean`. +``` +dev/scripts/bump-analysis-version.sh frame +dev/scripts/bump-analysis-version.sh index +dev/scripts/bump-analysis-version.sh module +``` -Also run the language sweep before the merge, because a large prose pass is where drift enters: +## 5. Run the full suite ``` -dev/scripts/americanize.py # report; --fix rewrites the safe ones +bash test/run_tests.sh ``` -`catalogue` and `analyses` are **not** errors and the script says so. Read every risky hit by hand. +A full run solves **both** `install/environment.yml` and `install/environment-analysis.yml` into a scratch pair, runs against those, and removes them on the way out. -## 4. Merge to `main` +Only a full run does this. `--fast` and `--suite` resolve this tree's exact version and skip what is not installed, so they stay cheap, and neither ever borrows another release's environment. -Open the merge request, review the diff as a whole, merge. +## 6. Merge to `main` -The release commits — the version bump and the CHANGELOG — are made on `main` after this, so `main` briefly holds the merge at the old version. That state is never tagged, so it costs nothing, and it keeps release-only commits off `dev` until step 11. +Open the merge request, review the diff as a whole, merge. -## 5. Run the analysis version gate -Three versions cover the analysis layer and nothing in the pipeline forces any of them to move: `analysis/frame.version` (the frame config and everything under `analysis/lib/`), the catalogue's `#!index-version`, and each module's own manifest version. +## 7. Bump the version ``` -dev/scripts/check-analysis-versions.sh --release +dev/scripts/bump-version.sh ``` -`--release` is stricter than the everyday run in one way that matters: **it refuses to answer when it cannot, rather than answering wrongly.** It stops on a shallow clone and on uncommitted changes under `analysis/`, and it treats a comparison it could not make as a failure. - -Mid-development a version is legitimately behind, which is why the plain run only reports and why none of this is in the test suite. +Abandoning the cycle after this: `dev/scripts/bump-version.sh --revert` undoes the bump and the CHANGELOG section it added. It refuses once the version has been tagged. -`release.yml` runs this on every tag, so a tag pushed without it is refused before anything is published. Running it by hand here is how you find out before the tag rather than after. - -## 6. Bump the version +**Then raise the environment floor of every module step 2 named.** A moved pin names a package version only this release's environment holds, so those modules require this release and nothing earlier -- and `environment` in their manifests is what tells a user on the older release to upgrade, instead of letting the install reach conda and fail there. ``` -dev/scripts/bump-version.sh +"environment": "" in each modules//manifest.json ``` -It rewrites the version in the wrapper (header comment and `VERSION=`) and in `nextflow.config`'s manifest, and prepends a CHANGELOG section listing every commit since the last release tag. It does not commit, tag or push — it prints those commands. +**It has to be here and not at step 2.** `analysis/lib/nf/modules.nf` refuses at run time any module whose floor is above the running release, so setting it before the bump makes every one of those modules unrunnable in its own repository: -**It does not touch a module or library manifest, and must not.** No module ships inside a release, so a module's `"environment"` is the oldest release its author says it needs — moved when its needs move, by whoever maintains it, not by a release bump acting on its behalf. `dev/scripts/bump-analysis-version.sh module ` is what moves a module's own version when you change it. +```text +'mds' v20261004.001 needs the analysis environment of PoolSeqFlow 3.3.0 or newer, and this is 3.2.0. +``` + +No second version bump is needed -- the module's version already moved at step 2 and nothing is committed yet, and the gate compares committed state. -## 7. Run the full suite +## 8. Run the full suite -**Install the new version first.** The suite is worthless without both environments and says so only in the skip count: +**Install the new version first**, and here that is the point rather than a prerequisite: this step is the only one that measures the environments a user actually receives, instead of a scratch solve of the files that describe them. ``` ./PoolSeqFlow install @@ -161,45 +176,40 @@ dev/scripts/check-host-floor.sh bash test/run_tests.sh ``` -On `main`, at the new version, with the frozen environments. This is the run that matters — everything before it tested a version string that is no longer the one shipping. The install is also what proves the files frozen at step 2 actually describe an installable environment, which nothing else checks. - -**The environments are named after the version, so step 6's bump renames them out from under the suite.** Discovery handles that: `find_release_env()` takes the pair matching the wrapper's own `VERSION`, and warns on stderr when it has to settle for another — so a tree at the new version cannot measure the old environment in silence. The run prints both paths it resolved in its first two lines, which is how it states what it measured. - -**Do not name them by hand.** This step used to set `TEST_CONDA_ENV` and `TEST_ANALYSIS_ENV` to literal `$HOME/.local/opt/miniconda3/...` paths, which is a fact about one machine written into the release protocol. A path that does not resolve is not an error: `have_tools` goes false, every case needing an environment skips, and the run reports success. Measured — pointing both at a nonexistent directory prints `none found - suites needing the pipeline will skip` and exits 0. Discovery either finds the pair or says which one it fell back to. - -That advice was the workaround for a discovery bug, kept after the bug was fixed. Discovery globbed `$HOME/.conda/envs/PoolSeqFlow-*` only, which finds nothing when conda keeps its environments under its own `envs/` — as miniconda does — so a machine with all four environments installed produced a full run in which every module case skipped for "no conda environment" and still reported success. The glob also took whatever sorted first, which is the oldest environment present. Both are fixed in `test/run_tests.sh`. +On `main`, at the new version. The suite resolves both environments and runs tests on them. -**A skip is a hole, not a pass.** `0 skipped` is the number that matters; the pass count moves with every commit, so compare it against the previous release's rather than against anything written here. Read the skip list, never the count alone: +**`0 skipped` and `O failed` are the numbers that matter.** -``` -bash test/run_tests.sh 2>&1 | grep SKIP -``` - -**Check the counts against the previous run, not only the exit status** — cases passed *and* cases skipped. A filter that matches nothing also reports success, and a skip is how a case that should have run says so quietly. - -**Lint without `modules/`**, at zero errors and zero warnings: +If it fails, nothing is published yet: undo the bump, remove what you just installed, and triage on `dev`. ``` -nextflow lint analysis analysis.nf dryrun.nf poolseqflow.nf scripts +dev/scripts/bump-version.sh --revert +PoolSeqFlow uninstall +git switch dev ``` -`nextflow lint .` cannot pass: a module's `main.nf` imports the frame as `'../../lib/nf/plan.nf'`, which resolves from the store it is installed into and not from `modules//`. `00_static` lints the modules, in an assembled store layout. - -## 8. Write the CHANGELOG +## 9. Write the CHANGELOG -`bump-version.sh` already generated the commit list under `### Commits`. **Write the release notes ABOVE that heading, never over it** — the commit list stays as the record of what landed. +- `bump-version.sh` already generated the commit list under `### Commits`. +- **Write the release notes ABOVE that heading**. +- Do not remove `### Commits`. They demonstrate the work. -What the notes owe a reader, beyond the commits: anything a user has to *do*, anything whose meaning changed, and every parameter that is gone or is now computed. +What the notes owe a reader, beyond the commits: +- anything a user has to *do*, +- anything whose meaning changed, and +- every parameter that is gone or is now computed. -Then check the section extracts, because `release.yml` refuses to publish a version the CHANGELOG does not describe, and this is the extractor that writes the release body. Cheaper here than as a deleted tag: +Then check the section extracts, because `release.yml` refuses to publish a version the CHANGELOG does not describe. Cheaper here than as a deleted tag: ```bash dev/scripts/changelog-section.sh 3.1.2 ``` -## 9. Finish the release +## 10. Finish the release -**Pushing the tag is the release.** `release.yml` fires on `v*`, rebuilds and verifies the archive, uses this version's CHANGELOG section as the release body, and publishes with both tarballs and `SHA256SUMS` attached. Nothing to assemble by hand. +**Pushing the tag is the release.** +- `release.yml` fires on `v*`, rebuilds and verifies the archive, uses this version's CHANGELOG section as the release body, and publishes with both tarballs and `SHA256SUMS` attached. +- Nothing to assemble by hand. ```bash git add -A && git commit -m "Version bump vX.X.X" @@ -210,44 +220,45 @@ git push origin vX.X.X Watch it at . -- **Zenodo mints a DOI for the version.** The citation machinery points at the all-versions DOI and tells a user to pick their version from it, so the version record has to exist for the citation the release prints to be answerable. -- Verify the published archive installs from scratch on a machine that has never had it: +Once the workflow has finished, verify what it published: -```bash -V=3.1.2 -cd "$(mktemp -d)" -curl -LO "https://github.com/ozankiratli/PoolSeqFlow/releases/download/v$V/PoolSeqFlow-$V.tar.gz" -curl -LO "https://github.com/ozankiratli/PoolSeqFlow/releases/download/v$V/SHA256SUMS" -sha256sum --ignore-missing -c SHA256SUMS -tar -xzf "PoolSeqFlow-$V.tar.gz" -cd "PoolSeqFlow-$V" -cp parameters.config.template parameters.config -./PoolSeqFlow install -./PoolSeqFlow analysis install -./PoolSeqFlow check install ``` +dev/scripts/check-release-archive.sh +``` + +- fetches the tarball and `SHA256SUMS`, checks them, and installs into a throwaway prefix it discards. +- **does not re-create the conda environment when one is already there**, because the name comes from the version alone. +- answers whether the archive is complete and deployable. +- not whether the environment files solve (done with `check-exported-floor.sh` at step 2) -## 10. Publish the modules and libraries this release runs +## 11. Publish the modules and libraries this release runs -No module ships inside a release, so a release on its own leaves users with an empty store. Publishing is what makes the modules installable, and it is separate on purpose: a module moves on its own timetable, and one published tomorrow is installable into this release without re-releasing anything. +No module ships inside a release, so a release on its own leaves users with an empty store. + +What the tree has that the catalogue does not: ``` -dev/scripts/publish-module.sh [ref] +dev/scripts/publish-module.sh --list ``` -It builds the tarball into `modules/repo/`, reads `kind`, `contract`, `frame`, `environment` and `summary` out of the thing's own manifest, appends the catalogue row and bumps `#!index-version`. It takes a module or a library by name and finds it in `modules/` or `modules/lib/`. +Then each one it names, module or library alike: -**Publish from a commit.** The script refuses a source with no commit timestamp, because the tarball's reproducibility depends on it — archiving a tree stamps *now* and two builds of one ref stop matching. So this comes after the release is committed, not before. +``` +dev/scripts/publish-module.sh +``` -**The tarball and the row it advertises go out in one commit.** The site deploys `modules/repo/` wholesale at the published address `/PoolSeqFlow/modules-repo/`, so a row committed without its file advertises a download that 404s until the next deploy. +It writes the tarball and the catalogue row together and commits neither. **Commit them together** -- the site deploys `modules/repo/` wholesale, so a row without its file advertises a download that 404s until the next deploy -- and push, which is what deploys. -**A published version is never rewritten.** Somebody may have installed it and its checksum is in the catalogue. Change means bumping the version and publishing that; `00_static` checks every row's file exists and its checksum matches. +**This is after the release, and a pin move is why it cannot move earlier.** A module whose pin step 2 moved names a package version only this release's environment holds, so its `environment` floor is this release, and that floor cannot be written before the version exists. The catalogue carries a row per module version and the wrapper installs the newest row whose floor an installation clears, so everyone on an earlier release keeps the row they were already running and sees the new one as `needs PoolSeqFlow `. -## 11. Return to `dev` +**Read what the moved pin does to the result.** A package version is an input to what a module computes. Step 2 moves the pin because there is only one version it may name, not because the new version is known to compute the same numbers -- so a module whose pin moved is worth a look before it is published, and worth a CHANGELOG line if the answer changed. + +## 12. Return to `dev` Sync `dev` with `main` so the version bump and the CHANGELOG come back, then carry on. The first commits after a release are usually the things this protocol found and deferred. ``` +git checkout dev git merge --ff-only main ``` @@ -255,41 +266,6 @@ git merge --ff-only main ## What to do when a step fails -**Do not work around it on the release branch.** Everything here is either a gate that found something or a gate that is wrong. If it found something, fix it on `dev` and start again from the step that would notice. If the gate is wrong, fix the gate — and add the case that would have caught what it missed, because a gate nobody trusts is worse than no gate. - -**A number in a message is worth reading twice.** The most common defect this project has shipped is not a broken feature — it is a gate reporting success over work it had not done. - -They look alike, and each one is cheap to spot once you know the shape: - -- **A filter that matched nothing still says PASS.** `--case "no package leaves"` selected zero cases and printed `PASS 0 passed`; the underscore form selected one. Read the count, not the word. -- **A fixture that failed to set itself up still says PASS.** Three cases wrote `modules-repo/index.tsv` into a directory their `mkdir -p` never created, and the checker skips its catalogue check when the file is absent. The tell was two stderr lines in a run that reported success. -- **A check that asks the developer's machine instead of the release still says PASS.** `have_report_tools` used `command -v typst`, and the maintainer had one; the shipped environment did not. -- **A literal that happens to be right still says PASS.** A case asserted `${EXPECTED_VERSION:-2.2.0}` against a variable set nowhere. Correct until the bump, then a failure that says nothing about what it tests. - -**A version bump is when this class surfaces**, because everything before it ran against a tree carrying the old version. That is why step 7 is after step 6 and not before it. - ---- - -## Post-release triage - -Things this protocol worked around rather than fixed. Each has a note saying what the workaround is, so a release is never blocked on one — and each is a gate that is weaker than it reads, so none of them should sit here long. - -**`run_tests.sh` reports success over cases it never ran.** Measured on 2026-09-10: a full run said `549 passed, 3 skipped` and exit 0, and the three that skipped — the PDF report, the compiled hot path, and the compiled-and-parallel agreement — are among the least trivial in the suite, F1's Rcpp worker bug having been caught by the combination of compiled *and* parallel and by nothing else. On 2026-09-23 the same shape was far worse: every module case skipped, and the run still exited 0. - -**The two discovery defects behind those particular runs are fixed** — `conda info --base` returning a plugin's error on stdout, and a glob that looked in one directory and sorted to the oldest match. The item itself is not, because the next thing to go wrong will also end in a skip. What the suite should do is **refuse**, the way `check-analysis-versions.sh --release` refuses rather than answering: asked for a full run, a suite that cannot find the environment its cases need should say so and exit non-zero. `--fast` and `--cost static` legitimately skip those, so the refusal belongs to the unfiltered run alone. - -Until then the workaround is step 7's and it is a person remembering: name both environments, and read the skip count. The header now prints both environments it resolved, so a run that found neither says so in its first two lines. - -**A release step that shells out to `conda` cannot assume `conda` works.** On a machine where the shell function is set up for an interactive shell of a different family, a non-interactive `bash -c 'conda env list'` fails with `__conda_exe: permission denied` — and `env_exists()` in `prep-version.sh` is `conda env list | grep -qxF`, so a broken function reads as *the environment is not there* and the script refuses a release that had nothing wrong with it. A misconfigured plugin is the milder version of the same thing: `anaconda-anon-usage` prints an error line on every invocation while conda still works, which is noise in a log that a person is being asked to read carefully. - -The workaround is to `source /etc/profile.d/conda.sh` before running anything that needs conda. What the scripts should do instead is source it themselves, or resolve the real binary and stop depending on the shell at all — and `env_exists()` in particular should tell *conda said no* apart from *conda did not run*, because those are opposite problems wearing the same message. - -**Step 2 is skippable when nothing has drifted, and the protocol should say how to know.** Its expensive half updates and re-freezes both environments; when the last freeze already describes them, that work produces an identical file and an hour of nothing. The check is two exports to a scratch path and a diff: - -``` -dev/scripts/export-environment.sh PoolSeqFlow- /tmp/a.yml -dev/scripts/export-environment.sh PoolSeqFlow--analysis /tmp/b.yml -diff /tmp/a.yml install/environment.yml && diff /tmp/b.yml install/environment-analysis.yml -``` +**Do not work around it on the release branch.** If a gate found something, fix it on `dev` and start again from the step that would notice. If the gate is wrong, fix the gate, and add the case that would have caught what it missed. -Identical both ways means the freeze still holds and the update-and-export cycle is redundant. **`check-module-packages.sh` is not part of that skip** — it asks a different question, whether the modules' own pins still solve against the frozen baseline, and a module's manifest can change on a day the environments do not. +**Read the count, not the word.** The most common defect this project has shipped is a gate reporting success over work it did not do. A `--case` filter matching nothing prints `PASS 0 passed`, and it matches the underscored function name rather than the displayed one. diff --git a/dev/logo/README.md b/dev/logo/README.md index 9152835..5fef133 100644 --- a/dev/logo/README.md +++ b/dev/logo/README.md @@ -1,26 +1,26 @@ # Logo source files -`dev/` is tracked in git but carries `export-ignore` in `.gitattributes`, so nothing here reaches a release archive — these files live in the repository without shipping to users. +`dev/` is tracked in git but carries `export-ignore` in `.gitattributes`, so nothing here reaches a release archive -- these files live in the repository without shipping to users. | File | Shipped as | Used at | |---|---|---| -| `logo-full.svg` | `manual/assets/logo-full.svg` | 48px and up — README hero, print, posters | -| `logo-small.svg` | `manual/assets/logo.svg`, and the mark inside `manual/assets/favicon.svg` | 16–32px — site header, favicon | +| `logo-full.svg` | `manual/assets/logo-full.svg` | 48px and up -- README hero, print, posters | +| `logo-small.svg` | `manual/assets/logo.svg`, and the mark inside `manual/assets/favicon.svg` | 16-32px -- site header, favicon | Both are the same mark. The vessel and the single helix are byte-identical between them; only the three pools differ. ## Editing -Edit `logo-full.svg` first, then regenerate `logo-small.svg` from it rather than changing the two independently — the point of the pair is that the silhouettes match exactly. +Edit `logo-full.svg` first, then regenerate `logo-small.svg` from it rather than changing the two independently -- the point of the pair is that the silhouettes match exactly. To rebuild the favicon after a change: take `logo-small.svg`, wrap the mark in ``` - … + ... ``` -and multiply every stroke width by 1.13 to cancel the 0.883 scale. The scale and offset come from the mark's bounding box (x 14.8–49.2, y 3.1–64.3, center 32,33.67) fitted into the tile with 5 units of padding. +and multiply every stroke width by 1.13 to cancel the 0.883 scale. The scale and offset come from the mark's bounding box (x 14.8-49.2, y 3.1-64.3, center 32,33.67) fitted into the tile with 5 units of padding. ## The helix recipe @@ -28,7 +28,7 @@ Each half-period is a single cubic: it starts and ends on the center line, with control offset = amplitude x 4/3 -The 4/3 is because a cubic Bézier never reaches its control points — it peaks at three quarters of the way there. So to bulge 6 units off center, put the controls 8 out. Mirror the offset for the second strand. Rungs sit at each half-period's midpoint, spaced by the **amplitude**, not the control offset. +The 4/3 is because a cubic Bézier never reaches its control points -- it peaks at three quarters of the way there. So to bulge 6 units off center, put the controls 8 out. Mirror the offset for the second strand. Rungs sit at each half-period's midpoint, spaced by the **amplitude**, not the control offset. Current values: @@ -39,7 +39,7 @@ Current values: ## Notes from getting here -**A downward fan reads as legs.** Earlier drafts had three strands meeting at an apex above a wide open base, and every one of them looked like a tripod. Closing the base into a vessel is what fixed it — a closed silhouette reads as an object. +**A downward fan reads as legs.** Earlier drafts had three strands meeting at an apex above a wide open base, and every one of them looked like a tripod. Closing the base into a vessel is what fixed it -- a closed silhouette reads as an object. **A double helix cannot survive 16px.** Rungs need roughly 3px of clear space between strands; a 16px favicon gives about 1. That is why `logo-small.svg` exists rather than just scaling the full mark down. diff --git a/dev/release-notes-tail.md b/dev/release-notes-tail.md index 5b3eda4..d9c51ab 100644 --- a/dev/release-notes-tail.md +++ b/dev/release-notes-tail.md @@ -29,7 +29,7 @@ scripted installs: **Upgrading an existing project?** Your `parameters.config` is not touched by a new version and can be missing parameters this release -expects. Run `./PoolSeqFlow migrate_config` and read what it reports — +expects. Run `./PoolSeqFlow migrate_config` and read what it reports -- see [Upgrading](https://ozankiratli.github.io/PoolSeqFlow/getting-started/upgrading/). Full documentation: diff --git a/dev/scripts/bench-compiled-paths.R b/dev/scripts/bench-compiled-paths.R index c3c9485..a205c2a 100755 --- a/dev/scripts/bench-compiled-paths.R +++ b/dev/scripts/bench-compiled-paths.R @@ -90,7 +90,7 @@ for (n in sizes) { # # ITS RATIO IS NOT THE PARSERS' RATIO AND SHOULD NOT BE AVERAGED WITH THEM. The two above # are string splits, memory-bandwidth bound, and land near ten. This one is k(k-1)/2 - # separate rowsum() passes over the allele matrix — fifteen at six pools — so what the + # separate rowsum() passes over the allele matrix -- fifteen at six pools -- so what the # compiled form removes is interpreted call overhead that scales with the PAIR count, not # with the site count. Expect it to climb with more pools where the parsers' will not. parsed <- allele_frequencies_cpp(cells) diff --git a/dev/scripts/build_docs.py b/dev/scripts/build_docs.py index 9b820d5..c7cd8b6 100644 --- a/dev/scripts/build_docs.py +++ b/dev/scripts/build_docs.py @@ -513,7 +513,10 @@ def format_entry(key: str, entry: dict[str, str], suffix: str = "") -> list[str] volume = entry.get("volume", "") if volume and entry.get("number"): volume += f"({entry['number']})" - locus = ", ".join(part for part in (volume, entry.get("pages", "").replace("--", "–")) if part) + # BibTeX writes a page range as `325--338`. Rendered as a plain hyphen, not an en dash: this + # line is written back into manual/PoolSeqFlow-manual.md, so what it produces is bytes in a + # tracked file and the repository is ASCII. + locus = ", ".join(part for part in (volume, entry.get("pages", "").replace("--", "-")) if part) year = f" ({entry['year']})." if entry.get("year") else "." head = f"**{format_authors(entry)}**{year} {detex(entry.get('title', ''))}." tail = " ".join(part for part in (where, locus) if part) diff --git a/dev/scripts/check-manual-parameters.sh b/dev/scripts/check-manual-parameters.sh new file mode 100755 index 0000000..5b52b61 --- /dev/null +++ b/dev/scripts/check-manual-parameters.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# +# Lay out what a release's read of the manual needs. +# +# Usage: dev/scripts/check-manual-parameters.sh [previous-tag] +# default: the newest tag, which during a release is the release being replaced +# +# Writes three scratch files into .tmp/release-review/. They are scratch: .tmp/ is gitignored and +# nothing reads them but a person. Delete the directory when the read is done. +# +# commits.md every commit since the previous tag, in the form bump-version.sh will prepend +# to the CHANGELOG, so the notes can be written against the real list. +# manual.diff the manual as published with that tag against the manual now. What changed +# since the last release is what has never been read in a release pass. +# parameters.txt every parameter a project may set, in the order parameters.config.template +# declares them, marked with whether the manual names it. The list to walk the +# manual against. +# +# WHAT IS ON THAT LIST IS WHAT A PROJECT SETS, and three kinds of assignment are therefore left +# off it: +# +# nextflow.config is the installation's, not a project's. A user edits parameters.config; a +# value here applies to every project on the machine and several of these +# lines exist precisely to set a default behind a template knob. +# derived values whose value references another parameter, so the pipeline computes them from +# what was set. The manual has a section for these as a class, and telling +# someone how to set one is the opposite of what it should say. +# scopes a key other keys nest under, which is a block and not a setting. +# +# A commented-out assignment stays on the list: uncommenting it is the user taking the value back, +# which is why it ships commented. +# +# THE MARK IS WHETHER THE MANUAL NAMES IT, which is not whether the manual documents it. Names are +# matched in backticks, because that is how the manual writes a setting and it is what tells +# `align` the parameter from "align" the verb. Whether the text says what the parameter does, and +# how to set it, is what you are reading for. +# +# dir.*, software.* and cores.* are each documented as a whole block rather than key by key, so a +# missing mark on one of those is not news and is not reported on stdout. Everything else is. +# +# No apostrophes in the awk below: the program is single-quoted and one would end it. + +set -euo pipefail + +ROOT="$(git rev-parse --show-toplevel)" +cd "$ROOT" + +MANUAL="manual/PoolSeqFlow-manual.md" +for f in "$MANUAL" parameters.config.template; do + [ -f "$f" ] || { echo "ERROR: $f not found" >&2; exit 1; } +done + +PREV="${1-}" +if [ -z "$PREV" ]; then + PREV="$(git describe --tags --abbrev=0 2>/dev/null || true)" +fi + +OUT=".tmp/release-review" +mkdir -p "$OUT" + +# --------------------------------------------------------------- what to read against --- + +if [ -z "$PREV" ]; then + echo "No tag to compare against; writing the whole history and no diff." + git log --no-merges --reverse --pretty='- (%h) %s' HEAD > "$OUT/commits.md" + : > "$OUT/manual.diff" +else + # The same range and the same --pretty as bump-version.sh, so this is the list that will + # appear rather than a second rendering of it. + git log --no-merges --reverse --pretty='- (%h) %s' "$PREV..HEAD" > "$OUT/commits.md" + git diff "$PREV" -- "$MANUAL" > "$OUT/manual.diff" || true +fi + +printf 'commits since %s : %s (%s commits)\n' "${PREV:-the start of history}" \ + "$OUT/commits.md" "$(wc -l < "$OUT/commits.md" | tr -d ' ')" +if [ -s "$OUT/manual.diff" ]; then + printf 'manual since %s : %s (%s changed lines)\n' "${PREV:-}" "$OUT/manual.diff" \ + "$(grep -cE '^[+-]' "$OUT/manual.diff" || true)" +else + printf 'manual since %s : unchanged\n' "${PREV:-}" +fi +echo "" + +# ------------------------------------------------------------------ what a project sets --- + +# Qualified by scope, in declaration order, derived values dropped. +settable() { + awk ' + function qualify(k, p, i) { + p = "" + for (i = 1; i <= depth; i++) p = p (p == "" ? "" : ".") stack[i] + sub(/^params\.?/, "", p) + return (p == "") ? k : p "." k + } + function literal(v) { return (index(v, "params.") == 0 && index(v, "${") == 0) } + { + line = $0; sub(/^[ \t]+/, "", line) + if (line ~ /^\}/) { if (depth > 0) depth--; next } + if (line ~ /^[A-Za-z_][A-Za-z0-9_]*[ \t]*\{[ \t]*(\/\/.*)?$/) { + name = line; sub(/[ \t]*\{.*$/, "", name); stack[++depth] = name; next + } + if (line ~ /^(\/\/[ \t]*)?[A-Za-z_][A-Za-z0-9_]*[ \t]*=/) { + commented = (line ~ /^\/\//) + k = line; sub(/^\/\/[ \t]*/, "", k); sub(/[ \t]*=.*$/, "", k) + v = line; sub(/^[^=]*=[ \t]*/, "", v); sub(/[ \t]*\/\/.*$/, "", v) + if (commented || literal(v)) print qualify(k) + } + }' parameters.config.template \ + | awk ' + # A key other keys nest under is a block, not a setting. + { all[NR] = $0; n = NR } + END { + for (i = 1; i <= n; i++) { + scope = 0 + for (j = 1; j <= n; j++) if (index(all[j], all[i] ".") == 1) { scope = 1; break } + if (!scope) print all[i] + } + }' +} + +# Named AS A PARAMETER: in backticks, by its own name or by its leaf, since the manual writes +# `adapterOptions` rather than `trim_galore.adapterOptions`. +named_in_manual() { # key + grep -qF "\`$1\`" "$MANUAL" && return 0 + grep -qF "\`params.$1\`" "$MANUAL" && return 0 + grep -qF "\`${1##*.}\`" "$MANUAL" && return 0 + return 1 +} + +MISSING=0 +: > "$OUT/parameters.txt" +while read -r key; do + [ -n "$key" ] || continue + if named_in_manual "$key"; then + printf ' %s\n' "$key" >> "$OUT/parameters.txt" + continue + fi + printf ' ? %s\n' "$key" >> "$OUT/parameters.txt" + case $key in + dir.*|software.*|cores.*) ;; + *) echo "not named in the manual: $key"; MISSING=$((MISSING + 1)) ;; + esac +done <<< "$(settable)" + +printf 'what a project may set : %s (%s parameters, %s unnamed)\n' "$OUT/parameters.txt" \ + "$(wc -l < "$OUT/parameters.txt" | tr -d ' ')" \ + "$(grep -c '^ ?' "$OUT/parameters.txt" || true)" +echo "Remove $OUT when the read is done." + +[ "$MISSING" -eq 0 ] || exit 1 diff --git a/dev/scripts/check-migration.sh b/dev/scripts/check-migration.sh new file mode 100755 index 0000000..aaa53a5 --- /dev/null +++ b/dev/scripts/check-migration.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# +# Is bin/config_migrate.sh ready for this release? +# +# Usage: dev/scripts/check-migration.sh [previous-tag] +# default: the newest v* tag, which during a release is the release being replaced +# +# Migrates the previous release's parameters.config.template onto the current one in a scratch +# directory and prints the report a user would see. Nothing in the working tree is touched. +# +# Read the categories. Two of them are wrong in ways only a person can see: a RENAMED parameter +# with no line in renamed() arrives as one DROPPED plus one NEW, and a knob that is merely +# commented out in the new template is not a parameter that is gone. +# +# The one thing checked mechanically is that every NEW parameter is mentioned in the migration's +# own notes. That is a reminder rather than a refusal: a parameter whose absence changes nothing +# needs no note, and only a person knows which those are. + +set -euo pipefail + +ROOT="$(git rev-parse --show-toplevel)" +cd "$ROOT" + +PREV="${1-}" +if [ -z "$PREV" ]; then + PREV=$(git tag -l 'v*' --sort=-v:refname | head -1) + [ -n "$PREV" ] || { echo "ERROR: no v* tag to migrate from; name one" >&2; exit 1; } +fi +git rev-parse --verify --quiet "$PREV^{commit}" > /dev/null \ + || { echo "ERROR: '$PREV' is not a commit this repository has" >&2; exit 1; } + +git show "$PREV:parameters.config.template" > /dev/null 2>&1 \ + || { echo "ERROR: '$PREV' carries no parameters.config.template" >&2; exit 1; } + +SCRATCH="$(mktemp -d)" +trap 'rm -rf "$SCRATCH"' EXIT + +git show "$PREV:parameters.config.template" > "$SCRATCH/parameters.config" +cp parameters.config.template "$SCRATCH/" + +echo "Migrating a $PREV config onto the current template." +echo "" + +REPORT="$SCRATCH/report.txt" +if ! ( cd "$SCRATCH" && bash "$ROOT/bin/config_migrate.sh" ) > "$REPORT" 2>&1; then + cat "$REPORT" + echo "" + echo "ERROR: the migration itself failed." >&2 + exit 1 +fi + +sed -n '/^Kept your value/,$p' "$REPORT" + +# Every parameter the report calls NEW, and whether the notes say anything about it. +NEW_KEYS=$(awk '/^New in this release/{f=1; next} /^$/{f=0} f && $1 != "none" {print $1}' "$REPORT") +if [ -n "$NEW_KEYS" ]; then + echo "New parameters, and whether the migration explains each:" + echo "" + MISSING=0 + while IFS= read -r key; do + [ -n "$key" ] || continue + if sed -n '/^Read these before your next run/,$p' "$REPORT" | grep -qF "$key"; then + printf ' explained %s\n' "$key" + else + printf ' NO NOTE %s\n' "$key" + MISSING=1 + fi + done <<< "$NEW_KEYS" + echo "" + if [ "$MISSING" -eq 1 ]; then + echo "A parameter whose absence changes nothing needs no note. For any that does, add one" + echo "to the \"Read these before your next run\" section of bin/config_migrate.sh, giving" + echo "the default and what to write for the other value." + fi +fi diff --git a/dev/scripts/check-release-archive.sh b/dev/scripts/check-release-archive.sh new file mode 100755 index 0000000..3093501 --- /dev/null +++ b/dev/scripts/check-release-archive.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# +# Does the published release archive install? +# +# Usage: dev/scripts/check-release-archive.sh [version] +# default: the version this working copy declares +# +# Downloads the tarball and SHA256SUMS from the GitHub release, verifies the checksum, extracts +# it, and installs into a throwaway prefix removed on every exit. Real network, minutes. Run it +# after the tag is pushed and the workflow has finished, because it fetches what was published +# rather than what is in the tree. +# +# POOLSEQFLOW_PREFIX is what keeps it out of the way: install_prefix() returns it when set and +# bindir is "$prefix/bin", so the payload and every command link land inside the sandbox and +# $HOME/.local/bin is untouched. +# +# WHAT IT DOES NOT ANSWER, because it cannot from here: +# +# The conda environment is reused when it is already present. `install` derives the name from +# the version alone - ENV_NAME="PoolSeqFlow-${VERSION}" - so a machine that has run the release +# suite already holds it and the install prints "already exists" rather than creating one. This +# answers whether the ARCHIVE is complete and deployable. Whether the environment FILES solve +# from nothing is answered by check-exported-floor.sh before the export. +# +# Nothing about a host unlike this one. Three of the four defects in +# .claude/development-notes/someone-elses-machine.md were found by running on a cluster and a +# server, and the suite was green for every one of them. + +set -euo pipefail + +ROOT="$(git rev-parse --show-toplevel)" +cd "$ROOT" + +V="${1-}" +if [ -z "$V" ]; then + V="$(sed -n 's/^VERSION="\(.*\)"$/\1/p' PoolSeqFlow | head -1)" + [ -n "$V" ] || { echo "ERROR: no VERSION= line in ./PoolSeqFlow; name a version" >&2; exit 1; } +fi + +command -v curl > /dev/null || { echo "ERROR: curl is not on PATH" >&2; exit 1; } + +SANDBOX="$(mktemp -d)" +trap 'rm -rf "$SANDBOX"' EXIT + +BASE="https://github.com/ozankiratli/PoolSeqFlow/releases/download/v$V" +echo "Fetching the published v$V archive into $SANDBOX" + +cd "$SANDBOX" +# -f so a 404 is a failure rather than a file containing GitHub's error page. +curl -fsSLO "$BASE/PoolSeqFlow-$V.tar.gz" \ + || { echo "ERROR: no published archive at $BASE/PoolSeqFlow-$V.tar.gz" >&2; exit 1; } +curl -fsSLO "$BASE/SHA256SUMS" \ + || { echo "ERROR: no SHA256SUMS at $BASE" >&2; exit 1; } + +echo "" +echo "Checksum:" +sha256sum --ignore-missing -c SHA256SUMS + +tar -xzf "PoolSeqFlow-$V.tar.gz" +cd "PoolSeqFlow-$V" +cp parameters.config.template parameters.config + +export POOLSEQFLOW_PREFIX="$SANDBOX/prefix" +echo "" +echo "Installing into $POOLSEQFLOW_PREFIX" +echo "" +./PoolSeqFlow install +./PoolSeqFlow analysis install +./PoolSeqFlow check install + +echo "" +echo "The published v$V archive installs. The sandbox is being discarded." diff --git a/dev/scripts/prep-version.sh b/dev/scripts/prep-version.sh index fd767eb..72b1f21 100755 --- a/dev/scripts/prep-version.sh +++ b/dev/scripts/prep-version.sh @@ -3,34 +3,54 @@ # Prepare the tool set for a release: update every package, prove the pipeline still works, # then record what was proven. # -# Usage: dev/scripts/prep-version.sh e.g. 2.3.0 -# dev/scripts/prep-version.sh --from --from-analysis +# Usage: dev/scripts/prep-version.sh [--no-cleanup] e.g. 2.3.0 # # A release ships two environments and both are prepared here: the one the pipeline runs in, # and the one an analysis module runs in. # +# BUILT FROM THE SHIPPED FILES, NOT CLONED FROM AN INSTALLATION. install/environment.yml and +# install/environment-analysis.yml are what a user installs from, so they are what this starts +# from. An installed environment was the wrong baseline twice over, and this script carried a +# guard against each: an installed module puts its own packages into the shared analysis +# environment, and an environment built before a line was added to the file does not hold that +# package, so cloning carried the gap into the export and the package left the release with +# nothing saying so. Neither can arise from a fresh solve of the file itself. +# # What it does, in order: # -# 1. Clones both environments for the version this working copy currently declares into -# scratch environments, PoolSeqFlow-update and PoolSeqFlow-update-analysis. -# 2. Runs `conda update --all` in each, so the tools move as one mutually consistent set -# rather than one package at a time. +# 1. Solves both shipped files into scratch environments, PoolSeqFlow-update and +# PoolSeqFlow-update-analysis, recording what each file resolves to today, then runs +# `conda update --all` in each, so the tools move as one mutually consistent set rather +# than one package at a time. +# 2. Moves any module package pin the update left behind, and bumps that module's version with +# it. A pin may name exactly one version - whatever the shared analysis environment holds - +# so the update decides it, and before the suite, so the suite runs on what will ship. It +# names the modules whose `environment` floor has to follow, and does not move it: the frame +# refuses a module whose floor is above the running release, which is still the old version +# here. That one waits for the bump. # 3. Runs the full test suite against both at once. -# 4. Only if that passes: exports them to install/environment.yml and -# install/environment-analysis.yml, then removes them. +# 4. Only if that passes: reads what each environment requires of its host, then exports them +# to install/environment.yml and install/environment-analysis.yml. +# 5. Proves the two files it just wrote, which is a different question from the environments +# above: check-exported-floor.sh solves each from nothing and reads its floor, and +# check-module-packages.sh puts the modules' pins through a baseline built from the new file. +# 6. Removes the scratch environments. +# +# Nothing is exported when the tests fail. Output lands in dev/logs/prep--/, +# including a table per environment of which packages moved. # -# Nothing is exported when the tests fail, and the scratch environments are left in place. -# Output lands in dev/logs/prep--/, including a table per environment of -# which packages moved. +# The conda package cache is left alone. `conda clean` reaches the packages every other +# environment on this machine shares, which is a decision about the whole machine. # -# This does not bump the version or touch the CHANGELOG - run dev/scripts/bump-version.sh -# afterwards. It does not commit anything. +# This does not bump the release version or touch the CHANGELOG - run dev/scripts/bump-version.sh +# afterwards. It commits nothing: the two exported files, and any manifest whose pin moved, are +# left in the tree to be reviewed and committed with the release. set -euo pipefail NEW="${1-}" if [[ ! "$NEW" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then - echo "Usage: $0 [--from ] [--from-analysis ] (e.g. 2.3.0)" >&2 + echo "Usage: $0 [--no-cleanup] (e.g. 2.3.0)" >&2 exit 1 fi shift @@ -38,23 +58,87 @@ shift ROOT="$(git rev-parse --show-toplevel)" cd "$ROOT" -CURRENT="$(sed -n 's/^VERSION="\(.*\)"$/\1/p' PoolSeqFlow | head -1)" -[ -n "$CURRENT" ] || { echo "ERROR: no VERSION= line in ./PoolSeqFlow" >&2; exit 1; } +PIPELINE_FILE="install/environment.yml" +ANALYSIS_FILE="install/environment-analysis.yml" +for f in "$PIPELINE_FILE" "$ANALYSIS_FILE"; do + [ -f "$f" ] || { echo "ERROR: $f is missing, so there is nothing to prepare from" >&2; exit 1; } +done + +# env_exists, store_packages and conda_conflicting_packages. INSTALL is what wrapper_lib +# resolves its own paths from; nothing here runs the wrapper. +INSTALL="$ROOT" +POOLSEQFLOW_INSTALLED_HOME="${POOLSEQFLOW_INSTALLED_HOME:-}" +# shellcheck source=../../lib/wrapper_lib.sh +. "$ROOT/lib/wrapper_lib.sh" + +# WHAT THIS SCRIPT NEEDS FROM THERE, NAMED, so a function that moves stops the run in its first +# second. conda_conflicting_packages is called at [2/6], on the far side of the solve and the +# update, so otherwise a rename would surface an hour in. +WRAPPER_LIB_NEEDS="env_exists store_packages module_packages conda_conflicting_packages" +for fn in $WRAPPER_LIB_NEEDS; do + declare -F "$fn" > /dev/null \ + || { echo "ERROR: lib/wrapper_lib.sh defines no $fn(), which this script calls." >&2 + exit 1; } +done -SOURCE_ENV="" -SOURCE_ANALYSIS_ENV="" +NO_CLEANUP=0 while [ $# -gt 0 ]; do case "$1" in - --from) SOURCE_ENV="${2-}"; shift ;; - --from-analysis) SOURCE_ANALYSIS_ENV="${2-}"; shift ;; + --no-cleanup) NO_CLEANUP=1 ;; *) echo "unknown option: $1" >&2; exit 1 ;; esac shift done -env_exists() { - conda env list | awk '{print $1}' | grep -qxF "$1" +# The scratch environments this run created, and only those. A conda environment outlives the +# process, so every exit path has to remove them or the next run finds them and refuses. +CREATED="" + +# Manifests this run rewrote. A conda environment is removed on the way out; a file edit is not, +# and after a failure these pin versions no export ever shipped - so they are named rather than +# reverted. Reverting would mean a git operation on files the maintainer may have edited too. +TOUCHED_MANIFESTS="" + +cleanup() { + local status=$? e log + trap - EXIT + if [ "$status" -ne 0 ] && [ -n "${TOUCHED_MANIFESTS:-}" ]; then + printf '\n' >&2 + printf 'These manifests were rewritten before this failed, and still are:\n' >&2 + for e in $TOUCHED_MANIFESTS; do printf ' %s\n' "$e" >&2; done + printf 'Each carries a moved package pin and the version bump that goes with it.\n' >&2 + printf 'Nothing was exported, so they now pin versions the shipped environment files\n' >&2 + printf 'do not hold. Keep them for the next attempt, or undo them:\n' >&2 + # shellcheck disable=SC2086 + printf ' git checkout --%s\n' "$(printf ' %s' $TOUCHED_MANIFESTS)" >&2 + printf '\n' >&2 + fi + [ -n "$CREATED" ] || exit "$status" + if [ "$NO_CLEANUP" -eq 1 ]; then + printf 'Scratch environments kept, --no-cleanup:\n' >&2 + if [ -n "${ENV_PREFIX:-}" ] && [ -n "${ANALYSIS_PREFIX:-}" ]; then + printf ' TEST_CONDA_ENV=%s \\\n' "$ENV_PREFIX" >&2 + printf ' TEST_ANALYSIS_ENV=%s \\\n' "$ANALYSIS_PREFIX" >&2 + printf ' ./test/run_tests.sh --suite \n' >&2 + printf '\n' >&2 + fi + printf 'Remove them when you are done:\n' >&2 + for e in $CREATED; do printf ' conda env remove -n %s\n' "$e" >&2; done + exit "$status" + fi + log=/dev/null + [ -n "${LOGDIR:-}" ] && [ -d "${LOGDIR:-}" ] && log="$LOGDIR/cleanup.log" + for e in $CREATED; do + conda env remove --name "$e" --yes >> "$log" 2>&1 \ + || printf "WARNING: could not remove scratch environment '%s'\n" "$e" >&2 + done + exit "$status" } +# INT and TERM route through EXIT rather than removing anything themselves, so there is one +# removal path whatever ends the run. +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM # One fixed name each, not PoolSeqFlow-. Neither outlives the run that made it. The # analysis one ends in -analysis, which is what export-environment.sh reads to decide which @@ -70,99 +154,12 @@ done if [ -n "$LEFTOVER" ]; then echo "ERROR: already present:$LEFTOVER" >&2 echo "" >&2 - echo "Left over from an earlier preparation run, most likely one whose tests failed." >&2 + echo "Left from an earlier run that used --no-cleanup, or one that was killed outright." >&2 echo "Investigate or discard before starting again:" >&2 for e in $LEFTOVER; do echo " conda env remove -n $e" >&2; done exit 1 fi -# Which environment to start from: the version this copy declares, unless told otherwise. An -# unversioned environment from an older release is accepted with a note. -if [ -z "$SOURCE_ENV" ]; then - if env_exists "PoolSeqFlow-$CURRENT"; then - SOURCE_ENV="PoolSeqFlow-$CURRENT" - elif env_exists "PoolSeqFlow"; then - SOURCE_ENV="PoolSeqFlow" - echo "Note: starting from the unversioned 'PoolSeqFlow' environment." - echo " That is an install from before environments were named per version." - echo "" - else - echo "ERROR: no environment to start from." >&2 - echo "Looked for 'PoolSeqFlow-$CURRENT' and 'PoolSeqFlow'." >&2 - echo "Install one first: ./PoolSeqFlow install" >&2 - exit 1 - fi -fi -env_exists "$SOURCE_ENV" || { echo "ERROR: no environment named '$SOURCE_ENV'" >&2; exit 1; } - -# The analysis environment is named per version too. It arrived with the analysis layer, so -# there is no unversioned form to fall back to the way the pipeline one has. -SOURCE_ANALYSIS_ENV="${SOURCE_ANALYSIS_ENV:-PoolSeqFlow-$CURRENT-analysis}" -if ! env_exists "$SOURCE_ANALYSIS_ENV"; then - echo "ERROR: no analysis environment named '$SOURCE_ANALYSIS_ENV'." >&2 - echo "" >&2 - echo "A release ships install/environment-analysis.yml as well, and it has to describe a" >&2 - echo "solve the suite passed against. Build one first:" >&2 - echo " ./PoolSeqFlow analysis install" >&2 - echo "or name an existing one with --from-analysis ." >&2 - exit 1 -fi - -# Whether the analysis export would be accepted at all. An installed module puts its own -# packages into the shared analysis environment, and export-environment.sh refuses to fold -# those into the baseline every project installs from. Asked now: the clone inherits them, so -# the export below reaches the same answer, an hour of solving and testing later. -if ! CHECK_OUT=$(bash dev/scripts/export-environment.sh --check "$SOURCE_ANALYSIS_ENV" 2>&1); then - if [ -n "$CHECK_OUT" ]; then - printf '%s\n' "$CHECK_OUT" >&2 - else - # It reads conda through a pipeline that discards conda's stderr, so a conda that - # fails outright ends the script with a status and nothing to read. - echo "ERROR: export-environment.sh --check '$SOURCE_ANALYSIS_ENV' failed silently." >&2 - echo "Run it by hand to see what conda says." >&2 - fi - exit 1 -fi - -# Every package a shipped environment file names, as bare names. Only the dependencies: block, -# so the channel list is not read as packages, and only up to the first `=`, because a version -# is what the update is about to move. -spec_packages() { - awk '/^dependencies:/ { d = 1; next } - /^[a-z]/ { d = 0 } - d && /^ *- / { sub(/^ *- */, ""); sub(/[=<> ].*/, ""); if ($0 != "") print }' "$1" | sort -u -} - -# What an environment holds that its own shipped file does not name is fine - that is what the -# solver added. What the file names and the environment does NOT hold means the environment was -# built before that line existed. Cloning it then carries the gap into the export, and the -# package leaves the release without anything saying so. -missing_from_env() { - local file="$1" env="$2" held pkg - held=$(conda list -n "$env" --export 2>/dev/null | sed -n 's/^\([^#=][^=]*\)=.*/\1/p') - while read -r pkg; do - [ -n "$pkg" ] || continue - printf '%s\n' "$held" | grep -qxF "$pkg" || printf '%s\n' "$pkg" - done < <(spec_packages "$file") -} - -STALE="" -for pair in "install/environment.yml:$SOURCE_ENV" \ - "install/environment-analysis.yml:$SOURCE_ANALYSIS_ENV"; do - f="${pair%%:*}"; e="${pair#*:}" - gone=$(missing_from_env "$f" "$e") - [ -n "$gone" ] && STALE="$STALE$e is missing, of what $f names:"$'\n'"$(printf '%s' "$gone" | sed 's/^/ /')"$'\n' -done -if [ -n "$STALE" ]; then - printf 'ERROR: a source environment is older than the file it was built from.\n\n' >&2 - printf '%s\n' "$STALE" >&2 - echo "Cloning it would carry that gap into the export, and the package would leave the" >&2 - echo "release with nothing saying so. Rebuild the environment before preparing a release:" >&2 - echo " ./PoolSeqFlow analysis uninstall && ./PoolSeqFlow analysis install" >&2 - echo "or add what is missing to the environment by hand and run this again." >&2 - exit 1 -fi - STAMP="$(date -u '+%Y%m%dT%H%M%SZ')" LOGDIR="dev/logs/prep-$NEW-$STAMP" mkdir -p "$LOGDIR" @@ -171,19 +168,30 @@ say() { printf '%s\n' "$*" | tee -a "$LOGDIR/summary.txt"; } say "PoolSeqFlow release preparation" say " target version : $NEW" -say " pipeline env : $SOURCE_ENV -> $UPDATE_ENV" -say " analysis env : $SOURCE_ANALYSIS_ENV -> $UPDATE_ANALYSIS_ENV" -say " scratch envs : removed once both exports succeed" +say " pipeline env : $PIPELINE_FILE -> $UPDATE_ENV" +say " analysis env : $ANALYSIS_FILE -> $UPDATE_ANALYSIS_ENV" +say " scratch envs : removed on every exit, unless --no-cleanup" say " logs : $LOGDIR" say "" -# Clone one environment into a scratch copy, update everything in it, and print what moved. -# $3 is the suffix that keeps the two environments' log files apart; the pipeline's is empty, -# so its file names are unchanged. +# Solve one shipped file into a scratch environment, update everything in it, and print what +# moved. $3 is the suffix that keeps the two environments' log files apart; the pipeline's is +# empty, so its file names are unchanged. prepare_env() { - local source="$1" scratch="$2" tag="$3" changed prefix floor - conda create --name "$scratch" --clone "$source" --yes > "$LOGDIR/clone$tag.log" 2>&1 \ - || { say " FAILED to clone '$source' - see $LOGDIR/clone$tag.log"; exit 1; } + local file="$1" scratch="$2" tag="$3" changed prefix floor + # Recorded before the status is judged: a solve that fails part way leaves an environment + # behind, and the trap has to know about it either way. + CREATED="$CREATED $scratch" + # -n, not the file's own name: the exported files carry no `name:` key, which 00_static + # asserts so that a user's install cannot be named by whoever ran the export. + conda env create --name "$scratch" --file "$file" --yes > "$LOGDIR/solve$tag.log" 2>&1 \ + || { say " FAILED to solve '$file' - see $LOGDIR/solve$tag.log"; exit 1; } + + # WHAT THE SHIPPED FILE RESOLVES TO TODAY, which is the baseline the update moves from. + # Read out of this solve rather than out of an installed environment, so the table below + # cannot report a drift that belongs to the maintainer's own machine. + conda list --name "$scratch" --export > "$LOGDIR/packages-before$tag.txt" + say " $file solves to $(grep -c '^[^#]' "$LOGDIR/packages-before$tag.txt") packages" # THE UPDATE IS TOLD THE HOST FLOOR BEFORE IT RUNS, NOT CORRECTED AFTERWARDS. # @@ -249,16 +257,10 @@ prepare_env() { fi } -# ---------------------------------------------------------------- snapshot and clone ---- -say "[1/5] Recording the current tool sets..." -conda list --name "$SOURCE_ENV" --export > "$LOGDIR/packages-before.txt" -conda list --name "$SOURCE_ANALYSIS_ENV" --export > "$LOGDIR/packages-before-analysis.txt" -say " $SOURCE_ENV: $(grep -c '^[^#]' "$LOGDIR/packages-before.txt") packages" -say " $SOURCE_ANALYSIS_ENV: $(grep -c '^[^#]' "$LOGDIR/packages-before-analysis.txt") packages" - -say "[2/5] Cloning both and updating everything..." -prepare_env "$SOURCE_ENV" "$UPDATE_ENV" "" -prepare_env "$SOURCE_ANALYSIS_ENV" "$UPDATE_ANALYSIS_ENV" "-analysis" +# ------------------------------------------------------------------ solve and update ---- +say "[1/6] Solving both shipped files, then updating everything in each..." +prepare_env "$PIPELINE_FILE" "$UPDATE_ENV" "" +prepare_env "$ANALYSIS_FILE" "$UPDATE_ANALYSIS_ENV" "-analysis" # Both environments carry Nextflow: the analysis layer is a pipeline of its own, and # install/environment-analysis.yml says it carries the pipeline environment's version. Two @@ -273,8 +275,124 @@ if [ "$NF_PIPELINE" != "$NF_ANALYSIS" ]; then say "" fi +# -------------------------------------------------------------------- module pins ------- +# +# DONE HERE, BEFORE THE SUITE, BECAUSE THE MODULES MOVE WITH THE ENVIRONMENT. +# +# A module pin has to name the version the shared analysis environment holds. conda_install_ +# packages refuses one that disagrees, because `--freeze-installed` does not cover a package +# NAMED ON THE COMMAND LINE: conda installs that at the version asked for and downgrades the +# baseline without a word. So an update that moves r-future leaves every module pinning the old +# one unable to install at all. +# +# WHICH MAKES THE NEW VALUE DERIVED RATHER THAN CHOSEN. There is exactly one version a pin may +# name, and the update has just decided it. Reporting that and stopping made every environment +# update a blocker on hand-editing manifests, so the pin is moved here and the module bumped with +# it, which is what makes the edit publishable. Both changes are left uncommitted, like the +# exported files, and each bumped module is republished at the end of the release. +# +# Before the suite, so the suite runs against the manifests that will ship. +say "[2/6] Checking the modules' pins against the updated baseline..." +pin_clashes() { + local shipped + shipped=$( { store_packages "$ROOT/modules"; store_packages "$ROOT/modules/lib"; } | sort -u ) + [ -n "$shipped" ] || return 0 + # shellcheck disable=SC2086 + conda_conflicting_packages "$UPDATE_ANALYSIS_ENV" $shipped +} + +SHIPPED=$( { store_packages "$ROOT/modules"; store_packages "$ROOT/modules/lib"; } | sort -u ) +if [ -z "$SHIPPED" ]; then + say " no module or library declares a package" +else + # shellcheck disable=SC2086 + PIN_COUNT=$(printf '%s\n' $SHIPPED | wc -l | tr -d ' ') + CLASH=$(pin_clashes) + if [ -z "$CLASH" ]; then + say " all $PIN_COUNT pins agree with what the updated baseline holds" + else + say " the update moved packages the modules pin. Moving each pin with it:" + say "" + MOVED="" + while read -r line; do + [ -n "$line" ] || continue + # conda_conflicting_packages prints `r-future=1.75.0 (installed 1.76.0)`. + OLD_SPEC="${line%% *}" + PKG="${OLD_SPEC%%=*}" + NOW=$(printf '%s' "$line" | sed -n 's/.*(installed \(.*\))$/\1/p') + [ -n "$NOW" ] || { say " could not read an installed version from: $line"; exit 1; } + # A package name carries dots - r-data.table - so the pattern is escaped rather than + # trusted to match itself. + ESCAPED=$(printf '%s' "$OLD_SPEC" | sed 's/[].[^$*\\]/\\&/g') + for manifest in "$ROOT"/modules/*/manifest.json "$ROOT"/modules/lib/*/manifest.json; do + [ -f "$manifest" ] || continue + module_packages "$manifest" | grep -qxF "$OLD_SPEC" || continue + NAME=$(basename "$(dirname "$manifest")") + sed -i "s|\"$ESCAPED\"|\"$PKG=$NOW\"|" "$manifest" + module_packages "$manifest" | grep -qxF "$PKG=$NOW" \ + || { say " FAILED to rewrite $OLD_SPEC in ${manifest#"$ROOT"/}"; exit 1; } + say " $NAME: $OLD_SPEC -> $PKG=$NOW" + case " $MOVED " in *" $NAME "*) ;; *) MOVED="$MOVED $NAME" ;; esac + REL="${manifest#"$ROOT"/}" + case " $TOUCHED_MANIFESTS " in + *" $REL "*) ;; *) TOUCHED_MANIFESTS="$TOUCHED_MANIFESTS $REL" ;; + esac + done + done <<< "$CLASH" + + say "" + for NAME in $MOVED; do + BUMP_OUT=$(bash dev/scripts/bump-analysis-version.sh module "$NAME" 2>&1) \ + || { say " FAILED to bump '$NAME':" + printf '%s\n' "$BUMP_OUT" | sed 's/^/ /' >&2; exit 1; } + printf '%s\n' "$BUMP_OUT" | head -1 | sed 's/^ / /;s/^/ /' \ + | tee -a "$LOGDIR/summary.txt" + done + + # THAT THE REWRITE TOOK, asked of the manifests again rather than assumed from the sed + # exiting 0. A pin left behind here is a module nobody can install, found at [5/6] after + # the suite instead of now. + CLASH=$(pin_clashes) + if [ -n "$CLASH" ]; then + say "" + say "STOPPED: a pin still disagrees with the baseline after the rewrite." + printf '%s\n' "$CLASH" | sed 's/^/ /' | tee -a "$LOGDIR/summary.txt" + exit 1 + fi + say "" + say " every pin now agrees with the baseline. The manifests are changed in the" + say " tree and uncommitted, like the exported files below." + say "" + # THE FLOOR THESE MODULES NOW NEED, AND WHY IT IS NOT SET HERE. + # + # A moved pin names a package version only this release's environment holds, so each of + # these modules now requires this release and nothing earlier. `environment` is the field + # that says so, and it has to move too - or a user on the previous release installs the + # module, reaches conda, and is refused with "already holds" instead of being told to + # upgrade. + # + # It CANNOT move now. analysis/lib/nf/modules.nf refuses at run time any module whose + # environment is above the running release, and the release is still the old version until + # bump-version.sh. Setting it here made every module in this repository unrunnable, which + # the suite found as six failures in the three module suites: + # + # 'mds' v... needs the analysis environment of PoolSeqFlow 3.3.0 or newer, and this is 3.2.0. + # + # So the pin moves here, where the solve decides it, and the floor moves after the bump. + say " AFTER dev/scripts/bump-version.sh, raise the environment floor in each:" + for NAME in $MOVED; do + say " modules/$NAME/manifest.json \"environment\": \"$NEW\"" + done + say "" + say " Not now: the frame refuses a module whose floor is above the running release," + say " and that is still the old version until the bump. Each module is republished" + say " at the end of the release, which is when the floor has to be true." + fi +fi +say "" + # ------------------------------------------------------------------------- test --------- -say "[3/5] Running the full test suite against both scratch environments..." +say "[3/6] Running the full test suite against both scratch environments..." ENV_PREFIX="$(conda env list | awk -v n="$UPDATE_ENV" '$1 == n {print $NF}')" if [ -z "$ENV_PREFIX" ] || [ ! -x "$ENV_PREFIX/bin/nextflow" ]; then say " ERROR: '$UPDATE_ENV' has no usable nextflow at $ENV_PREFIX/bin/nextflow" @@ -374,21 +492,13 @@ if [ "$TEST_STATUS" -ne 0 ]; then [ -n "$KEPT_XDEV" ] && say " $KEPT_XDEV" say " Delete them when you are done - they are not small." say "" - say " Both scratch environments have been kept so the failure can be reproduced:" - say " TEST_CONDA_ENV=$ENV_PREFIX \\" - say " TEST_ANALYSIS_ENV=$ANALYSIS_PREFIX \\" - say " ./test/run_tests.sh --suite " - say "" + say " $LOGDIR/tests.log is the run, and" say " $LOGDIR/packages-changed.tsv and" say " $LOGDIR/packages-changed-analysis.tsv say what moved, compared on" - say " version AND build. A package is one candidate among several: the same" - say " suite passing by hand against these very environments means the cause is" - say " the surroundings, not the packages, and the artifacts above are where" - say " that shows." + say " version AND build." say "" - say " Discard the attempt with:" - say " conda env remove -n $UPDATE_ENV" - say " conda env remove -n $UPDATE_ANALYSIS_ENV" + say " To reproduce against the environments themselves, run this again with" + say " --no-cleanup and use the TEST_CONDA_ENV / TEST_ANALYSIS_ENV it prints." exit 1 fi @@ -415,7 +525,7 @@ fi # would then carry a header promising a floor its own contents break. Checking the scratch # environment here catches that while the solve is still in hand, instead of after the file is # written and an install has been done from it. -say "[4/5] Tests passed. Checking what each environment requires of its host..." +say "[4/6] Tests passed. Checking what each environment requires of its host..." for e in "$UPDATE_ENV" "$UPDATE_ANALYSIS_ENV"; do if ! FLOOR_OUT=$(bash dev/scripts/check-host-floor.sh "$e" 2>&1); then printf '%s\n' "$FLOOR_OUT" | tee -a "$LOGDIR/summary.txt" | sed 's/^/ /' @@ -432,12 +542,45 @@ for e in "$UPDATE_ENV" "$UPDATE_ANALYSIS_ENV"; do || { say " FAILED to export '$e' - see $LOGDIR/summary.txt"; exit 1; } done -# Removed only after both exports have succeeded, so a failure there does not lose the solve. -say "[5/5] Removing the scratch environments..." -for e in "$UPDATE_ENV" "$UPDATE_ANALYSIS_ENV"; do - conda env remove --name "$e" --yes >> "$LOGDIR/cleanup.log" 2>&1 \ - || say " WARNING: could not remove '$e' - see $LOGDIR/cleanup.log" -done +# -------------------------------------------------------- prove what was written -------- +# +# EVERYTHING ABOVE THIS LINE REASONED ABOUT THE SCRATCH ENVIRONMENTS. These two read the FILES, +# which is a different question: an environment holds whatever solved into it once, while a file +# names constraints and lets the solver choose again, on a machine that is not this one. +# +# Both build environments of their own and remove them. Both are re-runnable on their own, which +# is what the failure messages below point at: the files are already written by this point, so a +# failure here does not cost the hour again. +say "[5/6] Proving the two files that were just written..." +say " Solving each from nothing and reading its floor (minutes)..." +if ! bash dev/scripts/check-exported-floor.sh > "$LOGDIR/exported-floor.log" 2>&1; then + tail -n 30 "$LOGDIR/exported-floor.log" | sed 's/^/ /' | tee -a "$LOGDIR/summary.txt" + say "" + say " A file that does not solve from nothing, or whose floor is above what the" + say " release promises, is not shippable. Both files are written, so this is" + say " re-runnable on its own once the cause is fixed:" + say " dev/scripts/check-exported-floor.sh" + exit 1 +fi +tail -n 4 "$LOGDIR/exported-floor.log" | sed 's/^/ /' | tee -a "$LOGDIR/summary.txt" + +say " Putting the modules' pins through a baseline built from the new file (minutes)..." +if ! bash dev/scripts/check-module-packages.sh > "$LOGDIR/module-packages.log" 2>&1; then + grep -E '^ (FAIL|ok)' "$LOGDIR/module-packages.log" | sed 's/^/ /' \ + | tee -a "$LOGDIR/summary.txt" + say "" + say " Every check in it must say ok. The full run is in" + say " $LOGDIR/module-packages.log" + say " and it is re-runnable on its own:" + say " dev/scripts/check-module-packages.sh" + exit 1 +fi +say " every check passed." + +# The removal itself is the exit trap's, which runs on every path out of here. +say "[6/6] Removing the scratch environments..." say "" -say "Done. The release steps are in dev/RELEASING.md." +say "Done. Read $LOGDIR/summary.txt, then the environment diffs:" +say " git diff install/environment.yml install/environment-analysis.yml" +say "The release steps are in dev/RELEASING.md." diff --git a/dev/scripts/publish-module.sh b/dev/scripts/publish-module.sh index d997e89..1ab8fcf 100755 --- a/dev/scripts/publish-module.sh +++ b/dev/scripts/publish-module.sh @@ -36,6 +36,7 @@ NAME="${1-}" REF="${2:-HEAD}" if [ -z "$NAME" ]; then echo "Usage: $0 [ref]" >&2 + echo " $0 --list what is in the tree and not yet in the catalogue" >&2 exit 1 fi @@ -51,6 +52,56 @@ PUBLISHED_PATH="modules-repo" INDEX="$REPO_DIR/index.tsv" [ -f "$INDEX" ] || { echo "ERROR: $INDEX not found" >&2; exit 1; } +# Is this name and version already a catalogue row? Prints DUPLICATE, BADHEADER, or nothing. +# +# THE COLUMNS ARE MATCHED BY NAME, from the header row, as module_index_rows() in +# lib/wrapper_lib.sh does. Read by position this compared the version against `kind`, which holds +# "module" or "library", so it never matched and the check passed over everything. +# +# The outcome is printed rather than carried in the exit status: a header naming neither column +# has to be told apart from a row that is simply absent, and both are non-zero. +catalogue_has() { # name version + awk -F'\t' -v n="$1" -v v="$2" ' + /^[[:space:]]*(#|$)/ { next } + !header { + header = 1 + for (i = 1; i <= NF; i++) at[$i] = i + if (!("name" in at) || !("version" in at)) { print "BADHEADER"; exit } + next + } + $(at["name"]) == n && $(at["version"]) == v { print "DUPLICATE"; exit } + ' "$INDEX" +} + +# Every module and library in the tree against the catalogue. Reads the working tree rather than a +# ref, because this answers what is left to publish; publishing itself refuses an uncommitted +# source and says so. +if [ "$NAME" = "--list" ]; then + LEFT=0 + for dir in modules/*/ modules/lib/*/; do + [ -f "$dir/manifest.json" ] || continue + read -r m_name m_version <&2 + exit 1 ;; + DUPLICATE) printf ' %-14s %s %s\n' "published" "$m_name" "$m_version" ;; + *) printf ' %-14s %s %s\n' "UNPUBLISHED" "$m_name" "$m_version" + LEFT=$((LEFT + 1)) ;; + esac + done + echo "" + if [ "$LEFT" -eq 0 ]; then + echo "Everything in the tree is in the catalogue." + else + echo "$LEFT to publish, each with: $0 " + fi + exit 0 +fi + # A module or a library: both are a folder with a manifest, published the same way, and the # manifest's own `kind` says which. Looked for in both places rather than taking a flag, so the # caller names the thing and the repository says what it is. @@ -99,11 +150,14 @@ if [ -e "$TARBALL" ]; then exit 1 fi -if awk -F'\t' -v n="$NAME" -v v="$VERSION" \ - '!/^#/ && NF > 1 && $1 == n && $2 == v { found = 1 } END { exit !found }' "$INDEX"; then - echo "ERROR: $INDEX already has a row for $NAME $VERSION" >&2 - exit 1 -fi +case "$(catalogue_has "$NAME" "$VERSION")" in + BADHEADER) + echo "ERROR: $INDEX has no header row naming 'name' and 'version'" >&2 + exit 1 ;; + DUPLICATE) + echo "ERROR: $INDEX already has a row for $NAME $VERSION" >&2 + exit 1 ;; +esac # The commit that last touched the module at this ref, for a timestamp that follows the content # rather than the clock. diff --git a/dev/scripts/select-tests.py b/dev/scripts/select-tests.py index 5f49ac0..4923090 100644 --- a/dev/scripts/select-tests.py +++ b/dev/scripts/select-tests.py @@ -20,6 +20,10 @@ IT ERRS WIDE. A file it cannot classify selects everything, and so does a change to the harness or to this script. A selection that is too big costs minutes; one that is too small costs a bug nobody was looking for. + +ERRING WIDE IS FOR A FILE IT CANNOT CLASSIFY, not for a request it cannot read. A named path +that does not exist, and an argument that looks like an option it does not have, both exit 2 +rather than selecting anything: "nothing to run" has to mean the graph was asked and answered. """ import os @@ -137,6 +141,27 @@ def main(): del args[i:i + 2] if args: + # A MALFORMED REQUEST IS AN ERROR, NOT AN EMPTY ANSWER. Both of these used to fall + # through to "nothing to run", which reads as "no suite covers this change" and is the + # one answer this script must never give by accident: an argument starting with - is a + # mistyped option rather than a path, and a path that does not exist matches no suite's + # footprint. `--changed` is run_tests.sh's flag and was read here as a filename. + unknown = [a for a in args if a.startswith("-")] + if unknown: + print("select-tests.py: unknown option %s" % unknown[0], file=sys.stderr) + print(" Arguments are paths. The options are --command and --ref , and with", + file=sys.stderr) + print(" no arguments it reads whatever git reports as changed.", file=sys.stderr) + return 2 + absent = [a for a in args + if not (os.path.exists(a) or os.path.exists(os.path.join(ROOT, a)))] + if absent: + print("select-tests.py: no such file: %s" % absent[0], file=sys.stderr) + print(" Paths are read from the repository root. One that does not exist is in no", + file=sys.stderr) + print(" suite's footprint, so the answer would be 'nothing to run'.", + file=sys.stderr) + return 2 changed = args else: cmd = ["git", "diff", "--name-only"] + ([ref] if ref else ["HEAD"]) diff --git a/dev/scripts/serve_docs.sh b/dev/scripts/serve_docs.sh index d85e364..82e488a 100755 --- a/dev/scripts/serve_docs.sh +++ b/dev/scripts/serve_docs.sh @@ -2,7 +2,8 @@ # Live preview of the manual. Regenerates docs/ whenever manual/ changes and serves the # site with live reload, so an edit to the manual shows in the browser a second later. # -# pip install -r manual/requirements.txt # once +# python3 -m venv dev/manual/.venv # once +# dev/manual/.venv/bin/pip install -r manual/requirements.txt # once # dev/scripts/serve_docs.sh # http://127.0.0.1:8055/PoolSeqFlow/ # # Not MkDocs' own default of 8000, which is the first port anything else takes. Override with @@ -10,16 +11,29 @@ # # PSF_DOCS_ADDR=0.0.0.0:9001 dev/scripts/serve_docs.sh # -# Set MKDOCS to use a particular interpreter's mkdocs rather than the one on PATH. +# POOLSEQFLOW_MKDOCS names a particular mkdocs and wins over both of the below. set -euo pipefail REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" cd "$REPO" -MKDOCS="${MKDOCS:-$(command -v mkdocs || true)}" -if [ -z "$MKDOCS" ]; then - echo "mkdocs not found. Install it with: pip install -r manual/requirements.txt" >&2 +# POOLSEQFLOW_MKDOCS first, then the environment this repository keeps, then PATH. +# +# THAT ENVIRONMENT LIVES UNDER dev/, NOT UNDER manual/, and three things make manual/ wrong for +# it: `manual` is a payload item, so deploy_payload's `cp -r` carries whatever is inside it into +# every installation made from a checkout; stamp() below walks manual/ once a second; and dev/ is +# already export-ignore, so nothing has to be added to keep it out of a release. +VENV_MKDOCS="$REPO/dev/manual/.venv/bin/mkdocs" +MKDOCS_BIN="${POOLSEQFLOW_MKDOCS:-}" +if [ -z "$MKDOCS_BIN" ] && [ -x "$VENV_MKDOCS" ]; then + MKDOCS_BIN="$VENV_MKDOCS" +fi +[ -n "$MKDOCS_BIN" ] || MKDOCS_BIN="$(command -v mkdocs || true)" +if [ -z "$MKDOCS_BIN" ]; then + echo "mkdocs not found. Build the environment this repository expects:" >&2 + echo " python3 -m venv dev/manual/.venv" >&2 + echo " dev/manual/.venv/bin/pip install -r manual/requirements.txt" >&2 exit 1 fi @@ -50,6 +64,6 @@ trap 'kill "$WATCHER" 2>/dev/null || true' EXIT INT TERM # An explicit --dev-addr wins; otherwise serve somewhere less contested than port 8000. case " $* " in - *" --dev-addr "*|*" -a "*) exec "$MKDOCS" serve "$@" ;; - *) exec "$MKDOCS" serve --dev-addr "${PSF_DOCS_ADDR:-127.0.0.1:8055}" "$@" ;; + *" --dev-addr "*|*" -a "*) exec "$MKDOCS_BIN" serve "$@" ;; + *) exec "$MKDOCS_BIN" serve --dev-addr "${PSF_DOCS_ADDR:-127.0.0.1:8055}" "$@" ;; esac diff --git a/dev/validation/README.md b/dev/validation/README.md index 16e3ee7..3a36ae4 100644 --- a/dev/validation/README.md +++ b/dev/validation/README.md @@ -6,7 +6,7 @@ Measurements of whether the numbers the analysis layer publishes mean what they `test/` asserts exact values over seventeen hand-computed sites: every count is written out, every expectation is derived by a plain Python loop, and a case fails when one number moves. That answers *is the arithmetic what we said it was*. -This directory answers a different question — *does the method behave* — and it cannot be answered by any exact value. A false-positive rate is a rate; a power curve is a curve; the limit at which an assumption stops holding is found by pushing on it until it breaks. So the assertions here are distributional, the replicate counts are in the hundreds of thousands, and a run takes minutes rather than seconds. Mixing the two would make the suite slow and make its failures ambiguous. +This directory answers a different question -- *does the method behave* -- and it cannot be answered by any exact value. A false-positive rate is a rate; a power curve is a curve; the limit at which an assumption stops holding is found by pushing on it until it breaks. So the assertions here are distributional, the replicate counts are in the hundreds of thousands, and a run takes minutes rather than seconds. Mixing the two would make the suite slow and make its failures ambiguous. Nothing here runs in `run_tests.sh` and nothing here gates a commit. **What ships is the chapter written from these numbers**, not the harness. @@ -23,10 +23,10 @@ The same rule holds for every section added later: the truth is constructed from | section | what it answers | |---|---| | `n_eff` | Is `n_eff` the variance of a two-stage pooled draw? Everything the analysis layer weights rests on this one identity | -| `parametric` | How often does the closed-form weighted *t* reject a site that has no association at all? Reported and never gated — these numbers are the evidence, and a gate would turn them into a test result | +| `parametric` | How often does the closed-form weighted *t* reject a site that has no association at all? Reported and never gated -- these numbers are the evidence, and a gate would turn them into a test result | | `permutation` | Does the published p reject at the rate it was asked for? Gated one-sided: over discrete counts a permutation test is legitimately conservative, so rejecting below alpha is safe and rejecting above it is not | | `units` | Pools that are not independent. Five estimators over identical sites, differing only in what they believe about how many independent observations there are | -| `weights` | The weight model itself being wrong — uneven pooling, and depths that line up with the phenotype. Averaged over many poolings, because how badly one arrangement misleads a fit is itself a draw | +| `weights` | The weight model itself being wrong -- uneven pooling, and depths that line up with the phenotype. Averaged over many poolings, because how badly one arrangement misleads a fit is itself a draw | | `dispersion` | Whether a better variance model repairs any of that. It does not, and the row that shows why is the one where the weights were already right | | `exchangeable` | Permuting the standardised residuals instead of the labels, with and without the estimated θ that makes them exchangeable | | `breakdown` | How far that carries, from even pooling to an effective pool of two individuals, looking for the value a guard should sit at | @@ -37,7 +37,7 @@ The same rule holds for every section added later: the truth is constructed from ## The two scripts -`calibrate.R` judges our arithmetic against our own idea of the truth. `external.R` judges it against BayPass, which shares none of our code and none of our statistics — and scores both against the sites that were planted rather than against each other, since two different statistics on two different scales cannot agree numerically and it would mean nothing if they did. +`calibrate.R` judges our arithmetic against our own idea of the truth. `external.R` judges it against BayPass, which shares none of our code and none of our statistics -- and scores both against the sites that were planted rather than against each other, since two different statistics on two different scales cannot agree numerically and it would mean nothing if they did. Both source `lib.R`, which holds every generator and estimator they share. They fit and permute identically or neither result says anything about the other, so nothing is defined twice. @@ -52,7 +52,7 @@ conda create -n poolseqflow-validation -c conda-forge -c bioconda baypass conda activate poolseqflow-validation ``` -**Not in the analysis environment**, and not in `install/environment-analysis.yml`. Nothing an analysis runs calls it — no module, no test, only the script above — so shipping it would put a Fortran binary into every user's environment for one command run once per release. It would also be the only line in that file no analysis touches, which is what E8's package manager exists to notice: the analysis environment matching its own record is the guarantee, and a hand-added package is exactly the mismatch it reports. +**Not in the analysis environment**, and not in `install/environment-analysis.yml`. Nothing an analysis runs calls it -- no module, no test, only the script above -- so shipping it would put a Fortran binary into every user's environment for one command run once per release. It would also be the only line in that file no analysis touches, which is what E8's package manager exists to notice: the analysis environment matching its own record is the guarantee, and a hand-added package is exactly the mismatch it reports. `external.R` checks for it and stops with the install line if it is absent, so a machine without it gets a sentence rather than a crash. @@ -66,4 +66,4 @@ Rscript dev/validation/calibrate.R modules/lib n_eff 500000 The third argument is the replicate count per cell; it defaults low enough to run in seconds and high enough to separate the hypotheses. Every run prints the seed it used and is reproducible from it. -Base R only, like the library — no packages to install, and it runs under either conda environment or a bare `Rscript`. +Base R only, like the library -- no packages to install, and it runs under either conda environment or a bare `Rscript`. diff --git a/install/environment-analysis.yml b/install/environment-analysis.yml index 28040a3..6210e5c 100644 --- a/install/environment-analysis.yml +++ b/install/environment-analysis.yml @@ -42,15 +42,15 @@ dependencies: - fonts-conda-forge=1=hc364b38_1 - freetype=2.14.3=ha770c72_2 - fribidi=1.0.17=h7cc23a3_0 - - gcc_impl_linux-64=16.2.0=hfe51025_5 + - gcc_impl_linux-64=16.2.0=h9891192_7 - gcc_linux-64=16.2.0=h0ab548f_2 - - gfortran_impl_linux-64=16.2.0=hf930d89_5 + - gfortran_impl_linux-64=16.2.0=hf930d89_7 - giflib=6.1.3=hebe6cf0_2 - graphite2=1.3.15=h54a6638_1 - gsl=2.7=he838d99_0 - - gxx_impl_linux-64=16.2.0=h0d273dc_5 + - gxx_impl_linux-64=16.2.0=h0d273dc_7 - gxx_linux-64=16.2.0=h23aca61_2 - - harfbuzz=14.5.0=ha770c72_0 + - harfbuzz=14.5.1=ha770c72_0 - htslib=1.24=ha79157c_0 - icu=78.3=py310h44b86e0_2 - kernel-headers_linux-64=3.10.0=he073ed8_18 @@ -66,20 +66,20 @@ dependencies: - libdeflate=1.25=hd45a770_1 - libedit=3.1.20250104=pl5321h373387f_1 - libev=4.33=h280c20c_3 - - libexpat=2.8.4=hd2095e1_0 + - libexpat=2.8.5=hd2095e1_0 - libffi=3.7.0=h81df57d_1 - libfreetype=2.14.3=ha770c72_2 - libfreetype6=2.14.3=h5e6c136_2 - - libgcc=16.2.0=ha9f2e26_5 - - libgcc-devel_linux-64=16.2.0=he3ce08f_105 - - libgcc-ng=16.2.0=h69a702a_5 - - libgfortran=16.2.0=h69a702a_5 - - libgfortran-ng=16.2.0=h69a702a_5 - - libgfortran5=16.2.0=h6b99dfc_5 + - libgcc=16.2.0=ha9f2e26_7 + - libgcc-devel_linux-64=16.2.0=he3ce08f_107 + - libgcc-ng=16.2.0=h69a702a_7 + - libgfortran=16.2.0=h69a702a_7 + - libgfortran-ng=16.2.0=h69a702a_7 + - libgfortran5=16.2.0=h6b99dfc_7 - libglib=2.90.0=h569388d_0 - - libgomp=16.2.0=he0feb66_5 - - libharfbuzz=14.5.0=h23af247_0 - - libharfbuzz-devel=14.5.0=h23af247_0 + - libgomp=16.2.0=he0feb66_7 + - libharfbuzz=14.5.1=h23af247_0 + - libharfbuzz-devel=14.5.1=h23af247_0 - libiconv=1.18=h0cb94f2_3 - libjpeg-turbo=3.2.0=hb03c661_1 - liblapack=3.11.0=11_h47877c9_openblas @@ -88,15 +88,15 @@ dependencies: - libmpdec=4.0.0=hb03c661_2 - libnghttp2=1.68.1=h74cf4be_1 - libopenblas=0.3.34=pthreads_hf13c14d_2 - - libpng=1.6.58=h922cc85_1 + - libpng=1.6.59=h922cc85_0 - libpsl=0.23.1=hd9e3e90_1 - libpython=3.14.7=hdc7f604_106_cp314 - - libsanitizer=16.2.0=h3048135_5 + - libsanitizer=16.2.0=h3048135_7 - libsqlite=3.53.4=h0737f62_1 - libssh2=1.11.1=h6154650_1 - - libstdcxx=16.2.0=h934c35e_5 - - libstdcxx-devel_linux-64=16.2.0=h86e191b_105 - - libstdcxx-ng=16.2.0=hdf11a46_5 + - libstdcxx=16.2.0=h934c35e_7 + - libstdcxx-devel_linux-64=16.2.0=h86e191b_107 + - libstdcxx-ng=16.2.0=hdf11a46_7 - libtiff=4.7.2=hcc2c06a_1 - libuuid=2.42.4=hcfc3c73_0 - libuv=1.52.1=h280c20c_1 @@ -109,11 +109,11 @@ dependencies: - ncurses=6.6=hdb14827_1 - nextflow=26.04.6=h2a3209d_1 - openjdk=25.0.2=h1602c4f_26 - - openssl=3.6.4=h781a0a9_0 - - pandoc=3.11=ha770c72_0 + - openssl=3.6.5=h781a0a9_0 + - pandoc=3.12=ha770c72_0 - pango=1.58.2=h7bb47b9_1 - pcre2=10.47=h8b3dc9c_1 - - perl=5.32.1=9_h7cc23a3_perl5 + - perl=5.44.0=1_h7cc23a3_perl5 - pip=26.2.1=pyh145f28c_0 - pixman=0.46.4=h54a6638_3 - popt=1.19=hcfc3c73_2 @@ -139,7 +139,7 @@ dependencies: - r-fontawesome=0.5.3=r45hc72bb7e_1 - r-foreach=1.5.2=r45hc72bb7e_4 - r-fs=2.1.0=r45h3697838_0 - - r-future=1.75.0=r45hc72bb7e_0 + - r-future=1.76.0=r45hacb3a79_0 - r-future.apply=1.20.2=r45hc72bb7e_0 - r-getopt=1.21.1=r45ha770c72_0 - r-ggplot2=4.0.3=r45h785f33e_0 @@ -178,7 +178,7 @@ dependencies: - r-vctrs=0.7.3=r45h3697838_0 - r-viridislite=0.4.3=r45hc72bb7e_0 - r-withr=3.0.3=r45hc72bb7e_0 - - r-xfun=0.60=r45h54b55ab_0 + - r-xfun=0.61=r45hddc5280_1 - r-yaml=2.3.12=r45h54b55ab_0 - readline=8.3=hd6e31c0_1 - rsync=3.4.4=hffd6c76_1 @@ -187,7 +187,7 @@ dependencies: - sysroot_linux-64=2.17=h0157908_18 - tk=8.6.13=noxft_h1df4ec4_4 - tktable=2.10=h0eb2ad0_11 - - typst=0.15.1=he64ecbb_0 + - typst=0.15.1=h2d90d6e_1 - tzdata=2026c=h151e31d_0 - xorg-libice=1.1.2=h280c20c_0 - xorg-libsm=1.2.6=h0d788c3_1 diff --git a/install/environment.yml b/install/environment.yml index 3bb83ce..2f3d523 100644 --- a/install/environment.yml +++ b/install/environment.yml @@ -23,7 +23,7 @@ dependencies: - backports.zstd=1.7.0=py311h6f959d0_1 - bamtools=2.5.3=he132191_0 - bash=5.2.37=h4be8908_0 - - bcftools=1.24=h118bc1c_2 + - bcftools=1.24=h118bc1c_4 - bwa=0.7.19=h577a1d6_1 - bzip2=1.0.8=hda65f42_10 - c-ares=1.34.8=hebe6cf0_2 @@ -35,7 +35,7 @@ dependencies: - cutadapt=5.2=py311hc303176_2 - diffutils=3.12=h73b1eb8_0 - dnaio=1.2.4=py311hc303176_0 - - fastqc=0.12.1=hdfd78af_0 + - fastqc=0.13.0=hdfd78af_0 - findutils=4.10.0=h86094a7_1 - font-ttf-dejavu-sans-mono=2.37=hab24e00_0 - font-ttf-inconsolata=3.000=h77eed37_0 @@ -52,7 +52,7 @@ dependencies: - grep=3.12=h09507ce_2 - gsl=2.6=he838d99_2 - gzip=1.14=hb9d3cd8_0 - - harfbuzz=14.5.0=ha770c72_0 + - harfbuzz=14.5.1=ha770c72_0 - htslib=1.24=ha79157c_0 - icu=78.3=h54a6638_2 - isa-l=2.32.1=hb03c661_0 @@ -70,31 +70,32 @@ dependencies: - libdeflate=1.25=hd45a770_1 - libedit=3.1.20250104=pl5321h373387f_1 - libev=4.33=h280c20c_3 - - libexpat=2.8.4=hd2095e1_0 + - libexpat=2.8.5=hd2095e1_0 - libffi=3.7.0=h81df57d_1 - libfreetype=2.14.3=ha770c72_2 - libfreetype6=2.14.3=h5e6c136_2 - - libgcc=16.2.0=ha9f2e26_5 - - libgcc-ng=16.2.0=h69a702a_5 + - libgcc=16.2.0=ha9f2e26_7 + - libgcc-ng=16.2.0=h69a702a_7 - libgettextpo=0.25.1=h0e23cf9_2 - - libgfortran=16.2.0=h69a702a_5 - - libgfortran5=16.2.0=h6b99dfc_5 + - libgfortran=16.2.0=h69a702a_7 + - libgfortran5=16.2.0=h6b99dfc_7 - libglib=2.90.0=h569388d_0 - - libgomp=16.2.0=he0feb66_5 - - libharfbuzz=14.5.0=h23af247_0 - - libharfbuzz-devel=14.5.0=h23af247_0 + - libgomp=16.2.0=he0feb66_7 + - libharfbuzz=14.5.1=h23af247_0 + - libharfbuzz-devel=14.5.1=h23af247_0 - libiconv=1.18=h0cb94f2_3 - libjpeg-turbo=3.2.0=hb03c661_1 - liblzma=5.8.3=hb03c661_1 + - liblzma-devel=5.8.3=hb03c661_1 - libnghttp2=1.68.1=h74cf4be_1 - libnsl=2.0.1=hb9d3cd8_1 - libopenblas=0.3.34=pthreads_hf13c14d_2 - - libpng=1.6.58=h922cc85_1 + - libpng=1.6.59=h922cc85_0 - libpsl=0.23.1=hd9e3e90_1 - libsqlite=3.53.4=h13e7031_1 - libssh2=1.11.1=h6154650_1 - - libstdcxx=16.2.0=h934c35e_5 - - libstdcxx-ng=16.2.0=hdf11a46_5 + - libstdcxx=16.2.0=h934c35e_7 + - libstdcxx-ng=16.2.0=hdf11a46_7 - libtiff=4.7.2=hcc2c06a_1 - libuuid=2.42.4=hcfc3c73_0 - libwebp-base=1.6.0=hd42ef1d_1 @@ -106,11 +107,11 @@ dependencies: - ncurses=6.6=hdb14827_1 - nextflow=26.04.6=h2a3209d_1 - openjdk=25.0.2=h1602c4f_26 - - openssl=3.6.4=h781a0a9_0 + - openssl=3.6.5=h781a0a9_0 - packaging=26.3=pyhc364b38_0 - pbzip2=1.1.13=h1fcc475_2 - pcre2=10.47=h8b3dc9c_1 - - perl=5.32.1=9_h7cc23a3_perl5 + - perl=5.32.1=10_h7cc23a3_perl5 - pigz=2.8=h421ea60_2 - pip=26.2.1=pyh8b19718_0 - pixman=0.46.4=h54a6638_3 @@ -147,6 +148,9 @@ dependencies: - xorg-libxt=1.3.1=h7cc23a3_1 - xorg-libxtst=1.2.5=h7cc23a3_4 - xxhash=0.8.3=hec8ed23_1 + - xz=5.8.3=ha02ee65_1 + - xz-gpl-tools=5.8.3=ha02ee65_1 + - xz-tools=5.8.3=hb03c661_1 - zlib=1.3.2=h25fd6f3_3 - zlib-ng=2.3.3=hceb46e0_1 - zstandard=0.25.0=py311h194be0f_4 diff --git a/lib/_PoolSeqFlow b/lib/_PoolSeqFlow new file mode 100644 index 0000000..1c6f3f8 --- /dev/null +++ b/lib/_PoolSeqFlow @@ -0,0 +1,145 @@ +#compdef PoolSeqFlow +# +# Tab completion for PoolSeqFlow, native to zsh. +# +# AUTOLOADED, never sourced. `PoolSeqFlow install` puts it in a site-functions directory and +# prints the one line that adds that directory to $fpath. zsh reads the `#compdef` line above +# and registers it for every command name given there. +# +# WHY THIS EXISTS BESIDE THE BASH ONE. zsh can run a bash completion through `bashcompinit`, and +# that was the first attempt: one file, both shells. It does not hold up. +# +# - `bashcompinit` emulates `compgen`, and the emulation IGNORES the `--` prefix argument. +# Measured: `compgen -W 'alpha beta' -- 'a'` answers `alpha beta`. A bash completion under +# zsh therefore offers every candidate whatever has been typed, and only zsh's own +# re-filtering in compadd hides it. +# - It has to be loaded after `compinit` and before anything that completes, which in an +# oh-my-zsh configuration is a placement question the user has to get right. +# - Descriptions are not available at all. +# +# 00_static checks the verbs below against the wrapper's own `case`, the same way it checks the +# bash completion, so the two cannot drift from it or from each other. +# +# NOTHING HERE RUNS THE WRAPPER. Starting it means starting conda, about 0.4s, which a keystroke +# cannot pay. Module names come from the filesystem. + +# The installation the command resolves to. $bindir/PoolSeqFlow is a symlink into +# $prefix/opt/PoolSeqFlow-/, so the resolved file's directory is the installation. +# $1 is the command word as typed. It cannot be read from $words here: `_arguments '*::'` +# re-slices that array for the sub-context, so inside it $words[1] is `analysis` rather than +# the command - which is why module discovery found nothing until this was passed in. +_poolseqflow_install_dir() { + local exe + exe=${commands[$1]:-$(whence -p "$1" 2>/dev/null)} + [[ -n $exe ]] || return 1 + exe=${exe:A} + print -r -- "${exe:h}" +} + +# The modules installed there. `lib` holds the shared libraries and is never named by a user. +_poolseqflow_modules() { + local dir entry + dir=$(_poolseqflow_install_dir "$1") || return 0 + [[ -d $dir/analysis/modules ]] || return 0 + for entry in $dir/analysis/modules/*(/N); do + [[ ${entry:t} == lib ]] && continue + print -r -- "${entry:t}" + done +} + +_PoolSeqFlow() { + local curcontext="$curcontext" state line ret=1 + local -a verbs analysis_verbs module_verbs modules + # Before _arguments re-slices $words for the sub-context. + local cmdname=$words[1] + + verbs=( + 'install:create this release'\''s conda environment and install the pipeline' + 'init:put a project'\''s starting files in the current directory' + 'check:verify an installation or a project' + 'run:start, or resume, the pipeline' + 'dryrun:show where the work would go without doing any of it' + 'dryclean:remove what a dry run left behind' + 'migrate_config:bring a parameters.config written for an earlier release up to date' + 'clean:remove the Nextflow work directory' + 'reset:remove the work directory and every output' + 'analysis:the analysis layer, which reads finished results' + 'version:print the version' + 'cite:print how to cite this release' + 'list:show the installed versions' + 'uninstall:remove this version' + ) + + analysis_verbs=( + 'install:build the analysis environment, which carries R' + 'check:verify the analysis installation' + 'modules:the module store' + 'complete:move finished analyses to permanent storage' + 'version:print the frame version' + 'cite:print what to cite for the analysis layer' + 'uninstall:remove this version'\''s analysis environment' + ) + + module_verbs=( + 'list:what is installed here' + 'available:what the catalogue offers' + 'install:install a module by name' + 'uninstall:remove an installed module' + ) + + _arguments -C '1: :->verb' '*:: :->rest' && ret=0 + + case $state in + verb) + _describe -t commands 'PoolSeqFlow command' verbs && ret=0 + ;; + rest) + case $words[1] in + check) + _describe -t targets 'check target' \ + '(install:the installation, every tool it is built to run + project:this project, as it is configured)' && ret=0 + ;; + init) + _describe -t modes 'init mode' \ + '(multi:for a project running several datasets)' && ret=0 + ;; + uninstall) + _describe -t scopes 'uninstall scope' \ + '(all:every installed version on this machine)' && ret=0 + ;; + analysis) + # $CURRENT is relative to the re-sliced $words that `*::` produces, where + # $words[1] is `analysis` itself - so completing the word after it is 2. + case $CURRENT in + 2) + modules=(${(f)"$(_poolseqflow_modules "$cmdname")"}) + _describe -t commands 'analysis command' analysis_verbs && ret=0 + (( $#modules )) && { _describe -t modules 'installed module' modules && ret=0 } + ;; + 3) + if [[ $words[2] == modules ]]; then + _describe -t commands 'modules command' module_verbs && ret=0 + else + _describe -t options 'option' \ + '(nocpp:run the plain-R path, which needs no compiler)' && ret=0 + fi + ;; + 4) + # `modules install ` offers nothing: those names come from the + # catalogue, over the network, and a keystroke may not fetch it. + if [[ $words[2] == modules && $words[3] == uninstall ]]; then + modules=(${(f)"$(_poolseqflow_modules "$cmdname")"}) + (( $#modules )) && { _describe -t modules 'installed module' modules && ret=0 } + fi + ;; + esac + ;; + esac + ;; + esac + + return ret +} + +_PoolSeqFlow "$@" diff --git a/lib/poolseqflow-completion.bash b/lib/poolseqflow-completion.bash index 2a17b78..04dc25c 100644 --- a/lib/poolseqflow-completion.bash +++ b/lib/poolseqflow-completion.bash @@ -3,11 +3,12 @@ # Tab completion for PoolSeqFlow, in bash and in zsh. # # SOURCED, never run. `PoolSeqFlow install` puts it in the user's bash-completion directory, -# where bash finds it by the command's name. zsh does not read that directory, so it needs two -# lines in ~/.zshrc, and install prints them: +# where bash finds it by the command's name. # -# autoload -U +X bashcompinit && bashcompinit -# . ~/.local/share/bash-completion/completions/PoolSeqFlow +# FOR BASH ONLY. zsh has its own, lib/_PoolSeqFlow, rather than running this one through +# `bashcompinit` - that emulates `compgen` and the emulation ignores the `--` prefix argument, +# so every candidate is offered whatever has been typed. Measured: `compgen -W 'alpha beta' +# -- 'a'` answers `alpha beta`. # # NOTHING HERE RUNS THE WRAPPER. It evaluates `conda shell.bash hook` before it dispatches, which # costs about 0.4s, and a keystroke cannot. Every list below is a literal or comes from the @@ -53,9 +54,8 @@ _poolseqflow_modules() { # Fill COMPREPLY with the words of $1 that start with $2, one array element each. # -# A read loop rather than `mapfile`, which is a bash builtin zsh does not have. It also keeps -# each match one element without relying on word splitting, which differs between the two shells -# outside the `emulate -L sh` that zsh's bashcompinit wraps this call in. +# A read loop rather than `COMPREPLY=( $(compgen ...) )`, so a candidate containing whitespace +# stays one element instead of being split into several. _poolseqflow_reply() { local line COMPREPLY=() @@ -70,8 +70,8 @@ _poolseqflow() { cur=${COMP_WORDS[COMP_CWORD]} cmd=${COMP_WORDS[0]} # Every verb the wrapper dispatches, which 00_static checks against its `case`. - top="install init init_multi check run dryrun dryclean migrate_config clean reset" - top="$top analysis version cite list uninstall uninstall_all" + top="install init check run dryrun dryclean migrate_config clean reset" + top="$top analysis version cite list uninstall" if [ "$COMP_CWORD" -eq 1 ]; then _poolseqflow_reply "$top" "$cur" @@ -82,6 +82,12 @@ _poolseqflow() { check) [ "$COMP_CWORD" -eq 2 ] && _poolseqflow_reply "install project" "$cur" ;; + init) + [ "$COMP_CWORD" -eq 2 ] && _poolseqflow_reply "multi" "$cur" + ;; + uninstall) + [ "$COMP_CWORD" -eq 2 ] && _poolseqflow_reply "all" "$cur" + ;; analysis) modules=$(_poolseqflow_modules "$cmd" | tr '\n' ' ') case $COMP_CWORD in diff --git a/manual/PoolSeqFlow-manual.md b/manual/PoolSeqFlow-manual.md index 14ef654..dce0ee7 100644 --- a/manual/PoolSeqFlow-manual.md +++ b/manual/PoolSeqFlow-manual.md @@ -2,7 +2,7 @@

-A Nextflow pipeline for allele frequency analysis from pooled Illumina sequencing data. It takes raw FASTQ files and a reference genome, and gives back allele frequency tables — quality control, adapter trimming, composition-aware clipping, alignment, BAM cleanup, variant calling and frequency conversion, in a single resumable run. +A Nextflow pipeline for allele frequency analysis from pooled Illumina sequencing data. It takes raw FASTQ files and a reference genome, and gives back allele frequency tables, running quality control, adapter trimming, composition-aware clipping, alignment, BAM cleanup, variant calling and frequency conversion in a single resumable run.

@@ -15,19 +15,19 @@ A Nextflow pipeline for allele frequency analysis from pooled Illumina sequencin !!! info "Platform support" - PoolSeqFlow is developed and tested on **Linux**. macOS and Windows are not supported: the shipped conda environments are pinned to `linux-64` builds, and the resume logic relies on symbolic links and Unix-style paths that do not behave correctly on native Windows filesystems. + PoolSeqFlow is developed and tested on **Linux**. macOS and Windows are not supported: the shipped conda environments are pinned to `linux-64` builds, and the resume logic relies on symbolic links and Unix-style paths that do not behave correctly on native Windows filesystems. macOS support is planned; Windows is not. --- ### What it is for -Pool-seq sequences many individuals together in one library. You give up the ability to say which individual carried which allele, and in exchange you get an allele frequency estimate for the whole pool at a fraction of the cost of sequencing each individual separately. The unit of analysis stops being a genotype and becomes a **frequency**. +Pool-seq sequences many individuals together in one library. You get an allele frequency snapshot of the whole population for a fraction of the cost of sequencing each individual, and you give up knowing which individual carried which allele. The unit of analysis stops being a genotype and becomes a **frequency**. -That difference runs through every stage of this pipeline. A variant caller built for individuals wants to assign one of three genotypes per site; here there is no genotype to assign, only a ratio of read counts that has to survive intact from the pileup to the final table. PoolSeqFlow is built around keeping that ratio honest: +That difference runs through every stage of this pipeline. A variant caller built for individual sequencing assigns a genotype per site; a pooled sample has no genotype to assign, only a ratio of read counts. PoolSeqFlow is built around keeping that ratio honest: - **No biallelic assumption.** Multiallelic sites are preserved through calling, filtering and conversion rather than being collapsed or dropped. - **Major-allele normalization.** VCFs are re-encoded so the most frequent allele is the reference, which makes frequencies comparable across samples and runs. -- **Pool-aware filtering.** The minimum credible frequency is derived from your pool size and ploidy, not from a fixed cutoff — see [The Filter Chain](#the-filter-chain). +- **Pool-aware filtering.** The minimum credible frequency is derived from your pool size and ploidy, not from a fixed cutoff. See [The Filter Chain](#the-filter-chain). If you are weighing Pool-seq against individual sequencing, or checking whether your design fits what this pipeline assumes, start with [When to use PoolSeqFlow](#when-to-use-poolseqflow). @@ -35,33 +35,22 @@ If you are weighing Pool-seq against individual sequencing, or checking whether ### The pipeline -```text -Raw FASTQ reads - │ - ▼ -[Step 0] Verify environment, parameters and folder structure - │ - ▼ -[Step 1] Build reference dictionaries (BWA, SAMtools, SnpEff) - │ - ▼ -[Step 2] QC & trimming (FastQC → Trim Galore → composition-aware clipping) - │ - ▼ -[Step 3] Alignment (BWA-MEM) - │ - ▼ -[Step 4] BAM cleanup (name-sort → fixmate → coord-sort → markdup → addRG → filter → index) - │ - ▼ -[Step 5] Alignment, coverage & depth reports (BAMtools, SAMtools) → depth ceiling per sample - │ - ▼ -[Step 6] Depth capping, then variant calling (BCFtools mpileup + call) - │ - ├────────────────────────────────────────────┐ - ▼ ▼ -[Step 7] VCF → allele frequency tables [Step 8] Annotation (SnpEff, optional) +```mermaid +graph TD + reads(["Raw FASTQ reads"]) + s0["Step 0
Verify environment, parameters
and folder structure"] + s1["Step 1
Build reference dictionaries
BWA, SAMtools, SnpEff"] + s2["Step 2
QC and trimming
FastQC → Trim Galore → composition-aware clipping"] + s3["Step 3
Alignment
BWA-MEM"] + s4["Step 4
BAM cleanup
name-sort → fixmate → coord-sort → markdup → addRG → filter → index"] + s5["Step 5
Alignment, coverage and depth reports
BAMtools, SAMtools → depth ceiling per sample"] + s6["Step 6
Depth capping, then variant calling
BCFtools mpileup + call"] + s7["Step 7
VCF → allele frequency tables"] + s8["Step 8
Annotation
SnpEff"] + + reads --> s0 --> s1 --> s2 --> s3 --> s4 --> s5 --> s6 + s6 --> s7 + s6 -.->|optional| s8 ``` Each step is an independent Nextflow DSL2 module. Full detail in [Pipeline Steps](#pipeline-steps). @@ -110,25 +99,27 @@ Each step is an independent Nextflow DSL2 module. Full detail in [Pipeline Steps ### What is different about it -**Configuration is a file, never a flag.** PoolSeqFlow rejects command-line parameter overrides. A run is therefore fully described by a file you can version, diff and publish. It also sidesteps a silent failure mode: Nextflow delivers `--param` values as strings, so `--annotate false` sets the *string* `"false"`, which Groovy evaluates as true and leaves annotation quietly switched on. [Why →](#configuration-is-a-file-never-a-flag) +**Clip points are measured for each sample.** Trimming reads the per-base composition table FastQC produced for that sample and derives where to clip from it, so you set a tolerance rather than a fixed number of bases. [How →](#composition-aware-clipping) -**Resume is filesystem-based, so there is no `-resume` to remember.** Every step checks whether its outputs already exist in permanent storage and skips itself if they do. That survives job timeouts, reboots and `work/` cleanups that would invalidate Nextflow's own cache. `PoolSeqFlow run` is both "start" and "resume". [Why →](#resume-is-filesystem-based) +**Each pool can state its own size.** Its detection limit follows from that size, so a design mixing a pool of 10 with a pool of 500 is one run and not two. [How →](#pools-of-different-sizes) -**Nothing is copied.** You give the pipeline two directories: `mainDir`, where you launch it and where everything it works on lives, and `storageDir`, where finished results are kept. Every file a step produces is *moved* out of `work/` and symlinked back, so nothing the pipeline touches — not just the big files — ever exists twice on disk. Results stay on `mainDir` while later steps still need them, and move to `storageDir` once nothing does. [Why →](#symbolic-links-instead-of-copies) +**A run that would mix two configurations stops before any work happens.** The pipeline compares `parameters.config` and `metadata.csv` against the records kept beside your existing outputs, and on any difference it names what to delete rather than adding to it. [Why →](#the-change-guard) -**Clipping thresholds are measured, not guessed.** Step 2 parses the FastQC per-base composition table and derives the clip points from where the A/T and G/C ratios actually settle, per sample. You set a tolerance, not a number of bases. [How →](#composition-aware-clipping) +**Every setting lives in `parameters.config`.** There are no command-line overrides, so a run is described entirely by a file you can version, diff and publish beside the results it produced. [Why →](#configuration-without-flags) -**Thread counts come from one number.** Set `threads` and every tool's core count follows a benchmarked ladder — including Trim Galore, which is costed on its real footprint of `N+4` threads rather than the `N` it advertises. [How →](#resources) +**One invocation can analyze the same reads under several configurations.** A run table lists them, and every step the runs agree on is carried out once rather than once per configuration. [How →](#multi-run) -**The run refuses to mix results from two settings.** Because steps skip on the existence of output files, an edited `parameters.config` or `metadata.csv` — or a different release of the pipeline — would otherwise leave one folder holding results produced under two configurations. Step 0 keeps a copy of each beside your results, compares them on every run, stops before any work happens, and names the outputs to delete. A version change is absolute: a project belongs to one release. [How →](#step-0-verify-environment) +**Resume is read from the filesystem rather than from a cached session.** Each step skips itself when its outputs are already in storage, so an interrupted run continues after a job timeout, a reboot, or a cleared `work/`. [Why →](#filesystem-based-resume) -**One invocation, many parameter sets.** A run table lets you analyze the same reads against several references, or under several filter settings, in one go. Work the runs have in common — trimming, alignment, a reference dictionary — is done once rather than once per run, and only what genuinely diverges gets a directory of its own. A single run is unaffected and keeps its plain results tree. [How →](#multi-run) +**Finished artifacts are moved to permanent storage and symlinked back.** Each byte crosses between the two storage roots exactly once, so a project holds one copy of a result rather than one per stage. [Why →](#symbolic-links-and-the-two-storage-roots) + +**Thread counts follow from a single setting.** `threads` is the ceiling and each tool's share of it comes from a benchmarked ladder, so moving a project to a different machine means changing one number. [How →](#resources) --- ### Citing -If PoolSeqFlow contributes to published work, please cite it — the DOI and a formatted reference are on the [Citation](#citation-license) page. +If PoolSeqFlow contributes to published work, please cite it; the DOI and a formatted reference are on the [Citation](#citation-license) page. # Getting Started @@ -137,12 +128,12 @@ If PoolSeqFlow contributes to published work, please cite it — the DOI and a f | | | |---|---| -| **Operating system** | Linux. macOS and Windows are not supported — see [below](#why-not-windows) | -| **glibc** | 2.28 or newer — RHEL and Rocky 8, Debian 10, Ubuntu 18.10, or anything later. `ldd --version` prints what you have. CentOS 7 is the one common machine below it, and is end of life. [Why →](#glibc-floor) | +| **Operating system** | Linux. macOS and Windows are not supported ([why →](#why-not-windows)) | +| **glibc** | 2.28 or newer: RHEL and Rocky 8, Debian 10, Ubuntu 18.10, or anything later. `ldd --version` prints what you have. CentOS 7 is the one common machine below it, and is end of life. [Why →](#glibc-floor) | | **Conda** | [Conda or Miniconda](https://docs.conda.io/en/miniconda.html) | | **Git** | Optional, for cloning | -| **Working directory** | `mainDir` — where you launch the pipeline. Holds your reads, the reference, this project's configuration and everything actively processed. It has to persist between runs; it is not scratch space | -| **Permanent storage** | `storageDir` — where finished results are kept. Must be a different path from `mainDir`, and neither may be the installation itself. The pipeline will refuse to start otherwise | +| **Working directory** | `mainDir`: where you launch the pipeline. Holds your reads, the reference, this project's configuration and everything actively processed. It has to persist between runs; it is not scratch space | +| **Permanent storage** | `storageDir`: where finished results are kept. Must be a different path from `mainDir`, and neither may be the installation itself. The pipeline will refuse to start otherwise | | **Disk** | `mainDir` needs room for your inputs and the files produced along the way; `storageDir` for the BAMs, VCFs and tables you keep | Everything else is installed for you. `./PoolSeqFlow install` builds an isolated conda environment from `install/environment.yml`, which pins every tool to an exact build: @@ -163,41 +154,23 @@ Everything else is installed for you. `./PoolSeqFlow install` builds an isolated Pinning is deliberate. Pool-seq results depend on the exact behavior of the pileup and filtering tools, and an unpinned environment would make two runs of the same config non-comparable. -#### Why Linux only { #why-not-windows } - -**Windows.** The pipeline moves each file it produces to where it belongs and leaves a symbolic link behind, and it relies on Unix path semantics throughout. Neither behaves correctly on native Windows filesystems, and WSL only works under some filesystem configurations — which is not a guarantee worth documenting. See [Symbolic links instead of copies](#symbolic-links-instead-of-copies). - -**macOS.** Every tool a release runs is pinned to an exact build in `install/environment.yml` and `install/environment-analysis.yml`, and those files are exports from a Linux machine: they name builds that exist for `linux-64` and for no other platform. Conda cannot solve either of them on a Mac, so the install fails before anything else is reached. Supporting macOS means a second set of pinned files exported and verified on a Mac — one for Apple Silicon and one for Intel, since conda treats them as different platforms — and that is a piece of work rather than a flag. Earlier releases listed macOS as supported; that was never true of the pinned environments, and saying so was the error. - -#### The glibc floor { #glibc-floor } - -**The floor is glibc 2.28, and it applies to both environments.** In distribution terms that is RHEL and Rocky 8, Debian 10, Ubuntu 18.10, and everything released after them. The one common machine it excludes is CentOS 7, which reached end of life in June 2024. - -`__glibc` is conda's name for the glibc your machine has, and `ldd --version` prints it. Below the floor, conda refuses the solve rather than installing something that would fail later, and the message names a virtual package instead of a distribution: - -```text -LibMambaUnsatisfiableError: Encountered problems while solving: - - nothing provides __glibc >=2.28 needed by rsync-3.4.4-hffd6c76_1 -``` - -**No flag works around it.** Conda offers one that forces the solve, and using it would install binaries the system cannot run — turning a refusal you get in seconds into a crash you get hours into a run. If your machine is below the floor, use a newer one. - -**What sets it is `rsync`, which both environments carry.** Every artifact the pipeline produces is moved to permanent storage by a copy that is verified before the original is removed, and `rsync` is what stages that copy. So the floor is a property of the whole release rather than of the analysis layer: the analysis environment additionally carries a C and C++ compiler, because every module builds its hot path on your machine rather than shipping a binary, but that toolchain reaches back further than `rsync` does and is not what decides this. - --- ### Ready -Two pages. [Install](#install) gets the tool onto your machine and verifies it, which is two steps and a wait. [Quick Start](#quick-start) is everything to do with your own data — making the project, filling in the three files that describe it, and starting the run. +Check out these two pages: + +- [Install](#install) gets the tool onto your machine and verifies it, which is two steps and a wait. +- [Quick Start](#quick-start) is everything to do with your own data: making the project, filling in the three files that describe it, and starting the run. [Install PoolSeqFlow](#install){ .md-button .md-button--primary } [Quick Start](#quick-start){ .md-button } -Already have it installed and upgrading from an earlier version? Read [Upgrading](#upgrading) first — your `parameters.config` is never touched by an update and can be missing parameters the new code expects. +Already have it installed and upgrading from an earlier version? Read [Upgrading](#upgrading) first. Your `parameters.config` is never touched by an update and can be missing parameters the new code expects. -## Install - +## Install { #install } + -Check the [requirements](#getting-started) first if you have not — in particular that you are on Linux, and that conda is available. +Check the [requirements](#getting-started) first if you have not, in particular that you are on Linux, and that conda is available. ### 1. Get PoolSeqFlow @@ -209,7 +182,7 @@ Check the [requirements](#getting-started) first if you have not — in particul cd PoolSeqFlow-*/ ``` - This is the recommended route. The archive is the pipeline, the analysis frame and this manual — no site sources, no CI config, no test suite, and **no analysis modules**. Every module is published on its own timetable and installed from the catalogue, so a new installation starts with an empty module store and you choose what goes into it. It extracts into a versioned directory, so you always know which release a working copy came from. + This is the recommended route. The archive is the pipeline, the analysis frame and this manual: no site sources, no CI config, no test suite, and **no analysis modules**. Every module is published on its own timetable and installed from the catalogue, so a new installation starts with an empty module store and you choose what goes into it. It extracts into a versioned directory, so you always know which release a working copy came from. Verify it if you like: download `SHA256SUMS` from the same release and run `sha256sum -c SHA256SUMS`. @@ -223,7 +196,7 @@ Check the [requirements](#getting-started) first if you have not — in particul chmod +x PoolSeqFlow ``` - Use this if you want to track development, work from a branch, or send a pull request. `main` is not guaranteed to be a released state, and the `chmod` is needed because a clone does not always preserve the executable bit — the release archive does. + Use this if you want to track development, work from a branch, or send a pull request. `main` is not guaranteed to be a released state, and the `chmod` is needed because a clone does not always preserve the executable bit; the release archive does. ### 2. Install it @@ -231,41 +204,59 @@ Check the [requirements](#getting-started) first if you have not — in particul ./PoolSeqFlow install ``` -This does two things at once, because they belong together: it creates a conda environment named `PoolSeqFlow-`, and it copies the pipeline itself to `~/.local/opt/PoolSeqFlow-`. The pinned tools are part of what produced a result, so code without its matching environment cannot reproduce anything. +This does several things: -Commands are then linked into `~/.local/bin`, each under two names: `PoolSeqFlow-`, which always means that exact release, and plain `PoolSeqFlow`, which points at the **newest version you have installed**. Releases install alongside each other rather than over each other, so an old project can keep running under the version that produced its results. `PoolSeqFlow list` shows what is installed; `PoolSeqFlow uninstall` removes one. +- It creates a conda environment named `PoolSeqFlow-`. +- It copies the pipeline itself to `~/.local/opt/PoolSeqFlow-`. +- Commands are then linked into `~/.local/bin`, each under two names: + - `PoolSeqFlow-`, which always means that exact release, and + - plain `PoolSeqFlow`, which points at the **newest version you have installed**. +- Releases install alongside each other rather than over each other, so an old project can keep running under the version that produced its results. A later one is installed the same way; see [Upgrading](#upgrading). +- `PoolSeqFlow list` shows what is installed; [`PoolSeqFlow uninstall`](#uninstalling) removes one. -The analysis layer is reached through the same command, as `PoolSeqFlow analysis `. It reads finished results and is **not enabled by this install**: it carries R in an environment of its own, which `PoolSeqFlow analysis install` builds and `PoolSeqFlow analysis uninstall` removes. Uninstalling a version removes its analysis environment along with everything else of that version's. +The analysis layer is reached through the same command but not installed by default. To install it: -Install somewhere else by setting `POOLSEQFLOW_PREFIX` — a shared location for a group, for instance: +```bash +PoolSeqFlow analysis install +``` + +- This creates a conda environment named `PoolSeqFlow--analysis`. + +To install somewhere else, such as a shared location for a group, set `POOLSEQFLOW_PREFIX`: ```bash POOLSEQFLOW_PREFIX=/opt/shared ./PoolSeqFlow install ``` -**Putting that `bin` directory on your `PATH` is yours to do.** The installer checks, tells you whether it is already there, and prints the exact line to add if it is not — but it does not edit your shell configuration. Until you add it, call the command by its full path. +### 3. After the install -Installing takes a while the first time; later installs reuse the conda package cache. It finishes by verifying itself and **fails if anything is missing** — an environment that was created but is short a tool is not an install, and the alternative is finding out hours into a run. +- **Make sure the `bin` directory it installed into is on your `PATH`**: `~/.local/bin`, or `$POOLSEQFLOW_PREFIX/bin` if you set one. +- Installing takes a while the first time; later installs reuse the conda package cache. +- It finishes by verifying itself and **fails if anything is missing**. Once this is done the folder you downloaded has served its purpose. Everything from here uses the installed command, from your own project directory. #### Tab completion { #tab-completion } -Installing also writes a completion for `PoolSeqFlow` into `~/.local/share/bash-completion/completions/`, or wherever `XDG_DATA_HOME` points. **In bash it works in your next shell and there is nothing to do.** - -**zsh does not read that directory**, so it needs two lines in `~/.zshrc` — the installer prints them with the right path filled in: +Installing writes a completion for each shell, under `XDG_DATA_HOME` or `~/.local/share`: -```bash -autoload -U +X bashcompinit && bashcompinit -. ~/.local/share/bash-completion/completions/PoolSeqFlow ``` +~/.local/share/bash-completion/completions/PoolSeqFlow for bash +~/.local/share/zsh/site-functions/_PoolSeqFlow for zsh +``` + +The installer prints whichever of these applies to your shell: -It completes every command, the two targets `check` takes, the analysis commands, and **the modules you actually have installed** — so `PoolSeqFlow analysis ` lists the modules in that installation's store rather than a fixed list. `analysis modules install ` deliberately offers nothing: those names come from the catalogue over the network, and a keystroke should not make a network request. +- **bash needs nothing**, as long as `bash-completion` is installed; that is what reads the directory. The installer checks and says which case you are in. +- **zsh needs one line in `~/.zshrc`**, **before** the line that runs `compinit` or sources oh-my-zsh, because every directory on its default `$fpath` belongs to root: -Uninstalling the last installed version removes the completion. Another version left installed keeps it, since the command it completes is still there. + ```zsh + fpath=(~/.local/share/zsh/site-functions $fpath) + ``` -It never runs `PoolSeqFlow` itself. Starting the wrapper means starting conda, which takes long enough to be felt on a keypress, so the completion reads what it needs from the filesystem instead. +Both complete every command, the two targets `check` takes, the analysis commands, and **the modules that installation has in its store**. `analysis modules install ` offers nothing on purpose: those names come from the catalogue over the network. The zsh one also shows what each command is for. +They go with the last installed version removed; another version left installed keeps them. --- @@ -285,7 +276,7 @@ It never runs `PoolSeqFlow` itself. Starting the wrapper means starting conda, w --- - Your existing `parameters.config` will be missing parameters the new code expects, and nothing detects that automatically. + Your existing `parameters.config` can be missing parameters the new code expects, and a project belongs to the release that produced its results. [Upgrading →](#upgrading) @@ -298,7 +289,7 @@ This page takes you from an installed PoolSeqFlow to a running pipeline. It assu ### 1. Make your project { #make-your-project } -A project is a directory of your own. It is **not** the installation — that is a tool, shared by any number of projects and replaced wholesale when you upgrade, so a project kept inside it would not survive one. The pipeline refuses to start if you point it at the installation. +A project is a directory of your own. Do not put any project into installation directory. The pipeline refuses to start if you point it at the installation. ```bash mkdir -p /path/to/project @@ -306,7 +297,7 @@ cd /path/to/project PoolSeqFlow init ``` -The paths are yours to choose; the structure inside them is not. A project ready to run looks like this — `init` makes everything except the reads, the reference and `metadata.csv`: +The paths are yours to choose; the structure inside them is not. A project ready to run looks like this (`init` makes everything except the reads, the reference and `metadata.csv`): ```text /path/to/project/ ← mainDir, and the directory you run from @@ -324,9 +315,9 @@ The paths are yours to choose; the structure inside them is not. A project ready └── reference.gff.gz ← gffFile, only if annotate = true ``` -The file names are examples. `readPattern` is what finds the reads — `*_R{1,2}.fq.gz` by default, which is what matches the pairs above — and `referenceFile` and `gffFile` name the two in `Reference/`. +The file names are examples. `readPattern` is what finds the reads (`*_R{1,2}.fq.gz` by default, which is what matches the pairs above), and `referenceFile` and `gffFile` name the two in `Reference/`. -**The reads may sit in subfolders of `Data/` instead, or as well.** However your sequencing facility handed them over is how you can leave them — one folder per sample, one per run or lane, nested as deep as you like, or all together as above. All three of these are read the same way: +**The reads may sit in subfolders of `Data/` instead, or as well.** However your sequencing facility handed them over is how you can leave them: one folder per sample, one per run or lane, nested as deep as you like, or all together as above. All three of these are read the same way: ``` Data/ Data/ Data/ @@ -338,33 +329,33 @@ Data/ Data/ Data/ └── Sample2_R2.fq.gz └── Sample2_R2.fq.gz ``` -**A sample is named by its file, never by its folder.** `Sample1_R1.fq.gz` is sample `Sample1` wherever it lies, so `metadata.csv` does not change and neither does `readPattern` — the folders are yours to arrange and the pipeline does not read meaning into them. A pair whose mates are in two *different* folders is still that pair, and is taken as one. +**A sample is named by its file, never by its folder.** `Sample1_R1.fq.gz` is sample `Sample1` wherever it lies, so rearranging the folders changes neither `metadata.csv` nor `readPattern`. A pair whose mates sit in two *different* folders is still one pair. -What step 0 checks is the pair itself: **every sample has exactly one of each mate.** Three ways to break it, all of which the reader would otherwise accept in silence: +What is checked is the pair: **every sample has exactly one of each mate.** Three ways to break it, none of which would announce itself: | in `Data/` | what it is | |---|---| -| `Sample1_R1.fq.gz` with no `Sample1_R2.fq.gz` anywhere | a sample that would be left out of the run without a word | -| `run_A/Sample1_R1.fq.gz` and `run_B/Sample1_R1.fq.gz` | the same mate twice, which would be aligned against itself | +| `Sample1_R1.fq.gz` with no `Sample1_R2.fq.gz` anywhere | a sample left out of the run | +| `run_A/Sample1_R1.fq.gz` and `run_B/Sample1_R1.fq.gz` | the same mate twice, aligned against itself | | a full pair under both `run_A/` and `run_B/` | one sample over again, each copy overwriting the other's results | -If two folders really do hold two different samples, give them two names. If they are one sample sequenced twice, that is two rows in `metadata.csv` sharing an `RG_Sample` — see [What a unit is](#what-a-unit-is). - -**Hidden folders are skipped, and the run says so.** Anything beginning with a dot is not searched — `.snapshot` on NetApp storage holds a copy of every file per snapshot, and searching it would find every sample many times over. If one of them does hold reads, step 0 names it so you know what was left out. +Two folders holding two different samples need two names. One sample sequenced twice is two rows in `metadata.csv` sharing an `RG_Sample`; see [What a unit is](#what-a-unit-is). -`init` never overwrites. Running it again in a project you have already filled in reports what is there and changes nothing. +**Hidden folders are skipped, and the run names any that held reads.** Nothing beginning with a dot is searched: `.snapshot` on NetApp storage holds a copy of every file per snapshot, and searching it would find every sample many times over. -It copies `parameters.config` for you because that is a settings file you edit in place. It does **not** write `metadata.csv`, because that is a table describing your experiment — which FASTQ pairs are one pool, how many individuals each holds, and the order your result columns come out in — and a copied one would describe someone else's. Write it yourself, starting from `metadata.csv.example`, and read [Metadata](#metadata) before your first run rather than after it. +Three things are then yours to fill in: -Finished results do not go here. They go to `storageDir`, which has to be a different directory. See [Requirements](#getting-started). +- **Edit `parameters.config`**: `mainDir`, `storageDir`, `readPattern`, `referenceFile`, `poolSize` and `ploidy` at minimum. The reference and the annotation may be gzipped or plain; the pipeline unpacks what it needs into `Reference/Dictionaries/`. +- **Point `storageDir` at a different directory.** Finished results go there rather than into the project. See [Requirements](#getting-started). +- **Write `metadata.csv` yourself.** `init` does not, because a copied table would describe someone else's experiment. Start from the `metadata.csv.example` it left beside you, which explains every column, and read [Metadata](#metadata) before your first run rather than after it. -Then edit `parameters.config`: `mainDir`, `storageDir`, `readPattern`, `referenceFile`, `poolSize` and `ploidy` at minimum. The reference and the annotation may be gzipped or plain — the pipeline takes either and unpacks what it needs into `Reference/Dictionaries/`. +`init` never overwrites: run it again in a project you have filled in and it reports what is there and changes nothing. -Analyzing one set of reads under several parameter sets — two reference genomes, say? Run `PoolSeqFlow init_multi` instead. It does everything `init` does, switches `multiRun` on, and copies `multi-run.csv.example` into the project. It does not write the run table itself, for the same reason it does not write `metadata.csv`. +Analyzing one set of reads under several parameter sets, two reference genomes say? `PoolSeqFlow init multi` does everything `init` does, switches `multiRun` on, and copies `multi-run.csv.example` in. The run table is yours to write, for the same reason `metadata.csv` is. ### 2. Verify it any time { #check } -**There are two checks and they answer different questions**, so `check` takes a word and refuses without one. An installation is a tool; a project is your configuration and your data. Neither answers the other, and a bare `check` would have to guess which you meant — leaving the other unchecked without saying so. +**There are two types of checks:** ```bash PoolSeqFlow check install # the tools and helpers this release is built to run @@ -393,19 +384,15 @@ Pipeline helpers … ``` -**Every command the release is built to run**, with the version each reports. The list is the canonical one — what this release expects its environment to provide. +**Every command the release is built to run**, with the version each reports, and **every helper in `bin/`**, present and executable; a lost executable bit would otherwise fail mid-run rather than at startup. -**And each has to come from that environment.** Every tool on the list is pinned in `install/environment.yml`, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in — at whatever version it happens to be, and only on this machine. That is reported, not passed over: +**Each tool has to come from that environment.** All of them are pinned in `install/environment.yml`, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in, at whatever version it happens to be and only on this machine: ```text samtools samtools OUTSIDE THE ENVIRONMENT /usr/bin/samtools ``` -It is the failure worth catching, because it is the quiet one: the pipeline runs, the results look fine, and nothing reproduces anywhere else. Reinstalling is the fix — `PoolSeqFlow install`. - -**Every helper in `bin/`**, present and executable. `nextflow.config` puts that directory on `PATH` and the process scripts call the helpers by bare name, so a lost executable bit fails mid-run rather than at startup. `bin/` is enumerated rather than listed, so a helper added to a release is checked without anyone remembering to say so. - -The three `check_*.sh` scripts live there too — `bin/` is where everything that is *run* rather than sourced lives — and are the one thing skipped: they are run by the command line, never by a process script, so a run does not depend on them. `lib/` is not enumerated at all, because what is in there is sourced rather than run, which is the whole reason the two directories are separate. +That is the quiet failure worth catching: the pipeline runs, the results look fine, and nothing reproduces anywhere else. Reinstalling with `PoolSeqFlow install` is the fix. **It reads no `parameters.config` and needs no project.** Run it from anywhere, including straight after installing and before you have a project at all. @@ -420,6 +407,7 @@ Configuration parameters.config PARSES metadata.csv PARSES runs.csv not in use + parameters NOTHING TO FLAG Tools, as this project configures them @@ -431,15 +419,13 @@ Tools, as this project configures them tool list from: params.software in parameters.config ``` -**Your files parse** — `parameters.config` through Nextflow itself, and `metadata.csv` and the run table through the same parsers step 0 uses, so what you are told here is what a run would tell you. It also says whether the config was written for this release, which is the one thing that stops a run before anything else is read. - -**Every command, as *you* configured it.** The list comes from `params.software`, so a command [repointed at a system binary](#using-system-tools) is checked the way the run will call it — and the second column shows what will actually be invoked. That override is the setting most likely to be wrong and least likely to announce itself. +**Your files parse**: `parameters.config` through Nextflow itself, `metadata.csv` and the run table through the same parsers a run uses. It also reports whether the config was written for this release, which is the one thing that stops a run before anything else is read. -**A path outside the environment is not a finding here.** `check install` treats one as a fault, because the installation is supposed to provide its own tools; `check project` does the opposite, because repointing one is a thing you are allowed to do and this is where you see the result. It reports what will be invoked and whether it runs, and leaves the judgment to you. +**Your settings are checked by the same rules a run applies**, so a project is never told one thing here and another during the run. [What is checked](#parameter-checks) lists them. `NOTHING TO FLAG` is the healthy answer and the common one: a finding appears only when a setting would make the run produce nothing, or would quietly change what a published number means. -A project that repoints nothing gets the same tool answers from both checks. **That is the point**: the difference between them is exactly your own configuration. +**Every command, as *you* configured it.** The list comes from `params.software`, so a command [repointed at a system binary](#using-system-tools) is checked the way the run will call it, and the second column shows what will actually be invoked. That override is the setting most likely to be wrong and least likely to announce itself. If `params.software` cannot be read, no tool is checked and it says so. -If `params.software` cannot be read, `check project` says so and checks no tool. It never falls back to the canonical list — the whole reason the section exists is that it reads yours. +**A path outside the environment is not a finding here**, where `check install` treats one as a fault. Repointing a command is yours to do, so this check reports what will be invoked and whether it runs, and leaves the judgment to you. ### 3. Fill in the essentials @@ -461,18 +447,16 @@ params { | Setting | How to choose it | |---|---| -| `mainDir` | The working directory you ran `init` in. Holds `Data/`, `Reference/`, your configuration and everything actively processed, so it has to persist between runs — not node scratch that is wiped between jobs. Put it on the fastest storage that satisfies that. | +| `mainDir` | The working directory you ran `init` in. Holds `Data/`, `Reference/`, your configuration and everything actively processed, so it has to persist between runs (not node scratch that is wiped between jobs). Put it on the fastest storage that satisfies that. | | `storageDir` | Where finished results are kept. **A different directory from `mainDir`**, and the pipeline refuses to start otherwise: the two are storage tiers, and outputs move from the first to the second as each step that needs them completes. | | `readPattern` | Must match **both** mates with a `{1,2}` group. If your files end `_1.fastq.gz`/`_2.fastq.gz`, write `"*_{1,2}.fastq.gz"`. | -| `poolSize` | Individuals in **one** pool, not the total across pools. Sets the smallest allele frequency worth believing — see [The Filter Chain](#where-s-comes-from). If your pools differ in size, give each its own value in `metadata.csv`; this setting is the default for any that do not. | +| `poolSize` | Individuals in **one** pool, not the total across pools. Sets the smallest allele frequency worth believing (see [The Filter Chain](#where-s-comes-from)). If your pools differ in size, give each its own value in `metadata.csv`; this setting is the default for any that do not. | | `ploidy` | Ploidy of the organism: `2` for diploid, `1` for haploid, `4` for tetraploid. | | `annotate` | `false` skips step 8 and makes `gffFile` unnecessary. | !!! warning "Set these through the file, never the command line" - PoolSeqFlow rejects command-line parameter overrides. Nextflow delivers `--param` values as strings, so `--annotate false` sets the string `"false"` — which Groovy evaluates as **true**, leaving annotation switched on with no warning. [Why →](#configuration-is-a-file-never-a-flag) - -The two directories cannot be the same path. They are storage tiers, not a preference: the pipeline works on `mainDir` and moves each output to `storageDir` once the last step that needed it has finished, which cannot mean anything if they are one place. On a cluster this is the difference between a node's fast disk and the archive it is backed by; on a laptop, make them two directories and the same reasoning still holds — one is churn, the other is what you keep. + PoolSeqFlow rejects command-line parameter overrides. Nextflow delivers `--param` values as strings, so `--annotate false` sets the string `"false"`, which Groovy evaluates as **true**, leaving annotation switched on with no warning. [Why →](#configuration-without-flags) ### 4. Size the run to your machine @@ -481,7 +465,7 @@ threads = 8 // cores a single task may use memory = '24 GB' // memory ceiling for a single task ``` -Set `threads` to the cores you actually have — on HPC, the size of one node. Every tool's thread count follows from this one number; do not set the per-tool counts by hand. A request larger than the machine fails immediately: +Set `threads` to the cores you actually have (on HPC, the size of one node). Every tool's thread count follows from this one number; do not set the per-tool counts by hand. A request larger than the machine fails immediately: ```text Process requirement exceeds available CPUs -- req: 12; avail: 8 @@ -491,7 +475,7 @@ Details and the full ladder: [Resources](#resources). ### 5. Write `metadata.csv` -One row per FASTQ pair. `SampleID` must match the sample name `readPattern` takes from your filenames — with `*_R{1,2}.fq.gz`, the file `Sample1T1Rep1_R1.fq.gz` gives `Sample1T1Rep1`. +One row per FASTQ pair. `SampleID` is **the whole name before the read tag, exactly as your sequencer delivered it**: with `*_R{1,2}.fq.gz`, the file `Sample1T1_S19_L001_R1.fq.gz` gives `Sample1T1_S19_L001`. Do not shorten it, because it is what finds your files. The name you want your *results* under goes in `RG_Sample`, and [you should always write that one out](#the-read-group-tags). ```csv SampleID,RG_Sample,RG_Library,RG_Platform,param_poolSize,exp_population,exp_time,pt_resistance,cov_temperature @@ -507,23 +491,26 @@ Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,40,Pop2,T2,resistant,19.4 Three things this file decides: -- **`RG_Sample` decides what counts as a sample.** Rows sharing one are merged into a single VCF column and their read depths add together. The eight rows above produce **four** columns, not eight — two populations at two timepoints, each sequenced twice. +- **`RG_Sample` decides what counts as a sample.** Rows sharing one are merged into a single VCF column and their read depths add together. The eight rows above produce **four** columns, not eight: two populations at two timepoints, each sequenced twice. - **`param_poolSize` sets that pool's detection limit.** It describes the pool rather than the row, so rows sharing an `RG_Sample` have to agree on it. Leave the column out and every pool uses the global `poolSize`. - **Row order decides column order** in the VCF and the frequency tables. -**The prefix is what a column means.** No pipeline step reads any of the three below — [analysis modules](#the-experimental-design) do, and they build their whole design from them: +**The prefix is what a column means.** No pipeline step reads any of the three below; [analysis modules](#the-experimental-design) do, and they build their whole design from them: | Prefix | What it holds | Example above | |---|---|---| -| `exp_` | something you **set** — the experiment's own structure | `exp_population`, `exp_time` | +| `exp_` | something you **set**: the experiment's own structure | `exp_population`, `exp_time` | | `pt_` | a phenotype you **measured** on the pool, and are testing against | `pt_resistance` | | `cov_` | measured too, but neither set nor the thing being tested | `cov_temperature` | -Add as many of each as your experiment needs; the names after the prefix are yours. A column with **no** prefix is recorded just as faithfully and read by nothing, so a variable you might analyze later is worth prefixing now. Editing any of them never invalidates results you already have. All three describe the **pool** — what differs between two rows of one pool takes no prefix, because once their reads are merged nothing can tell them apart again. [Metadata](#kinds-of-metadata-column) has the full account. +- Add as many of each as your experiment needs; the names after the prefix are yours. +- A column with **no** prefix is recorded just as faithfully and read by nothing, so a variable you might analyze later is worth prefixing now. +- Editing any of them never invalidates results you already have. +- All three describe the **pool**. What differs between two rows of one pool takes no prefix, because once their reads are merged nothing can tell them apart again. -**`exp_time` is the one name treated specially.** A project that has it must set `analysis.timeVar.kind` before any analysis will run — the pipeline itself does not care. Leave the column out if this is not a time course. See [The time axis](#the-time-axis). +**`exp_time` is the one name treated specially.** A project that has it must set `analysis.timeVar.kind` before any analysis will run; the pipeline itself does not care. Leave the column out if this is not a time course. See [The time axis](#the-time-axis). -All of it is covered in [Metadata](#metadata). Getting `RG_Sample` wrong is the most common way to end up with results that are valid but not what you meant. +[Metadata](#metadata) has the full account, and getting `RG_Sample` wrong is the most common way to end up with results that are valid but not what you meant. ### 6. Run @@ -533,11 +520,11 @@ PoolSeqFlow run That is also the resume command. Every step checks whether its outputs already exist and skips itself if they do, so an interrupted run picks up where it left off with no extra flag. There is no `-resume`. -Run it from your project directory — the one holding `parameters.config`. The pipeline reads its settings from wherever you launch, so running from anywhere else stops with a message saying so. +**Run it from your project directory**, the one holding `parameters.config`: that is where the settings are read from, and anywhere else stops with a message saying so. To see where a run's results would go before spending any compute on it, `PoolSeqFlow dryrun` builds that directory tree empty and changes nothing else; `PoolSeqFlow dryclean` removes the preview. -To start genuinely from scratch, use `PoolSeqFlow reset` first — it requires typing `DELETE_MY_ANALYSIS` to confirm. +To start genuinely from scratch, use `PoolSeqFlow reset` first. It requires typing `DELETE_MY_ANALYSIS` to confirm. ### 7. Check the output @@ -557,7 +544,7 @@ storageDir/ Start with `Output/Reports/Depth/` and `Output/run_parameters.txt`. The first says what depth ceiling each sample was given and why; the second is a read-only record of exactly which settings produced these files. -Your inputs stay where they were, under `mainDir` — `Data/`, `Reference/` and the dictionaries built from your reference are working material, not results, and none of them are copied here. +Your inputs stay where they were, under `mainDir`. `Data/`, `Reference/` and the dictionaries built from your reference are not copied here. If you are running a table of several runs, `Output/` and `Logs/` gain a level: work every run shared sits under `All_Runs/`, work some of them shared under `Shared_/`, and whatever one run did alone under its own `RunID`. A single run keeps the plain tree above. @@ -571,29 +558,27 @@ How to read the tables: [Interpreting Results](#interpreting-results). |---|---| | `PoolSeqFlow install` | Create this release's conda environment, install the pipeline, then verify both | | `PoolSeqFlow init` | Populate the current directory as a project ([what it writes](#make-your-project)) | -| `PoolSeqFlow init_multi` | The same, for a project running several parameter sets over one set of reads | -| `PoolSeqFlow check install` | Verify an installation — the tools and helpers it is built to run ([what it covers](#check-install)) | -| `PoolSeqFlow check project` | Verify a project — its configuration, and the commands it names ([what it covers](#check-project)) | -| `PoolSeqFlow run` | Start — or resume — the pipeline | +| `PoolSeqFlow init multi` | The same, for a project running several parameter sets over one set of reads | +| `PoolSeqFlow check install` | Verify an installation: the tools and helpers it is built to run ([what it covers](#check-install)) | +| `PoolSeqFlow check project` | Verify a project: its configuration, and the commands it names ([what it covers](#check-project)) | +| `PoolSeqFlow run` | Start (or resume) the pipeline | | `PoolSeqFlow dryrun` | Create the directory tree the run would write, empty, so the layout can be approved before any compute is spent. Records nothing and changes none of your files | | `PoolSeqFlow dryclean` | Remove the preview `dryrun` made | | `PoolSeqFlow migrate_config` | Carry an older `parameters.config` onto the current template ([details](#upgrading)) | | `PoolSeqFlow clean` | Remove Nextflow work directories | | `PoolSeqFlow reset` | Remove all progress and start fresh (requires typed confirmation) | -| `PoolSeqFlow analysis ` | The analysis layer, which carries a word of its own — listed below | +| `PoolSeqFlow analysis ` | The analysis layer, which carries a word of its own (listed below) | | `PoolSeqFlow version` | Print the version of this copy | | `PoolSeqFlow cite` | Print how to cite this copy, and which DOI to use ([why it matters](#which-doi-to-use)) | | `PoolSeqFlow list` | List the pipelines and conda environments installed on this machine | | `PoolSeqFlow uninstall` | Remove one installed version, environment and pipeline together, after confirmation | -| `PoolSeqFlow uninstall_all` | Remove every PoolSeqFlow environment and installation, after confirmation | +| `PoolSeqFlow uninstall all` | Remove every PoolSeqFlow environment and installation, after confirmation | Before anything is installed there is no `PoolSeqFlow` on your `PATH`, so the first command is `./PoolSeqFlow install`, run from the folder you downloaded. Everything after that uses the installed command. -**Each subcommand takes no arguments of its own**, with one exception: `analysis` carries a word — the analysis command, or the module to run — and `analysis modules` carries a second. Anything else is rejected. Two environment variables adjust the wrapper instead: `POOLSEQFLOW_PREFIX`, where `install` puts things and where `list` and `uninstall` look, and `POOLSEQFLOW_HOME`, to run a checkout without installing it. - -**Naming a version.** Every installed release is also on your `PATH` under its own name, so `PoolSeqFlow-2.1.0 run` uses that release and plain `PoolSeqFlow` uses the newest. This matters most for `uninstall`: with several installations present it lists them and asks which to remove, and if nothing is attached to ask — a script, a CI job — it refuses and tells you to name one, rather than guessing at which installation to delete. `PoolSeqFlow-2.1.0 uninstall` names it and is never asked. +**Each subcommand takes no arguments of its own**, with one exception: `analysis` carries a word (the analysis command, or the module to run) and `analysis modules` carries a second. Anything else is rejected. Two environment variables adjust the wrapper instead: `POOLSEQFLOW_PREFIX`, where `install` puts things and where `list` and `uninstall` look, and `POOLSEQFLOW_HOME`, to run a checkout without installing it. -**What that list contains, and what removing one takes with it.** Every installed version, plus the unversioned `PoolSeqFlow` environment if a release from before per-version environments left one behind. A version whose analysis layer is installed is marked, because **an installation is removed whole** — its environment, its pipeline, and its analysis environment together: +**Naming a version.** Every installed release is also on your `PATH` under its own name, so `PoolSeqFlow-2.1.0 run` uses that release and plain `PoolSeqFlow` uses the newest. `PoolSeqFlow list` shows what is installed, marks a version whose analysis layer is installed, and includes the unversioned environment if a release from before per-version environments left one behind: ```text Installed under /home/you/.local: @@ -602,18 +587,14 @@ Installed under /home/you/.local: 3) PoolSeqFlow- (this wrapper, analysis installed) ``` -Choosing 3 removes `PoolSeqFlow-` and `PoolSeqFlow--analysis`. To remove only an analysis layer and keep the pipeline that produced your results, use `PoolSeqFlow analysis uninstall`, which never touches anything else. - -**Both then list exactly what will go and ask before removing any of it**, every time — including when there is only one installation and nothing to choose between. Choosing *which* is not the same as agreeing to the removal. Answering anything but `y` removes nothing, and with no terminal attached to ask — a script, a CI job — the command refuses rather than proceeding unasked. That is the same rule `uninstall_all` has always followed. - -`PoolSeqFlow resume` still works as a deprecated alias for `run` and prints a notice. +Naming a version matters most when removing one; see [Uninstalling](#uninstalling). **The analysis layer hangs off `analysis`**, the one subcommand that carries a word of its own. It ships with the same `PoolSeqFlow install` and is versioned with it: | Command | Description | |---|---| | `PoolSeqFlow analysis install` | Create this release's analysis conda environment, which carries R, then verify it | -| `PoolSeqFlow analysis check` | Verify an existing analysis installation — tools, R packages, entry point | +| `PoolSeqFlow analysis check` | Verify an existing analysis installation: tools, R packages, entry point | | `PoolSeqFlow analysis modules available` | List the modules published for this release's table contract | | `PoolSeqFlow analysis modules install [version]` | Install one, newest readable version unless you name it. Also installs [what it runs on](#module-packages) | | `PoolSeqFlow analysis modules list` | List the modules installed for this release, and name any that are broken | @@ -623,63 +604,47 @@ Choosing 3 removes `PoolSeqFlow-` and `PoolSeqFlow--analysis`. | `PoolSeqFlow analysis cite` | Print how to cite this copy, plus R and every package a module runs on | | `PoolSeqFlow analysis uninstall` | Remove the analysis environment, after confirmation. The pipeline, its environment, and your results are untouched | -The environment is separate from the pipeline's and is built only when you ask for it. `PoolSeqFlow uninstall` removes both environments of the version it is removing. - -`cite` reads R's own citation records, so what it prints is the version of each package actually installed rather than a list kept in the documentation. +The analysis environment is separate from the pipeline's and is built only when you ask for it. `analysis cite` reads R's own citation records, so it prints the version of each package actually installed. -`PoolSeqFlow analysis` also accepts the name of an installed module. Modules are installed separately from the pipeline and each is documented with itself; the commands in the table above are the ones a release ships with. +`PoolSeqFlow analysis` also accepts the name of an installed module. Modules are installed separately and each is documented with itself; the table above is what a release ships with. ## Upgrading -Two separate things upgrade here, and it helps to keep them apart. **The installation** is a tool: a new release installs alongside the ones you already have and disturbs none of them. **A project** is your results — and a project belongs to one release for as long as it exists. +Two separate things upgrade here. + +- **The installation**: a new release installs alongside the ones you already have and disturbs none of them. +- **A project**: your results, and a project belongs to one release for as long as it exists. -### Installing a new release +Coming from 2.2.0 or older, read [Coming from 2.2.0](#coming-from-220) alongside this: 3.0 moved where things live, not just the settings. -Download and install it exactly as you did the first time. Nothing is replaced: `~/.local/opt` gains a directory, `~/.local/bin` gains a `PoolSeqFlow-` command, and plain `PoolSeqFlow` starts meaning the new one. Every earlier release still runs, under its own name. +### 1. Install the new release { #install-a-new-release } -Your projects are untouched by this. `parameters.config` lives in your project directory, not in the installation, so nothing an install or an uninstall does can reach it. +Get it and install it exactly as you did the first time ([Install](#install)). -**The new release's module store starts empty.** No module ships inside a release, and the store belongs to the installation that runs it — so the old release keeps everything you installed into it and the new one has nothing. Ask the old one what to reinstall, then install each into the new one: +- Nothing is replaced. `~/.local/opt` gains a directory, `~/.local/bin` gains a `PoolSeqFlow-` command, and plain `PoolSeqFlow` starts meaning the new one. Every earlier release still runs under its own name. +- Your projects are untouched: `parameters.config` lives in your project directory, not in the installation. +- **The new release's module store starts empty.** No module ships inside a release, and the store belongs to the installation that runs it, so the old release keeps what you installed into it and the new one has nothing. + +Ask the old release what to reinstall, then install each into the new one: ```bash PoolSeqFlow-3.0.0 analysis modules list # what the old release has PoolSeqFlow analysis modules install mds # and again, into the new one ``` -Nothing is lost by this. A module installs the libraries it needs along with it, and the analyses it has already produced are results — no install or uninstall reaches them. - -### A project belongs to one release - -This is the part that changes how upgrading works. Completed steps are skipped by looking for output files, not by checking what produced them, so continuing an existing project under a new release would leave one set of results built by two versions of the pipeline, with nothing on disk to say which is which. - -The pipeline refuses. On the first run after upgrading it stops before any work happens: - -```text -PIPELINE VERSION: These results were produced by 2.2.0, -PIPELINE VERSION: and this is 3.0.0. -``` - -There are two ways forward, and both are deliberate choices rather than defaults: - -- **Finish the project on the release that started it** — `PoolSeqFlow-2.2.0 run`. The version that produced your results is still installed and still works, which is the reason releases sit side by side. -- **Start the project again under the new one** — `PoolSeqFlow reset`, then run. This deletes the existing results. - -There is deliberately no third option. A project that changed version midway has no answer to the question of which version to cite. +Nothing is lost by this. A module installs the libraries it needs along with it, and the analyses it has already produced are results; no install or uninstall reaches them. -### Bringing your configuration forward +### 2. Bring each project's configuration forward { #migrate-config } -`parameters.config` belongs to you and is never touched by an update — an update must not silently change your analysis settings. The consequence is that after installing a new release your file can be **missing parameters the newer code expects**, and `migrate_config` is the tool for exactly that. Run it before anything else. - -Skip it and the run stops before anything is computed. A configuration from an older release does not set `storageDir`, the pipeline recognizes that, and it refuses — naming the parameters it found that this release renamed or removed, and telling you to run `migrate_config`. The refusal covers `run`, `resume`, `dryrun`, `reset`, running an analysis module, and `analysis complete`. It never applies to `migrate_config` itself, which is the fix, nor to `clean` and `dryclean`, which read nothing it covers. - -### The assisted route +Run this in each project directory, before the first run under the new release: ```bash PoolSeqFlow migrate_config ``` -Run it in your project directory. It backs your file up, rebuilds it from the current template, carries across every setting whose parameter still exists, and reports what happened to each one: +- An update never touches `parameters.config`, because it must not silently change your analysis settings. So after a new release your file can be **missing parameters the newer code expects**. +- `migrate_config` saves your file as `parameters.config.bak`, rebuilds it from the current template, carries across every setting whose parameter still exists, and reports what happened to each one. | Report | Meaning | |---|---| @@ -687,21 +652,21 @@ Run it in your project directory. It backs your file up, rebuilds it from the cu | `Renamed this release` | The parameter was renamed and your value followed it to the new name | | `Now computed by the pipeline` | This release derives the value; yours was ignored | | `Format changed this release` | The value's meaning or format changed, so the template's wins | -| `New in this release` | The template has a parameter your file did not — review the default | -| `Still yours to set` | The pipeline works the value out itself now, and your new config carries the parameter **commented out** — uncomment it to take it back | +| `New in this release` | The template has a parameter your file did not; review the default | +| `Still yours to set` | The pipeline works the value out itself now, and your new config carries the parameter **commented out**; uncomment it to take it back | | `No longer used` | Your file had a parameter this release does not use | -`Still yours to set` is the one people mistake for a loss. Coming from 2.2.0 it covers thirteen parameters — the eight `cores` values and the five `options` strings your file already had — none of which is gone. They are computed by default and sit commented out in your new config, so setting one is a matter of removing a `//`. It is distinct from `Now computed by the pipeline`, where the value is derived from other parameters and there is no line to uncomment. +**`Still yours to set` is not a loss**, and is distinct from `Now computed by the pipeline`, where the value is derived from other parameters and there is no line to uncomment. Coming from 2.2.0 it covers thirteen parameters: the eight `cores` values and the five `options` strings your file already had. -It also ends with a list of **files to move yourself**, and moves none of them. If you are upgrading from 2.2.0 or older, there will be several, because the layout changed: your reads, reference and sample table used to live under the storage directory and now belong on `mainDir`. It prints the exact `mv` commands, having checked which files are actually there — read them before running them. +**Move the files it names.** It ends with a list and moves none of them, printing the exact `mv` commands for the files it found. From 2.2.0 or older there will be several, because your reads, reference and sample table used to live under the storage directory and now belong on `mainDir`. -One group in that list matters more than the rest. `.poolseqflow_params` and its neighbors are the records the change guard compares against — they are how "has anything changed since these results were produced" gets answered. Leave them behind and the next run finds no record, decides the project is new, and writes down your *current* configuration as though it had produced the results already on disk. The guard would then report that nothing has changed, having quietly stopped guarding. +**`.poolseqflow_params` and its neighbors matter most in that list.** They are the records the change guard compares against. Leave them behind and the next run finds no record, decides the project is new, and writes your *current* configuration down as though it had produced the results already on disk. The guard then reports that nothing has changed, having quietly stopped guarding. -**Treat the migrated config as a starting point, not an answer.** Migration can only recognize a parameter that still exists *and still means the same thing*. A parameter whose behavior changed while its value still looks like an ordinary number or string is carried across and is silently wrong. Always read the report, and compare afterwards. +**Treat the migrated config as a starting point, not an answer.** Migration can only recognize a parameter that still exists *and still means the same thing*. One whose behavior changed while its value still looks like an ordinary number or string is carried across and is silently wrong. -### The manual route +#### Rebuilding by hand -Every release adds parameters, and rebuilding by hand is often the safer choice — it is the only way to be certain you have actually looked at the new ones. The template ships inside the installation, and `init` is what puts a copy of it in front of you: +Often the safer choice, and the only way to be certain you have looked at the new parameters: ```bash mv parameters.config parameters.config.bak # keep your settings @@ -709,35 +674,60 @@ PoolSeqFlow init # writes a fresh parameter diff parameters.config.bak parameters.config # see what changed, then re-apply yours ``` -`init` never overwrites, so move your own file aside first, as above — otherwise it reports the config as already present and leaves it alone. +`init` never overwrites, so move your own file aside first; otherwise it reports the config as already present and leaves it alone. -### Coming from 2.2.0 +### A project belongs to one release + +Completed steps are skipped by looking for output files, not by checking what produced them, so continuing a project under a new release would leave one set of results built by two versions, with nothing on disk to say which is which. The pipeline refuses, before any work happens: + +```text +PIPELINE VERSION: These results were produced by 2.2.0, +PIPELINE VERSION: and this is 3.0.0. +``` + +`migrate_config` does not clear this. There are two ways forward and deliberately no third, because a project that changed version midway has no answer to the question of which version to cite. + +- **Finish the project on the release that started it**: `PoolSeqFlow-2.2.0 run`. That version is still installed and still works, which is the reason releases sit side by side. +- **Start it again under the new one**: `PoolSeqFlow reset`, then run. This deletes the existing results. + +A project started fresh under the new release is governed by the ordinary [change guard](#the-change-guard) from then on, and the [Changelog](#changelog) has what each release changed. -3.0 is a major release that brings in a complete new design and more reproducibility options, and the changes are structural rather than a handful of new settings. -**Where things live has changed.** Before 3.0 there was one storage directory holding your inputs and your results together. There are now two, and they must be different paths: `mainDir` holds your reads, reference and configuration and is where you run, `storageDir` holds finished results. `migrate_config` prints the `mv` commands for the files that need to move. +## Uninstalling + -**`projectDir` is now `storageDir`.** A straight rename, and your value is carried over — note that `projectDir` is also a name Nextflow defines for itself, which is why it could not stay. +An installation is removed whole: its conda environment, its analysis environment where one is installed, the pipeline under `~/.local/opt`, and the links in `~/.local/bin`. Versions are separate, so removing one leaves the others working. `PoolSeqFlow list` shows what is on the machine. -**`RGTags.csv` is replaced by `metadata.csv`.** This one is not a rename and cannot be migrated: the old file held raw SAM read-group tags and nothing else, while the new one has [seven kinds of column](#kinds-of-metadata-column) and carries your pool sizes as well. `migrate_config` reports `rgTagsFile` as no longer used, tells you to move the file, and prints a note saying plainly that the two are not the same file under a new name — but rewriting it into the new schema is yours to do. Start from `$POOLSEQFLOW_HOME/metadata.csv.template`, which documents every column, and read [Metadata](#metadata). **Until `metadata.csv` exists the run stops at step 0**, so this is not a step you can defer. It is also the change that buys the most: the experiment itself — populations, timepoints, replicates — finally has somewhere to live. +### One version -**The depth ceiling moved, and your old value is deliberately not carried.** In 2.2.0 `bcftools.maxDepth = 2000` was the only depth control there was: one number for every pileup in the run. From 3.0, step 5 measures a ceiling for each sample from its own depth histogram and step 6 applies it to the BAM before calling, so a sample is capped where its own coverage says to rather than at one number for the whole cohort — see [Depth capping](#depth-capping). `variantCall.maxDepth` is now a second ceiling on top of that one and ships as `0`, which `mpileup` reads as no limit at all. +```bash +PoolSeqFlow uninstall # the only installation, or it asks which +PoolSeqFlow-3.1.1 uninstall # that version, named +``` -Carrying `2000` across would leave you capped at a number this release never chose, on top of a per-sample cap that cannot see it. So `migrate_config` reports it under `Format changed this release` and explains the change in full. Automatic capping is `capBAM.maxDepth = -1`, which is what you now have. **To reproduce results from 2.2.0 exactly, set `variantCall.maxDepth` back to your old value and `capBAM.maxDepth = 0`.** +- With several versions installed, plain `PoolSeqFlow uninstall` lists them and asks which. In a script, where there is nothing attached to ask, it refuses rather than guessing; name the version instead. +- Plain `PoolSeqFlow` is re-pointed at the newest version that remains, and removed once none does. +- `PoolSeqFlow analysis uninstall` removes only that release's analysis environment, leaving the pipeline installed. +- If you installed with `POOLSEQFLOW_PREFIX` set, set it again: an uninstall looks only under that prefix. -**The installation is separate from your project now.** Earlier releases were run from the folder you unpacked, with `parameters.config` beside the pipeline. From 3.0 you install once, releases sit side by side, and you run the installed command from your own project directory. If your project *is* the old unpacked folder, move it out — the pipeline refuses to run inside its own installation. +### Every version -**Some parameters are gone**: `params.gff`, `params.dir.scripts` and `params.dir.output.temp`, along with `rgTagsFile` and `rgTagsPath` from the section above. `migrate_config` reports each under `No longer used`. The `cores` block and the `options` strings look gone too and are not — they are computed now and ship commented out, which the report says under `Still yours to set`. +```bash +PoolSeqFlow uninstall all +``` -**Multi-run is new**, and off by default — `multiRun = false` changes nothing about how an existing project behaves. +Every PoolSeqFlow conda environment on the machine, every installed pipeline, and the links in `bin`. -### After upgrading +### What is not touched -Once your configuration is current, the first run will still stop, because your existing results were produced by an older release. That is the version block described above, and `migrate_config` does not clear it: choose between finishing on the old release and starting again on the new one. +- **Your projects.** `parameters.config`, `Data/`, `Reference/` and everything in `storageDir` sit outside every installation, including the analyses a module has already produced. +- **Another version's modules.** A module store belongs to the installation that holds it, so modules go with that version and with no other. -For a project started fresh under 3.0, the ordinary guard applies from then on — a change to an analysis-affecting parameter stops the next run, and the report names the folders to delete. +Both forms list what will go and ask before anything does. The tab completions go with the last version removed; another version left installed keeps them. -See [The run refuses to mix settings](#the-run-refuses-to-mix-settings) for what is and is not tracked, and the [Changelog](#changelog) for what each release changed. +**Removing an environment means results produced with that version can no longer be reproduced without reinstalling it.** + +**On a network filesystem the directory can outlive its contents.** Files the running command still has open leave `.nfs*` placeholders behind, so the pipeline directory cannot remove itself from under itself. It says so, and prints the one `rm -rf` that finishes the job after the command has exited. # When to use PoolSeqFlow @@ -750,9 +740,9 @@ In a pooled library, DNA from many individuals is combined before sequencing. Ev | You gain | You give up | |---|---| -| Frequency estimates for many individuals at a fraction of the per-individual cost | Individual genotypes — and therefore heterozygosity, relatedness, and anything phased | +| Frequency estimates for many individuals at a fraction of the per-individual cost | Individual genotypes, and therefore heterozygosity, relatedness, and anything phased | | Depth concentrated where it matters: on the frequency estimate, not on calling a genotype confidently | The ability to separate sampling noise from real low-frequency variation without care | -| A design that scales to large populations and many time points | Straightforward variant filtering — the usual genotype-based heuristics do not apply | +| A design that scales to large populations and many time points | Straightforward variant filtering: the usual genotype-based heuristics do not apply | The second row is the one that shapes this pipeline. In individual sequencing, a variant seen on two reads out of a hundred is almost certainly an error. In a pool of 50 diploids, one chromosome out of 100 is a real allele at frequency 0.01, and it looks exactly the same. Distinguishing the two is not something a fixed cutoff can do, which is why PoolSeqFlow derives its threshold from your pool size and ploidy rather than hard-coding one. See [The Filter Chain](#the-filter-chain). @@ -763,9 +753,9 @@ These are structural assumptions. If your data does not match, the pipeline will | Assumption | Where it comes from | If you do not match | |---|---|---| | Paired-end Illumina reads | `readPattern` matches an `R1`/`R2` pair; step 4 requires properly-paired alignments (`0x2`) | Single-end data will not survive the pairing filter | -| One reference genome per run | Variant calling is a single joint `bcftools mpileup` over all BAMs | Samples on different references cannot be called together. Running one set of reads *against* several references is a different thing, and supported — that is what a run table is for | -| One ploidy per run | `ploidy` sets the detection threshold for every pool in the run | Split the work into runs, each with its own `ploidy` — see [below](#sequences-with-a-different-ploidy) | -| Several comparable pools per run | The false-positive filter keeps a variant only if a **fraction of samples** support it | A single-sample run needs `sampleThreshold` reconsidered — see [below](#single-sample-runs) | +| One reference genome per run | Variant calling is a single joint `bcftools mpileup` over all BAMs | Samples on different references cannot be called together. Running one set of reads *against* several references is a different thing, and supported: that is what a run table is for | +| One ploidy per run | `ploidy` sets the detection threshold for every pool in the run | Split the work into runs, each with its own `ploidy` (see [below](#sequences-with-a-different-ploidy)) | +| Several comparable pools per run | The false-positive filter keeps a variant only if a **fraction of samples** support it | A single-sample run needs `sampleThreshold` reconsidered (see [below](#single-sample-runs)) | | A reference FASTA, and a GFF if you annotate | `referenceFile`, `gffFile` | Either may be gzipped or plain; the pipeline takes both and unpacks what it needs | | Compute and storage reachable from one process | `mainDir` and `storageDir` can be on different filesystems, and both must be mounted | Cloud object storage without a filesystem mount is not supported | @@ -801,7 +791,7 @@ nuclear,chromosomes.fasta.gz,chromosomes.gff.gz,2 mitochondrial,mito.fasta.gz,mito.gff.gz,1 ``` -Both runs read the same FASTQ files, and the work they share — trimming and quality control — is done once rather than twice. Changing `ploidy` re-derives that run's sensitivity threshold automatically, so there is nothing else to keep in step. +Both runs read the same FASTQ files, and the work they share (trimming and quality control) is done once rather than twice. Changing `ploidy` re-derives that run's sensitivity threshold automatically, so there is nothing else to keep in step. The same argument applies to anything else whose copy number differs from the nuclear genome: chloroplast sequences, and unplaced scaffolds whose ploidy you are not confident about. Splitting them out costs one row in the table and makes each threshold defensible. @@ -815,7 +805,7 @@ At the default `sampleThreshold = 0.2`, an allele present in only one pool out o #### Very high depth -Deep is not a problem in itself: no flat depth ceiling ships any more. Each sample gets one measured from its own coverage, and a library that is uniformly deep is left alone — see [Depth capping](#depth-capping). What a very deep run does change is what an *anomaly* looks like, so read `Output/Reports/Depth/` rather than assuming: a sample reported uncapped had nothing separable to cut, which on a deeply and unevenly sequenced library is worth knowing. +Deep is not a problem in itself: no flat depth ceiling ships any more. Each sample gets one measured from its own coverage, and a library that is uniformly deep is left alone (see [Depth capping](#depth-capping)). What a very deep run does change is what an *anomaly* looks like, so read `Output/Reports/Depth/` rather than assuming: a sample reported uncapped had nothing separable to cut, which on a deeply and unevenly sequenced library is worth knowing. ### Choosing between run layouts @@ -825,7 +815,7 @@ Deep is not a problem in itself: no flat depth ceiling ships any more. Each samp | One pool split across lanes or runs to reach depth | One run, one row per FASTQ pair, **sharing** an `RG_Sample` so the reads are combined | | Technical replicates you want treated as one observation | Share an `RG_Sample` | | Technical replicates you want to compare against each other | Distinct `RG_Sample` values | -| Pools of different sizes | One run — give each pool its own `param_poolSize` in `metadata.csv` | +| Pools of different sizes | One run: give each pool its own `param_poolSize` in `metadata.csv` | | To be compared against different reference genomes | One project, one row per reference in a run table | | Nuclear and organellar sequences together | One project, one row per ploidy in a run table | @@ -836,7 +826,7 @@ The `RG_Sample` decision is the one most often got wrong by accident, because tw ## Scientific Rationale -This page is about the **tools this pipeline chose and why**, not about pooled sequencing as a method. That method is established and well described elsewhere — see [Futschik & Schlötterer 2010](#ref-futschik2010markers) for the estimation problem and [Kofler et al. 2011a](#ref-kofler2011popoolation) for the toolkit most people met it through. +This page is about the **tools this pipeline chose and why**, not about pooled sequencing as a method. That method is established and well described elsewhere. See [Futschik & Schlötterer 2010](#ref-futschik2010markers) for the estimation problem and [Kofler et al. 2011a](#ref-kofler2011popoolation) for the toolkit most people met it through. What follows matters because a pooled allele frequency is a measurement, and every stage between the reads and that number can bias it. A choice that is harmless when you are calling genotypes in one individual is not necessarily harmless when the quantity of interest is a fraction. @@ -844,9 +834,9 @@ Every run writes its own `CITATIONS.md` and `references.bib` beside the results, ### Trimming: Trim Galore, then cutadapt -Trim Galore is an excellent adapter trimmer, and it is at its best when it is given the adapters rather than left to detect them. Its thread handling is the awkward part — `--cores N` runs rather more than N threads — which is why the pipeline works out the core allocation for you rather than passing a number straight through. +Trim Galore is an excellent adapter trimmer, and it is at its best when it is given the adapters rather than left to detect them. Its thread handling is the awkward part (`--cores N` runs rather more than N threads), which is why the pipeline works out the core allocation for you rather than passing a number straight through. -Running cutadapt afterwards costs time the pipeline would otherwise save. It is worth it because trimming can leave the retained portion of a read skewed in base composition, and the clipping step exists to keep the A/T and G/C ratios near 1 across the length of the read. **For frequency estimation a compositional bias is not cosmetic** — it moves the numbers, and it moves them in a way nothing downstream can detect. +Running cutadapt afterwards costs time the pipeline would otherwise save. It is worth it because trimming can leave the retained portion of a read skewed in base composition, and the clipping step exists to keep the A/T and G/C ratios near 1 across the length of the read. **For frequency estimation a compositional bias is not cosmetic**: it moves the numbers, and it moves them in a way nothing downstream can detect. ### Alignment: bwa, with a fixed batch size @@ -856,9 +846,9 @@ bwa is the standard, and there is little to argue about. One detail is worth sta This is a considered preference rather than a default. Working through GATK and Picard in earlier versions of this pipeline, the read-group handling in particular was a recurring source of trouble; the samtools equivalents are less forgiving to use but they produce a cleaner result. Everything Picard was doing in the earlier pipelines, here is now samtools. -The same goes for calling. Recent comparisons of germline callers put samtools/bcftools among the accurate ones and consistently the most efficient — short runtimes and low memory where the deep-learning callers need far more of both. That is the basis for using it here, and call quality matters more in this setting than it does when the output is a genotype: an allele frequency inherits every error in the underlying counts rather than rounding it away. +The same goes for calling. Recent comparisons of germline callers put samtools/bcftools among the accurate ones and consistently the most efficient (short runtimes and low memory where the deep-learning callers need far more of both). That is the basis for using it here, and call quality matters more in this setting than it does when the output is a genotype: an allele frequency inherits every error in the underlying counts rather than rounding it away. -**Those comparisons do not measure what this pipeline needs, and it is worth being clear about that.** They benchmark genotype calls on a single diploid individual against a truth set. This pipeline wants allele frequencies from a pool, and precision on a genotype call says little directly about the bias in a frequency derived from allele depths. The choice here rests on efficiency, on the `AD` handling being what the rest of the pipeline reads, and on long familiarity with the tool's failure modes — not on a benchmark that measured a different quantity. +**Those comparisons do not measure what this pipeline needs, and it is worth being clear about that.** They benchmark genotype calls on a single diploid individual against a truth set. This pipeline wants allele frequencies from a pool, and precision on a genotype call says little directly about the bias in a frequency derived from allele depths. The choice here rests on efficiency, on the `AD` handling being what the rest of the pipeline reads, and on long familiarity with the tool's failure modes. It does not rest on a benchmark that measured a different quantity. The cleaning step is deliberately strict: anything that is not a properly paired read is discarded. That costs a little depth, which is the right trade when the quantity being measured is a ratio. @@ -872,13 +862,13 @@ The reason is that in pooled data **the reference allele carries no special stat These two are simple to state and take a moment to sit with. -Each pool has a **detection limit** set by how many chromosomes went into it: `1 / (ploidy × poolSize)` is the smallest allele fraction that pool can physically produce — one chromosome out of all of them. The limit is set to half of this value, (`1 / (2 x ploidy × poolSize)`) as a statistical threshold. Anything below it is noise rather than a rare allele, because the pool contains nothing that could generate it. +Each pool has a **detection limit** set by how many chromosomes went into it: `1 / (ploidy × poolSize)`, one chromosome out of all of them, is the smallest allele fraction that pool can physically produce. The limit is set to half of this value, (`1 / (2 × ploidy × poolSize)`) as a statistical threshold. Anything below it is noise rather than a rare allele, because the pool contains nothing that could generate it. The filter therefore applies a threshold **per pool**, not one threshold for the whole file. A pool of ten individuals and a pool of a hundred have different resolutions, and judging both at the same cut-off either discards real variation in the large pool or keeps noise in the small one. -Clearing that limit in one pool is not by itself a reason to keep a site, so a second choice sits beside it: **`sampleThreshold`**, the proportion of pools in which an allele has to clear its own detection limit before the site survives. It defaults to `0.2`, and it is converted to a count — with six pools, an allele must be detectable in at least two of them. +Clearing that limit in one pool is not by itself a reason to keep a site, so a second choice sits beside it: **`sampleThreshold`**, the proportion of pools in which an allele has to clear its own detection limit before the site survives. It defaults to `0.2`, and it is converted to a count. With six pools, an allele must be detectable in at least two of them. -**This one is deliberately yours to set, because the right value depends on what you are asking.** A high threshold keeps variation that is shared across the cohort and discards anything private to one population — which is correct if you are estimating differentiation over standing variation, and exactly wrong if you are looking for a sweep or a locally adapted allele, since the sites carrying your signal are the ones a high threshold removes first. A low threshold keeps those, and admits more noise with them. The default is permissive on purpose: it is easier to notice that a table is noisy than to notice that the site you were looking for was discarded before you saw it. +**This one is deliberately yours to set, because the right value depends on what you are asking.** A high threshold keeps variation that is shared across the cohort and discards anything private to one population, which is correct if you are estimating differentiation over standing variation, and exactly wrong if you are looking for a sweep or a locally adapted allele, since the sites carrying your signal are the ones a high threshold removes first. A low threshold keeps those, and admits more noise with them. The default is permissive on purpose: it is easier to notice that a table is noisy than to notice that the site you were looking for was discarded before you saw it. The point that takes longer to absorb is that neither of these is a **confidence filter**. It is not saying an allele is probably wrong. It is saying that the experiment as designed could not have detected it, so a number reported there describes the sequencing rather than the population. Depths are recomputed alongside it because once the surviving allele set changes, the counts a frequency is derived from have to change with it. @@ -888,7 +878,7 @@ Standard, optional, and unchanged in approach. It runs only when you ask for it, ### The analysis layer -The modules that read these tables — what each one estimates, what it assumes and what it cannot tell you — are documented with the modules themselves in [Modules](#modules). This section will grow as they do. +The modules that read these tables are documented with the modules themselves in [Modules](#modules): what each one estimates, what it assumes and what it cannot tell you. This section will grow as they do. ## Design Decisions @@ -897,7 +887,7 @@ PoolSeqFlow departs from stock Nextflow practice in several places. Each departu --- -### Configuration is a file, never a flag +### Configuration without flags **The decision.** Every setting lives in `parameters.config`. The wrapper takes a subcommand, not settings. There is no `--poolSize 50`. @@ -923,30 +913,42 @@ it is a real boolean and behaves as expected. This class of failure is silent an --- -### Resume is filesystem-based +### Filesystem-based resume **The decision.** Every step checks whether its own outputs already exist, and skips itself if they do. It looks in permanent storage first and then on the working volume, so an output that has been produced but not yet moved to `storageDir` counts as done. This replaces Nextflow's `-resume` entirely; the wrapper never passes that flag. `PoolSeqFlow run` is both "start" and "resume". **Why.** Nextflow's cache lives in `work/`. It is invalidated by anything that removes or changes those directories, which for this pipeline is routine: - `cleanup = true` in `nextflow.config` deletes task working directories once a run completes. `-resume` replays task outputs *from* those directories, so after a successful run there is nothing left to replay. -- Several steps delete their own inputs once the next stage has consumed them — the trimmed reads are removed after clipping, and each VCF is removed after the next filter produces its successor. That leaves the upstream task's recorded outputs dangling, which invalidates the cache entry regardless. +- Several steps delete their own inputs once the next stage has consumed them: the trimmed reads are removed after clipping, and each VCF is removed after the next filter produces its successor. That leaves the upstream task's recorded outputs dangling, which invalidates the cache entry regardless. - On HPC, jobs hit walltime, nodes reboot, and scratch is purged on a schedule. A cache that lives in scratch does not survive the failure modes that actually interrupt long runs. A check for "does this output file already exist" survives all of that, because it depends on nothing but the storage the results are already in. **What it costs.** Two things worth knowing. -Step-skipping happens *inside* each task rather than before it, so a re-run still submits every process to the scheduler. Those jobs exit almost immediately — they test for a file, create a symlink and copy two log files — but they are real submissions. Expect roughly one short job per process per sample on a fully resumed run. +Step-skipping happens *inside* each task rather than before it, so a re-run still submits every process to the scheduler. Those jobs exit almost immediately (they test for a file, create a symlink and copy two log files), but they are real submissions. Expect roughly one short job per process per sample on a fully resumed run. -More importantly, "the output exists" is not the same as "the output is correct for your current settings". A file produced under `poolSize = 50` looks identical to one produced under `poolSize = 100`. That gap is closed separately, by the guardrails [below](#the-run-refuses-to-mix-settings). +More importantly, "the output exists" is not the same as "the output is correct for your current settings". A file produced under `poolSize = 50` looks identical to one produced under `poolSize = 100`. That gap is closed separately, by the change guard [below](#the-change-guard). --- -### Symbolic links instead of copies +### Symbolic links and the two storage roots **The decision.** Each output is **moved** out of the task's working directory to where it belongs, and a **symbolic link** is left behind pointing at it. Nothing is copied, and no file the pipeline produces exists twice. +```mermaid +graph LR + task["task directory
under work/"] + main["mainDir
your inputs, and
everything in progress"] + store["storageDir
what you keep"] + + task -->|"moved, symlink left behind"| main + main -->|"moved once the last step
that needed it has finished,
symlink left behind"| store +``` + +An output therefore crosses between the two roots exactly once, and a reader that opens it at the old path follows a link to wherever it is now. + **Why.** Pool-seq intermediates are large. A run with a dozen pools moves through hundreds of gigabytes of BAMs and VCFs. Nextflow's default is to publish outputs by copying them out of `work/`, which means every large file exists twice for as long as `work/` survives. An output is not moved to its final home immediately, though, and that is the second half of the design. It first lands on `mainDir`, the working volume, and moves to `storageDir` only once the last step that needed it has finished. A BAM is read several times on its way to a frequency table, and reading it repeatedly across a network mount is the slowest thing a run does. @@ -958,21 +960,21 @@ The two directories therefore have distinct jobs, and **cannot be the same path* On a cluster this maps onto a node's local disk and the network volume behind it; on a laptop it is one directory that churns and one you back up. Either way an output crosses between them exactly once, at the point where it stops being working material and becomes a result. -**What it costs.** Symbolic links, which is why Windows is not supported — including WSL under some filesystem configurations. It also means a task's working directory is not self-contained: deleting either directory while a run is in flight breaks links that are already in use. And `mainDir` is not scratch — it holds your inputs and your configuration, so it has to survive between runs. +**What it costs.** Symbolic links, which is why Windows is not supported, including WSL under some filesystem configurations. It also means a task's working directory is not self-contained: deleting either directory while a run is in flight breaks links that are already in use. And `mainDir` is not scratch: it holds your inputs and your configuration, so it has to survive between runs. --- -### Threads are budgeted, not divided +### The thread budget **The decision.** One `threads` value sizes the whole run. Every tool's core count is derived from it through a fixed ladder, and each process reserves what it actually uses. -**Why.** The obvious alternative — divide the available cores evenly among concurrent tasks — assumes tools scale linearly with threads. They do not. Each tool here is quantized to the point where its published scaling flattens out, so extra cores go to another task instead of into diminishing returns. +**Why.** The obvious alternative, dividing the available cores evenly among concurrent tasks, assumes tools scale linearly with threads. They do not. Each tool here is quantized to the point where its published scaling flattens out, so extra cores go to another task instead of into diminishing returns. -The sharper reason is that a tool's advertised thread count is not always what it spawns. Trim Galore's `--cores N` runs **N+4** threads: N workers, two decompressors, a batcher and a writer. A process that declares `cpus 4` and then passes `--cores 4` is really using eight. Nextflow decides how many tasks to run concurrently by comparing `cpus` against available resources, so an under-declared task causes oversubscription — the machine ends up running twice the work it thinks it is. +The sharper reason is that a tool's advertised thread count is not always what it spawns. Trim Galore's `--cores N` runs **N+4** threads: N workers, two decompressors, a batcher and a writer. A process that declares `cpus 4` and then passes `--cores 4` is really using eight. Nextflow decides how many tasks to run concurrently by comparing `cpus` against available resources, so an under-declared task causes oversubscription. The machine ends up running twice the work it thinks it is. PoolSeqFlow reserves Trim Galore's full footprint and maps back to the worker count in the script, so the declaration and the reality agree. The full ladder is in [Resources](#resources). -**What it costs.** Honest accounting is slower than optimistic accounting on a small machine. At `threads = 8`, a single trimming task reserves all eight cores, so samples are trimmed one at a time. Earlier behavior ran three concurrently at twelve threads each on an eight-core box — faster in wall-clock, and a 4.5× oversubscription. A request larger than the machine now fails immediately rather than quietly degrading: +**What it costs.** Honest accounting is slower than optimistic accounting on a small machine. At `threads = 8`, a single trimming task reserves all eight cores, so samples are trimmed one at a time. Earlier behavior ran three concurrently at twelve threads each on an eight-core box, which was faster in wall-clock, and a 4.5× oversubscription. A request larger than the machine now fails immediately rather than quietly degrading: ```text Process requirement exceeds available CPUs -- req: 12; avail: 8 @@ -980,58 +982,58 @@ Process requirement exceeds available CPUs -- req: 12; avail: 8 --- -### The run refuses to mix settings +### The change guard **The decision.** The run stops when the pipeline version, the analysis parameters, the run table or `metadata.csv` have changed since the existing outputs were produced. -**Why.** This is the direct consequence of [filesystem-based resume](#resume-is-filesystem-based). Because a step skips itself when its output file exists, and the file carries no record of what produced it, changing `poolSize` and re-running would leave one `Frequencies/` folder holding tables computed under two different thresholds. Nothing downstream could detect that, and the mixture would be invisible in the output. +**Why.** This is the direct consequence of [filesystem-based resume](#filesystem-based-resume). Because a step skips itself when its output file exists, and the file carries no record of what produced it, changing `poolSize` and re-running would leave one `Frequencies/` folder holding tables computed under two different thresholds. Nothing downstream could detect that, and the mixture would be invisible in the output. So a record of what produced them is kept beside the results: | Record | Covers | |---|---| -| `.poolseqflow_version` | The release that produced these results. **A mismatch is a hard stop**, not a warning — see [A project belongs to one release](#a-project-belongs-to-one-release) | +| `.poolseqflow_version` | The release that produced these results. **A mismatch is a hard stop**, not a warning; see [A project belongs to one release](#a-project-belongs-to-one-release) | | `.poolseqflow_params` | The analysis-affecting parameters, mirrored to a readable `run_parameters.txt` beside it | | `.parameters.config` | Your configuration file, copied verbatim as you wrote it | | `.multirun.csv` | Your run table, copied verbatim, if you used one | -| `.poolseqflow_metadata` | The parts of `metadata.csv` that can change a result — the read-group tags, the pool sizes, and the row order that sets column order. Kept beside the results it describes | +| `.poolseqflow_metadata` | The parts of `metadata.csv` that can change a result: the read-group tags, the pool sizes, and the row order that sets column order. Kept beside the results it describes | -That last one is also why most of your own columns cost nothing to change. A column recording experimental design, or a result from somewhere else — how the samples were grouped, when they were collected, what was measured about them — is not in the record, so adding or editing one never invalidates results you already have. **That holds for now.** Once the analysis layer arrives, the columns it reads will start to matter, and that freedom will narrow. +That last one is also why most of your own columns cost nothing to change. A column recording experimental design, or a result from somewhere else (how the samples were grouped, when they were collected, what was measured about them), is not in the record, so adding or editing one never invalidates results you already have. **That holds for now.** Once the analysis layer arrives, the columns it reads will start to matter, and that freedom will narrow. Path and resource parameters are excluded, since they change where and how fast the work happens rather than what the answer is; so are the `software` entries that name the commands to run. Anything added in a later release counts as analysis-affecting until decided otherwise, which is the conservative direction to err in. -The two verbatim copies serve a different purpose from the rest. They are not what the comparison is made against — they are the citable record, the exact files you ran kept next to the numbers they produced. +The two verbatim copies serve a different purpose from the rest. They are not what the comparison is made against; they are the citable record, the exact files you ran kept next to the numbers they produced. -**What it costs.** You cannot change a threshold and re-run to see the difference in place. The report names the folders to delete, and deleting them is what clears it — that is deliberate, because the alternative is a folder of results you can no longer attribute to a setting. +**What it costs.** You cannot change a threshold and re-run to see the difference in place. The report names the folders to delete, and deleting them is what clears it. That is deliberate, because the alternative is a folder of results you can no longer attribute to a setting. --- -### Moves across filesystems are atomic +### Atomic moves **The decision.** All cross-filesystem moves stage inside a temporary directory of the mover's own and rename into place, via `bin/atomic_mv.sh`. -**Why.** A plain `mv` across a filesystem boundary is a copy followed by an unlink, not an atomic rename. A job killed mid-move — walltime, preemption, a node failure — leaves a **truncated file under its final name**. Combined with existence-based resume, that is the worst possible failure: the next run sees the file, concludes the step is done, and builds everything downstream on a partial BAM. +**Why.** A plain `mv` across a filesystem boundary is a copy followed by an unlink, not an atomic rename. A job killed mid-move (walltime, preemption, a node failure) leaves a **truncated file under its final name**. Combined with existence-based resume, that is the worst possible failure: the next run sees the file, concludes the step is done, and builds everything downstream on a partial BAM. Staging under a temporary name and renaming means an interrupted move leaves nothing any existence check looks for, and the step simply runs again. The staging directory belongs to the one move: two jobs racing for the same destination stage separately, so neither can be seen writing into the other's copy. --- -### Steps delete their own inputs +### Deleted intermediates -**The decision.** Once a stage's output is safely in permanent storage, several steps delete the input they consumed — trimmed reads after clipping, each VCF after the next filter produces its successor. +**The decision.** Once a stage's output is safely in permanent storage, several steps delete the input they consumed: trimmed reads after clipping, each VCF after the next filter produces its successor. **Why.** Peak disk usage on a Pool-seq run is dominated by intermediates that nobody needs once the next stage has run. Keeping every one of them would roughly multiply the storage requirement by the number of filter stages, for files that exist only to be consumed. **What it costs.** You cannot inspect an intermediate after the fact without re-running from an earlier point, and it is part of why Nextflow's own cache cannot be used. -Note that this applies to the permanent copies too, not just the scratch ones: the step deletes the file the symlink resolves to. After a complete run, `Output/VCF/` holds the raw call set, the fully filtered VCF, and the annotated VCF if you enabled it — the per-stage intermediates between them are gone. Exactly which files survive is listed in [Directory Layout](#what-survives-a-completed-run). +Note that this applies to the permanent copies too, not just the scratch ones: the step deletes the file the symlink resolves to. After a complete run, `Output/VCF/` holds the raw call set, the fully filtered VCF, and the annotated VCF if you enabled it. The per-stage intermediates between them are gone. Exactly which files survive is listed in [Directory Layout](#what-survives-a-completed-run). ## The Filter Chain -A variant that reaches a frequency table has come through nine stages spread across four steps — six that remove something and three that only rewrite or split. A **read** is subject to the first three; after that the chain works on sites and alleles. This page walks the whole chain in order: what each stage removes, which parameter controls it, and what you are trading when you move that parameter. +A variant that reaches a frequency table has come through nine stages spread across four steps: six that remove something and three that only rewrite or split. A **read** is subject to the first three; after that the chain works on sites and alleles. This page walks the whole chain in order: what each stage removes, which parameter controls it, and what you are trading when you move that parameter. -If you are trying to work out why a variant you expected is missing, read this page top to bottom — the answer is usually earlier in the chain than people look. +If you are trying to work out why a variant you expected is missing, read this page top to bottom; the answer is usually earlier in the chain than people look. ### The chain at a glance @@ -1041,13 +1043,13 @@ If you are trying to work out why a variant you expected is missing, read this p | 2 | Depth capping | Reads at a position | Reads above a ceiling measured from the sample's own coverage | `capBAM.maxDepth` or `param_capMaxDepth` | | 3 | Pileup filter | Reads at a position | Low mapping quality, low base quality; optionally caps depth again | `variantCall.baseQualMin`, `variantCall.varQualMin`, `variantCall.maxDepth` | | 4 | Variant calling | Sites | Non-variant sites | `variantCall.callOptions` | -| 5 | Major-allele normalization | Allele order | Nothing — it rewrites | — | +| 5 | Major-allele normalization | Allele order | Nothing: it rewrites | none | | 6 | False-positive filter | Alternate alleles | Alleles without cross-sample support | `poolSize` or `param_poolSize`, `ploidy`, `filterFalsePositives.sampleThreshold` | -| 7 | Depth & quality filter | Sites | Sites where any sample is under-covered, and low-QUAL sites | `vcffilter.minDP`, `vcffilter.minQUAL` | -| 8 | SNP/INDEL split | Sites | Splits into two files; nothing is lost | — | -| 9 | Frequency conversion | — | Nothing | — | +| 7 | Depth & quality filter | Sites | Sites where any sample is under-covered or has no reads at all, and low-QUAL sites | `vcffilter.minDP`, `vcffilter.minQUAL`, `vcffilter.dropZeroDepth` | +| 8 | SNP/INDEL split | Sites | Splits into two files; nothing is lost | none | +| 9 | Frequency conversion | none | Nothing | none | -Stages 1–4 happen in [steps 4, 5 and 6](#pipeline-steps); stages 5–9 are the five sub-steps of step 7. +Stages 1-4 happen in [steps 4, 5 and 6](#pipeline-steps); stages 5-9 are the five sub-steps of step 7. --- @@ -1065,7 +1067,7 @@ samtools view -F 0xF0C -f 0x2 -q 30 -b | `-f 0x2` | `cleanBAM.required` | **Requires** the read to be properly paired | | `-q 30` | `cleanBAM.mapq` | **Excludes** reads with mapping quality below 30 | -The MAPQ floor is easy to miss because it is not part of either flag word. At `30` it is a strict filter — it discards reads that map ambiguously, which in a repetitive genome can be a substantial fraction. That is usually the right call for Pool-seq, because an ambiguously placed read contributes a read count to the wrong position and frequencies are read counts. But if your coverage reports from step 5 show much less depth than you sequenced for, this is the first place to look. +The MAPQ floor is easy to miss because it is not part of either flag word. At `30` it is a strict filter: it discards reads that map ambiguously, which in a repetitive genome can be a substantial fraction. That is usually the right call for Pool-seq, because an ambiguously placed read contributes a read count to the wrong position and frequencies are read counts. But if your coverage reports from step 5 show much less depth than you sequenced for, this is the first place to look. Duplicate removal happens just upstream (`samtools markdup -r`) and matters more here than in individual sequencing: a PCR duplicate is a second vote from a molecule that should only vote once, and in a frequency estimate every vote counts directly. @@ -1075,33 +1077,33 @@ Full flag reference: [Alignment & Cleaning](#alignment-cleaning). Step 5 measures a per-position depth histogram for every sample and publishes it to `Reports/Depth/`. From that histogram it decides a ceiling for that sample, and step 6 applies the ceiling to the BAM before calling. Nothing is deleted: a position deeper than the ceiling is truncated to it, and the sample's ready BAM is untouched on disk. -**Why the ceiling is measured rather than set.** Pooled coverage is uneven by design, so there is no depth that is "too deep" in the abstract. What is worth truncating is a *second population* of positions at high depth — a collapsed repeat, or a PCR hill, where reads belonging to several places in the genome stack up on one. Those positions carry read counts that no single locus produced, and a frequency is a read count. A single hand-set number cannot find them: it cuts legitimate coverage in a deep sample and lets the pile-up through in a shallow one, and nothing in the output says which happened. +**Why the ceiling is measured rather than set.** Pooled coverage is uneven by design, so there is no depth that is "too deep" in the abstract. What is worth truncating is a *second population* of positions at high depth: a collapsed repeat, or a PCR hill, where reads belonging to several places in the genome stack up on one. Those positions carry read counts that no single locus produced, and a frequency is a read count. A single hand-set number cannot find them: it cuts legitimate coverage in a deep sample and lets the pile-up through in a shallow one, and nothing in the output says which happened. **What the detector looks for.** One rise and fall is a library. A rise, a fall to nothing, and a *second* rise is two populations, and the ceiling goes where coverage ran out: -![A collapsed repeat: coverage at 120x, a second population at 4000x, and the cap between them](assets/depth-hill-small.svg) +![A collapsed repeat: coverage at 120×, a second population at 4000×, and the cap between them](assets/depth-hill-small.svg) -Blue is the sample before capping, green after. Everything above the ceiling collapses onto it, which is the spike at 447. Both axes are logarithmic and both have to be — the second population here holds two thousandths of the covered genome, so on a linear count axis it is invisible. +Blue is the sample before capping, green after. Everything above the ceiling collapses onto it, which is the spike at 447. Both axes are logarithmic and both have to be: the second population here holds two thousandths of the covered genome, so on a linear count axis it is invisible. The same shape at a different scale is the same decision. Organelle contamination is a small number of positions two orders of magnitude above the library: -![Organelle contamination: 16 kb at 30000x above a 150x library, capped at 502](assets/depth-mito.svg) +![Organelle contamination: 16 kb at 30000× above a 150× library, capped at 502](assets/depth-mito.svg) -**A long tail is not a second population.** This is the distinction the whole stage turns on, and it is why nothing here uses a mean, a median or a maximum — the maximum is precisely the thing being cut. An overdispersed library reaching 3000× has one population, and is left alone: +**A long tail is not a second population.** This is the distinction the whole stage turns on, and it is why nothing here uses a mean, a median or a maximum. The maximum is precisely the thing being cut. An overdispersed library reaching 3000× has one population, and is left alone: -![An overdispersed library reaching 3000x, uncapped](assets/depth-heavy-tail.svg) +![An overdispersed library reaching 3000×, uncapped](assets/depth-heavy-tail.svg) -So is a library on a reference that most of the genome does not map to. Two thirds of the covered positions here sit at depth 1–5. That is not coverage and it is not an anomaly either, and the ceiling has to stay above the real lobe at 60× rather than below it: +So is a library on a reference that most of the genome does not map to. Two thirds of the covered positions here sit at depth 1-5. That is not coverage and it is not an anomaly either, and the ceiling has to stay above the real lobe at 60× rather than below it: -![A poor reference: a junk population at depth 1-5, a trough, then real coverage at 60x](assets/depth-bad-reference.svg) +![A poor reference: a junk population at depth 1-5, a trough, then real coverage at 60×](assets/depth-bad-reference.svg) -**The three outcomes, and the third is the one to read.** Under the default `capBAM.maxDepth = -1` a sample is capped where the detector finds a second population, and left **uncapped** where it does not — including when the histogram is one the detector cannot read. Every sample's decision is published as a sentence beside its histogram, whether it was capped or not, because "nothing was done" is the one outcome you cannot see in the output. +**The three outcomes, and the third is the one to read.** Under the default `capBAM.maxDepth = -1` a sample is capped where the detector finds a second population, and left **uncapped** where it does not, including when the histogram is one the detector cannot read. Every sample's decision is published as a sentence beside its histogram, whether it was capped or not, because "nothing was done" is the one outcome you cannot see in the output. The detector declines deliberately when the deep population is too large to call an artefact: -![A pile-up at 20000x outweighing the real coverage at 200x, left uncapped](assets/depth-hill-dominant.svg) +![A pile-up at 20000× outweighing the real coverage at 200×, left uncapped](assets/depth-hill-dominant.svg) -Four fifths of the covered genome is at 20000× and one fifth at 200×. Which of those is the artefact cannot be read off a depth histogram, and capping at 500 would truncate most of a real genome on a guess. PoolSeqFlow reports it and does nothing. If you know which population is real, `param_capMaxDepth` in `metadata.csv` sets that sample's ceiling by hand — see [Metadata](#kinds-of-metadata-column). +Four fifths of the covered genome is at 20000× and one fifth at 200×. Which of those is the artefact cannot be read off a depth histogram, and capping at 500 would truncate most of a real genome on a guess. PoolSeqFlow reports it and does nothing. If you know which population is real, `param_capMaxDepth` in `metadata.csv` sets that sample's ceiling by hand; see [Metadata](#kinds-of-metadata-column). **The knob.** @@ -1111,29 +1113,97 @@ Four fifths of the covered genome is at 20000× and one fifth at 200×. Which of | a positive number | Cap every sample at that depth | | `0` | Do not cap at all | -`param_capMaxDepth` overrides it for one sample and takes the same three values. The histogram is published on every run whichever you choose — it costs little and it is what tells you whether the sample needed capping. +`param_capMaxDepth` overrides it for one sample and takes the same three values. The histogram is published on every run whichever you choose: it costs little and it is what tells you whether the sample needed capping. ### 3. Pileup filter (step 6) ```bash -bcftools mpileup -B -C 50 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou +bcftools mpileup -B -C 100 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou ``` | Flag | Parameter | Effect | |---|---|---| | `-q 30` | `variantCall.varQualMin` | Minimum **mapping** quality for a read to be counted | | `-Q 30` | `variantCall.baseQualMin` | Minimum **base** quality for a base to be counted | -| `-C 50` | `variantCall.scaleMapQ` | Downgrades mapping quality for reads with excessive mismatches | +| `-C 100` | `variantCall.scaleMapQ` | Downgrades mapping quality for reads with excessive mismatches. [What this costs you ->](#scalemapq-measured) | | `-d 0` | `variantCall.maxDepth` | A second, flat cap on reads per file per position. `0` is **no limit** | | `-B` | fixed | Disables BAQ (base alignment quality) recalculation | | `-a AD,DP,SP,INFO/AD` | fixed | Emits the allelic-depth fields everything downstream depends on | **`variantCall.maxDepth` is not the same setting as `capBAM.maxDepth`, and the two zeros do not mean the same thing.** `capBAM.maxDepth = 0` means *do not cap*; `variantCall.maxDepth = 0` means *impose no ceiling*, because that is what `-d 0` means to `bcftools mpileup`. Both ship as a state where nothing is truncated flatly, and the measured per-sample ceiling from stage 2 does the work instead. -It ships as `0` because a flat number applied to every sample is what stage 2 exists to replace. Setting it to a positive value restores a backstop under the measured ceiling — one number for every sample of every run, applied after capping. If you set it, compare it against your step 5 coverage reports first: a cap that bites truncates the read counts frequencies are computed from, and nothing downstream flags it. See [maxDepth](#maxdepth). +It ships as `0` because a flat number applied to every sample is what stage 2 exists to replace. Setting it to a positive value restores a backstop under the measured ceiling (one number for every sample of every run, applied after capping). If you set it, compare it against your step 5 coverage reports first: a cap that bites truncates the read counts frequencies are computed from, and nothing downstream flags it. See [maxDepth](#maxdepth). `-B` is a deliberate choice for pooled data. BAQ downweights bases near indels to suppress false positives that arise from misalignment in a single diploid genome. In a pool, the same signal may be a genuine low-frequency indel, and BAQ's correction assumes a genotype model that does not apply. Disabling it keeps the raw evidence and leaves the decision to the cross-sample filter at stage 6. +#### What `-C` costs, measured { #scalemapq-measured } + +`-C` exists because a read carrying many mismatches may be in the wrong place, and a misplaced read puts its bases at the wrong position. In a pool that reasoning is only half right: excess mismatches are also what **diversity** looks like, and the reads carrying them are the ones a frequency estimate is made of. So the setting was measured rather than inherited. + +Three pools of *Drosophila melanogaster*, whole genome, called as a cohort exactly as step 6 does, with every option but `-C` as this release composes it. Each candidate value is compared against `-C 0`, and **every statistic is read from the `-C 0` call set**, so both groups are scored under identical settings. + +| `-C` | sites called | Ti/Tv of the sites it keeps | Ti/Tv of the sites it drops | frequency shift | +|---|---|---|---|---| +| 30 | 94,681 | **0.608** | **1.090** | +0.0612 | +| 40 | 738,393 | 1.190 | 1.036 | -0.0479 | +| 50 | 1,178,812 | 1.168 | 0.978 | -0.0353 | +| 70 | 1,568,464 | 1.134 | 0.890 | -0.0154 | +| **100** | **1,716,020** | 1.113 | **0.848** | **-0.0076** | +| 0 | 1,854,369 | 1.113 | n/a | 0 | + +**Ti/Tv is the test.** Transitions outnumber transversions in real polymorphism; alignment artefacts approach the 0.5 that random substitution would give, because eight of the twelve possible base changes are transversions. A well-aimed filter therefore drops a set with a *low* Ti/Tv and keeps one with a high one. The ratios here are low in absolute terms because this is the raw call set, before stages 5 to 7. + +**`-C 100` aims best.** The 138,349 sites it removes have Ti/Tv 0.848 and nearly four times the soft-clip bias of the sites it keeps: they are artefacts, and leaving `-C` off entirely would keep them. At 50 the discarded set is five times larger and diluted with real variants (Ti/Tv 0.978), and the 537,942 sites that 50 drops but 100 keeps score 1.015, which places them with the variants. + +**`-C 30` is inverted**, and it is a warning about reaching for a harsher setting. The set it keeps scores 0.608 while the set it discards scores 1.090: a value that severe only passes a site where nearly every read matches the reference, so what survives is dominated by sequencing error rather than by polymorphism. + +##### The bias it leaves behind + +At the sites a setting keeps, `-C` still shifts the frequencies, because it penalises reads carrying the alternate allele more often than reads carrying the reference. Under the previous default of 50, measured across 3.4 million sample-cells: + +| frequency | mean shift | relative | pushed down : up | +|---|---|---|---| +| 0.00-0.10 | +0.010 | - | 0.01:1 | +| 0.10-0.25 | -0.001 | -0.7% | 0.70:1 | +| 0.25-0.50 | -0.030 | -8.1% | 2.00:1 | +| **0.50-0.75** | **-0.081** | **-12.9%** | **4.05:1** | +| 0.75-1.00 | -0.047 | -5.3% | 5.88:1 | + +**Rare alleles are untouched**, so the detection limit that [the false-positive filter](#6-false-positive-filter-step-7) rests on is unaffected. The compression lands where a pool is most non-reference, and it **scales with the quantity being measured** rather than being a flat offset. At `-C 100` the same figure is -0.0076 against 50's -0.0353. + +##### `scaleMapQ` below `varQualMin` produces nothing at all { #scalemapq-and-varqualmin } + +**At or below 10, `-C` does nothing** - output is byte-identical to `-C 0`. A small number does not mean a gentle adjustment; it means no adjustment. + +**Above 10 and below `varQualMin`, it discards every read.** This is not a fixed range. `-C` caps each read's adjusted mapping quality near its own value, and `-q` (`variantCall.varQualMin`) then rejects anything below the minimum - so whenever `scaleMapQ` is under `varQualMin`, nothing survives the pileup. + +Measured on three pools over `2L:5000000-5050000`, calling exactly as step 6 does. Sites called, against 695 with the adjustment off: + +| `scaleMapQ` | `varQualMin` 10 | 20 | 30 (default) | 40 | +|---|---|---|---|---| +| off, 0, 5, 10 | 695 | 695 | 695 | 695 | +| 11 | 17 | **0** | **0** | **0** | +| 15 | 65 | **0** | **0** | **0** | +| 20 | 112 | 32 | **0** | **0** | +| 25 | 205 | 163 | **0** | **0** | +| 29 | 279 | 203 | **0** | **0** | +| 30 | 297 | 217 | 44 | **0** | +| 35 | 374 | 368 | 251 | **0** | +| 40 | 460 | 441 | 305 | 52 | +| 50 | 545 | 545 | 490 | 316 | +| 70 | 620 | 617 | 612 | 573 | +| 100 | 648 | 644 | 644 | 641 | + +**The first column is the proof.** At `varQualMin` 10 there is no dead band at all, because 11 already clears the minimum. The band is not a property of `-C`; it is the relationship between the two settings. + +Checked directly at six thresholds - `varQualMin` 12, 15, 20, 30, 40 and 50. In every case `scaleMapQ` one below returned **zero** sites and `scaleMapQ` equal to it returned sites. The boundary is exact. + +**Clearing the minimum is not the same as being safe.** At the default `varQualMin` 30, a `scaleMapQ` of 30 returns 44 sites out of 695 - **6%** - because only the best-placed reads clear `-q` once their quality has been capped. Recovery is gradual: 36% at 35, 44% at 40, 70% at 50, 88% at 70, 93% at 100. This is the same shape as the whole-genome measurement above, where `-C 30` kept 94,681 sites against `-C 100`'s 1,716,020. + +So if you are tempted by something "gentler than 100", the number runs the other way - **lower is harsher** - and the region just above `varQualMin` is where it is most severe, not least. + +**Nothing silently succeeds any more.** A run that calls no variants is [refused at step 6](#empty-result-refusals), and one whose filters remove every site is refused in step 7. `PoolSeqFlow check project` reports the `scaleMapQ` / `varQualMin` relationship before you start. + ### 4. Variant calling (step 6) ```bash @@ -1143,7 +1213,7 @@ bcftools call -m -A -v -Ov | Flag | Effect | Why it matters here | |---|---|---| | `-m` | Multiallelic caller | Handles sites with more than one alternate allele, which biallelic-only calling would collapse | -| `-A` | Keep **all** alternate alleles from the pileup | Without this, bcftools discards alternates it considers unlikely under a genotype model — exactly the low-frequency alleles a pool is meant to detect | +| `-A` | Keep **all** alternate alleles from the pileup | Without this, bcftools discards alternates it considers unlikely under a genotype model: exactly the low-frequency alleles a pool is meant to detect | | `-v` | Variant sites only | Invariant sites are dropped | `-A` is the flag that makes this a Pool-seq caller rather than a general one. The default behavior prunes alternate alleles that no plausible genotype supports, which is sound for an individual and wrong for a pool, where a true allele at frequency 0.01 supports no genotype at all. @@ -1152,17 +1222,17 @@ bcftools call -m -A -v -Ov `bin/MajorAlleleToRef.py` re-encodes the VCF so the **most-read allele is the reference**. It does not remove anything; it rewrites. -For each site it sorts the alleles by `INFO/AD` — the read count summed across every sample — and reorders `REF`, `ALT`, `INFO/AD`, `INFO/DP`, `FORMAT/AD` and `FORMAT/DP` to match. `DP` is recomputed as the sum of the reordered `AD`, so depth and allelic depth cannot disagree. +For each site it sorts the alleles by `INFO/AD` (the read count summed across every sample) and reorders `REF`, `ALT`, `INFO/AD`, `INFO/DP`, `FORMAT/AD` and `FORMAT/DP` to match. `DP` is recomputed as the sum of the reordered `AD`, so depth and allelic depth cannot disagree. **Why do it at all.** The reference genome is one individual's assembly. There is no reason its allele should be the common one in your population, and when it is not, every frequency in that row is reported against a rare baseline. Two studies on the same species then report mirror-image frequencies for the same site. Normalizing to the major allele makes rows comparable across samples, across runs and across projects. **Two consequences worth knowing.** -The ordering is **cohort-wide**, not per-sample. `REF` is the allele most read across all samples combined. In an individual sample where the cohort-minor allele is locally dominant, that sample's frequency for `REF` will be below 0.5 — which is meaningful, not an error. +The ordering is **cohort-wide**, not per-sample. `REF` is the allele most read across all samples combined. In an individual sample where the cohort-minor allele is locally dominant, that sample's frequency for `REF` will be below 0.5, which is meaningful, not an error. `FORMAT/GT` is set to `./.` on every genotype. A pool has no genotype, and leaving bcftools' diploid call in place would invite downstream tools to read it as one. Making it explicitly missing is the honest encoding. Keep it in mind if you point a genotype-based tool at these VCFs: it will find nothing, by design. -The script runs **twice** — once before the false-positive filter and again after it, because splitting and rejoining multiallelic records can change allele order. +The script runs **twice**: once before the false-positive filter and again after it, because splitting and rejoining multiallelic records can change allele order. ### 6. False-positive filter (step 7) @@ -1199,7 +1269,7 @@ This is a **cross-sample corroboration** filter, not a frequency cutoff. A site Earlier releases did the counting inside `bcftools view -i`, which can only apply **one** threshold across every sample. Per-pool sensitivity does not fit in that expression: a per-sample threshold would have to become a per-sample FORMAT field, annotated onto every record and stripped off again afterwards. -Doing the counting in `awk` instead buys a property worth more than the tidiness it costs. **Each threshold is bound to the sample's name, not to its column position.** A column that changes place — because you reordered `metadata.csv`, which you are free to do — still gets its own pool's threshold. Binding by position would have applied the wrong pool's threshold after a reorder and produced a perfectly ordinary-looking table. +Doing the counting in `awk` instead buys a property worth more than the tidiness it costs. **Each threshold is bound to the sample's name, not to its column position.** A column that changes place (because you reordered `metadata.csv`, which you are free to do) still gets its own pool's threshold. Binding by position would have applied the wrong pool's threshold after a reorder and produced a perfectly ordinary-looking table. It also fails loudly rather than guessing. If the VCF contains a sample column that your pool sizes do not mention, the filter stops rather than judging that column at some other pool's threshold. @@ -1211,11 +1281,11 @@ Each filtered VCF carries a header line per pool, stating the size used and the ##PoolSeqFlowPool= ``` -These are provenance, written beside the data they were applied to. Note that the filtered VCFs are intermediates and most do not survive a completed run — the durable record of your pool sizes is `.poolseqflow_metadata`, kept beside the results. +These are provenance, written beside the data they were applied to. Note that the filtered VCFs are intermediates and most do not survive a completed run; the durable record of your pool sizes is `.poolseqflow_metadata`, kept beside the results. #### Where S comes from -A pool of `poolSize` individuals, each carrying `ploidy` copies of the genome, contains $\text{ploidy} \times \text{poolSize}$ chromosomes, so a single chromosome carries a frequency of $1 / (\text{ploidy} \times \text{poolSize})$. The extra factor of two puts `S` at **half** that — a true singleton clears the threshold with margin rather than sitting exactly on it, which matters because the observed fraction of a singleton is itself noisy. +A pool of `poolSize` individuals, each carrying `ploidy` copies of the genome, contains $\text{ploidy} \times \text{poolSize}$ chromosomes, so a single chromosome carries a frequency of $1 / (\text{ploidy} \times \text{poolSize})$. The extra factor of two puts `S` at **half** that: a true singleton clears the threshold with margin rather than sitting exactly on it, which matters because the observed fraction of a singleton is itself noisy. | `poolSize` | `ploidy` | One chromosome is | `S` (threshold) | |---|---|---|---| @@ -1225,9 +1295,9 @@ A pool of `poolSize` individuals, each carrying `ploidy` copies of the genome, c | 100 | 2 | 0.0050 | 0.0025 | | 200 | 2 | 0.0025 | 0.00125 | -Larger pools give a smaller `S`, so the filter admits rarer alleles — appropriately, since a larger pool really can contain rarer ones. +Larger pools give a smaller `S`, so the filter admits rarer alleles, which is appropriate, since a larger pool really can contain rarer ones. -Each pool can be set to get its own row of that table. `param_poolSize` in `metadata.csv` can set the size per pool, and any pool without one falls back to the global `poolSize`. `ploidy` can be set per run rather than per pool, so a design mixing ploidies is split into runs — see [Sequences with a different ploidy](#sequences-with-a-different-ploidy). +Each pool can be set to get its own row of that table. `param_poolSize` in `metadata.csv` can set the size per pool, and any pool without one falls back to the global `poolSize`. `ploidy` can be set per run rather than per pool, so a design mixing ploidies is split into runs; see [Sequences with a different ploidy](#sequences-with-a-different-ploidy). #### Where M comes from @@ -1246,11 +1316,11 @@ Each pool can be set to get its own row of that table. `param_poolSize` in `meta !!! danger "This filter removes population-private alleles" - At the default, an allele found in only one pool out of eight is discarded no matter how frequent it is in that pool — one sample does not reach the two the threshold requires. If private variation is what you are studying, lower `sampleThreshold` before your first run, not after. See [sampleThreshold](#samplethreshold). + At the default, an allele found in only one pool out of eight is discarded no matter how frequent it is in that pool: one sample does not reach the two the threshold requires. If private variation is what you are studying, lower `sampleThreshold` before your first run, not after. See [sampleThreshold](#samplethreshold). #### Why the splitting and rejoining -`FORMAT/AD[:1]` indexes the **first** alternate allele. On a multiallelic record, a rare third allele would never be tested — it would simply ride along on whatever the second allele did. Splitting to one line per alternate (`norm -m -`) makes each allele stand on its own evidence; `norm -m+` puts the survivors back together. +`FORMAT/AD[:1]` indexes the **first** alternate allele. On a multiallelic record, a rare third allele would never be tested: it would simply ride along on whatever the second allele did. Splitting to one line per alternate (`norm -m -`) makes each allele stand on its own evidence; `norm -m+` puts the survivors back together. The `*` → `X` substitution around the rejoin masks the spanning-deletion allele, which `norm -m+` does not handle in this position. It is restored immediately afterwards. @@ -1264,18 +1334,21 @@ vcftools --vcf _dp.vcf --minQ 30 --recode --recode-INFO-all --out _d ``` `bcftools view -e "FMT/DP<20"` (`vcffilter.minDP`) -: Removes a site if **any** sample falls below the depth. A site survives only when every sample meets the floor, so **the weakest library sets the threshold for the whole cohort** — one under-sequenced pool removes sites for all of them. +: Removes a site if **any** sample falls below the depth. A site survives only when every sample meets the floor, so **the weakest library sets the threshold for the whole cohort**. One under-sequenced pool removes sites for all of them. + +`|| FMT/DP==0` (`vcffilter.dropZeroDepth`, on by default) +: Removes a site where any sample has no reads, whatever `minDP` says. Above `minDP` 1 it changes nothing; it exists for when you lower `minDP` to keep shallow sites. Turn it off and a pool with no reads at a site publishes [`NA` rather than a frequency](#unmeasured-cells), so what you are keeping is a table with holes in it rather than a table of zeros. `vcftools --minQ 30` (`vcffilter.minQUAL`) : Removes sites whose `QUAL` falls below 30. -The depth test has to be site-level rather than per-sample. Both vcftools and bcftools express a genotype-level verdict by rewriting `FORMAT/GT` and nothing else — `AD` and `DP` survive untouched — and stage 9 reads `AD`. Stage 5 has already set every `GT` to `./.` besides, so a genotype filter would have nothing left to mark. +The depth test has to be site-level rather than per-sample. Both vcftools and bcftools express a genotype-level verdict by rewriting `FORMAT/GT` and nothing else (`AD` and `DP` survive untouched), and stage 9 reads `AD`. Stage 5 has already set every `GT` to `./.` besides, so a genotype filter would have nothing left to mark. Alternative expressions, what each trades, and how to pick a value: [Depth and quality](#depth-and-quality). ### 8. SNP/INDEL split (step 7) -Two `vcftools` passes over the same input — `--remove-indels` and `--keep-only-indels` — produce a SNP VCF and an INDEL VCF. Nothing is discarded; every surviving site lands in exactly one of the two files, and each is converted to its own frequency table. +Two `vcftools` passes over the same input (`--remove-indels` and `--keep-only-indels`) produce a SNP VCF and an INDEL VCF. Nothing is discarded; every surviving site lands in exactly one of the two files, and each is converted to its own frequency table. ### 9. Frequency conversion (step 7) @@ -1290,17 +1363,17 @@ Work from the outside in. A variant lost at stage 1 cannot be recovered by loose | Symptom | Most likely stage | Parameter to examine | |---|---|---| | Far less depth than sequenced | 1 | `cleanBAM.mapq`, then duplicate rate in the step 5 reports | -| Depth plateaus at one number in one sample | 2 | That sample's `Reports/Depth/` report — a measured ceiling was applied | +| Depth plateaus at one number in one sample | 2 | That sample's `Reports/Depth/` report: a measured ceiling was applied | | Depth plateaus at the same number in every sample | 3 | `variantCall.maxDepth`, or `capBAM.maxDepth` set to a fixed depth | | A sample you expected to be capped was not | 2 | Its `Reports/Depth/` report says why; `param_capMaxDepth` overrules it | | Low-frequency alleles absent everywhere | 6 | `poolSize`, or `param_poolSize` for the pool in question, and `ploidy` | -| Low-frequency alleles absent from one pool only | 6 | That pool's `param_poolSize` — a size set too low raises its threshold alone | +| Low-frequency alleles absent from one pool only | 6 | That pool's `param_poolSize`: a size set too low raises its threshold alone | | Alleles present in one pool only, absent from output | 6 | `filterFalsePositives.sampleThreshold` | | Whole sites missing despite good depth | 7 | `vcffilter.minQUAL` | -| Almost every site gone after filtering | 7 | `vcffilter.minDP` — one under-covered sample removes sites for all of them | -| Multiallelic sites reduced to two alleles | 4 | `variantCall.callOptions` — confirm `-A` is still present | +| Almost every site gone after filtering | 7 | `vcffilter.minDP`: one under-covered sample removes sites for all of them | +| Multiallelic sites reduced to two alleles | 4 | `variantCall.callOptions`: confirm `-A` is still present | -Changing any of these invalidates existing outputs, and step 0 will stop the next run rather than mix results. That is covered in [Design Decisions](#the-run-refuses-to-mix-settings). +Changing any of these invalidates existing outputs, and step 0 will stop the next run rather than mix results. That is covered in [Design Decisions](#the-change-guard). ## Interpreting Results @@ -1334,9 +1407,9 @@ CHROM POS REF ALLELE TOTAL_AD … | `REF` | The reference allele **for the site**, repeated on every row of that site | | `ALLELE` | The allele this row reports on | | `TOTAL_AD` | Frequency of `ALLELE` across **all samples combined** | -| *sample columns* | Frequency of `ALLELE` in that sample | +| *sample columns* | Frequency of `ALLELE` in that sample, or [`NA`](#unmeasured-cells) where that sample has no reads at the site | -Sample columns appear in `metadata.csv` row order — see [Row order decides column order](#row-order-decides-column-order). +Sample columns appear in `metadata.csv` row order. See [Row order decides column order](#row-order-decides-column-order). #### One row per allele @@ -1368,6 +1441,23 @@ chr1 1000 A T 0.05 0 0.1 Every column within one site sums to 1. A zero means the allele was not observed in that sample, not that the site was missing there. +#### A cell that was never measured reads `NA` { #unmeasured-cells } + +A pool with no reads at all at a site has no frequency to report, and the table says `NA` on every allele of that site rather than a number: + +```text +CHROM POS REF ALLELE TOTAL_AD sample1 sample2 +chr1 1000 A A 0.8 0.8 NA +chr1 1000 A G 0.15 0.2 NA +chr1 1000 A T 0.05 0 NA +``` + +**`0` and `NA` are different answers and the difference matters.** `sample1`'s `0` on the `T` row says the pool was read 500 times and none of those reads carried `T`. `sample2`'s `NA` says the pool was not read here, so nothing is known about `T` either way. Averaging the second in as a zero drags a frequency toward the reference by however many pools happened to be thin. + +The depth table is where the distinction comes from and it is unambiguous there: the cell holds `0,0,0`. `NA` is what that becomes once you divide by a total of zero. + +**By default no such cell reaches a published table.** `vcffilter.minDP` removes a site where any sample falls below the depth floor, and `vcffilter.dropZeroDepth` removes one where any sample has no reads even if you set `minDP` to zero. You see `NA` only after turning both off, which is a deliberate choice to keep sites that some pools missed. R reads `NA` natively; in Python, `pandas.read_table` gives `NaN`; in awk, test the string. + #### The depth tables { #depth-tables } `_snp_depth.tsv` and `_indel_depth.tsv` are the input that conversion was applied to: the same sites, holding **read counts instead of frequencies**. They are what the worked example above calls the AD columns, exactly as they came out of the VCF. @@ -1379,7 +1469,7 @@ CHROM POS REF ALT TOTAL_AD sample1 sample2 chr1 1000 A G,T 800,150,50 400,100,0 400,50,50 ``` -Two consequences. A depth table has fewer rows than its frequency table — one per site rather than one per allele. And its fourth column is headed `ALT`, listing only the alternates, where the frequency table's is `ALLELE` and names one allele per row including `REF`. +Two consequences. A depth table has fewer rows than its frequency table: one per site rather than one per allele. And its fourth column is headed `ALT`, listing only the alternates, where the frequency table's is `ALLELE` and names one allele per row including `REF`. Use these when a frequency alone is not enough: a frequency of 0.5 from 400 reads and one from 2 reads are the same number and very different evidence. They are also what the analysis layer reads to compute each site's effective sample size. @@ -1389,9 +1479,9 @@ Use these when a frequency alone is not enough: a frequency of 0.5 from 400 read **"Most-read" is decided across the whole cohort.** A sample where the cohort-minor allele dominates locally will show a `REF` frequency below 0.5. That is a real signal, not a defect. -**A missing row means the variant did not survive the chain, not that it is absent.** Sites are removed at nine points between the FASTQ and the table. Before concluding a variant is absent from your population, check it against [The Filter Chain](#tuning-the-chain) — in particular the cross-sample requirement at stage 6, which discards alleles seen in too few pools regardless of how frequent they are in those pools. +**A missing row means the variant did not survive the chain, not that it is absent.** Sites are removed at nine points between the FASTQ and the table. Before concluding a variant is absent from your population, check it against [The Filter Chain](#tuning-the-chain), in particular the cross-sample requirement at stage 6, which discards alleles seen in too few pools regardless of how frequent they are in those pools. -**Frequencies are read proportions, not estimates with error bars.** The table reports the fraction of reads carrying each allele. Sampling error in that fraction depends on depth at the position and on the pool size, and the pipeline does not propagate it. If your analysis needs uncertainty, compute it from the depth — which is why the VCFs retain `AD` and `DP`. +**Frequencies are read proportions, not estimates with error bars.** The table reports the fraction of reads carrying each allele. Sampling error in that fraction depends on depth at the position and on the pool size, and the pipeline does not propagate it. If your analysis needs uncertainty, compute it from the depth. That is why the VCFs retain `AD` and `DP`. **There is no genotype to read.** `FORMAT/GT` is `./.` throughout, deliberately. Genotype-based tools pointed at these VCFs will find nothing. @@ -1401,10 +1491,10 @@ Use these when a frequency alone is not enough: a frequency of 0.5 from 400 read | File | What it is | |---|---| -| `.vcf` | Raw output of step 6 — every called site, no step 7 filtering, original reference encoding | +| `.vcf` | Raw output of step 6: every called site, no step 7 filtering, original reference encoding | | `_annotated.vcf` | SnpEff annotation of the **raw** call set, if `annotate = true` | -Those two are the whole of it. Every VCF step 7 produces — `_sort`, `_sort_fp`, `_sort_fp_dq`, and the split `_snp` and `_indel` files — is consumed by the next process and deleted, so none of them survives a finished run. See [Steps delete their own inputs](#steps-delete-their-own-inputs). +Those two are the whole of it. Every VCF step 7 produces (`_sort`, `_sort_fp`, `_sort_fp_dq`, and the split `_snp` and `_indel` files) is consumed by the next process and deleted, so none of them survives a finished run. See [Deleted intermediates](#deleted-intermediates). **The filtered call set is therefore not kept as a VCF.** What it becomes is the frequency tables. If you need the filtered sites in VCF form, the raw call set plus the positions in the tables is what you have to work from. @@ -1431,7 +1521,7 @@ Those two are the whole of it. Every VCF step 7 produces — `_sort`, `_sort_fp` | `PoolSeqFlow_pipeline_trace.txt` | Nextflow | Machine-readable task trace | | `PoolSeqFlow_pipeline_dag.html` | Nextflow | Workflow graph | -`run_parameters.txt` is **not** in here — it sits one level up, at the root of `Output/`, beside the results rather than among the reports about them. It is a readable record of the analysis parameters these outputs were built from. +`run_parameters.txt` is **not** in here. It sits one level up, at the root of `Output/`, beside the results rather than among the reports about them. It is a readable record of the analysis parameters these outputs were built from. The three worth reading on every run are the coverage report, the depth report and `run_parameters.txt`. The depth report is the one that is easy to skip and should not be: the capped BAM it describes is transient, so this file is the only record of which reads reached bcftools for that sample. @@ -1443,7 +1533,7 @@ After a run finishes, four checks catch most problems: 1. **Sample columns.** Does the table have the number of columns you expect, in the order you laid out in `metadata.csv`? A count lower than expected means rows were merged by a shared `RG_Sample` ([why](#rg_sample-decides-what-counts-as-a-sample)). 2. **What was capped.** `grep -H 'ceiling applied' Output/Reports/Depth/*_depth_report.txt`. A ceiling far below the sample's typical depth, or a sample left uncapped when you expected otherwise, are both worth a look at its histogram before you trust the frequencies. -3. **Row counts.** Compare the site count in `.vcf` with the distinct positions that reached the tables — `tail -n +2 _snp_freq.tsv | cut -f1,2 | sort -u | wc -l`, and the same for the indel table. A very large drop points at the cross-sample filter; check `sampleThreshold` against your sample count in [the table above](#where-m-comes-from). +3. **Row counts.** Compare the site count in `.vcf` with the distinct positions that reached the tables: `tail -n +2 _snp_freq.tsv | cut -f1,2 | sort -u | wc -l`, and the same for the indel table. A very large drop points at the cross-sample filter; check `sampleThreshold` against your sample count in [the table above](#where-m-comes-from). 4. **Column sums.** Frequencies within a site should sum to 1 in every column. # Pipeline Overview @@ -1451,33 +1541,25 @@ After a run finishes, four checks catch most problems: PoolSeqFlow is a set of Nextflow DSL2 modules, each an independent file under `scripts/` and each responsible for its own resume logic. Steps 0 to 8 are the analysis; a tenth module handles promotion, moving each finished artifact to permanent storage once nothing needs it any more. This page is the map; the detail is in [Steps](#pipeline-steps). -```text -Raw FASTQ reads - │ - ▼ -[Step 0] Verify environment, parameters and folder structure - │ - ▼ -[Step 1] Build reference dictionaries (BWA, SAMtools, SnpEff) - │ - ▼ -[Step 2] QC & trimming (FastQC → Trim Galore → composition-aware clipping) - │ - ▼ -[Step 3] Alignment (BWA-MEM) - │ - ▼ -[Step 4] BAM cleanup (name-sort → fixmate → coord-sort → markdup → addRG → filter → index) - │ - ▼ -[Step 5] Reports, and the depth ceiling for each sample - │ - ▼ -[Step 6] Cap each BAM, then variant calling (BCFtools mpileup + call) - │ - ├────────────────────────────────┐ - ▼ ▼ -[Step 7] VCF → frequency tables [Step 8] Annotation (optional) +```mermaid +graph TD + reads(["Raw FASTQ reads"]) + s0["Step 0
Verify environment, parameters
and folder structure"] + s1["Step 1
Build reference dictionaries
BWA, SAMtools, SnpEff"] + s2["Step 2
QC and trimming
FastQC → Trim Galore → composition-aware clipping"] + s3["Step 3
Alignment
BWA-MEM"] + s4["Step 4
BAM cleanup
name-sort → fixmate → coord-sort → markdup → addRG → filter → index"] + s5["Step 5
Reports, and the depth ceiling for each sample"] + s6["Step 6
Cap each BAM, then variant calling
BCFtools mpileup + call"] + s7["Step 7
VCF → frequency tables"] + s8["Step 8
Annotation"] + promote(["Promotion
each finished artifact to permanent storage"]) + + reads --> s0 --> s1 --> s2 --> s3 --> s4 --> s5 --> s6 + s6 --> s7 + s6 -.->|optional| s8 + s7 --> promote + s8 -.-> promote ``` ### What runs per sample and what runs once @@ -1506,19 +1588,19 @@ After variant calling the workflow forks, and the branches never rejoin: **Step 7** takes the raw VCF through major-allele normalization, the cross-sample false-positive filter, depth and quality filtering, a SNP/INDEL split, and conversion to frequency tables. This is the analytical path. -**Step 8** takes the *same raw VCF* — not step 7's output — splits multiallelic sites, and annotates with SnpEff. So the annotated VCF contains sites step 7 removed, in the original reference encoding rather than the major-allele one. Joining the two requires matching on `CHROM`/`POS` and expecting unmatched rows ([details](#the-vcf-files)). +**Step 8** takes the *same raw VCF* (not step 7's output), splits multiallelic sites, and annotates with SnpEff. So the annotated VCF contains sites step 7 removed, in the original reference encoding rather than the major-allele one. Joining the two requires matching on `CHROM`/`POS` and expecting unmatched rows ([details](#the-vcf-files)). ### Where the work happens Every step follows the same pattern: -1. Check whether its output already exists — in `storageDir`, or on the working volume waiting to be promoted. If so, symlink it and exit. +1. Check whether its output already exists, in `storageDir` or on the working volume waiting to be promoted. If so, symlink it and exit. 2. Otherwise do the work in `mainDir/work/`. 3. Move the result out atomically: to `mainDir/Utilized/` if a later step will read it, or straight to `storageDir` if nothing will. 4. Symlink it back into the task's working directory. 5. Copy `.command.log` and `.command.err` into `Logs/`. -Then, separately: when the last step that needed an artifact has succeeded, it is moved from `Utilized/` to `storageDir`. Each byte crosses between the two volumes exactly once, and something enters `Utilized/` precisely when it will be read again — the unpaired reads, the trimming reports and the FastQC output have no consumer at all, so they go straight to permanent storage and never appear there. +Then, separately: when the last step that needed an artifact has succeeded, it is moved from `Utilized/` to `storageDir`. Each byte crosses between the two volumes exactly once, and something enters `Utilized/` precisely when it will be read again. The unpaired reads, the trimming reports and the FastQC output have no consumer at all, so they go straight to permanent storage and never appear there. That pattern is what makes the pipeline resumable without Nextflow's cache, keeps files from being duplicated, and survives an interrupted move. The reasoning is in [Design Decisions](#design-decisions); the mechanics are in [Resume Logic](#resume-logic). @@ -1526,11 +1608,11 @@ That pattern is what makes the pipeline resumable without Nextflow's cache, keep `nextflow.config` sets `errorStrategy = 'finish'`: on a failure, running tasks are allowed to complete and no new ones start. Nothing is left half-written, and a re-run picks up from whatever genuinely finished. -`ClipReads` is the exception, with `errorStrategy 'retry'` and `maxRetries 3` — it is the one step whose failures are commonly transient. +`ClipReads` is the exception, with `errorStrategy 'retry'` and `maxRetries 3`. It is the one step whose failures are commonly transient. `cleanup = true` removes task working directories after a successful run, leaving only empty hash-prefix folders under `work/`. `PoolSeqFlow clean` clears those. -**These five are yours to change, from `parameters.config`** — `errorStrategy`, `maxRetries`, `maxErrors`, `cleanup` and `conda.enabled`. They are defaults the installation sets *before* it reads your config, so anything you write wins. Nextflow scopes go outside the `params { }` block: +**These five are yours to change, from `parameters.config`**: `errorStrategy`, `maxRetries`, `maxErrors`, `cleanup` and `conda.enabled`. They are defaults the installation sets *before* it reads your config, so anything you write wins. Nextflow scopes go outside the `params { }` block: ```groovy params { @@ -1544,7 +1626,7 @@ process { } ``` -Two settings in `nextflow.config` are **not** yours to change, and a value you write for either is ignored. `workDir` is `mainDir/work`, which is where `clean` and `reset` look for it — move `mainDir` if you need the work directory elsewhere. `env.PATH` is what puts `dir.bin` in front of every task, and it is how the process scripts find the helpers they call by bare name; both of those follow the installation, and a run that lost them would fail at its first `atomic_mv.sh`. +Two settings in `nextflow.config` are **not** yours to change, and a value you write for either is ignored. `workDir` is `mainDir/work`, which is where `clean` and `reset` look for it. Move `mainDir` if you need the work directory elsewhere. `env.PATH` is what puts `dir.bin` in front of every task, and it is how the process scripts find the helpers they call by bare name; both of those follow the installation, and a run that lost them would fail at its first `atomic_mv.sh`. ## Pipeline Steps @@ -1573,7 +1655,7 @@ The gate for everything else. Nine stages run in parallel, each writing its own | `CheckDirectories` | `DIRECTORY CHECK` | `mainDir` and `storageDir` are the same path, or either is the installation itself | | `CheckMultiRun` | `MULTI-RUN CHECK` | The run table is missing, unparseable, or names a column that is not a parameter | -`CheckMetadataFile` reports more than it refuses. It prints the pooling it worked out — which rows merged into which VCF column — and the pool sizes it will apply, before any compute is spent. A mistake there is visible in seconds rather than in a result months later. +`CheckMetadataFile` reports more than it refuses. It prints the pooling it worked out (which rows merged into which VCF column) and the pool sizes it will apply, before any compute is spent. A mistake there is visible in seconds rather than in a result months later. #### The consistency guards @@ -1581,13 +1663,13 @@ Three of those stages are not validating your input; they are checking that what | Record | Contents | |---|---| -| `.poolseqflow_version` | The release that produced these results. A mismatch is a **hard stop** — nothing else is compared, because the parameter set itself moves between releases | +| `.poolseqflow_version` | The release that produced these results. A mismatch is a **hard stop**: nothing else is compared, because the parameter set itself moves between releases | | `.poolseqflow_params` | The analysis-affecting parameters, mirrored to a readable `run_parameters.txt` | | `.parameters.config` | Your configuration file, copied verbatim | | `.multirun.csv` | Your run table, copied verbatim, if you used one | | `.poolseqflow_metadata` | The `RG_*` and `param_*` columns and the row order, kept beside the results they describe | -Path and resource parameters are excluded, along with the `software` entries — they change where and how fast work happens rather than what the answer is. +Path and resource parameters are excluded, along with the `software` entries; they change where and how fast work happens rather than what the answer is. When a guard trips, the run stops **before any work happens** and the report names what to delete. Deleting it is what clears the check. How much has to go depends on what changed: a reordered `metadata.csv` invalidates less than an edited tag value, and a changed pool size less again ([table](#editing-metadatacsv-after-a-run)). @@ -1597,7 +1679,7 @@ When a guard trips, the run stops **before any work happens** and the report nam `scripts/1_build_dictionaries.nf` -Puts an uncompressed reference into `Reference/Dictionaries/` — decompressing it if it was gzipped, copying it if it was not — and builds three index sets there, in parallel: +Puts an uncompressed reference into `Reference/Dictionaries/` (decompressing it if it was gzipped, copying it if it was not) and builds three index sets there, in parallel: | Sub-step | Produces | |---|---| @@ -1607,7 +1689,7 @@ Puts an uncompressed reference into `Reference/Dictionaries/` — decompressing All of it lives under `mainDir`, beside the reference it was built from, and none of it is copied to `storageDir`. Dictionaries are working material: they are derived from your reference and can be rebuilt from it, so they are not a result to keep. Under a run table, runs sharing a reference build them **once**. -The SnpEff database name is derived from the GFF filename — `reference.gff.gz` becomes database `reference.gff`. The build copies the reference and GFF into a SnpEff `data/` layout, generates a minimal config, and verifies that `.bin` files were produced before declaring success. Completion is marked by `.build_complete`, which is what the resume check looks for. +The SnpEff database name is derived from the GFF filename: `reference.gff.gz` becomes database `reference.gff`. The build copies the reference and GFF into a SnpEff `data/` layout, generates a minimal config, and verifies that `.bin` files were produced before declaring success. Completion is marked by `.build_complete`, which is what the resume check looks for. Build options are `-gff3 -noCheckCds -noCheckProtein -v`. The two `-noCheck` flags suppress SnpEff's protein and CDS consistency checks, which fail on many non-model GFFs for reasons that do not affect variant annotation. @@ -1641,7 +1723,7 @@ bwa mem -K 10000000 -T 30 -t reference R1_clipped R2_clipped \ | Option | Parameter | Effect | |---|---|---| | `-T 30` | `bwa.minScoreOutput` | Minimum alignment score to report | -| `-K 10000000` | `bwa.batchSize` | Fixed bases per batch — makes output **deterministic** regardless of thread count | +| `-K 10000000` | `bwa.batchSize` | Fixed bases per batch: makes output **deterministic** regardless of thread count | `-K` is worth knowing about. Without it, BWA processes a batch sized by thread count, so the same input aligned with different `threads` can produce slightly different output. Fixing the batch size makes a run reproducible across machines. @@ -1651,9 +1733,9 @@ bwa mem -K 10000000 -T 30 -t reference R1_clipped R2_clipped \ // options = "-K ${params.bwa.batchSize} -T ${params.bwa.minScoreOutput}" ``` -Uncomment it only when you want flags those two cannot express — a different `-A`/`-B` scoring, say. From then on the string is used exactly as you write it and the two values above stop being read, so **`-K` is gone unless you put it back yourself**, and the run quietly loses the determinism the section above is about. `-t` is the exception: it is not in the string at either end, so the thread count still comes from the [`cores` ladder](#resources) whatever you pin here. +Uncomment it only when you want flags those two cannot express, such as a different `-A`/`-B` scoring. From then on the string is used exactly as you write it and the two values above stop being read, so **`-K` is gone unless you put it back yourself**, and the run quietly loses the determinism the section above is about. `-t` is the exception: it is not in the string at either end, so the thread count still comes from the [`cores` ladder](#resources) whatever you pin here. -Like the values it replaces, it is analysis-affecting — step 0 refuses a run whose string differs from the one recorded beside your existing alignments. +Like the values it replaces, it is analysis-affecting: step 0 refuses a run whose string differs from the one recorded beside your existing alignments. Output goes to `Output/Aligned/` as unsorted BAM. @@ -1683,7 +1765,7 @@ Three independent sub-steps per sample: | `CoverageReport` | `samtools coverage` | `Output/Reports/Coverage/_coverage_report.txt` | | `DepthProfile` | `samtools stats` | `Output/Reports/Depth/_depth_histogram.tsv` and `_depth_report.txt` | -`DepthProfile` also decides the depth ceiling step 6 applies to that sample, which is why this step is no longer a leaf: it passes the BAM, its index and the chosen ceiling on to variant calling. It runs whatever `capBAM.maxDepth` is set to — the histogram is worth having even when nothing is capped, and the reports are what [step 6](#depth-capping) is judged by. +`DepthProfile` also decides the depth ceiling step 6 applies to that sample, which is why this step is no longer a leaf: it passes the BAM, its index and the chosen ceiling on to variant calling. It runs whatever `capBAM.maxDepth` is set to. The histogram is worth having even when nothing is capped, and the reports are what [step 6](#depth-capping) is judged by. Reading the depth report on every run is the habit worth forming. The capped BAM step 6 builds is transient, so this file is the only record of which reads were actually called for that sample. @@ -1694,15 +1776,15 @@ Reading the depth report on every run is the habit worth forming. The capped BAM `scripts/6_variant_call.nf` ```bash -bcftools mpileup -B -C 50 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou -f reference \ +bcftools mpileup -B -C 100 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou -f reference \ | bcftools call -m -A -v -Ov -o .vcf ``` -Before that, `CapBAM` truncates each BAM to the ceiling step 5 chose for it. It runs **only for samples that have one**: a sample the detector left alone, or a run with `capBAM.maxDepth = 0`, goes to bcftools with its ready BAM unchanged, and no `CapBAM` task appears in the trace. The capped BAM never reaches either storage root — it is built here, read by the pileup, and discarded, which is why step 5's reports are the durable record. +Before that, `CapBAM` truncates each BAM to the ceiling step 5 chose for it. It runs **only for samples that have one**: a sample the detector left alone, or a run with `capBAM.maxDepth = 0`, goes to bcftools with its ready BAM unchanged, and no `CapBAM` task appears in the trace. The capped BAM never reaches either storage root: it is built here, read by the pileup, and discarded, which is why step 5's reports are the durable record. One joint task over every BAM, producing a multi-sample VCF with `AD` and `DP` FORMAT fields. Every flag and its consequence: [Variant Calling](#variant-calling). -**The BAMs are sorted before being handed to bcftools**, in `metadata.csv` row order. bcftools orders VCF columns by command-line order, so without this the column order would follow task-completion order — three consecutive runs on identical input gave three different orders. +**The BAMs are sorted before being handed to bcftools**, in `metadata.csv` row order. bcftools orders VCF columns by command-line order, so without this the column order would follow task-completion order. Three consecutive runs on identical input gave three different orders. A header fix is applied afterwards: `INFO/MQ` is declared `Integer` by bcftools but can carry a float, so the declaration is rewritten to `Float`. Without it, strict VCF parsers reject the file. @@ -1726,7 +1808,7 @@ Sub-step 2 is the pool-aware core of the pipeline and is documented in full in [ $$f_{\min} = \frac{1}{2 \times \text{ploidy} \times \text{poolSize}}$$ -but the test is not a plain cutoff in either direction. An allele must reach that frequency in a *fraction of samples*, which is what lets the threshold sit so low — and each sample is judged against **its own** pool's threshold, taken from `param_poolSize`, rather than against one number for the whole run. +but the test is not a plain cutoff in either direction. An allele must reach that frequency in a *fraction of samples*, which is what lets the threshold sit so low. And each sample is judged against **its own** pool's threshold, taken from `param_poolSize`, rather than against one number for the whole run. Output: `Output/Frequencies/_snp_freq.tsv` and `_indel_freq.tsv`, each beside a `_depth.tsv` holding the read counts it was computed from. Formats: [Interpreting Results](#the-frequency-tables) and [The depth tables](#depth-tables). @@ -1734,13 +1816,13 @@ Output: `Output/Frequencies/_snp_freq.tsv` and `_indel_freq.tsv`, ea ### Step 8: Annotate Variants {: #step-8-annotate-variants } -`scripts/8_annotate_variants.nf` — optional, controlled by `annotate`. +`scripts/8_annotate_variants.nf` (optional, controlled by `annotate`). ```bash bcftools norm -m - .vcf | snpEff -v -stats snpeff_summary.html ``` -Splits multiallelic sites onto separate lines — SnpEff annotates one alternate allele per record — and annotates against the database built in step 1. +Splits multiallelic sites onto separate lines (SnpEff annotates one alternate allele per record) and annotates against the database built in step 1. !!! warning "This runs on the raw call set" @@ -1748,9 +1830,9 @@ Splits multiallelic sites onto separate lines — SnpEff annotates one alternate Output: `Output/VCF/_annotated.vcf`, `Output/Reports/snpeff_summary.html` and `Output/Reports/snpeff_summary.genes.txt`. SnpEff names the gene table after the summary and writes the two together, so they arrive and are replaced as a pair. -Setting `annotate = false` skips this step and makes `gffFile` unnecessary — step 1 also stops building the SnpEff database. +Setting `annotate = false` skips this step and makes `gffFile` unnecessary. Step 1 also stops building the SnpEff database. -The parameters that control it — `annotate`, `gffFile` and the `snpEff` block — are in [Annotations](#annotations). +The parameters that control it (`annotate`, `gffFile` and the `snpEff` block) are in [Annotations](#annotations). --- @@ -1760,31 +1842,31 @@ The parameters that control it — `annotate`, `gffFile` and the `snpEff` block Not an analysis step, and it produces nothing of its own. It moves each finished artifact from the working volume to permanent storage once nothing needs it any more, so every byte crosses between the two exactly once. -An artifact enters `mainDir/Utilized/` precisely when a later step will read it again. Things with no consumer at all — the unpaired reads, the trimming reports, the FastQC output, both step 5 reports — go straight to `storageDir` and never appear there. +An artifact enters `mainDir/Utilized/` precisely when a later step will read it again. Things with no consumer at all (the unpaired reads, the trimming reports, the FastQC output, both step 5 reports) go straight to `storageDir` and never appear there. What moves it is a **completion signal from the last step that reads it**, not the artifact itself. Which step that is can depend on your configuration: `.vcf` is read by step 7 always and by step 8 only when `annotate = true`, with no ordering between them, so the gate has to wait for whichever set applies to this run. -If a run is interrupted, artifacts stay in `Utilized/` and the skip checks find them there — a resumed run counts them as done and picks up where it stopped, rather than repeating the work because the file is not in its final place yet. +If a run is interrupted, artifacts stay in `Utilized/` and the skip checks find them there. A resumed run counts them as done and picks up where it stopped, rather than repeating the work because the file is not in its final place yet. ## Resume Logic -PoolSeqFlow implements its own resume strategy rather than using Nextflow's. This page covers how it behaves; the reasoning behind replacing `-resume` is in [Design Decisions](#resume-is-filesystem-based). +PoolSeqFlow implements its own resume strategy rather than using Nextflow's. This page covers how it behaves; the reasoning behind replacing `-resume` is in [Design Decisions](#filesystem-based-resume). ### Two directories | | | |---|---| -| **`mainDir`** | Where the pipeline runs and where everything it works on lives — your reads, your reference, the dictionaries built from it, `work/`, and outputs still being read. Not scratch: it holds your inputs, so it has to survive between runs | +| **`mainDir`** | Where the pipeline runs and where everything it works on lives (your reads, your reference, the dictionaries built from it, `work/`, and outputs still being read). Not scratch: it holds your inputs, so it has to survive between runs | | **`storageDir`** | Where finished results are kept. Network storage, a group volume, a different mount | -**They cannot be the same path**, and the run stops if they are. The two are storage tiers with different jobs, and an output moving from one to the other is the event that marks it finished — which cannot mean anything if they are one place. +**They cannot be the same path**, and the run stops if they are. The two are storage tiers with different jobs, and an output moving from one to the other is the event that marks it finished, which cannot mean anything if they are one place. Each output is **moved** out of `work/` and a symlink left behind. Consequences: - **No duplication.** Nothing the pipeline produces exists twice on disk. - **One crossing.** An artifact moves between the two volumes exactly once, when the last step that needed it has finished. -- **Automatic step-skipping.** Every step looks for its own outputs — in `storageDir` first, then on the working volume — and skips itself if they are there, regardless of the state of `work/`. +- **Automatic step-skipping.** Every step looks for its own outputs (in `storageDir` first, then on the working volume) and skips itself if they are there, regardless of the state of `work/`. - **Resilience.** The check depends on nothing but the storage the results are already in, so it survives cluster timeouts, reboots and `work/` cleanups. The order of that search matters. Permanent storage is consulted first so that a stray copy left on the working volume can never outrank a promoted one. @@ -1796,7 +1878,7 @@ This strategy **replaces** Nextflow's `-resume`, and the wrapper never passes th - `cleanup = true` deletes task working directories once a run completes. `-resume` replays task outputs *from* those directories; after a successful run there is nothing to replay. - Several steps delete their own inputs once consumed. That leaves the upstream task's recorded outputs dangling, which invalidates the cache entry regardless. -So `PoolSeqFlow run` is both "start" and "resume". `PoolSeqFlow resume` survives as a deprecated alias and prints a notice. +So `PoolSeqFlow run` is both "start" and "resume". To start genuinely from scratch: @@ -1810,7 +1892,7 @@ Your reads, your reference and your two configuration files are not touched. ### What a resumed run looks like -Every process is still submitted. Step-skipping happens *inside* each task, not before it, so a fully resumed run submits roughly one job per process per sample. Those jobs test for a file, create a symlink, copy two log files and exit — but on a scheduler they are real submissions with real queue time. +Every process is still submitted. Step-skipping happens *inside* each task, not before it, so a fully resumed run submits roughly one job per process per sample. Those jobs test for a file, create a symlink, copy two log files and exit, but on a scheduler they are real submissions with real queue time. The log lines to look for are the `COMPLETED` messages that follow a "Found existing" line: @@ -1841,21 +1923,21 @@ That gap is closed by the consistency guards at the start of a run, which record | Command | Removes | |---|---| -| `PoolSeqFlow clean` | Nextflow work directories — the empty hash-prefix folders `cleanup = true` leaves behind — and any staging directory left by a killed transfer | +| `PoolSeqFlow clean` | Nextflow work directories (the empty hash-prefix folders `cleanup = true` leaves behind) and any staging directory left by a killed transfer | | `PoolSeqFlow dryclean` | The empty directory tree `dryrun` created as a preview | | `PoolSeqFlow reset` | All progress, across both directories, after listing it and asking you to type a confirmation | -`clean` is safe at any time and does not affect resume — nothing in `work/` is consulted by the skip logic. `reset` deletes results. +`clean` is safe at any time and does not affect resume: nothing in `work/` is consulted by the skip logic. `reset` deletes results. -Every file the pipeline and the analysis layer publish is written into a staging directory beside its destination, verified there, and only then renamed into place, so that a partly-written file is never visible under a final name. That staging directory is removed on every path the transfer can reach, including failure — but a process that is *killed* reaches none of them, and leaves one behind. `clean` collects them from both `mainDir` and `storageDir`, naming each one before it goes. They are named `.atomic_mv.*`, `.restore.*` and `.analysis_results.*`, and finding one is a sign a run was killed rather than allowed to stop. +Every file the pipeline and the analysis layer publish is written into a staging directory beside its destination, verified there, and only then renamed into place, so that a partly-written file is never visible under a final name. That staging directory is removed on every path the transfer can reach, including failure. But a process that is *killed* reaches none of them, and leaves one behind. `clean` collects them from both `mainDir` and `storageDir`, naming each one before it goes. They are named `.atomic_mv.*`, `.restore.*` and `.analysis_results.*`, and finding one is a sign a run was killed rather than allowed to stop. The preview itself is built in `dryRunDir`, which is `dryrun/` in the directory you launch from, beside `parameters.config`, unless you point it somewhere else. It is outside the `Output/` tree on both roots, so a preview is never mixed in among results. `dryclean` asks the same parameter where to look, and both commands check what they are about to delete first: a directory holding anything other than empty folders, a `README.txt` and a `members.txt` is not a preview, so they list what is in it and remove nothing. `dryrun` replaces a preview that passes that check without asking, because there is nothing in one to lose. -`dryRun` is the other half, and it is the pipeline's own flag rather than a setting for you to make: the `dryrun` command sets it, and it is what stops step 0 writing. Every check still runs and every comparison is still made — the report says *would record* where a real run says *recording*. That is what keeps a preview from leaving a baseline behind for results that are never produced, and it is why `dryRun` and `dryRunDir` are among the handful of parameters kept out of the recorded manifest: a preview must not be able to look like a run. +`dryRun` is the other half, and it is the pipeline's own flag rather than a setting for you to make: the `dryrun` command sets it, and it is what stops step 0 writing. Every check still runs and every comparison is still made, and the report says *would record* where a real run says *recording*. That is what keeps a preview from leaving a baseline behind for results that are never produced, and it is why `dryRun` and `dryRunDir` are among the handful of parameters kept out of the recorded manifest: a preview must not be able to look like a run. !!! danger "Do not delete either directory's contents while a run is in flight" - Task working directories contain symlinks into the volume the output was moved to. Removing the target breaks links that are actively in use, and the failure will not be obvious. That applies to the working volume as well as to permanent storage — an artifact waiting to be promoted is being read from where it is. + Task working directories contain symlinks into the volume the output was moved to. Removing the target breaks links that are actively in use, and the failure will not be obvious. That applies to the working volume as well as to permanent storage: an artifact waiting to be promoted is being read from where it is. ## Directory Layout @@ -1918,7 +2000,7 @@ There are three directories, and keeping them apart is most of understanding the │ ├── analysis.config.template │ ├── citations.json │ ├── references.bib -│ └── modules/ # THE MODULE STORE — created empty, and yours to fill +│ └── modules/ # THE MODULE STORE, created empty and yours to fill ├── manual/ # This manual ├── nextflow.config ├── parameters.config.template @@ -1930,9 +2012,9 @@ There are three directories, and keeping them apart is most of understanding the └── PoolSeqFlow # CLI wrapper, pipeline and analysis layer alike ``` -One copy serves any number of projects, and it is replaced wholesale when you upgrade — which is why nothing of yours belongs in it. `bin/` is prepended to `PATH` by `nextflow.config`, which is how the helper scripts are callable by bare name inside process scripts. +One copy serves any number of projects, and it is replaced wholesale when you upgrade, which is why nothing of yours belongs in it. `bin/` is prepended to `PATH` by `nextflow.config`, which is how the helper scripts are callable by bare name inside process scripts. -**`analysis/modules/` is the one part of the installation you add to.** No module arrives with a release, so a fresh copy has an empty store; `analysis modules install ` puts a module there along with the libraries it declares, which live together under `analysis/modules/lib/`. It is still part of the installation and not part of a project — replaced wholesale on upgrade, like everything else here, which is why [upgrading](#upgrading) means installing your modules again. +**`analysis/modules/` is the one part of the installation you add to.** No module arrives with a release, so a fresh copy has an empty store; `analysis modules install ` puts a module there along with the libraries it declares, which live together under `analysis/modules/lib/`. It is still part of the installation and not part of a project. It is replaced wholesale on upgrade, like everything else here, which is why [upgrading](#upgrading) means installing your modules again. ### What you provide, on `mainDir` @@ -1964,7 +2046,7 @@ Either reference file may be gzipped or plain. └── work/ # Nextflow's task directories ``` -`Utilized/` empties itself as the run proceeds — each artifact moves to `storageDir` once the last step that needed it has finished, so a completed run leaves it empty. Under a run table each run gets its own, `Utilized_`, because the runs share `mainDir` and would otherwise write to one path. +`Utilized/` empties itself as the run proceeds: each artifact moves to `storageDir` once the last step that needed it has finished, so a completed run leaves it empty. Under a run table each run gets its own, `Utilized_`, because the runs share `mainDir` and would otherwise write to one path. `work/` is emptied by `cleanup = true` after a successful run, and `PoolSeqFlow clean` removes what is left. The dictionaries stay: they are derived from your reference and rebuilding them costs time for no gain. @@ -2056,7 +2138,7 @@ Three things are worth reading off it: - **`Shared_1` needs `members.txt`**, because a group number says nothing about which runs are in it. Numbers are assigned in order of appearance in the table, so reordering rows can move them. - **`run_a1` and `run_a2` hold only `Frequencies/`.** Everything earlier was identical between them and lives in `Shared_1`; the filtered VCFs that differ are intermediates and do not survive. -`.poolseqflow_metadata` is the one record that does not sit at the root with the others. It describes the column order of a particular VCF, so it is kept beside that VCF — here, one in `Shared_1/` and one in `run_b/`. +`.poolseqflow_metadata` is the one record that does not sit at the root with the others. It describes the column order of a particular VCF, so it is kept beside that VCF. Here that means one in `Shared_1/` and one in `run_b/`. The Nextflow reports describe the whole invocation rather than any one run, so there is a single set under `All_Runs/Reports/`. @@ -2064,7 +2146,7 @@ A run that sets its own `storageDir` gets none of this. It has a results tree to ### What survives a completed run {: #what-survives-a-completed-run } -Several steps delete their inputs once the next stage has consumed them, and the deletion follows the symlink — the permanent copy goes too ([why](#steps-delete-their-own-inputs)). So the contents of `Output/VCF/` mid-run and after a completed run are not the same. +Several steps delete their inputs once the next stage has consumed them, and the deletion follows the symlink: the permanent copy goes too ([why](#deleted-intermediates)). So the contents of `Output/VCF/` mid-run and after a completed run are not the same. With `vcf.fileName = 'Test'`: @@ -2080,9 +2162,9 @@ With `vcf.fileName = 'Test'`: | `Test_snp_freq.tsv` | **Yes** | In `Frequencies/` | | `Test_indel_freq.tsv` | **Yes** | In `Frequencies/` | -So `Output/VCF/` holds exactly two files after a completed run: the raw call set, and the annotated one if you enabled annotation. **Every VCF step 7 produces is an intermediate**, including the fully filtered one — what that becomes is the frequency tables. +So `Output/VCF/` holds exactly two files after a completed run: the raw call set, and the annotated one if you enabled annotation. **Every VCF step 7 produces is an intermediate**, including the fully filtered one. What that becomes is the frequency tables. -Likewise, `Output/Trimmed//` keeps only the `_clipped.fq.gz` files after a completed run — the intermediate `_val_1`/`_val_2` files are removed once clipping has used them. +Likewise, `Output/Trimmed//` keeps only the `_clipped.fq.gz` files after a completed run. The intermediate `_val_1`/`_val_2` files are removed once clipping has used them. `Output/Aligned/` and `Output/Ready/` are both kept. Nothing deletes a BAM. @@ -2096,20 +2178,20 @@ Rough guidance for planning, per sample: | `Trimmed/` | Slightly under input, after clipping | Yes | | `Unpaired/` | Small | Yes | | `Aligned/` | Comparable to trimmed input | Yes | -| `Ready/` | Smaller — duplicates and filtered reads removed | Yes | +| `Ready/` | Smaller: duplicates and filtered reads removed | Yes | | `VCF/` | Depends on variant density, not read count | Partly | -| `Frequencies/` | Larger than the VCF it came from — one row per allele, not per site | Yes | +| `Frequencies/` | Larger than the VCF it came from: one row per allele, not per site | Yes | The two BAM directories dominate. If disk is tight, `Output/Aligned/` is the safe thing to remove after a completed run: step 4 has consumed it, and only a re-run from alignment would need it back. -**`mainDir` needs room for more than scratch.** It holds your reads and reference permanently, the dictionaries built from them, and — while the run is going — every output that a later step still has to read. At peak that is most of a run's intermediates at once. `work/` itself stays small, because it holds symlinks rather than copies. +**`mainDir` needs room for more than scratch.** It holds your reads and reference permanently, the dictionaries built from them, and (while the run is going) every output that a later step still has to read. At peak that is most of a run's intermediates at once. `work/` itself stays small, because it holds symlinks rather than copies. The peak on `mainDir` falls as the run proceeds, since each artifact leaves for `storageDir` as soon as the last step needing it finishes. A completed run leaves `Utilized/` empty. # Configuration -Everything the pipeline does is set in `parameters.config`. There are no command-line overrides ([why](#configuration-is-a-file-never-a-flag)). The optional analysis layer has settings of its own, in a file of its own — see [Analysis Layer](#analysis-configuration). +Everything the pipeline does is set in `parameters.config`. There are no command-line overrides ([why](#configuration-without-flags)). The optional analysis layer has settings of its own, in a file of its own. See [Analysis Layer](#analysis-configuration). This page sorts the parameters by what they actually affect, which is the distinction that matters most: some change your numbers, some change only where files land or how fast the run goes, and some are computed for you and should not be edited at all. @@ -2132,7 +2214,7 @@ Change one of these and your output changes. Step 0 records them and **refuses t | `cutadapt.at_gc_error` | Composition tolerance driving the clip points | [Trimming & Clipping](#trimming-clipping) | | `trim_galore.quality`, `.autodetect`, `.adapter1/2` | What is trimmed off the reads | [Trimming & Clipping](#trimming-clipping) | | `annotate`, `gffFile`, `snpEff.*` | Whether step 8 runs, against what, and with which SnpEff options | [Annotations](#annotations) | -| `metadata.csv` | Which FASTQ pairs are one pool, each pool's size and depth ceiling, and column order — the `RG_*` and `param_*` columns only | [Metadata](#metadata) | +| `metadata.csv` | Which FASTQ pairs are one pool, each pool's size and depth ceiling, and column order: the `RG_*` and `param_*` columns only | [Metadata](#metadata) | | The run table | Whatever it varies, per run. Every column in it is a parameter | [Multi-run](#multi-run) | #### Parameters that change speed, not answers @@ -2148,14 +2230,14 @@ Safe to tune between runs. Step 0 does not track them, precisely because they ca | `software.*` | Paths to executables, if not using the conda environment | [below](#using-system-tools) | | `capBAM.histogramMax` | How deep step 5's depth histogram looks | [Variant Calling](#capping-each-bam) | -`capBAM.histogramMax` is the odd one here: it changes nothing about speed. It is in this group because it bounds what step 5 *looks at* rather than what it decides — a run whose histogram would be truncated stops instead of choosing, so every value a run completes at gives the same ceiling. That is what makes it safe to raise on a project that already holds results. +`capBAM.histogramMax` is the odd one here: it changes nothing about speed. It is in this group because it bounds what step 5 *looks at* rather than what it decides. A run whose histogram would be truncated stops instead of choosing, so every value a run completes at gives the same ceiling. That is what makes it safe to raise on a project that already holds results. #### Parameters that change where files go | Parameter | Effect | |---|---| -| `mainDir` | Working directory — your inputs, `work/`, and everything in progress. Where you run from | -| `storageDir` | Permanent storage — the finished results. Must be a different path from `mainDir` | +| `mainDir` | Working directory: your inputs, `work/`, and everything in progress. Where you run from | +| `storageDir` | Permanent storage: the finished results. Must be a different path from `mainDir` | | `dataSource` | Subdirectory of `mainDir` holding the FASTQs | | `readPattern` | Glob matching paired FASTQs; needs a `{1,2}` group | | `referenceFile`, `gffFile` | Input filenames within `mainDir/Reference` | @@ -2172,32 +2254,65 @@ One parameter changes only what a published *report* contains, and nothing that |---|---|---| | `fastqc.options` | What FastQC reports on the clipped reads | [Trimming & Clipping](#trimming-clipping) | -Step 0 does not refuse a change to these, because your results do not depend on them. It *does* refuse two runs of a table that **share a step and disagree** about one — only one of the values can have produced the single report sitting in the shared directory, so the run stops rather than publishing an ambiguous file. +Step 0 does not refuse a change to these, because your results do not depend on them. It *does* refuse two runs of a table that **share a step and disagree** about one: only one of the values can have produced the single report sitting in the shared directory, so the run stops rather than publishing an ambiguous file. + +#### What is checked, and what it will not check { #parameter-checks } + +Some settings are wrong in a way that costs you a run and says nothing. `PoolSeqFlow check project` and step 0 both apply the same rules, from one implementation, so a project cannot be told one thing before a run and another during it. Silence is the healthy answer; a finding carries one of three levels: + +| Level | What happens | +|---|---| +| **FAIL** | The run stops at step 0. The setting would make it produce nothing at all | +| **WARN** | The run continues. The setting works and costs you far more than it looks like it does | +| **NOTE** | The run continues. The setting changes what a published number means, and you should know which | + +What is checked today: + +| Setting | Level | Why | +|---|---|---| +| `scaleMapQ` below `varQualMin` | FAIL | Every read is discarded and the call set is empty. [Measured at six thresholds](#scalemapq-and-varqualmin) | +| `filterFalsePositives.sampleThreshold` above 1 | FAIL | It is a fraction of your samples, so no site can satisfy it | +| `ploidy` or `poolSize` below 1 | FAIL | Every detection limit is computed from their product | +| `fastqc.memory` carrying a unit | FAIL | FastQC takes megabytes as a bare number and refuses `2G` | +| `scaleMapQ` at or below 10 | WARN | No adjustment is applied at all, whatever you wrote | +| `scaleMapQ` under twice `varQualMin` | WARN | It works, and at the boundary keeps 6% of the sites an unadjusted run called | +| `sampleThreshold` at or below 0 | WARN | The cross-sample requirement asks for nothing | +| `ploidy` times `poolSize` equal to 1 | WARN | The pipeline runs, but no diversity computed over that pool means anything | +| `variantCall.maxDepth` above 0 with `capBAM.maxDepth` at `-1` | NOTE | Two ceilings, and the smaller one decides | + +**A setting that merely gives you fewer sites is never flagged.** `minDP 20` against `minDP 5` is your scientific judgment and the pipeline has no opinion on it. The line is whether a setting produces *nothing*, or silently changes what a number *means*. + +**An older config is judged on what it will actually do.** A parameter your `parameters.config` does not mention falls back to the pipeline's own default rather than to whatever an unset value happens to mean: `nextflow.config` resolves it below the point your file is read, so a config that never mentions `vcffilter.dropZeroDepth` runs at the shipped `true`, and one that does set it still wins. Running `PoolSeqFlow migrate_config` is what brings a config forward, and it reports every parameter that is new; see [Bring each project's configuration forward](#migrate-config). + +**Two things it cannot check**, both because the answer is not in your configuration: + +- `cutadapt.min_length` above the read length clipping computes will discard every read and exit successfully. That length is measured per sample at run time, so no check of a config can see it. +- A pinned `variantCall.mpileupOptions` with no `-C` or `-q` in it is reported as unjudged rather than guessed at. Pinning the string is allowed, and it makes the two settings it was built from inert. #### Do not edit: derived values A large part of `parameters.config` is computed. The `cores` block derives every tool's thread count from `threads`; the `dir` block builds every path from `mainDir` and `storageDir`; `filterFalsePositives.sensitivity` is computed from `poolSize` and `ploidy`; `snpEff.db` is derived from `gffFile`. -Beyond those, eight named values are assembled from the filenames and roots you set. They appear in the config as ordinary assignments, so they can be overridden — but each has an input that is the thing you actually mean to change: +Beyond those, eight named values are assembled from the filenames and roots you set. They appear in the config as ordinary assignments, so they can be overridden, but each has an input that is the thing you actually mean to change: | Parameter | What it is | Set instead | |---|---|---| -| `referenceFa` | Your reference filename with `.gz` removed — the decompressed name step 1 works with | `referenceFile` | +| `referenceFa` | Your reference filename with `.gz` removed: the decompressed name step 1 works with | `referenceFile` | | `referencePath` | Full path to the reference, under `mainDir/Reference` | `referenceFile`, `dir.references` | | `gffPath` | Full path to the annotation file, alongside it | `gffFile` | | `metadataPath` | Full path to the sample table, in `mainDir` | `metadataFile` | | `multiRunPath` | Full path to the run table, in `mainDir` | `multiRunFile` | | `dir.allOutputs` | Results root for things belonging to the whole invocation rather than one run. `storageDir/Output`, or `storageDir/Output/All_Runs` when `multiRun` is on | `storageDir`, `multiRun` | | `dir.allLogs` | The same for logs: `storageDir/Logs`, or `storageDir/Logs/All_Runs` | `storageDir`, `multiRun` | -| `dir.sessionReports` | Where the four Nextflow session reports go — `PoolSeqFlow_pipeline_report.html`, `_timeline.html`, `_trace.txt` and `_dag.html`, named in `nextflow.config` | `storageDir`, `dir.subpath.reports` | +| `dir.sessionReports` | Where the four Nextflow session reports go: `PoolSeqFlow_pipeline_report.html`, `_timeline.html`, `_trace.txt` and `_dag.html`, named in `nextflow.config` | `storageDir`, `dir.subpath.reports` | The three `All_Runs` values are why a multi-run project does not scatter session-level output through the individual run directories: a report describing the whole invocation has one home, and it is chosen by `multiRun` rather than by each step guessing. -Editing these by hand breaks the invariant that makes the pipeline predictable — that one number sizes the run, and one pair of paths places everything. Change the input, not the derivation. +Editing these by hand breaks the invariant that makes the pipeline predictable: one number sizes the run, and one pair of paths places everything. Change the input, not the derivation. -Setting one is supported rather than forbidden, and how you do it depends on which: the `cores` block and the `options` strings ship commented out, so uncommenting a line is what pins it, while `filterFalsePositives.sensitivity`, `referenceFa` and `snpEff.db` are written out as formulas, so pinning one means replacing the formula with a value. Either way **a derived value you set by hand is used exactly as written, and nothing is derived from its inputs any more.** Pin `variantCall.mpileupOptions` and `variantCall.maxDepth` stops meaning anything for that run. That is a reasonable thing to want when you need full control of a command line — it is only a trap when it happens by accident. +Setting one is supported rather than forbidden, and how you do it depends on which: the `cores` block and the `options` strings ship commented out, so uncommenting a line is what pins it, while `filterFalsePositives.sensitivity`, `referenceFa` and `snpEff.db` are written out as formulas, so pinning one means replacing the formula with a value. Either way **a derived value you set by hand is used exactly as written, and nothing is derived from its inputs any more.** Pin `variantCall.mpileupOptions` and `variantCall.maxDepth` stops meaning anything for that run. That is a reasonable thing to want when you need full control of a command line; it is only a trap when it happens by accident. -**The paths are the exception, and a value written over one of them is replaced rather than used.** `referencePath`, `gffPath`, `metadataPath`, `multiRunPath` and the `dir` block are rebuilt for every run out of the roots and filenames that run holds — under a run table `dir.outputs` and `dir.logs` carry the run's own name, and a pin there would send two runs to one directory. Change the root or the filename instead, which is what the table above names for each. +**The paths are the exception, and a value written over one of them is replaced rather than used.** `referencePath`, `gffPath`, `metadataPath`, `multiRunPath` and the `dir` block are rebuilt for every run out of the roots and filenames that run holds. Under a run table `dir.outputs` and `dir.logs` carry the run's own name, and a pin there would send two runs to one directory. Change the root or the filename instead, which is what the table above names for each. Where the pipeline can tell, it says so. The verification step at the beginning of a run reports when your trimming options have been pinned rather than derived, and if you are using a run table it names any column that sets a computed value directly. @@ -2208,7 +2323,7 @@ The tables above sort parameters by *what they affect*. The other axis is *how w | Level | Where you write it | Applies to | |---|---|---| | **Global** | `parameters.config` | Every sample of every run | -| **Per run** | a column in the run table | Every sample of one run — [Multi-run](#multi-run) | +| **Per run** | a column in the run table | Every sample of one run ([Multi-run](#multi-run)) | | **Per sample** | a `param_*` column in `metadata.csv` | The rows that carry a value | **Any parameter can be set globally or per run.** The run table takes any parameter name as a column, spelled as `parameters.config` spells it. @@ -2221,7 +2336,7 @@ The tables above sort parameters by *what they affect*. The other axis is *how w | `param_capMaxDepth` | `capBAM.maxDepth` | step 5 | | `param_adapter1`, `param_adapter2` | `trim_galore.adapter1/2` | step 2 | -A `param_` column that is not on this list is refused rather than ignored — a name the pipeline cannot act on is a typo, not a preference, and silently recording it would leave you with a setting you can see in your own file and that never took effect. +A `param_` column that is not on this list is refused rather than ignored: a name the pipeline cannot act on is a typo, not a preference, and silently recording it would leave you with a setting you can see in your own file and that never took effect. #### What a per-sample value costs you in a run table @@ -2233,19 +2348,19 @@ A `param_` column that is not on this list is refused rather than ignored — a | `param_capMaxDepth` | trimming, alignment, BAM cleanup | reports, capping, calling, and everything after | | `param_adapter1/2` | nothing | the whole analysis | -Two runs can only differ in a `param_*` value by using **different metadata files** — `metadataFile` is itself a parameter, so it can be a column in the run table. Step 0 prints the resulting split before any compute is spent, so check there rather than inferring it: it names each results directory, which runs share it, and which steps it holds. +Two runs can only differ in a `param_*` value by using **different metadata files**. `metadataFile` is itself a parameter, so it can be a column in the run table. Step 0 prints the resulting split before any compute is spent, so check there rather than inferring it: it names each results directory, which runs share it, and which steps it holds. ### What to decide before your first run In rough order of how expensive it is to get wrong: -1. **`metadata.csv`** — which FASTQ pairs share an `RG_Sample`. Wrong here means valid results that answer a different question, and fixing it invalidates every BAM. [→](#rg_sample-decides-what-counts-as-a-sample) -2. **`poolSize` and `ploidy`** — these set the frequency floor. `poolSize` can be given per pool in `metadata.csv`, and `ploidy` applies to a whole run. [→](#poolsize-and-ploidy) -3. **`filterFalsePositives.sampleThreshold`** — decides whether alleles seen in few pools survive. The default removes them. [→](#samplethreshold) -4. **`capBAM.maxDepth`** — leave it at `-1` unless you know your libraries need otherwise. It is the one item here you can safely decide *after* the first run, because the depth reports tell you what it did. [→](#capping-each-bam) -5. **`threads`** — must fit the machine, or the run fails at submission. [→](#resources) +1. **`metadata.csv`**: which FASTQ pairs share an `RG_Sample`. Wrong here means valid results that answer a different question, and fixing it invalidates every BAM. [->](#rg_sample-decides-what-counts-as-a-sample) +2. **`poolSize` and `ploidy`**: these set the frequency floor. `poolSize` can be given per pool in `metadata.csv`, and `ploidy` applies to a whole run. [->](#poolsize-and-ploidy) +3. **`filterFalsePositives.sampleThreshold`**: decides whether alleles seen in few pools survive. The default removes them. [->](#samplethreshold) +4. **`capBAM.maxDepth`**: leave it at `-1` unless you know your libraries need otherwise. It is the one item here you can safely decide *after* the first run, because the depth reports tell you what it did. [->](#capping-each-bam) +5. **`threads`**: must fit the machine, or the run fails at submission. [->](#resources) -Changing any of items 1–4 after outputs exist means deleting those outputs. That is enforced, not advisory. +Changing any of items 1-4 after outputs exist means deleting those outputs. That is enforced, not advisory. ### Using system tools @@ -2261,7 +2376,7 @@ software { Replacing a command with an absolute path makes the pipeline use a system installation instead of the conda environment. This is supported but not recommended: the environment pins exact builds because Pool-seq results depend on the precise behavior of the pileup and filtering tools, and a version mismatch will not announce itself. Use it to work around a genuine packaging problem, not as a default. -**Three of them cannot be repointed.** `rsync`, `diff` and `find` are checked at the start of a run along with everything else, so a missing one is reported before any work begins — but a path you give for them is not used. They are called by name from the small scripts that move a finished file into permanent storage, and those scripts read none of your settings. If you need a different one, change what is on your `PATH`. +**Three of them cannot be repointed.** `rsync`, `diff` and `find` are checked at the start of a run along with everything else, so a missing one is reported before any work begins, but a path you give for them is not used. They are called by name from the small scripts that move a finished file into permanent storage, and those scripts read none of your settings. If you need a different one, change what is on your `PATH`. ## Metadata @@ -2282,11 +2397,11 @@ Sample2T2Rep1,Sample2T2,Lib1,ILLUMINA,Unit1,40,Pop2,T2,resistant,19.4,Run1 Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,Unit2,40,Pop2,T2,resistant,19.4,Run2 ``` -*Eight FASTQ pairs, four distinct `RG_Sample` values — this file produces a VCF with **four** columns: `Sample1T1`, `Sample1T2`, `Sample2T1`, `Sample2T2`. Each pool was sequenced twice, so its two rows share an `RG_Sample` and repeat everything that belongs to the pool.* +*Eight FASTQ pairs, four distinct `RG_Sample` values. This file produces a VCF with **four** columns: `Sample1T1`, `Sample1T2`, `Sample2T1`, `Sample2T2`. Each pool was sequenced twice, so its two rows share an `RG_Sample` and repeat everything that belongs to the pool.* -*Read the columns left to right and they group: what the sample **is**, what the **pipeline** should do with it, what was **set** (`exp_`), what was **measured as the response** (`pt_`), what was **measured alongside** (`cov_`), and last `sequencing_run` — which differs between the two rows of a pool, and so carries no prefix at all.* +*Read the columns left to right and they group: what the sample **is**, what the **pipeline** should do with it, what was **set** (`exp_`), what was **measured as the response** (`pt_`), what was **measured alongside** (`cov_`), and last `sequencing_run`, which differs between the two rows of a pool, and so carries no prefix at all.* -`PoolSeqFlow init` does not write this file, because its content is your experiment and a copied one would describe someone else's. It leaves `metadata.csv.example` beside you to write it from — the same table with every column explained in comments. Blank lines and lines beginning with `#` are ignored, so those comments can stay in the file you keep. A value containing a comma must be quoted: `"Pop1, coastal"`. +`PoolSeqFlow init` does not write this file, because its content is your experiment and a copied one would describe someone else's. It leaves `metadata.csv.example` beside you to write it from: the same table with every column explained in comments. Blank lines and lines beginning with `#` are ignored, so those comments can stay in the file you keep. A value containing a comma must be quoted: `"Pop1, coastal"`. ### The seven kinds of column { #kinds-of-metadata-column } @@ -2294,21 +2409,21 @@ The **name** of a column is what decides how it is treated. There is no second s | Column | What it is | |---|---| -| `SampleID` | **Required and unique.** Matched against the sample name `readPattern` takes from your FASTQ filenames, and becomes the read group's `ID` in the BAM | +| `SampleID` | **Required and unique.** The whole name before the read tag, as delivered: any `_S` or `_L00` your sequencer added is part of it. Matched against the sample name `readPattern` takes from your filenames, and becomes the read group's `ID` in the BAM | | `RG_*` | A read-group tag. Eight are known, listed below. A blank cell omits that tag rather than writing an empty one | | `param_*` | A setting from `parameters.config`, overridden for these samples only. Four are known: `param_poolSize`, `param_capMaxDepth`, `param_adapter1`, `param_adapter2`. A blank cell means "use the global value" | -| `exp_*` | An experimental variable — something you **set**. Read by [analysis modules](#the-experimental-design) and by no pipeline step. Any name you like after the prefix | -| `pt_*` | A phenotype — something you **measured** on the pool, and the thing being tested against. Also read only by analysis modules, and also open. See [The phenotype](#the-phenotype) | -| `cov_*` | A covariate — measured on the pool, but neither set nor the response. A cage temperature, an altitude, a collection site. See [Covariates](#covariates) | -| anything else | Yours. A lane, a batch, a note, a measurement from another experiment — whatever this study needs recorded | +| `exp_*` | An experimental variable: something you **set**. Read by [analysis modules](#the-experimental-design) and by no pipeline step. Any name you like after the prefix | +| `pt_*` | A phenotype: something you **measured** on the pool, and the thing being tested against. Also read only by analysis modules, and also open. See [The phenotype](#the-phenotype) | +| `cov_*` | A covariate: measured on the pool, but neither set nor the response. A cage temperature, an altitude, a collection site. See [Covariates](#covariates) | +| anything else | Yours. A lane, a batch, a note, a measurement from another experiment; whatever this study needs recorded | **`RG_` and `param_` are closed lists, and an unrecognized one is refused rather than ignored.** For `RG_` that stops a typo quietly losing a tag. For `param_` the reason is sharper: a `param_` column the pipeline did not recognize would be a setting you had written down, could see in your own file, and that was never applied to anything. -**`exp_`, `pt_` and `cov_` are three prefixes because only `exp_` says what the experiment set up.** The analysis layer works out which pools are independent of each other — and, where there is a time course, which are one thing measured repeatedly — from your `exp_` columns. A trait value differs from pool to pool, and so does a cage temperature: either one recorded as an `exp_` column would make every pool its own unit and split every series into single timepoints, **quietly**, because a design with no repeated measurements is a legal design. Keeping them apart is what stops that, structurally, rather than by remembering a setting. +**`exp_`, `pt_` and `cov_` are three prefixes because only `exp_` says what the experiment set up.** From your `exp_` columns, the analysis layer works out which pools are independent of each other and, where there is a time course, which are one thing measured repeatedly. A trait value differs from pool to pool, and so does a cage temperature: either one recorded as an `exp_` column would make every pool its own unit and split every series into single timepoints, **quietly**, because a design with no repeated measurements is a legal design. Keeping them apart is what stops that, structurally, rather than by remembering a setting. -**All three describe the POOL.** What differs between two rows of one pool — the lane, the run, the technician who handled one library — takes no prefix, and **not because it does not matter**. Once two libraries' reads are merged into one column, nothing downstream can attribute a read to the row it came from, so a row-level factor is not something the analysis layer has yet to support: it is unrecoverable. Record it unprefixed for the record. If every library of a pool *does* share one — one technician per pool — then it is pool-level after all, and it is a `cov_`. +**All three describe the POOL.** What differs between two rows of one pool (the lane, the run, the technician who handled one library) takes no prefix, and **not because it does not matter**. Once two libraries' reads are merged into one column, nothing downstream can attribute a read to the row it came from, so a row-level factor is not something the analysis layer has yet to support: it is unrecoverable. Record it unprefixed for the record. If every library of a pool *does* share one (one technician per pool), then it is pool-level after all, and it is a `cov_`. -Everything else is free. **You can add, remove and edit your own columns — `exp_` and `pt_` ones included — without invalidating results you already have.** The change guard does not look at them, so no analysis you have already published stops being valid because you named a variable better. +Everything else is free. **You can add, remove and edit your own columns (`exp_` and `pt_` ones included) without invalidating results you already have.** The change guard does not look at them, so no analysis you have already published stops being valid because you named a variable better. ### The experimental design { #the-experimental-design } @@ -2316,11 +2431,11 @@ An `exp_` column says what an experimental variable was for each pool: `exp_popu Two conventions and one rule: -- **`exp_time` is time**, and it is the one variable the analysis layer treats specially — see [The time axis](#the-time-axis). A project that has an `exp_time` column must say how to read it or every analysis refuses. +- **`exp_time` is time**, and it is the one variable the analysis layer treats specially. See [The time axis](#the-time-axis). A project that has an `exp_time` column must say how to read it or every analysis refuses. - **The prefix is open.** Unlike `RG_` and `param_` there is no list to check a name against, so `exp_tiempoint` is a new variable rather than an error. Nothing can catch that for you; the verification report prints the variables it found at the start of every analysis, which is where you will see it. -- **An `exp_` column describes the POOL, not the row.** Rows sharing an `RG_Sample` are one pool — their reads are merged and their depths added into one column of every published table — so they must give the same value for every `exp_` column, and **an analysis refuses a file where they disagree**, naming the column, the pool and each value. A blank cell means no value, which is a third answer rather than agreement with either. +- **An `exp_` column describes the POOL, not the row.** Rows sharing an `RG_Sample` are one pool (their reads are merged and their depths added into one column of every published table), so they must give the same value for every `exp_` column, and **an analysis refuses a file where they disagree**, naming the column, the pool and each value. A blank cell means no value, which is a third answer rather than agreement with either. -That last rule is the one that catches people, and the fix is always the same: what differs *between* two rows of one pool — the lane, the batch, the sequencing run, the extraction date — is not an experimental variable. Record it in a column with no prefix. +That last rule is the one that catches people, and the fix is always the same: what differs *between* two rows of one pool (the lane, the batch, the sequencing run, the extraction date) is not an experimental variable. Record it in a column with no prefix. The refusal is a hard one, and it stops **every** analysis rather than only the ones that read a design. Each published analysis records the design the project was in at the time; a project whose design contradicts itself has nothing true to record. @@ -2328,20 +2443,58 @@ The refusal is a hard one, and it stops **every** analysis rather than only the | Column | SAM tag | Meaning | |---|---|---| -| `RG_Sample` | `SM` | The pool. **Decides VCF columns** — see [below](#rg_sample-decides-what-counts-as-a-sample) | +| `RG_Sample` | `SM` | The pool. **Decides VCF columns**. See [below](#rg_sample-decides-what-counts-as-a-sample) | | `RG_Library` | `LB` | Library identifier | | `RG_Platform` | `PL` | Sequencing platform, e.g. `ILLUMINA` | -| `RG_PlatformUnit` | `PU` | Platform unit — flowcell, lane | +| `RG_PlatformUnit` | `PU` | Platform unit: flowcell, lane | | `RG_Description` | `DS` | Free text | | `RG_Center` | `CN` | Sequencing center | | `RG_Date` | `DT` | Run date, ISO 8601, e.g. `2024-03-07` | | `RG_FlowOrder` | `FO` | Flow order | -You never write the two-letter tags yourself; the prefix is what marks a column as one. `RG_Sample` is optional and defaults to `SampleID`, which makes every row its own pool — what you want when each sample was sequenced once. +You never write the two-letter tags yourself; the prefix is what marks a column as one. + +**Fill `RG_Sample` in on every row, even when it repeats `SampleID`.** The two columns answer different questions and it is worth being deliberate about both: + +| | answers | comes from | +|---|---|---| +| `SampleID` | *which pair of files is this row?* | your sequencer. Whatever it wrote, in full | +| `RG_Sample` | *what name do I want this under in my results?* | you | + +`RG_Sample` is what names the VCF column, every frequency table header, and everything an analysis keys on. + +The column is optional. However, if **left out, it is filled from `SampleID`.** Every such row becomes its own pool, named exactly what its files were named. + +Filling it in with the value it was already defaulting to changes nothing and invalidates nothing: the run records the rows as resolved, so a project that adds the column and repeats `SampleID` in it passes [the consistency guards](#the-consistency-guards) untouched. Only a change that really does move the pooling stops a run, which is the point of them. + +**Every `SampleID` must appear exactly once.** A row is looked up by it and only the first match would be read, so a repeat would quietly give a sample the wrong tags and produce a perfectly valid BAM that nothing downstream could flag. The run stops and lists the offending values, along with every other problem in the file, reported together with line numbers rather than one at a time. + +Line endings are not your problem, and neither is the byte-order mark Excel adds when it saves as "CSV UTF-8". Both are read correctly as they stand, and **nothing rewrites the file you wrote**. What is not read is a file that is not comma-separated: a spreadsheet saved in a locale where the comma is the decimal mark writes semicolons instead, and the whole header then arrives as a single column. When a file cannot be used, the message lists the column names it actually read, which is what tells that apart from a misspelled heading. + +### When your file names carry a sequencer suffix + +Illumina's demultiplexing adds a suffix you did not choose. `bcl2fastq` and BCL Convert append `_S`, where `n` is the sample's **row number in the sample sheet**, so a sample you call `A08781` is delivered as `A08781_S58_R1.fq.gz`. The sample name `readPattern` takes from that file is `A08781_S58`. -**Every `SampleID` must appear exactly once.** A row is looked up by it and only the first match would be read, so a repeat would quietly give a sample the wrong tags and produce a perfectly valid BAM that nothing downstream could flag. The run stops and lists the offending values — along with every other problem in the file, reported together with line numbers rather than one at a time. +That is what `SampleID` has to say, because `SampleID` is what joins the row to the files. Put the name you actually use in `RG_Sample`: -Line endings are not your problem. The file is parsed as CSV, so a file saved from Excel on Windows is read correctly as it stands, and **nothing rewrites the file you wrote**. +```csv +SampleID,RG_Sample +A08781_S58,A08781 +``` + +The suffix stops at the join. `RG_Sample` is what names the VCF column and what every result is keyed on, so `_S58` never reaches a number you publish. + +This is the same mechanism that makes lanes work. A sample split across two lanes arrives as `A08781_S58_L001` and `A08781_S58_L002`, which are two rows sharing one `RG_Sample` and merging into one pool. A rule that stripped the suffix from `SampleID` would give both rows the same identifier and lose one of the pairs. That is why nothing here strips anything, and why carrying the suffix into `SampleID` is yours to do. + +Both ways of getting this wrong are in Troubleshooting: [a sample with reads but no row](#reads-but-no-row) when `SampleID` is too short, and [a pool named after your files](#pool-named-after-files) when `RG_Sample` is blank. + +To save typing the identifiers, list them from the files themselves rather than from the sample sheet: + +```bash +find Data -name '*_R1.fq.gz' | sed 's|.*/||; s|_R1\.fq\.gz$||' | sort +``` + +That prints exactly the `SampleID` column for the default `readPattern`, in the form step 0 will check it against. ### `RG_Sample` decides what counts as a sample @@ -2352,20 +2505,20 @@ Line endings are not your problem. The file is parsed as CSV, so a file saved fr | `Sample1`, `Sample2`, `Sample3` | `Sample1` `Sample2` `Sample3` | | `Population1`, `Population1`, `Sample3` | `Population1` `Sample3` | -**Give every pool its own `RG_Sample`** when you want them analyzed separately. This is what most runs want, and it is the safe default — and it is what you get by leaving the column out, since it falls back to `SampleID`. +**Give every pool its own `RG_Sample`** when you want them analyzed separately. This is what most runs want, it is the safe default, and it is what you get by leaving the column out, since it falls back to `SampleID`. **Share an `RG_Sample` deliberately** when several FASTQ pairs are really the same biological pool: -- **One pool sequenced more than once** — split across lanes or runs to reach the depth Pool-seq needs. Each run arrives as its own FASTQ pair, but they describe one set of individuals, and the allele frequencies are only correct once the reads are combined. +- **One pool sequenced more than once**: split across lanes or runs to reach the depth Pool-seq needs. Each run arrives as its own FASTQ pair, but they describe one set of individuals, and the allele frequencies are only correct once the reads are combined. - **Technical replicates** of the same library that you want treated as one observation rather than compared with each other. -Because merging happens at variant calling, it changes the numbers: read depths add together and each frequency is computed across the pooled reads. Leaving one pool split across two `RG_Sample` values instead gives you **two under-powered estimates of the same thing** — which is easy to do by accident, since the FASTQ files look like two ordinary samples. +Because merging happens at variant calling, it changes the numbers: read depths add together and each frequency is computed across the pooled reads. Leaving one pool split across two `RG_Sample` values instead gives you **two under-powered estimates of the same thing**, which is easy to do by accident, since the FASTQ files look like two ordinary samples. The pooling that was worked out is printed before any compute is spent, so a mistake here is visible in seconds rather than in a result months later. !!! warning "This interacts with the cross-sample filter" - Merging changes the sample count, and the false-positive filter requires an allele to appear in a *fraction* of samples. Eight pairs as eight samples require two supporting samples; the same eight merged into four require one. Deciding `RG_Sample` is therefore also deciding how strict your filtering is — see [The Filter Chain](#where-m-comes-from). + Merging changes the sample count, and the false-positive filter requires an allele to appear in a *fraction* of samples. Eight pairs as eight samples require two supporting samples; the same eight merged into four require one. Deciding `RG_Sample` is therefore also deciding how strict your filtering is. See [The Filter Chain](#where-m-comes-from). ### Pool size belongs to the pool @@ -2373,7 +2526,7 @@ The pooling that was worked out is printed before any compute is spent, so a mis $$S = \frac{1}{2 \times \text{ploidy} \times \text{poolSize}}$$ -A pool of 10 and a pool of 500 have very different limits, so giving the whole run one number judges the small pool at the large one's resolution. The column exists so each pool can be judged at its own — see [Where S comes from](#where-s-comes-from). +A pool of 10 and a pool of 500 have very different limits, so giving the whole run one number judges the small pool at the large one's resolution. The column exists so each pool can be judged at its own. See [Where S comes from](#where-s-comes-from). Because the size describes the **pool** rather than the row, two rules follow: @@ -2382,13 +2535,13 @@ Because the size describes the **pool** rather than the row, two rules follow: Leave the column out entirely and every pool uses the global `poolSize` from `parameters.config`, which is the right answer when your pools really are all the same size. -The threshold each column was filtered at is recorded in the filtered VCF's header. Those VCFs are intermediates and do not survive a completed run — the durable record is `.poolseqflow_metadata`, kept beside the results. +The threshold each column was filtered at is recorded in the filtered VCF's header. Those VCFs are intermediates and do not survive a completed run. The durable record is `.poolseqflow_metadata`, kept beside the results. ### A depth ceiling belongs to the library `param_capMaxDepth` overrides `capBAM.maxDepth` for one row. Unlike pool size, it is a property of that **sequencing run** rather than of the pool, so rows sharing an `RG_Sample` are free to differ: two libraries of one pool can have been amplified quite differently, and each gets its own ceiling. -It takes the same three values as the global setting — `-1` to measure a ceiling from that sample's own coverage, a positive number to fix one, `0` to leave the sample uncapped — and a blank cell means the global `capBAM.maxDepth`. See [Depth capping](#depth-capping) for what the measurement does and when it declines to act. +It takes the same three values as the global setting (`-1` to measure a ceiling from that sample's own coverage, a positive number to fix one, `0` to leave the sample uncapped), and a blank cell means the global `capBAM.maxDepth`. See [Depth capping](#depth-capping) for what the measurement does and when it declines to act. Reach for this column when you have read a sample's depth report and disagree with it. The usual case is a library where the deep population is too large for the detector to call an artefact, so it reports the sample uncapped: if you know that population is a PCR hill rather than real coverage, pin the ceiling here. @@ -2398,7 +2551,7 @@ Reach for this column when you have read a sample's depth report and disagree wi ### Row order decides column order -**The order of the rows in `metadata.csv` is the order of the sample columns** in the VCF and in the frequency tables. Put the rows in whatever order you want to read your results in — treatment before control, or by time point — and the output follows. +**The order of the rows in `metadata.csv` is the order of the sample columns** in the VCF and in the frequency tables. Put the rows in whatever order you want to read your results in (treatment before control, or by time point), and the output follows. ```csv SampleID,RG_Sample,RG_Library,RG_Platform @@ -2411,11 +2564,11 @@ When several rows share an `RG_Sample`, the merged column appears where the **fi Reordering rows only moves columns; it never changes a value. Nothing else about the file is positional. -This exists because the alternative is worse. `collect()` alone emits BAMs in task-completion order, so whichever sample finished first landed first on the bcftools command line — and three consecutive runs on identical input gave three different column orders. Sorting on the file path is no better, since Nextflow's paths begin with a random work-directory hash. Row order is the only ordering that is both stable and meaningful. +This exists because the alternative is worse. `collect()` alone emits BAMs in task-completion order, so whichever sample finished first landed first on the bcftools command line; three consecutive runs on identical input gave three different column orders. Sorting on the file path is no better, since Nextflow's paths begin with a random work-directory hash. Row order is the only ordering that is both stable and meaningful. ### Editing `metadata.csv` after a run -Completed steps are skipped by looking for their output files, not by checking what produced them. So once this file has been consumed, editing it does **not** update anything that already exists — the tags are inside the BAMs, the column order is inside the VCF, and the pool sizes are inside the filter that produced the tables. +Completed steps are skipped by looking for their output files, not by checking what produced them. So once this file has been consumed, editing it does **not** update anything that already exists: the tags are inside the BAMs, the column order is inside the VCF, and the pool sizes are inside the filter that produced the tables. The analysis-affecting part of the file is recorded the first time it is used and compared on every later run. **If it has changed, the run stops before any work happens** and names what is now stale. How much that is depends on what you changed: @@ -2424,13 +2577,13 @@ The analysis-affecting part of the file is recorded the first time it is used an | A read-group tag value | The BAMs, and everything called from them | `Output/Ready/`, `Output/VCF/`, `Output/Frequencies/` | | Row order only | The VCF sample column order | `Output/VCF/`, `Output/Frequencies/` | | `param_poolSize` only | The false-positive filter's output, and nothing before it | `Output/Frequencies/` and the filtered VCFs | -| One of your own columns | Nothing | — | +| One of your own columns | Nothing | Nothing | -The pool-size row is the one worth noticing. A changed pool size moves a threshold inside step 7, and step 7's input is the called VCF — so the BAMs and the call set are still valid and **you are not asked to delete them.** Re-running redoes the filtering and the tables, which is minutes rather than hours. +The pool-size row is the one worth noticing. A changed pool size moves a threshold inside step 7, and step 7's input is the called VCF, so the BAMs and the call set are still valid and **you are not asked to delete them.** Re-running redoes the filtering and the tables, which is minutes rather than hours. Deleting what is named is what clears the check; the edit becomes the new baseline on the next run. Or discard everything and start over with `PoolSeqFlow reset`. -The record is `.poolseqflow_metadata`, kept beside the results it describes. It holds only the read-group tags, the pool sizes and the row order — never your own columns, which is why the last row of that table says nothing. Line endings and trailing whitespace are ignored when comparing; row order is not. +The record is `.poolseqflow_metadata`, kept beside the results it describes. It holds only the read-group tags, the pool sizes and the row order. Your own columns are never recorded, which is why the last row of that table says nothing. Line endings and trailing whitespace are ignored when comparing; row order is not. ### Checklist @@ -2441,7 +2594,7 @@ Before your first run: - [ ] `RG_Sample` shared only where pairs are genuinely the same pool - [ ] `param_poolSize` the same on every row of a pool, or absent everywhere - [ ] Rows in the order you want your result columns -- [ ] Any column of your own you will want later — adding them now costs nothing, and adding them afterwards costs nothing either +- [ ] Any column of your own you will want later (adding them now costs nothing, and adding them afterwards costs nothing either) ## Multi-run @@ -2453,7 +2606,7 @@ The reason to do it this way rather than copying the project is that the runs ha ### Turning it on ```bash -PoolSeqFlow init_multi +PoolSeqFlow init multi ``` in a new project, or in an existing one set it yourself: @@ -2463,7 +2616,7 @@ multiRun = true multiRunFile = 'runs.csv' ``` -`init_multi` also copies `multi-run.csv.example` into the project. It does not write the table: the runs, and the parameters that differ between them, are the whole content of that file, and only you know them. +`init multi` also copies `multi-run.csv.example` into the project. It does not write the table: the runs, and the parameters that differ between them, are the whole content of that file, and only you know them. ### The run table @@ -2471,11 +2624,11 @@ Four rules, and they are all of them. **One column must be `RunID`.** It names the run and becomes a directory, so keep it to letters, digits, dot, dash and underscore. -**Every other column is a parameter name, spelled exactly as `parameters.config` spells it, with no `params.` prefix.** Nested ones are dotted: `trim_galore.quality`, `variantCall.maxDepth`. Any parameter may be varied, including ones the pipeline normally computes for itself. A column that does not name a parameter is refused rather than ignored — that is a typo, not a preference. +**Every other column is a parameter name, spelled exactly as `parameters.config` spells it, with no `params.` prefix.** Nested ones are dotted: `trim_galore.quality`, `variantCall.maxDepth`. Any parameter may be varied, including ones the pipeline normally computes for itself. A column that does not name a parameter is refused rather than ignored. That is a typo, not a preference. **A blank cell means "take it from `parameters.config`".** That is what keeps a table as short as the difference between the runs. One consequence worth knowing: there is no way to set a parameter to an empty string here, because blank already means something else. -**A value containing a comma must be quoted** — `"-a AD,DP,SP,INFO/AD"`. The default `readPattern`, `*_R{1,2}.fq.gz`, is exactly such a value. +**A value containing a comma must be quoted**: `"-a AD,DP,SP,INFO/AD"`. The default `readPattern`, `*_R{1,2}.fq.gz`, is exactly such a value. Blank lines and lines starting with `#` are ignored, so a run can be kept in the file with a `#` in front of it rather than deleted. @@ -2489,11 +2642,11 @@ reference_b,reference_b.fasta.gz,reference_b.gff.gz,30 ### Setting a value the pipeline would compute -Some parameters are derived from others — `filterFalsePositives.sensitivity` from `poolSize` and `ploidy`, `variantCall.mpileupOptions` from the four `bcftools` values, the whole `cores` ladder from `threads`. Varying either end of that relationship is allowed, and the two behave differently. +Some parameters are derived from others: `filterFalsePositives.sensitivity` from `poolSize` and `ploidy`, `variantCall.mpileupOptions` from the four `bcftools` values, the whole `cores` ladder from `threads`. Varying either end of that relationship is allowed, and the two behave differently. **Set an input and the computed value follows it.** A run with `poolSize = 200` gets the sensitivity that a pool of 200 implies; one with `trim_galore.quality = 30` gets trimming options built around 30. You do not have to keep the derived values in step by hand. -**Set a computed value directly and it is used exactly as written** — and nothing is derived from its inputs any more. Pin `variantCall.mpileupOptions` and `variantCall.maxDepth` stops meaning anything for that run. That is a legitimate thing to want when you need a command line under your own control, and any column doing it is named in the report, so it is a choice rather than a surprise. +**Set a computed value directly and it is used exactly as written**; nothing is derived from its inputs any more. Pin `variantCall.mpileupOptions` and `variantCall.maxDepth` stops meaning anything for that run. That is a legitimate thing to want when you need a command line under your own control, and any column doing it is named in the report, so it is a choice rather than a surprise. ### Where the results go @@ -2516,7 +2669,25 @@ Group numbers are assigned in order of appearance in the table, so reordering ro ### What is shared, and what is not -Sharing is decided by comparing the parameters each step actually reads. Two runs share a step when every value that step depends on is the same for both — so runs differing only in `vcffilter.minDP` share everything up to and including variant calling, and diverge at the filtering. +Sharing is decided by comparing the parameters each step actually reads. Two runs share a step when every value that step depends on is the same for both, so runs differing only in `vcffilter.minDP` share everything up to and including variant calling, and diverge at the filtering. + +```mermaid +graph TD + reads(["Raw FASTQ reads"]) + trim["All_Runs/
QC and trimming"] + shared["Shared_1/
alignment, BAM cleanup, reports
and variant calling, on reference A"] + a1["run_a1/
frequency conversion
at its own minDP"] + a2["run_a2/
frequency conversion
at its own minDP"] + b["run_b/
everything past trimming,
on reference B"] + + reads --> trim + trim --> shared + trim --> b + shared --> a1 + shared --> a2 +``` + +*Three runs. `run_a1` and `run_a2` name the same reference and differ only in `vcffilter.minDP`, so alignment and calling happen once for both. `run_b` names a reference of its own and shares nothing past the trimming.* One thing switches it off entirely. **A run that sets its own `storageDir` shares nothing**, because sharing means writing one artifact into one results tree, and a run with its own tree has nowhere to put a shared one. It repeats every step alone. That is allowed and occasionally what you want; the report names any run in that position so it is not discovered from a task count. @@ -2536,7 +2707,7 @@ threads = 8 // cores a single task may use memory = '24 GB' // memory ceiling for a single task ``` -Every tool's thread count is derived from `threads`. **Do not set the per-tool counts by hand** — they live in the `cores` block, which exists to be computed, not edited. +Every tool's thread count is derived from `threads`. **Do not set the per-tool counts by hand**: they live in the `cores` block, which exists to be computed, not edited. ### The ladder @@ -2551,9 +2722,9 @@ Every tool's thread count is derived from `threads`. **Do not set the per-tool c Three details explain the shape of that table. -**Tools are quantized to where their scaling flattens.** BWA and cutadapt both take `cores.ladder` — the largest power of two at or below `threads`, capped at 8. Past that point the published scaling for these tools returns very little, so the cores are better spent on another task. FastQC is given `cores.fastqc`, which is two, because its `-t` counts *files* rather than threads per file and step 2 only ever hands it a pair. Measured on a pair of 2M-read files: `-t 2` is 1.93× faster than `-t 1`, and `-t 4`, `-t 6` and `-t 8` are no faster at all while each thread past the second costs roughly 250 MB of resident memory. Given eight files instead it scales to 4.8×, which is what the number is sized against. +**Tools are quantized to where their scaling flattens.** BWA and cutadapt both take `cores.ladder` (the largest power of two at or below `threads`, capped at 8). Past that point the published scaling for these tools returns very little, so the cores are better spent on another task. FastQC is given `cores.fastqc`, which is two, because its `-t` counts *files* rather than threads per file and step 2 only ever hands it a pair. Measured on a pair of 2M-read files: `-t 2` is 1.93× faster than `-t 1`, and `-t 4`, `-t 6` and `-t 8` are no faster at all while each thread past the second costs roughly 250 MB of resident memory. Given eight files instead it scales to 4.8×, which is what the number is sized against. -**Trim Galore's `--cores N` really runs N+4 threads** — N workers, two decompressors, a batcher and a writer. The ladder picks the largest N whose *full footprint* still fits in `threads`, which is why 4 cores yields `--cores 1` rather than `--cores 4`. The `--cores 1` case is the exception: it bypasses the worker pool entirely and is genuinely single-threaded. +**Trim Galore's `--cores N` really runs N+4 threads** (N workers, two decompressors, a batcher and a writer). The ladder picks the largest N whose *full footprint* still fits in `threads`, which is why 4 cores yields `--cores 1` rather than `--cores 4`. The `--cores 1` case is the exception: it bypasses the worker pool entirely and is genuinely single-threaded. **SAMtools' `-@` counts additional threads**, so `0` means one core and `1` means two. @@ -2584,7 +2755,7 @@ process Align { Inside `SortCleanBam`, every stage of the streamed pipeline is given `task.cpus - 1` except `samtools fixmate`, which gets the run's `threads - 1`. That is intentional: fixmate's algorithm scales further than the sort and markdup stages around it, so it is allowed more of the machine than the task reserves. -This is more than bookkeeping. **Nextflow decides how many tasks to run at once by comparing `cpus` against the resources available**, so an under-declared task leads to oversubscription — the machine runs more work than it thinks it is. Overriding `cpus` in a profile automatically changes what the tool is told, because both come from `task.cpus`. +This is more than bookkeeping. **Nextflow decides how many tasks to run at once by comparing `cpus` against the resources available**, so an under-declared task leads to oversubscription: the machine runs more work than it thinks it is. Overriding `cpus` in a profile automatically changes what the tool is told, because both come from `task.cpus`. `TrimReads` is the one place the number is not passed through unchanged. Its reservation is Trim Galore's *footprint*, so the script maps back to the worker count: @@ -2603,9 +2774,9 @@ Because tasks reserve what they really use, a request larger than the available Process requirement exceeds available CPUs -- req: 12; avail: 8 ``` -Set `threads` to the cores you actually have — on HPC, the size of one node. +Set `threads` to the cores you actually have (on HPC, the size of one node). -Note the consequence on a small machine. At `threads = 8`, a single `TrimReads` task reserves all eight, so samples are trimmed one at a time instead of three at once. That is slower in wall-clock than running three concurrently at twelve threads each on eight cores — and it is also the only version of that arrangement which respects the machine. See [Threads are budgeted, not divided](#threads-are-budgeted-not-divided). +Note the consequence on a small machine. At `threads = 8`, a single `TrimReads` task reserves all eight, so samples are trimmed one at a time instead of three at once. That is slower in wall-clock than running three concurrently at twelve threads each on eight cores, and it is also the only version of that arrangement which respects the machine. See [The thread budget](#the-thread-budget). ### `resourceLimits` is a ceiling, not an allocation @@ -2617,7 +2788,7 @@ process { } ``` -If a task requests more than this, Nextflow reduces the request before submitting it, which prevents a job that no node can satisfy from queueing forever. It does **not** reserve anything and does **not** limit concurrency on its own — that is what `cpus` does. Set `threads` and `memory` to match the node you are running on. +If a task requests more than this, Nextflow reduces the request before submitting it, which prevents a job that no node can satisfy from queueing forever. It does **not** reserve anything and does **not** limit concurrency on its own. That is what `cpus` does. Set `threads` and `memory` to match the node you are running on. ### Java {: #java } @@ -2633,7 +2804,7 @@ fastqc { } ``` -`fastqc.memory` must be a bare number — FastQC rejects `2G`. +`fastqc.memory` must be a bare number: FastQC rejects `2G`. `-XX:ParallelGCThreads` is **not** set here. It is applied per process from `task.cpus`, so the JVM always gets the cores that task actually reserved rather than a figure fixed in the config. @@ -2641,21 +2812,21 @@ fastqc { | Situation | `threads` | `memory` | |---|---|---| -| Laptop or workstation | Physical cores, minus one or two if you want the machine usable | Comfortably under total RAM — one task can use all of it | +| Laptop or workstation | Physical cores, minus one or two if you want the machine usable | Comfortably under total RAM (one task can use all of it) | | HPC node, exclusive | Cores on one node | Node memory | | HPC node, shared | Cores your allocation guarantees | Memory your allocation guarantees | | Debugging a failure | `1` | Generous | `threads = 1` forces every tool to a single core, which makes a failing run reproducible and its logs readable. It is slow, but it removes concurrency as a variable. -Neither value can change your results, which is exactly why the change guard does not track them. Tune them freely between runs — you will not be asked to delete anything. +Neither value can change your results, which is exactly why the change guard does not track them. Tune them freely between runs; you will not be asked to delete anything. ## Trimming & Clipping Step 2 runs in two stages: Trim Galore removes adapters and low-quality tails, then a second pass clips a fixed number of cycles from both ends based on what FastQC measured about base composition. The second stage is unusual and worth understanding before you change its one tunable. -### Stage 1 — Trim Galore +### Stage 1: Trim Galore ```groovy trim_galore { @@ -2671,10 +2842,10 @@ The assembled command is `--fastqc --paired --retain_unpaired -q 25`, plus adapt | Setting | Meaning | |---|---| | `quality` | Phred score below which bases are trimmed from the 3′ end | -| `autodetect` | `true` — no adapter is passed and Trim Galore detects it (Illumina, Nextera or smallRNA) | +| `autodetect` | `true`: no adapter is passed and Trim Galore detects it (Illumina, Nextera or smallRNA) | | `adapter1`, `adapter2` | Used only when `autodetect = false`, and then **both** are required | -`--retain_unpaired` keeps reads whose mate was discarded, in `Output/Unpaired/`. They are not used downstream — step 4 requires properly-paired alignments — but they are kept so you can see what was lost rather than having it disappear. +`--retain_unpaired` keeps reads whose mate was discarded, in `Output/Unpaired/`. They are not used downstream (step 4 requires properly-paired alignments), but they are kept so you can see what was lost rather than having it disappear. #### When to turn autodetect off @@ -2684,7 +2855,7 @@ Autodetection samples the first reads of a file and matches against known adapte - the trimming reports in `Output/Reports/Trimming/` disagree between samples of the same library, which means detection is not landing on a consistent answer; - you need the run to be exactly reproducible against a specific adapter regardless of what the first reads happen to contain. -Setting only one of `adapter1`/`adapter2` is not valid — the option string is built from both. +Setting only one of `adapter1`/`adapter2` is not valid: the option string is built from both. #### `adapterOptions`, if the three settings above are not enough @@ -2694,7 +2865,7 @@ Setting only one of `adapter1`/`adapter2` is not valid — the option string is // adapterOptions = autodetect ? '' : "-a ${params.trim_galore.adapter1} -a2 ${params.trim_galore.adapter2}" ``` -Uncomment it and what you write is passed to Trim Galore verbatim; `autodetect`, `adapter1` and `adapter2` then set nothing at all. That is the way to reach adapter handling the three settings cannot express — `--nextera` by name, a three-prime clip, more than one `-a`: +Uncomment it and what you write is passed to Trim Galore verbatim; `autodetect`, `adapter1` and `adapter2` then set nothing at all. That is the way to reach adapter handling the three settings cannot express (`--nextera` by name, a three-prime clip, more than one `-a`): ```groovy trim_galore { @@ -2702,7 +2873,7 @@ trim_galore { } ``` -**Per-sample adapters still work over a pinned `adapterOptions`.** A row setting both `param_adapter1` and `param_adapter2` replaces the adapter fragment for that sample, whatever you pinned here. That is not true of the *other* knob in the same block: pinning `trim_galore.options` replaces the whole command line, leaving nowhere for a per-sample adapter to go, and step 0 **fails the run** rather than ignoring the metadata — naming the samples and telling you to remove one or the other. +**Per-sample adapters still work over a pinned `adapterOptions`.** A row setting both `param_adapter1` and `param_adapter2` replaces the adapter fragment for that sample, whatever you pinned here. That is not true of the *other* knob in the same block: pinning `trim_galore.options` replaces the whole command line, leaving nowhere for a per-sample adapter to go, and step 0 **fails the run** rather than ignoring the metadata. It names the samples and tells you to remove one or the other. So of the two, `adapterOptions` is the one to reach for: it takes over adapter handling and leaves quality trimming, FastQC and the per-sample overrides working. @@ -2716,13 +2887,13 @@ Sample1T1Rep1,Sample1T1,, Sample1T1Rep2,Sample1T1,AGATCGGAAGAGC,AGATCGGAAGAGC ``` -A row that sets both replaces the run's adapter setting for that sample, whatever `autodetect` is set to — the adapters you name are used. A row that leaves both blank uses the run's settings, which is the ordinary case. +A row that sets both replaces the run's adapter setting for that sample, whatever `autodetect` is set to; the adapters you name are used. A row that leaves both blank uses the run's settings, which is the ordinary case. Give both or neither. One adapter on its own is a setting that cannot be acted on, so it is refused rather than half-applied. There is one combination the pipeline cannot honor, and it stops rather than guessing: if you have pinned `trim_galore.options` by hand, that string is used exactly as written and a per-sample adapter has nowhere to go. The run stops, names the samples that set adapter columns, and asks you to remove either the pin or the columns. -### Stage 2 — Composition-aware clipping {: #composition-aware-clipping } +### Stage 2: Composition-aware clipping {: #composition-aware-clipping } ```groovy cutadapt { @@ -2736,7 +2907,7 @@ Rather than clipping a fixed number of bases, the pipeline reads what FastQC mea #### What it measures -In an unbiased library, each cycle should show roughly equal A and T, and roughly equal G and C. Departures at the read ends are a well-known artefact — residual adapter, priming bias, and end-of-read quality decay all show up as a composition skew before they show up as a base-quality failure. +In an unbiased library, each cycle should show roughly equal A and T, and roughly equal G and C. Departures at the read ends are a well-known artefact: residual adapter, priming bias, and end-of-read quality decay all show up as a composition skew before they show up as a base-quality failure. For Pool-seq that matters more than usual. Allele frequencies are read counts, so a systematic bias in which base gets called at a given cycle propagates directly into the frequency estimate. Cycles where composition has not settled are cycles you cannot trust to count alleles. @@ -2748,7 +2919,7 @@ $$1 - \varepsilon \;\le\; \frac{A}{T} \;\le\; 1 + \varepsilon \qquad 1 - \varepsilon \;\le\; \frac{G}{C} \;\le\; 1 + \varepsilon$$ -with $\varepsilon$ = `at_gc_error`. At the default of `0.025` that is a ratio between 0.975 and 1.025. The first and last qualifying cycle give a usable range per read — `Min1`–`Max1` for R1 and `Min2`–`Max2` for R2 — and the two are combined: +with $\varepsilon$ = `at_gc_error`. At the default of `0.025` that is a ratio between 0.975 and 1.025. The first and last qualifying cycle give a usable range per read (`Min1` to `Max1` for R1 and `Min2` to `Max2` for R2), and the two are combined: ```text Clip5 = max(Min1, Min2) @@ -2765,9 +2936,9 @@ cutadapt -u Clip5 -U Clip5 -l readLengthLimit -o R1_clipped.fq.gz -p R2_clipped. #### Two deliberate asymmetries -**Both mates get the same treatment.** `Clip5` is the *larger* of the two 5′ bounds and the length limit applies to both files, so R1 and R2 come out the same length. The alternative — clipping each mate to its own measured range — would leave mates of different lengths for no downstream benefit. +**Both mates get the same treatment.** `Clip5` is the *larger* of the two 5′ bounds and the length limit applies to both files, so R1 and R2 come out the same length. The alternative, clipping each mate to its own measured range, would leave mates of different lengths for no downstream benefit. -**The length limit is the more permissive of the two.** Because `readLengthLimit` takes the `max`, a read whose own usable range ended earlier is kept to the longer mate's length, retaining a few cycles past its own bound. This favors read length over strict adherence to the tolerance. If that trade is wrong for your data, tighten `at_gc_error` — which pulls both bounds in — rather than trying to change the rule. +**The length limit is the more permissive of the two.** Because `readLengthLimit` takes the `max`, a read whose own usable range ended earlier is kept to the longer mate's length, retaining a few cycles past its own bound. This favors read length over strict adherence to the tolerance. If that trade is wrong for your data, tighten `at_gc_error` (which pulls both bounds in) rather than trying to change the rule. #### When it refuses to run @@ -2785,7 +2956,7 @@ CLIPPING READS : ERROR: no usable clip range in CLIPPING READS : exit 3 = unexpected FastQC header; 4 = no cycle within at_gc_error (0.025) ``` -Cycles where T or C is zero are skipped rather than divided by — that division would abort awk mid-pipeline, which plain `set -e` does not catch, and the bounds would be silently derived from a truncated table. +Cycles where T or C is zero are skipped rather than divided by: that division would abort awk mid-pipeline, which plain `set -e` does not catch, and the bounds would be silently derived from a truncated table. #### Tuning `at_gc_error` @@ -2793,10 +2964,10 @@ This is the only value here you would normally change, and it trades data volume | Direction | Effect | Risk | |---|---|---| -| **Tighter** (e.g. `0.01`) | Fewer cycles qualify, so more is clipped from both ends | Exit 4 — no cycle qualifies and the run stops. Shorter reads map less uniquely | +| **Tighter** (e.g. `0.01`) | Fewer cycles qualify, so more is clipped from both ends | Exit 4: no cycle qualifies and the run stops. Shorter reads map less uniquely | | **Looser** (e.g. `0.05`) | More cycles qualify, so reads stay longer | Retains cycles with real composition bias, which feeds into your frequencies | -Exit 4 on a library that is otherwise fine usually means the tolerance is too tight for its natural composition — GC-skewed genomes will not produce a G/C ratio near 1 anywhere. In that case raising `at_gc_error` is the correct response, not a workaround. +Exit 4 on a library that is otherwise fine usually means the tolerance is too tight for its natural composition. GC-skewed genomes will not produce a G/C ratio near 1 anywhere. In that case raising `at_gc_error` is the correct response, not a workaround. Check the before/after FastQC reports in `Output/Reports/Fastqc//` after changing it. The clipped-read report is the one that tells you whether the value you chose did what you wanted. @@ -2809,9 +2980,9 @@ Check the before/after FastQC reports in `Output/Reports/Fastqc//` after // options = "-m ${params.cutadapt.min_length}" ``` - Swap the two and reads shorter than `min_length` are discarded after clipping. This is an analysis-affecting change — existing trimmed output has to be removed before it takes effect. + Swap the two and reads shorter than `min_length` are discarded after clipping. This is an analysis-affecting change: existing trimmed output has to be removed before it takes effect. - Mind the order the two settings apply in. Cutadapt truncates to the computed read length limit **first**, and only then drops reads shorter than `min_length` — so a `min_length` above that limit discards every pair, and cutadapt still exits 0. That would leave empty FASTQ files which the existence checks would happily treat as a finished step, for good. The pipeline checks for it and stops with the two numbers in the message rather than letting it happen. The limit is computed per sample from its own FastQC report, so the value that is safe for one library is not automatically safe for another. + Mind the order the two settings apply in. Cutadapt truncates to the computed read length limit **first**, and only then drops reads shorter than `min_length`, so a `min_length` above that limit discards every pair, and cutadapt still exits 0. That would leave empty FASTQ files which the existence checks would happily treat as a finished step, for good. The pipeline checks for it and stops with the two numbers in the message rather than letting it happen. The limit is computed per sample from its own FastQC report, so the value that is safe for one library is not automatically safe for another. ### What this step writes @@ -2822,12 +2993,12 @@ Check the before/after FastQC reports in `Output/Reports/Fastqc//` after | `Output/Reports/Fastqc//` | FastQC on trimmed and on clipped reads | | `Output/Reports/Trimming//` | Trim Galore reports (`.txt` and, on 2.x, `.json`) | -The trimmed reads are **deleted** once clipping has consumed them ([why](#steps-delete-their-own-inputs)); the clipped reads are what step 3 aligns. +The trimmed reads are **deleted** once clipping has consumed them ([why](#deleted-intermediates)); the clipped reads are what step 3 aligns. ## Alignment & Cleaning -Step 4 turns a raw BWA alignment into the BAM that variant calling reads. The last stage of that is a filter, and it is the first place a variant can be lost — nothing later in the pipeline can recover a read discarded here. +Step 4 turns a raw BWA alignment into the BAM that variant calling reads. The last stage of that is a filter, and it is the first place a variant can be lost: nothing later in the pipeline can recover a read discarded here. ### The cleanup pipeline @@ -2855,7 +3026,7 @@ cleanBAM { } ``` -#### `filter` — what is excluded +#### `filter`: what is excluded `0xF0C` is the sum of six flag bits: @@ -2870,13 +3041,13 @@ cleanBAM { Secondary and supplementary alignments are excluded because a read that aligns in more than one place would otherwise contribute its bases to several positions. In a frequency estimate that is double-counting, not extra evidence. -#### `required` — what must be true +#### `required`: what must be true `0x2` requires the read to be **properly paired**: both mates mapped, in the expected orientation, at a plausible insert size. This is a strict requirement, and it is why single-end data does not work with the pipeline as configured. -#### `mapq` — the quality floor +#### `mapq`: the quality floor -This one is not part of either flag word and is easy to overlook. At `30`, reads whose mapping quality falls below the threshold are discarded — meaning reads that could plausibly have come from more than one location in the genome. +This one is not part of either flag word and is easy to overlook. At `30`, reads whose mapping quality falls below the threshold are discarded, meaning reads that could plausibly have come from more than one location in the genome. For Pool-seq this matters more than usual. An ambiguously placed read still adds a read count wherever it lands, and allele frequencies *are* read counts. A repetitive region that collects mismapped reads produces frequencies that look real and are not. @@ -2890,15 +3061,15 @@ The cost is coverage. In a repeat-rich or recently duplicated genome, a MAPQ 30 ### Why duplicates are removed, not just marked -`markdup -r` removes duplicate reads rather than flagging them. Downstream callers usually respect the duplicate flag anyway, so the two are close to equivalent — but removal is the safer default here, and duplicates matter more in a pool than in an individual. +`markdup -r` removes duplicate reads rather than flagging them. Downstream callers usually respect the duplicate flag anyway, so the two are close to equivalent. But removal is the safer default here, and duplicates matter more in a pool than in an individual. A PCR duplicate is a second read from a molecule that should only be counted once. In individual sequencing that inflates confidence in a genotype you would have called anyway. In Pool-seq it changes the answer: the duplicate adds a vote to one allele, and the frequency shifts. A library with uneven amplification produces a frequency estimate biased toward whichever molecules amplified best. -`-s` prints duplicate statistics into the step's log, which is worth reading — a high duplicate rate is a library-prep signal that no amount of filtering fixes. +`-s` prints duplicate statistics into the step's log, which is worth reading: a high duplicate rate is a library-prep signal that no amount of filtering fixes. ### Changing these values -All three are analysis-affecting. Changing any of them means the existing BAMs no longer match your configuration, and the next run stops before doing any work ([why](#the-run-refuses-to-mix-settings)). Clearing it means deleting `Output/Ready/`, `Output/VCF/` and `Output/Frequencies/`. +All three are analysis-affecting. Changing any of them means the existing BAMs no longer match your configuration, and the next run stops before doing any work ([why](#the-change-guard)). Clearing it means deleting `Output/Ready/`, `Output/VCF/` and `Output/Frequencies/`. Because the alignment step itself is unaffected, `Output/Aligned/` is preserved and the re-run starts from BAM cleanup rather than from BWA. @@ -2944,17 +3115,17 @@ capBAM { |---|---| | `-1` *(default)* | Step 5 measures a ceiling for each sample from its own depth histogram; step 6 truncates that sample's BAM to it. A sample with nothing worth cutting is left uncapped | | positive `N` | Every sample is capped at `N`, measured or not | -| `0` | No capping at all — the BAMs reach the pileup as step 4 left them | +| `0` | No capping at all: the BAMs reach the pileup as step 4 left them | -`param_capMaxDepth` in `metadata.csv` overrides this for a single sample and takes the same three values, which is the way to handle one library that needs a different answer from the rest — see [Metadata](#kinds-of-metadata-column). +`param_capMaxDepth` in `metadata.csv` overrides this for a single sample and takes the same three values, which is the way to handle one library that needs a different answer from the rest. See [Metadata](#kinds-of-metadata-column). **Nothing is deleted.** A position deeper than the ceiling is truncated to it on the way into the pileup, and the sample's ready BAM is untouched on disk. Reads are dropped whole rather than trimmed, so a pair may lose one mate. -**`histogramMax` is how deep step 5 looks, and a sample deeper than it stops the run.** The histogram reaches `100000×` by default, and everything above that would land in a single open bin — so a ceiling read from it would be read from a partial picture. Rather than choose on partial evidence, step 5 fails and names the sample and the depth it found. Raise `histogramMax` past that depth and run again. An organelle in a well-covered library is the case that reaches it in practice. **Raising it costs nothing already produced**: `samtools stats` reports only the depths that actually occur, so every value the run completes at gives the same histogram and the same ceiling, and step 0 does not compare it against previous runs. +**`histogramMax` is how deep step 5 looks, and a sample deeper than it stops the run.** The histogram reaches `100000×` by default, and everything above that would land in a single open bin, so a ceiling read from it would be read from a partial picture. Rather than choose on partial evidence, step 5 fails and names the sample and the depth it found. Raise `histogramMax` past that depth and run again. An organelle in a well-covered library is the case that reaches it in practice. **Raising it costs nothing already produced**: `samtools stats` reports only the depths that actually occur, so every value the run completes at gives the same histogram and the same ceiling, and step 0 does not compare it against previous runs. -**Why the ceiling is measured rather than set.** Pooled coverage is uneven by design, so no single depth is "too deep" in the abstract. What is worth truncating is a *second population* of positions — a collapsed repeat, or a PCR hill — carrying read counts no single locus produced. A hand-set number cannot find those: it cuts legitimate coverage in a deep sample and lets the pile-up through in a shallow one. [The Filter Chain](#depth-capping) shows the histograms this is read off, including the two cases where the detector deliberately declines. +**Why the ceiling is measured rather than set.** Pooled coverage is uneven by design, so no single depth is "too deep" in the abstract. What is worth truncating is a *second population* of positions (a collapsed repeat, or a PCR hill) carrying read counts no single locus produced. A hand-set number cannot find those: it cuts legitimate coverage in a deep sample and lets the pile-up through in a shallow one. [The Filter Chain](#depth-capping) shows the histograms this is read off, including the two cases where the detector deliberately declines. -**Every sample's decision is published**, capped or not, beside its histogram in `Output/Reports/Depth/` — "nothing was done" is the outcome you cannot otherwise see: +**Every sample's decision is published**, capped or not, beside its histogram in `Output/Reports/Depth/`; "nothing was done" is the outcome you cannot otherwise see: ```bash grep -H 'ceiling applied' Output/Reports/Depth/*_depth_report.txt @@ -2964,22 +3135,22 @@ grep -H 'ceiling applied' Output/Reports/Depth/*_depth_report.txt ```groovy variantCall { - scaleMapQ = 50 // -C downgrade coefficient for mismatch-heavy reads + scaleMapQ = 100 // -C downgrade coefficient for mismatch-heavy reads varQualMin = 30 // -q minimum mapping quality baseQualMin = 30 // -Q minimum base quality maxDepth = 0 // -d a flat depth cap on top of capBAM; 0 is no limit } ``` -`-C 50` (`scaleMapQ`) -: Downgrades mapping quality for reads carrying excessive mismatches. Reads that align poorly are more likely to be misplaced, and a misplaced read contributes its bases to the wrong position — which in a frequency estimate is a direct error, not just noise. +`-C 100` (`scaleMapQ`) +: Downgrades mapping quality for reads carrying excessive mismatches. Reads that align poorly are more likely to be misplaced, and a misplaced read contributes its bases to the wrong position, which in a frequency estimate is a direct error, not just noise. **The number runs the other way from what it looks like: lower is harsher, at or below 10 it does nothing, and anywhere below `varQualMin` it discards every read.** That last one is a relationship between two settings rather than a fixed range, and it is [measured at six different thresholds](#scalemapq-and-varqualmin). The value was chosen by measurement on pooled data rather than inherited; what each setting costs is in [What `-C` costs, measured](#scalemapq-measured). `-B` (fixed, not configurable) : Disables BAQ recalculation. BAQ downweights bases near indels to suppress misalignment artefacts under a single-genome model. In a pool, the same signal may be a genuine low-frequency indel, so the raw evidence is kept and the decision is left to the cross-sample filter. #### `maxDepth` -`-d` caps the reads considered **per file per position**, and it ships as `0`, which to `bcftools mpileup` means *no limit*. The BAMs reaching this point were already capped above, from each sample's own coverage — a second flat number on top of a measured one is what that stage exists to remove. +`-d` caps the reads considered **per file per position**, and it ships as `0`, which to `bcftools mpileup` means *no limit*. The BAMs reaching this point were already capped above, from each sample's own coverage. A second flat number on top of a measured one is what that stage exists to remove. **The two are not the same knob**, and they do not even take the same values: @@ -2989,11 +3160,11 @@ variantCall { | `0` | Do not cap | No limit | | positive `N` | Cap every sample at `N` | Cap the pileup at `N` | -**When to set it.** Leave it at `0` unless you want a hard backstop under the measured ceiling — a memory limit on a shared machine is the usual reason, since a very deep pile-up costs mpileup memory whether or not it was capped. If you do set it, size it against your data rather than guessing: the depth reports named above say what each sample actually looked like. +**When to set it.** Leave it at `0` unless you want a hard backstop under the measured ceiling. A memory limit on a shared machine is the usual reason, since a very deep pile-up costs mpileup memory whether or not it was capped. If you do set it, size it against your data rather than guessing: the depth reports named above say what each sample actually looked like. -A cap that bites truncates the read counts your frequencies are computed from, and nothing downstream flags it — which was the whole problem with the fixed `2000` this replaced. +A cap that bites truncates the read counts your frequencies are computed from, and nothing downstream flags it, which was the whole problem with the fixed `2000` this replaced. -**Coming from 2.2.0 or earlier**, your old `bcftools.maxDepth` is **not** carried across. It was the only depth control there was, and here it would sit as a flat ceiling under a measured one; `migrate_config` reports it under `Format changed this release` and explains the change. You get `capBAM.maxDepth = -1` and `variantCall.maxDepth = 0` — measured capping, no flat backstop. To reproduce older results exactly, put your old number back in `variantCall.maxDepth` and set `capBAM.maxDepth = 0`. +**Coming from 2.2.0 or earlier**, your old `bcftools.maxDepth` is **not** carried across. It was the only depth control there was, and here it would sit as a flat ceiling under a measured one; `migrate_config` reports it under `Format changed this release` and explains the change. You get `capBAM.maxDepth = -1` and `variantCall.maxDepth = 0`: measured capping, no flat backstop. To reproduce older results exactly, put your old number back in `variantCall.maxDepth` and set `capBAM.maxDepth = 0`. ### Calling @@ -3003,12 +3174,12 @@ callOptions = "-m -A -v -Ov" | Flag | Effect | |---|---| -| `-m` | Multiallelic caller — required for sites with more than one alternate allele | +| `-m` | Multiallelic caller (required for sites with more than one alternate allele) | | `-A` | Keep **all** alternate alleles from the pileup | | `-v` | Output variant sites only | | `-Ov` | Uncompressed VCF | -**`-A` is what makes this a Pool-seq caller.** Without it, bcftools prunes alternate alleles that no plausible genotype supports — sound for an individual, and wrong for a pool, where a true allele at frequency 0.01 supports no genotype at all. If you edit `callOptions`, keep `-A` and `-m`. +**`-A` is what makes this a Pool-seq caller.** Without it, bcftools prunes alternate alleles that no plausible genotype supports. That is sound for an individual and wrong for a pool, where a true allele at frequency 0.01 supports no genotype at all. If you edit `callOptions`, keep `-A` and `-m`. Variant calling is a **single joint task** over all BAMs, not one task per sample. That is what produces a multi-sample VCF with comparable columns, and it is why every sample must share a reference. @@ -3017,16 +3188,17 @@ Variant calling is a **single joint task** over all BAMs, not one task per sampl Step 7 takes the joint VCF that step 6 produced and turns it into the frequency tables you actually read. Between the two sit the filters that decide which of the called sites survive. -The order is fixed — stages 5 to 9 of [The Filter Chain](#the-chain-at-a-glance): normalize to the major allele, drop alleles too few samples support, drop sites too thin or too poorly called, split SNPs from indels, then convert allelic depths to frequencies. Only two of those stages have parameters, and both are analysis-affecting. +The order is fixed (stages 5 to 9 of [The Filter Chain](#the-chain-at-a-glance)): normalize to the major allele, drop alleles too few samples support, drop sites too thin or too poorly called, split SNPs from indels, then convert allelic depths to frequencies. Only two of those stages have parameters, and both are analysis-affecting. | Parameter | What it does | |---|---| | `poolSize`, `ploidy` | Set the smallest allele frequency worth believing, which is what the false-positive filter tests against. Both can be set per sample or per run | | `filterFalsePositives.sampleThreshold` | Removes alternate alleles without support across enough samples. The filter that makes the pipeline pool-aware, and the one most worth understanding before changing anything | | `vcffilter.minDP`, `vcffilter.minQUAL` | Remove whole sites that are too shallow in any one sample, or too poorly called | -| `vcf.fileName` | Removes nothing — it names the files this step writes | +| `vcffilter.dropZeroDepth` | Removes a site where any sample has no reads at all, independently of `minDP`. On by default, and what keeps a published table free of cells that were never measured | +| `vcf.fileName` | Removes nothing; it names the files this step writes | -`filterFalsePositives.sensitivity` is **computed** from `poolSize` and `ploidy` rather than set, and can be overridden per sample in `metadata.csv` — see [Metadata](#metadata) for the per-sample form. +`filterFalsePositives.sensitivity` is **computed** from `poolSize` and `ploidy` rather than set, and can be overridden per sample in `metadata.csv`; see [Metadata](#metadata) for the per-sample form. For the stage-by-stage account with counts, read [The Filter Chain](#the-filter-chain). @@ -3037,7 +3209,7 @@ poolSize = 50 // individuals in one pool ploidy = 2 // copies of the genome per individual ``` -These two are the pipeline's most consequential settings. They are not used to model anything — they set the **minimum credible allele frequency**: +These two are the pipeline's most consequential settings. They are not used to model anything; they set the **minimum credible allele frequency**: $$S = \frac{1}{2 \times \text{ploidy} \times \text{poolSize}}$$ @@ -3045,7 +3217,7 @@ An alternate allele must reach `S` in a sample for that sample to count as suppo **`poolSize` is per pool, not per run.** With eight pools of 50 individuals each, it is `50`, not `400`. -**The factor of two is deliberate.** A pool of 50 diploids holds 100 chromosomes, so one chromosome is frequency 0.01. `S` comes out at 0.005 — half of that — so a genuine singleton clears the threshold with margin rather than sitting exactly on it. The observed fraction of a singleton is itself noisy, and a threshold placed exactly at the expected value would reject half of them. +**The factor of two is deliberate.** A pool of 50 diploids holds 100 chromosomes, so one chromosome is frequency 0.01. `S` comes out at 0.005 (half of that), so a genuine singleton clears the threshold with margin rather than sitting exactly on it. The observed fraction of a singleton is itself noisy, and a threshold placed exactly at the expected value would reject half of them. | `poolSize` | `ploidy` | One chromosome is | `S` | |---|---|---|---| @@ -3059,7 +3231,7 @@ An alternate allele must reach `S` in a sample for that sample to count as suppo **If your pools differ in size**, give each one its own `param_poolSize` in `metadata.csv` and every column is judged at its own threshold. The global `poolSize` here stays as the default for pools that do not state one. See [Pool size belongs to the pool](#pool-size-belongs-to-the-pool). -**`ploidy` applies to a whole run.** It describes the sequence rather than the sample, so a diploid animal's haploid mitochondrial genome belongs in a separate run with its own value — see [Sequences with a different ploidy](#sequences-with-a-different-ploidy). +**`ploidy` applies to a whole run.** It describes the sequence rather than the sample, so a diploid animal's haploid mitochondrial genome belongs in a separate run with its own value. See [Sequences with a different ploidy](#sequences-with-a-different-ploidy). ### `sampleThreshold` @@ -3086,7 +3258,7 @@ The comparison is `count >= n_samples × sampleThreshold`, so the effective requ !!! danger "The default removes population-private alleles" - With eight samples, an allele found in **one** pool is discarded regardless of how frequent it is there — one does not reach two. If private or population-specific variation is the subject of your study, this default is wrong for you. + With eight samples, an allele found in **one** pool is discarded regardless of how frequent it is there: one does not reach two. If private or population-specific variation is the subject of your study, this default is wrong for you. | Value | Behavior | Suits | |---|---|---| @@ -3094,7 +3266,7 @@ The comparison is `count >= n_samples × sampleThreshold`, so the effective requ | Low enough to require 1 sample | Any single pool can carry an allele | Private variants, small sample counts, discovery runs | | Higher, e.g. `0.5` | Needs half the samples | Conservative core-variant sets; high-noise data | -Remember that the denominator is the number of **VCF columns**, which is the number of distinct `RG_Sample` values — not the number of FASTQ pairs. Merging replicates by sharing an `RG_Sample` changes this threshold as a side effect ([see Metadata](#rg_sample-decides-what-counts-as-a-sample)). +Remember that the denominator is the number of **VCF columns**, which is the number of distinct `RG_Sample` values and not the number of FASTQ pairs. Merging replicates by sharing an `RG_Sample` changes this threshold as a side effect ([see Metadata](#rg_sample-decides-what-counts-as-a-sample)). ### Depth and quality @@ -3137,13 +3309,13 @@ If "every sample" is too strict for your design, the alternatives are one-line s | `-i "COUNT(FMT/DP>=20)>=2"` | At least 2 samples pass | 2 of 3 | | `-i "MEAN(FMT/DP)>=20"` | Mean depth across samples | 2 of 3 | | `-i "INFO/DP>=20"` | Cohort total depth | 3 of 3 | -| `-i "FMT/DP>=20"` | **Any one** sample passes — not a depth floor | 3 of 3 | +| `-i "FMT/DP>=20"` | **Any one** sample passes, not a depth floor | 3 of 3 | The last row is worth noting as a trap: `-i "FMT/DP>=20"` reads like the obvious inverse of the current expression and is not, because bcftools evaluates a `FORMAT` condition per sample and keeps the site if it holds for any of them. !!! note "Genotype-level filtering is not available here" - A per-sample depth floor — blanking one pool's frequency while keeping the row — cannot be done in the VCF, because vcftools and bcftools both express a genotype-level verdict by rewriting `GT`, and `GT` is set to `./.` throughout by major-allele normalization ([why](#major-allele-normalization)). Frequency conversion reads `AD`. If you need per-sample blanking rather than whole-site removal, it has to happen in `bin/depth2freq.awk`, which already computes each sample's depth as the denominator. + A per-sample depth floor (blanking one pool's frequency while keeping the row) cannot be done in the VCF, because vcftools and bcftools both express a genotype-level verdict by rewriting `GT`, and `GT` is set to `./.` throughout by major-allele normalization ([why](#major-allele-normalization)). Frequency conversion reads `AD`. If you need per-sample blanking rather than whole-site removal, it has to happen in `bin/depth2freq.awk`, which already computes each sample's depth as the denominator. ### Output naming @@ -3153,14 +3325,14 @@ vcf { } ``` -Base name for every VCF and frequency table. With the default, a finished run leaves `Test.vcf`, `Test_snp_freq.tsv`, `Test_indel_freq.tsv`, and `Test_annotated.vcf` if annotation ran. Worth setting to something descriptive — it is the name your results carry from here on. +Base name for every VCF and frequency table. With the default, a finished run leaves `Test.vcf`, `Test_snp_freq.tsv`, `Test_indel_freq.tsv`, and `Test_annotated.vcf` if annotation ran. Worth setting to something descriptive, because it is the name your results carry from here on. Changing it after a run does not rename anything; it makes the resume checks look for files that do not exist, and the whole VCF and frequency branch runs again alongside the old files. ## Annotations -Step 8 annotates the called variants with what they do to genes — missense, synonymous, intergenic — using SnpEff against a database built from your GFF back in step 1. The annotation is a **parallel branch**, not a filter: it never changes the frequency tables, and nothing downstream depends on it. +Step 8 annotates the called variants with what they do to genes (missense, synonymous, intergenic) using SnpEff against a database built from your GFF back in step 1. The annotation is a **parallel branch**, not a filter: it never changes the frequency tables, and nothing downstream depends on it. | Parameter | Default | What it is | |---|---|---| @@ -3170,7 +3342,7 @@ Step 8 annotates the called variants with what they do to genes — missense, sy | `snpEff.buildOptions` | `-gff3 -noCheckCds -noCheckProtein -v` | Passed to `snpEff build` in step 1 | | `snpEff.runOptions` | `-v` | Passed to `snpEff` in step 8 | -`snpEff.db` is **computed** from `gffFile` — the same name with `.gz` removed — and names the database SnpEff builds and then annotates against. Change `gffFile`, not this. +`snpEff.db` is **computed** from `gffFile` (the same name with `.gz` removed) and names the database SnpEff builds and then annotates against. Change `gffFile`, not this. `annotate` is a **per-run** parameter, so a run table may switch annotation on for some runs and off for others; step 1 builds the database only for the runs that asked for it. @@ -3178,7 +3350,7 @@ Step 8 annotates the called variants with what they do to genes — missense, sy The `snpEff` defaults are not arbitrary, and the build ones in particular are worth understanding before you change them. -`buildOptions` says the annotation file is GFF3 and turns off two consistency checks SnpEff runs by default: that every CDS translates to the protein SnpEff expects, and that protein sequences match. Non-model references routinely fail both — a GFF with slightly different coordinates, a genome with a non-standard genetic code — and the failure aborts the build rather than degrading it. They are off so that a usable database is built from an imperfect annotation, which is the normal case here. Turn them back on by removing the flags if you want the check. +`buildOptions` says the annotation file is GFF3 and turns off two consistency checks SnpEff runs by default: that every CDS translates to the protein SnpEff expects, and that protein sequences match. Non-model references routinely fail both (a GFF with slightly different coordinates, a genome with a non-standard genetic code), and the failure aborts the build rather than degrading it. They are off so that a usable database is built from an imperfect annotation, which is the normal case here. Turn them back on by removing the flags if you want the check. `runOptions` is verbosity only. Both are plain strings passed straight through, so anything `snpEff` accepts can go in them: @@ -3195,12 +3367,12 @@ Changing either changes what step 8 writes, so step 0 tracks them and refuses a Output is `Output/VCF/_annotated.vcf`, `Output/Reports/snpeff_summary.html` and `Output/Reports/snpeff_summary.genes.txt`. -**The annotated VCF is not your frequency tables with a column added.** Step 8 reads step 6's output, so it carries sites the step 7 filters removed and encodes them against the original reference rather than the major allele. Joining the two is yours to do, on `CHROM`/`POS`, expecting unmatched rows on the annotation side — see [Step 8](#step-8-annotate-variants). +**The annotated VCF is not your frequency tables with a column added.** Step 8 reads step 6's output, so it carries sites the step 7 filters removed and encodes them against the original reference rather than the major allele. Joining the two is yours to do, on `CHROM`/`POS`, expecting unmatched rows on the annotation side; see [Step 8](#step-8-annotate-variants). # Analysis Layer -The pipeline stops at `Output/Frequencies`. The analysis layer is a second, **optional** layer that reads what the pipeline published and produces analyses from it. It is reached as `PoolSeqFlow analysis `, and it has a conda environment carrying R and an entry point of its own — it is not steps 9 and up, and running it never changes, moves or re-runs anything the pipeline made. +The pipeline stops at `Output/Frequencies`. The analysis layer is a second, **optional** layer that reads what the pipeline published and produces analyses from it. It is reached as `PoolSeqFlow analysis `, and it has a conda environment carrying R and an entry point of its own. It is not steps 9 and up, and running it never changes, moves or re-runs anything the pipeline made. ### Installing it { #installing-the-analysis-layer } @@ -3215,14 +3387,14 @@ The pipeline is complete without it, and a machine that only ever analyzes resul ### Running a module -A **module** is one analysis. You run one at a time, from your project directory — the same one you run the pipeline from: +A **module** is one analysis. You run one at a time, from your project directory, which is the same one you run the pipeline from: ```bash cd /path/to/project PoolSeqFlow analysis verify ``` -**Only `verify` comes with the pipeline.** It is part of the analysis frame itself, reports what the layer can see and produces nothing. Every other module — `basicstats`, `association` and `mds` among them — is published on its own timetable and installed from the catalogue, so a fresh installation has an empty store and you choose what goes into it. Each has a page of its own under [Modules](#modules), which is where what it computes and what it assumes is written down. +**Only `verify` comes with the pipeline.** It is part of the analysis frame itself, reports what the layer can see and produces nothing. Every other module (`basicstats`, `association` and `mds` among them) is published on its own timetable and installed from the catalogue, so a fresh installation has an empty store and you choose what goes into it. Each has a page of its own under [Modules](#modules), which is where what it computes and what it assumes is written down. A module is installed with the **libraries** it declares: the shared arithmetic more than one module wants, each one published and versioned like a module and installed into `analysis/modules/lib/`. You never ask for a library by name; it arrives with whatever needs it, and leaves when nothing installed still declares it. @@ -3238,21 +3410,21 @@ PoolSeqFlow analysis modules uninstall mds # remove one, after confirming All of them read the installation rather than your project, so they work from anywhere and none needs a `parameters.config`. `list` and `available` need no environment. **`install` does**, because installing a module also installs what it runs on; `uninstall` uses the environment when it is there and says so when it is not. -**`install` pins by name.** Without a version it takes the newest one this release can read; with one it takes exactly that. **Name the version in your methods section**, and install that version to reproduce the analysis — a module carries its own version precisely so it can move without the pipeline moving, which means two runs of "the same module" are not necessarily the same code. Every analysis prints the module version it ran in its header, and installing writes a `.source` file beside the module recording where it came from and the checksum it matched. +**`install` pins by name.** Without a version it takes the newest one this release can read; with one it takes exactly that. **Name the version in your methods section**, and install that version to reproduce the analysis. A module carries its own version precisely so it can move without the pipeline moving, which means two runs of "the same module" are not necessarily the same code. Every analysis prints the module version it ran in its header, and installing writes a `.source` file beside the module recording where it came from and the checksum it matched. -**The same list is on the website**, at [Published modules](https://ozankiratli.github.io/PoolSeqFlow/modules-repo/) — the page and the catalogue are generated from the same rows and published in the same deploy, so what it shows is what `available` will tell you. +**The same list is on the website**, at [Published modules](https://ozankiratli.github.io/PoolSeqFlow/modules-repo/). The page and the catalogue are generated from the same rows and published in the same deploy, so what it shows is what `available` will tell you. **`available` reads a catalogue over the network** and a release carries no copy of it, so a module published long after a release is still installable into it. What it lists is filtered to the table contract this release speaks; a module written against a later contract is shown and marked rather than hidden, so being told to install something that cannot work here gives you a reason instead of a blank. -**One line per module, and it is the version `install` would take.** The catalogue holds a row for every published version, so several rows name one module — a listing of them would be that module's history rather than something to act on. What you see is the newest version *your* release can run, which is what `install ` picks, so the two commands cannot tell you different things. Older versions stay published and stay installable by naming one. If your machine has no route to the internet, or your institution keeps a mirror, `POOLSEQFLOW_MODULE_INDEX` points at a URL or a file instead. +**One line per module, and it is the version `install` would take.** The catalogue holds a row for every published version, so several rows name one module, and a listing of them would be that module's history rather than something to act on. What you see is the newest version *your* release can run, which is what `install ` picks, so the two commands cannot tell you different things. Older versions stay published and stay installable by naming one. If your machine has no route to the internet, or your institution keeps a mirror, `POOLSEQFLOW_MODULE_INDEX` points at a URL or a file instead. **A download is verified before it is unpacked.** A module is code that runs on your machine, so the checksum in the catalogue is checked first, and a mismatch stops the install having written nothing. -**Every module declares its own license, and the report prints it.** PoolSeqFlow is Apache-2.0 and imposes nothing on a module, so a module that builds on a GPL package is GPL itself and the results it produces are produced under those terms. The verification report at the top of every analysis carries the line — `published under GPL-3.0-or-later` — before anything runs, which is where to look if you are about to redistribute what comes out. +**Every module declares its own license, and the report prints it.** PoolSeqFlow is Apache-2.0 and imposes nothing on a module, so a module that builds on a GPL package is GPL itself and the results it produces are produced under those terms. The verification report at the top of every analysis carries the line (`published under GPL-3.0-or-later`) before anything runs, which is where to look if you are about to redistribute what comes out. -**`basicstats`, `association` and `mds` are GPL-3.0-or-later**, and the pipeline they run beside is not. Each compiles its hot path with `Rcpp`, which is GPL, and does so by default — `nocpp` turns it off for one run but the module ships expecting it. `verify` is the exception: it belongs to the frame, runs no R at all, and is Apache-2.0 with the rest of the pipeline. +**`basicstats`, `association` and `mds` are GPL-3.0-or-later**, and the pipeline they run beside is not. Each compiles its hot path with `Rcpp`, which is GPL, and does so by default. `nocpp` turns it off for one run but the module ships expecting it. `verify` is the exception: it belongs to the frame, runs no R at all, and is Apache-2.0 with the rest of the pipeline. -**A module also declares the frame and the release it needs**, and a module that needs a newer one is refused by name rather than run. The frame is the library a module imports — `analysis/frame.version` in the installation, versioned separately from the pipeline because a module runs against it and not against the pipeline's own steps. The release is named because the analysis environment belongs to it: there is one R environment per release, shared by every module installed into it. +**A module also declares the frame and the release it needs**, and a module that needs a newer one is refused by name rather than run. The frame is the library a module imports (`analysis/frame.version` in the installation), versioned separately from the pipeline because a module runs against it and not against the pipeline's own steps. The release is named because the analysis environment belongs to it: there is one R environment per release, shared by every module installed into it. #### What installing a module does to your environment { #module-packages } @@ -3263,7 +3435,7 @@ Installing what fst runs on, into 'PoolSeqFlow--analysis': r-poolfstat=3.0.0 ``` -**Only what is missing is installed, so that list is usually shorter than the manifest.** A package already there at the version asked for is not work, and conda is not asked about it. Most of what a manifest names is normally already in the environment — the release's own analysis layer carries `ggplot2`, `data.table`, `Rcpp` and the rest — so installing a module that needs nothing new says so and stops: +**Only what is missing is installed, so that list is usually shorter than the manifest.** A package already there at the version asked for is not work, and conda is not asked about it. Most of what a manifest names is normally already in the environment (the release's own analysis layer carries `ggplot2`, `data.table`, `Rcpp` and the rest), so installing a module that needs nothing new says so and stops: ``` Everything basicstats runs on is already in 'PoolSeqFlow--analysis'. @@ -3271,7 +3443,7 @@ Everything basicstats runs on is already in 'PoolSeqFlow--analysis'. That is the common case and not a sign that something was skipped: the manifest still states everything the module needs, and uninstalling it still reasons over that whole list. -**Nothing already in the environment is allowed to move.** A pin naming a package the environment already holds at a different version is refused before conda is asked at all, so a module can never quietly downgrade something another module — or the release itself — is running on: +**Nothing already in the environment is allowed to move.** A pin naming a package the environment already holds at a different version is refused before conda is asked at all, so a module can never quietly downgrade something another module (or the release itself) is running on: > ERROR: these pins disagree with what 'PoolSeqFlow--analysis' already holds: > r-poolfstat=2.9.0 (installed 3.0.0) @@ -3284,9 +3456,9 @@ Either way it is a compatibility question between a release and a module, settle **Uninstalling takes back only what nothing else asks for.** If two modules both name `r-poolfstat=3.0.0`, removing one leaves it installed for the other. And because `conda remove` takes everything that depends on what it is given, the removal is planned before it is run: if taking a package out would take something else with it, nothing is removed and the module stays installed. -**Two things follow from the store living inside the installation.** Reinstalling the pipeline over itself wipes the store, so the packages its modules added are taken out of the environment first, while the manifests declaring them still exist — afterwards both are back to what a fresh installation is, which is empty, and the modules you want are installed again. And `analysis uninstall` removes the environment while leaving the store, so `analysis install` puts back what the modules still there need. Neither is something to manage by hand. +**Two things follow from the store living inside the installation.** Reinstalling the pipeline over itself wipes the store, so the packages its modules added are taken out of the environment first, while the manifests declaring them still exist. Afterwards both are back to what a fresh installation is, which is empty, and the modules you want are installed again. And `analysis uninstall` removes the environment while leaving the store, so `analysis install` puts back what the modules still there need. Neither is something to manage by hand. -`list` is worth knowing about before you need it. Modules live inside the release's own installation, so each release has its own set and a module installed for one is never picked up by another — reinstalling the same version wipes them, and one command puts each back. More usefully: **a module directory that has lost its pipeline stops every analysis run, not only its own**, and `list` is what names the one at fault. It also tells you where the store is, which is the directory a module is installed into. +`list` is worth knowing about before you need it. Modules live inside the release's own installation, so each release has its own set and a module installed for one is never picked up by another. Reinstalling the same version wipes them, and one command puts each back. More usefully: **a module directory that has lost its pipeline stops every analysis run, not only its own**, and `list` is what names the one at fault. It also tells you where the store is, which is the directory a module is installed into. Versions are not in that listing. A module reports its own version in the header of every analysis it produces, and that is the copy worth reading, because it comes from the same manifest the layer validated before it ran. @@ -3302,11 +3474,11 @@ There are two, both optional, and the second wins over the first: | File | Where | Applies to | |---|---|---| | `analysis.config` | your project, beside `parameters.config` | every module in this project | -| `.config` | your project | that one module — `mds.config` for `mds` | +| `.config` | your project | that one module (`mds.config` for `mds`) | -Copy `analysis/analysis.config.template` out of the installation to start one — until you do, the report at the top of every module prints the path. **The default for every setting below is built into PoolSeqFlow**, so a setting you do not write has exactly one source, and there is no file anywhere you could edit to change it for every project at once. +Copy `analysis/analysis.config.template` out of the installation to start one. Until you do, the report at the top of every module prints the path. **The default for every setting below is built into PoolSeqFlow**, so a setting you do not write has exactly one source, and there is no file anywhere you could edit to change it for every project at once. -There is a third file, `analysis/frame.config` in the installation, and it is **not** a settings file: it holds none of the settings in this section and is installed read-only. It is how a module reaches conda, the helpers in `bin/` and its resource ceiling — the things the pipeline gets from `nextflow.config` and a module, being its own entry script, does not get at all. It is listed here so that seeing it in a verification report is not a surprise; nothing in it is yours to change, and anything you set in `analysis.config` wins over it in any case. +There is a third file, `analysis/frame.config` in the installation, and it is **not** a settings file: it holds none of the settings in this section and is installed read-only. It is how a module reaches conda, the helpers in `bin/` and its resource ceiling (the things the pipeline gets from `nextflow.config` and a module, being its own entry script, does not get at all). It is listed here so that seeing it in a verification report is not a surprise; nothing in it is yours to change, and anything you set in `analysis.config` wins over it in any case. `parameters.config` is read as well, and is where the pipeline's own settings stay. The analysis layer does not repeat them. @@ -3316,7 +3488,7 @@ Everything the analysis layer takes: |---|---|---| | `analysis.runs` | Which of your runs this invocation covers | [below](#analysis-runs) | | `analysis.folderName` | Which folder under `Analysis/Results` it writes to | [Output Layout](#analysis-folder-name) | -| `analysis.metadata.missingValueEncoding` | What a cell means when it is not a value — `NA` and its spellings | [below](#missing-values) | +| `analysis.metadata.missingValueEncoding` | What a cell means when it is not a value: `NA` and its spellings | [below](#missing-values) | | `analysis.metadata.timeVar.*` | How the time column is read and ordered | [below](#the-time-axis) | | `analysis.design.*` | What the experiment was: which pools are independent of each other, which repeats are which, and which covariates a module may fit | [below](#experimental-design) | | `analysis.design.series.incomplete` | What a time axis makes of that, and what to do with a trajectory that skips a timepoint | [below](#time-series) | @@ -3324,13 +3496,13 @@ Everything the analysis layer takes: | `analysis.metadata.covariates.*` | What each `cov_` column holds, so a module can compute with it | [below](#covariates) | | `analysis.modules..*` | One module's own settings | [below](#module-settings) | -**The three inner scopes are namespaces, and the split between them is a real one.** `analysis.metadata` says how `metadata.csv` is **read** — what a blank means, how a date parses, what scale a column carries. `analysis.design` says what the file **describes** — which pools are independent, which repeats are which. `analysis.modules` holds one scope per installed module. Keeping them apart means a module called `design` cannot read the frame's design settings, and it lets each level **refuse a key it does not have** — without which `analysis { timevar { … } }`, with a small `v`, would sit in your file unread while the project ran on defaults you never chose. +**The three inner scopes are namespaces, and the split between them is a real one.** `analysis.metadata` says how `metadata.csv` is **read**: what a blank means, how a date parses, what scale a column carries. `analysis.design` says what the file **describes**: which pools are independent, which repeats are which. `analysis.modules` holds one scope per installed module. Keeping them apart means a module called `design` cannot read the frame's design settings, and it lets each level **refuse a key it does not have**. Without that, `analysis { timevar { ... } }`, with a small `v`, would sit in your file unread while the project ran on defaults you never chose. -A scope you write only part of keeps the rest of its defaults — `timeVar { kind = 'numerical' }` still takes `column` from `exp_time`. A key a scope does not have is refused, naming the ones it does. +A scope you write only part of keeps the rest of its defaults. `timeVar { kind = 'numerical' }` still takes `column` from `exp_time`. A key a scope does not have is refused, naming the ones it does. ### Missing values { #missing-values } -A **blank cell always means "no value"**, everywhere, with no setting required. `analysis.metadata.missingValueEncoding` is for the *other* spellings of it — the ones your export wrote. +A **blank cell always means "no value"**, everywhere, with no setting required. `analysis.metadata.missingValueEncoding` is for the *other* spellings of it, the ones your export wrote. ```groovy params { @@ -3346,7 +3518,7 @@ Each entry is matched **whole**, so `NA` does not catch `NAive`. `*` stands for **Matching is case sensitive, and that is deliberate.** `NA` does not catch `na`; list both if your file has both. Folding case would be convenient right up to the project whose population codes include `na`, where it would silently delete a real level and leave a smaller design with no error attached. No setting can tell those two situations apart, so the safe one is the default. -**It applies to `exp_` and `pt_` columns only** — the ones the analysis layer reads. Read-group tags and `param_` overrides were consumed by the pipeline long before any of this, and changing what they mean now would say a completed run had been configured differently than it was. +**It applies to `exp_` and `pt_` columns only**: the ones the analysis layer reads. Read-group tags and `param_` overrides were consumed by the pipeline long before any of this, and changing what they mean now would say a completed run had been configured differently than it was. **An entry that would match everything is refused.** `'*'` on its own would turn every experimental and phenotype cell in the project into a blank, leaving an analysis with no design and nothing to say so. @@ -3380,26 +3552,26 @@ params { | `kind` | ordered by | spacing | a module may | |---|---|---|---| -| `numerical` | the number | **real** | fit a rate — per generation, per day | +| `numerical` | the number | **real** | fit a rate (per generation, per day) | | `datetime` | the parsed date | **real**, in days | fit a rate per day | -| `categorical` | `order`, else alphabetically | **none** | rank and sequence only — **not** a rate | +| `categorical` | `order`, else alphabetically | **none** | rank and sequence only, **not** a rate | Each level gets a `position`: the number itself for `numerical`, days from the earliest for `datetime`, and **nothing at all** for `categorical`. That absence is what lets a module refuse to fit a slope against an axis that has no spacing, instead of doing it anyway. #### `unit`, required for `numerical` -The numbers are distances, so a rate is meaningful and has to be labeled — `0.003 per unit` is not an answer. One of: +The numbers are distances, so a rate is meaningful and has to be labeled. `0.003 per unit` is not an answer. One of: | | | |---|---| | exact durations | `millisecond` `second` `minute` `hour` | | calendar | `day` `week` `month` `year` | | counts | `generation` `passage` `cycle` | -| unnamed | `step` — evenly spaced in something you have not named | +| unnamed | `step`: evenly spaced in something you have not named | `step` is the escape: `1, 2, 5` under `step` still means the second gap is three times the first. If the numbers are only ranks with no spacing at all, that is `categorical`, not `step`. -**A `unit` on `categorical` time is refused.** Setting one asserts a spacing that categorical time does not have; if the spacing is real, the kind is `numerical`. `month` and `year` are calendar units and not fixed durations — 28 to 31 days, 365 or 366 — which costs nothing here because the position stays in your unit and is never converted. +**A `unit` on `categorical` time is refused.** Setting one asserts a spacing that categorical time does not have; if the spacing is real, the kind is `numerical`. `month` and `year` are calendar units and not fixed durations (28 to 31 days, 365 or 366), which costs nothing here because the position stays in your unit and is never converted. #### `order`, for `categorical` @@ -3407,7 +3579,7 @@ The numbers are distances, so a rate is meaningful and has to be labeled — `0. timeVar { kind = 'categorical'; order = ['pre', 'during', 'post'] } ``` -Matching is exact: `Pre` and `pre` are different. A level in your metadata that `order` does not include is **refused**, naming it. An entry in `order` that no pool has is **allowed** and reported — a timepoint you have planned but not yet sequenced is an ordinary state. +Matching is exact: `Pre` and `pre` are different. A level in your metadata that `order` does not include is **refused**, naming it. An entry in `order` that no pool has is **allowed** and reported. A timepoint you have planned but not yet sequenced is an ordinary state. **Without `order` the levels are sorted alphabetically**, and alphabetical is wrong more often than it is right: @@ -3417,7 +3589,7 @@ Matching is exact: `Pre` and `pre` are different. A level in your metadata that | `baseline week2 week12` | `baseline week12 week2` | `baseline week2 week12` | | `pre post` | `post pre` | `pre post` | -When alphabetical is in use, PoolSeqFlow compares it against a sort that reads embedded digits as numbers, and **warns when the two disagree** — that catches the first two rows above. It cannot catch `pre`/`post`: there is nothing in those strings to disagree about, and no software can know which came first. **That one is yours**, and the defense is that the verification report prints the levels in the order it will use them, immediately above your results. +When alphabetical is in use, PoolSeqFlow compares it against a sort that reads embedded digits as numbers, and **warns when the two disagree**, which catches the first two rows above. It cannot catch `pre`/`post`: there is nothing in those strings to disagree about, and no software can know which came first. **That one is yours**, and the defense is that the verification report prints the levels in the order it will use them, immediately above your results. #### `format` and `locale`, for `datetime` @@ -3428,7 +3600,7 @@ Patterns are `java.time`'s. The letters you will actually use: | letter | means | example | |---|---|---| | `uuuu` / `yyyy` | four-digit year | `2024` | -| `uu` / `yy` | two-digit year — **always 2000–2099** | `26` → 2026 | +| `uu` / `yy` | two-digit year: **always 2000-2099** | `26` -> 2026 | | `MM` | month as two digits | `03` | | `M` | month as one or two digits | `3` | | `MMM` | short month name | `Mar` | @@ -3436,7 +3608,7 @@ Patterns are `java.time`'s. The letters you will actually use: | `dd` | day as **exactly two** digits | `07` | | `d` | day as one or two digits | `7` or `07` | | `HH` | hour, **24-hour clock** | `14` | -| `hh` | hour, **12-hour clock — requires `a`** | `02` | +| `hh` | hour, **12-hour clock (requires `a`)** | `02` | | `mm` | minute | `30` | | `ss` | second | `00` | | `a` | AM/PM marker | `PM`, `pm` | @@ -3444,7 +3616,7 @@ Patterns are `java.time`'s. The letters you will actually use: **The failures, explicitly.** Each of these was measured, and each is a real way to lose an afternoon: -- **`hh` without `a` is refused.** `hh:mm` on afternoon samples has no way to know they are afternoon. PoolSeqFlow stops rather than reading `02:30 PM` as half past two in the morning — twelve hours early on every one of them. Write `hh:mm a`, or `HH:mm` for a 24-hour clock. +- **`hh` without `a` is refused.** `hh:mm` on afternoon samples has no way to know they are afternoon. PoolSeqFlow stops rather than reading `02:30 PM` as half past two in the morning, twelve hours early on every one of them. Write `hh:mm a`, or `HH:mm` for a 24-hour clock. - **`dd` does not accept a one-digit day.** `2024-03-7` fails under `yyyy-MM-dd` and works under `yyyy-MM-d`. `d` accepts both `7` and `07`, so prefer it if your file is inconsistent. - **An impossible date is refused, never corrected.** `2024-02-31` stops the run naming the value. A lenient parser would quietly make it `2024-02-29`, which then sorts perfectly and is not the date anybody wrote. - **`yyyy` and `uuuu` both work.** They differ only for dates before year 1, which yours are not. @@ -3453,7 +3625,7 @@ Patterns are `java.time`'s. The letters you will actually use: - **Time zones are not supported.** Dates are read as local, without an offset. If your values carry one, strip it. - **A pattern with no date is refused.** A time of day alone has no order across days. -`locale` is the language your month names are written in — it defaults to `en` and only matters for `MMM`, `MMMM` and `a`. A numeric pattern like `yyyy-MM-dd` never needs it. +`locale` is the language your month names are written in. It defaults to `en` and only matters for `MMM`, `MMMM` and `a`. A numeric pattern like `yyyy-MM-dd` never needs it. ```groovy timeVar { kind = 'datetime'; format = 'd MMMM yyyy'; locale = 'fr' } // 5 décembre 2011 @@ -3463,7 +3635,7 @@ timeVar { kind = 'datetime'; format = 'd MMMM yyyy'; locale = 'fr' } // 5 déc #### What the report tells you, and why to read it -Two mistakes are impossible to catch — an ambiguous date pattern, and a categorical order with no lexical clue. The defense is that every run prints what it did, in the order it will use, immediately above the results: +Two mistakes are impossible to catch: an ambiguous date pattern, and a categorical order with no lexical clue. The defense is that every run prints what it did, in the order it will use, immediately above the results: ``` TIME VARIABLE: exp_time, datetime, 'dd/MM/yyyy' (fr) @@ -3493,18 +3665,18 @@ params { **This applies with or without a time course.** A one-off comparison of three treated pools against three controls has conditions and independent units exactly as a five-generation time series does, and reads them from the same three settings. -**`by` defaults to every `exp_` variable except time**, and the report always prints the key it used. Set it explicitly when one of your `exp_` columns is recorded **on** each pool rather than saying what the pool is — a cage temperature, a census count. Such a column differs from pool to pool, so leaving it in the key makes every pool its own unit and, in a time series, splits every series into singletons with no error at all. +**`by` defaults to every `exp_` variable except time**, and the report always prints the key it used. Set it explicitly when one of your `exp_` columns is recorded **on** each pool rather than saying what the pool is: a cage temperature, a census count. Such a column differs from pool to pool, so leaving it in the key makes every pool its own unit and, in a time series, splits every series into singletons with no error at all. #### Replicates: which repeats are independent { #replicates } "Replicate" covers two things that are handled in two different places and have **opposite statistical standing**. -**Technical replicates** are one biological sample measured more than once — two lanes, two libraries, a second sequencing run for validation. They are not independent, and treating them as though they were is pseudo-replication. PoolSeqFlow supports both ways of handling them, and **either is a valid choice**: +**Technical replicates** are one biological sample measured more than once: two lanes, two libraries, a second sequencing run for validation. They are not independent, and treating them as though they were is pseudo-replication. PoolSeqFlow supports both ways of handling them, and **either is a valid choice**: -- **Merge them.** Give the rows the same `RG_Sample`. The pipeline pools their reads and adds their depths, and they become one column of every published table. Anything that differs between those rows — the lane, the run — then goes in an unprefixed column, because it is no longer an experimental variable for that pool. +- **Merge them.** Give the rows the same `RG_Sample`. The pipeline pools their reads and adds their depths, and they become one column of every published table. Anything that differs between those rows (the lane, the run) then goes in an unprefixed column, because it is no longer an experimental variable for that pool. - **Keep them apart.** Give the rows different `RG_Sample` values, so they stay distinct pools, and an `exp_lane` column that tells them apart is then a perfectly good experimental variable. Name it in `technicalRep`. -**Biological replicates** are independent repeats of the same condition — two cages of flies under one selection regime. They carry drift, they are the strata a test conditions on, and they are where degrees of freedom come from. Give them an `exp_` column and name it in `biologicalRep`. +**Biological replicates** are independent repeats of the same condition: two cages of flies under one selection regime. They carry drift, they are the strata a test conditions on, and they are where degrees of freedom come from. Give them an `exp_` column and name it in `biologicalRep`. **Both settings take any number of columns.** Two lanes crossed with three sequencing runs is `technicalRep = ['exp_lane', 'exp_seqrun']` and six technical replicates per unit; nested and crossed designs both work, because forming the unit drops all of those columns either way. @@ -3527,7 +3699,7 @@ That is the whole rule, and it cuts both ways: - **Three control pools with nothing but `exp_treatment` in the file are three units**, not one. They are three separate RG_Samples and you did not say otherwise. - **Three cages sequenced on two lanes each, with `technicalRep = ['exp_lane']`, are three units** from six pools. Leave the lane out of `technicalRep` and they are six, which roughly halves every standard error in the project. -`biologicalRep` and `technicalRep` both default to empty, so a project that declares nothing gets one unit per pool. **That is not a claim that your pools are independent — it is the absence of a claim that they are not**, and a module whose test needs declared biological replication refuses by name rather than inventing it. +`biologicalRep` and `technicalRep` both default to empty, so a project that declares nothing gets one unit per pool. **That is not a claim that your pools are independent; it is the absence of a claim that they are not**, and a module whose test needs declared biological replication refuses by name rather than inventing it. Once `technicalRep` **is** declared it has to resolve the pools it applies to. A group where some pools are told apart by the technical column and others are not is neither one unit nor several, and that refuses, naming the pools and offering the same remedies as above. @@ -3535,7 +3707,7 @@ Once `technicalRep` **is** declared it has to resolve the pools it applies to. A `covariates` names the `cov_` columns a module may **adjust for**, as opposed to only report. Left empty it is every `cov_` column that has a [declared scale](#covariates); naming any restricts it to those, and a column named here without a scale refuses, because a column with no scale has no value a model could take. -**Set it when you record covariates you do not want in any model.** A collection site kept for provenance is not a term anybody wants fitted, and each covariate that *is* in the design costs a degree of freedom — with six pools there are four to spend. +**Set it when you record covariates you do not want in any model.** A collection site kept for provenance is not a term anybody wants fitted, and each covariate that *is* in the design costs a degree of freedom; with six pools there are four to spend. Every covariate is resolved, reported and published either way. The verification report marks each one, and a covariate you left out is named in the design notes, because leaving one out is as much a decision as putting one in and neither is visible from the values: @@ -3558,15 +3730,15 @@ REPLICATION: 2 conditions, 3 biological replicates each, 6 technic REPLICATION: 6 independent units from 144 pools ``` -Read those three lines. A technical column left out of `technicalRep` is read as a **condition** — one treatment silently becomes several, and a test is handed strata that are the same DNA. Nothing can detect that, for the same reason nothing can detect `dd/MM` against `MM/dd`: both readings are internally consistent. Printing the partition is the whole defense. +Read those three lines. A technical column left out of `technicalRep` is read as a **condition**. One treatment silently becomes several, and a test is handed strata that are the same DNA. Nothing can detect that, for the same reason nothing can detect `dd/MM` against `MM/dd`: both readings are internally consistent. Printing the partition is the whole defense. The counts are per unit and given as a range when they vary, because a design where one sample was sequenced twice for validation and the rest once is perfectly ordinary and a single number would be a plausible-looking lie. -**And a design PoolSeqFlow cannot see through:** three cages sequenced on two lanes each, with the lanes neither merged nor declared technical but labeled `exp_replicate = 1..6`, has three independent units and claims six. The frame reports what you declared. The line `6 pools from 6 libraries` — where a merged design would say `from 12` — is the number to check. +**And a design PoolSeqFlow cannot see through:** three cages sequenced on two lanes each, with the lanes neither merged nor declared technical but labeled `exp_replicate = 1..6`, has three independent units and claims six. The frame reports what you declared. The line `6 pools from 6 libraries` (where a merged design would say `from 12`) is the number to check. ### Time series { #time-series } -A trajectory needs more than an order: it needs to know which pools are **one thing measured repeatedly**. A series is the pools that share every identifying variable and differ only in time — the same key the [experimental design](#experimental-design) is built from, followed through the [time axis](#the-time-axis). +A trajectory needs more than an order: it needs to know which pools are **one thing measured repeatedly**. A series is the pools that share every identifying variable and differ only in time. That is the same key the [experimental design](#experimental-design) is built from, followed through the [time axis](#the-time-axis). Where a project has no time axis there are no series, and the units and conditions above are unaffected. The only setting a series adds is what to do when one has a gap in it, and it lives **inside** the design scope, because a series is what a time axis makes of the design rather than a thing of its own. @@ -3620,7 +3792,7 @@ keepLeft └───────┘ -> g0 g5 (discards g15 g20, keepRight └───┘ -> g15 g20 (discards g0 g5, which B has) ``` -That last one is the case worth thinking about: **both work and they keep different data.** The choice is early drift against late response — a scientific one, not a mechanical one, and picking the wrong one silently answers a different question. +That last one is the case worth thinking about: **both work and they keep different data.** The choice is early drift against late response. It is a scientific one, not a mechanical one, and picking the wrong one silently answers a different question. If a truncation leaves a **single** timepoint, that is reported loudly: every series then has one measurement and there is no time axis at all. `drop` is usually what such a design wanted. @@ -3633,9 +3805,9 @@ Four things are structural rather than preferences, and no setting turns them of - **Two pools with the same key at the same timepoint.** The series is then not a function of time. What to do depends on which kind they are: give them an `exp_` column and declare it in [`biologicalRep` or `technicalRep`](#replicates) if they are genuinely separate measurements, or give the rows the same `RG_Sample` if they are one pool sequenced twice that you meant to merge. - **`by` naming the time column**, a column that is not an `exp_` column, or one your metadata does not have. - **Two series that would carry the same label.** -- **Two spellings of one point in time** — `5 December 2011` and `05 December 2011`, or `1` and `1.0`. Which level a pool belongs to would have no answer. +- **Two spellings of one point in time**: `5 December 2011` and `05 December 2011`, or `1` and `1.0`. Which level a pool belongs to would have no answer. -A pool whose `exp_time` cell is **blank** joins no series. It is counted and named in the report, and whether that matters is the module's to say — a project can legitimately hold one pool that was sampled once. +A pool whose `exp_time` cell is **blank** joins no series. It is counted and named in the report, and whether that matters is the module's to say: a project can legitimately hold one pool that was sampled once. ### A module's own settings { #module-settings } @@ -3653,13 +3825,13 @@ params { } ``` -What a module has is listed in its own section of this manual, and the defaults are the module's own. **A key it does not have is refused, naming what it does have** — so `chromosome` for `chromosomes` stops the run rather than leaving the module quietly running on a default you did not choose. The verification report echoes every setting you did write, as you wrote it. +What a module has is listed in its own section of this manual, and the defaults are the module's own. **A key it does not have is refused, naming what it does have**, so `chromosome` for `chromosomes` stops the run rather than leaving the module quietly running on a default you did not choose. The verification report echoes every setting you did write, as you wrote it. -Inside `analysis`, always. A scope written at the top level of a configuration file becomes part of the record this project is checked against, and every analysis would then refuse with a message about a changed project rather than about the file you just wrote — so that mistake is caught and named where it happens. +Inside `analysis`, always. A scope written at the top level of a configuration file becomes part of the record this project is checked against, and every analysis would then refuse with a message about a changed project rather than about the file you just wrote. That mistake is therefore caught and named where it happens. ### Phenotypes { #the-phenotype } -A `pt_` column records something **measured on the pool** — a trait value, a resistance score, a case/control status. No pipeline step reads one. Declare a scale for as many as you record, one block per column: +A `pt_` column records something **measured on the pool**: a trait value, a resistance score, a case/control status. No pipeline step reads one. Declare a scale for as many as you record, one block per column: ```groovy params { @@ -3674,39 +3846,39 @@ params { } ``` -**Declaring a phenotype does not choose it.** This block says what each column *holds*; which one an analysis tests against is the module's own setting, listed in that module's section. So a project with three phenotypes writes one declaration block and runs three analyses from it — set [`folderName`](#analysis-folder-name) so each lands somewhere of its own. It is the same split [covariates](#covariates) use, and the same one that separates [`analysis.metadata`](#analysis-configuration) from `analysis.design`: what a column holds is one question, what it is *for* is another. +**Declaring a phenotype does not choose it.** This block says what each column *holds*; which one an analysis tests against is the module's own setting, listed in that module's section. So a project with three phenotypes writes one declaration block and runs three analyses from it. Set [`folderName`](#analysis-folder-name) so each lands somewhere of its own. It is the same split [covariates](#covariates) use, and the same one that separates [`analysis.metadata`](#analysis-configuration) from `analysis.design`: what a column holds is one question, what it is *for* is another. **A `pt_` column you do not declare is not an error.** It is recorded, checked and reported like any pool-level column; it simply carries no scale, so no module can test against it. The report names the columns in that state, because "kept for the record" and "forgot to declare it" look identical in the file. -**It has to be a `pt_` column, and that is a rule rather than a convention.** `pt_` columns are checked for agreeing across the rows of one pool, exactly as `exp_` ones are — so declaring any other column would let one pool carry two phenotype values with nothing to stop it. Anything outside the prefix is refused by name. +**It has to be a `pt_` column, and that is a rule rather than a convention.** `pt_` columns are checked for agreeing across the rows of one pool, exactly as `exp_` ones are, so declaring any other column would let one pool carry two phenotype values with nothing to stop it. Anything outside the prefix is refused by name. #### `kind` is a measurement scale, declared and never detected { #phenotype-kind } | `kind` | What the column holds | `levels` | What a module may fit | |---|---|---|---| | `quantitative` | a measurement. Every value must parse as a number | refused | a slope, signed | -| `binary` | **presence and absence** — affected or not. Exactly two levels | **required** | a signed effect, and case/control methods | +| `binary` | **presence and absence** (affected or not). Exactly two levels | **required** | a signed effect, and case/control methods | | `ordinal` | groups whose **order** means something and whose spacing does not | **required** | a trend across the order | | `nominal` | groups with **no order** | **required** | a comparison between groups, and no trend | Nothing is inferred, for the same reason `analysis.metadata.timeVar.kind` is not: `0` and `1` read as numbers as readily as they encode two groups, and which of `case` and `control` you mean by *affected* is not in your file at all. -**`binary` is not "the two-group case".** It is a claim that one level is the *absence* of the other — affected or unaffected, resistant or susceptible — which is what makes a design a case/control design rather than a comparison of two arbitrary groups. Two groups that are neither, such as coastal and inland or two host plants, are **`nominal` with two levels**. The published result records which you declared. +**`binary` is not "the two-group case".** It is a claim that one level is the *absence* of the other (affected or unaffected, resistant or susceptible), which is what makes a design a case/control design rather than a comparison of two arbitrary groups. Two groups that are neither, such as coastal and inland or two host plants, are **`nominal` with two levels**. The published result records which you declared. **`ordinal` and `nominal` are two kinds because they permit different things.** Scoring low/medium/high as 0/1/2 and fitting a slope asserts that low→medium is the same distance as medium→high, which an ordinal scale does not claim. Treating it as unordered instead throws the ordering away and loses the power to see a trend at all. So you say which you have, and the module fits accordingly. -For a `nominal` phenotype the frame carries **no number at all** for each pool — only which group it is in. That is deliberate, and it is the same mechanism [categorical time](#the-time-axis) uses when it sets `position` to null: it makes *"you may not fit a rate on this"* something a module can check rather than something its author has to remember. Wing types `spotted`, `striped` and `curly` have an index each, and a slope fitted on that index would assert curly is twice as far from spotted as striped is. +For a `nominal` phenotype the frame carries **no number at all** for each pool, only which group it is in. That is deliberate, and it is the same mechanism [categorical time](#the-time-axis) uses when it sets `position` to null: it makes *"you may not fit a rate on this"* something a module can check rather than something its author has to remember. Wing types `spotted`, `striped` and `curly` have an index each, and a slope fitted on that index would assert curly is twice as far from spotted as striped is. #### Counts, percentages, and what is not a phenotype kind { #phenotype-not-kinds } -**Counts and percentages are `quantitative`.** A phenotype is a *predictor*, and a predictor carries no distributional assumption — that a bristle count is really Poisson changes nothing here. +**Counts and percentages are `quantitative`.** A phenotype is a *predictor*, and a predictor carries no distributional assumption; that a bristle count is really Poisson changes nothing here. -**A trait that varies WITHIN a pool is not a nominal phenotype, it is a composition.** If a pool is 30% spotted, 50% striped and 20% curly, its phenotype is not "striped" — labeling it by the majority morph discards the other 70%. Record it as one quantitative column per morph, `pt_spotted = 0.3` and so on, and analyze them one at a time. **`nominal` is correct only when each pool was built homogeneous for the trait**, which is a real design but a different one. +**A trait that varies WITHIN a pool is not a nominal phenotype, it is a composition.** If a pool is 30% spotted, 50% striped and 20% curly, its phenotype is not "striped"; labeling it by the majority morph discards the other 70%. Record it as one quantitative column per morph, `pt_spotted = 0.3` and so on, and analyze them one at a time. **`nominal` is correct only when each pool was built homogeneous for the trait**, which is a real design but a different one. **Two kinds that are real and are not supported**, named here so that nobody declares one by accident: -- **Circular** measurements — time of day of eclosion, a direction. 359° and 1° are two degrees apart, and any linear fit on them is simply wrong. Nothing in the values reveals it, so this cannot be refused for you: do not declare a circular phenotype `quantitative`. -- **Censored** measurements — time to death where some individuals survived. This needs survival analysis, which no module here does. +- **Circular** measurements: time of day of eclosion, a direction. 359° and 1° are two degrees apart, and any linear fit on them is simply wrong. Nothing in the values reveals it, so this cannot be refused for you: do not declare a circular phenotype `quantitative`. +- **Censored** measurements: time to death where some individuals survived. This needs survival analysis, which no module here does. #### `levels` decides the sign of every result { #phenotype-levels } @@ -3722,7 +3894,7 @@ phenotype { **Write those the wrong way round and every effect reported against the phenotype has the opposite sign.** `['affected', 'unaffected']` is just as legal, produces no error, and inverts the entire result. Nothing in your data says which you meant. -For `ordinal` the order **is** the scale, so it carries the same weight. For `nominal` it only sets which group the others are reported against, and a comparison between groups does not depend on that choice — which is the one case here where getting the order wrong costs you nothing. +For `ordinal` the order **is** the scale, so it carries the same weight. For `nominal` it only sets which group the others are reported against, and a comparison between groups does not depend on that choice. That is the one case here where getting the order wrong costs you nothing. **A level no pool has is kept, not refused.** A group you have not sequenced yet is legitimate, so the run continues and the report names the unused level. A misspelling looks exactly the same, which is why that line is worth reading. @@ -3742,13 +3914,13 @@ Read those two lines before you read any result. They are where a reversed encod **If every pool has the same value** there is nothing to associate anything with, and the report says so. A module that fits against it refuses rather than reporting an effect of zero as a finding. -**A group holding one pool is reported too.** It contributes no within-group variance, so any comparison against it rests on that single pool — legitimate, occasionally unavoidable, and not something a table of group means shows you. +**A group holding one pool is reported too.** It contributes no within-group variance, so any comparison against it rests on that single pool. That is legitimate, occasionally unavoidable, and not something a table of group means shows you. Record and declare as many `pt_` columns as you like. **What a module does with them is its own setting**, and a module that fits one phenotype at a time is run once per phenotype with [`analysis.folderName`](#analysis-folder-name) set, so each result lands in a folder of its own and says which phenotype produced it. A folder holding three answers under one name is a folder nobody can cite. ### Covariates { #covariates } -A `cov_` column records something **measured on the pool that you neither set nor are testing against** — a cage temperature, an altitude, a collection site, a technician when every library of a pool shares one. It is the third pool-level prefix and it exists for one mechanical reason: an `exp_` column says what the experiment set up, so a temperature recorded on each pool would make every pool its own unit and leave every series a single point long. A `cov_` column never identifies a unit or a series, so adding one cannot do that. +A `cov_` column records something **measured on the pool that you neither set nor are testing against** (a cage temperature, an altitude, a collection site, a technician when every library of a pool shares one). It is the third pool-level prefix and it exists for one mechanical reason: an `exp_` column says what the experiment set up, so a temperature recorded on each pool would make every pool its own unit and leave every series a single point long. A `cov_` column never identifies a unit or a series, so adding one cannot do that. **Declaring a covariate is optional.** Undeclared, it is recorded, checked for agreeing across the rows of its pool, and printed in the verification report like any other pool-level column. Declaring it gives it a **scale**, so a module can compute with it: @@ -3770,11 +3942,11 @@ params { } ``` -The kinds are the phenotype's — `quantitative`, `binary`, `ordinal`, `nominal` — and they mean [the same things](#phenotype-kind), including that a `nominal` covariate carries no number a module could fit a slope on. The report names any `cov_` column you recorded and did not declare, because "kept for the record" and "forgot to declare it" look identical in the file. +The kinds are the phenotype's (`quantitative`, `binary`, `ordinal`, `nominal`) and they mean [the same things](#phenotype-kind), including that a `nominal` covariate carries no number a module could fit a slope on. The report names any `cov_` column you recorded and did not declare, because "kept for the record" and "forgot to declare it" look identical in the file. #### What adjusting for one costs, and why it is a decision { #covariates-not-adjusted } -**The frame never adjusts for anything.** It resolves each covariate, says which are [part of the design](#design-covariates), and publishes both. Whether a module puts a covariate in its model is that module's business, declared in its own section — and no module is published without saying. +**The frame never adjusts for anything.** It resolves each covariate, says which are [part of the design](#design-covariates), and publishes both. Whether a module puts a covariate in its model is that module's business, declared in its own section. And no module is published without saying. **The arithmetic is why it has to be your decision.** *n* is the number of **pools**, typically six to twenty, so every covariate a model fits is a degree of freedom the effect you came for does not get. At six pools a comparison starts with four; a single covariate makes it three; a repeated-measures design of three units has one left before any covariate at all. @@ -3789,7 +3961,7 @@ If a covariate turns out to explain your result, the answer is usually a better ```groovy params { analysis { - runs = 'all' // every run in the project — the default + runs = 'all' // every run in the project, the default // runs = 'lenient' // one, by the RunID you gave it // runs = ['lenient', 'strict'] // several } @@ -3798,7 +3970,7 @@ params { `all` is a keyword and a list is always run names, so a run you have actually named `all` is selected by writing `['all']`. -A single-run project has no run names — there is no run table — so `analysis.runs` must be left at `'all'`. Setting anything else is refused rather than ignored, because a name that selects nothing would otherwise look like it had worked. +A single-run project has no run names (there is no run table), so `analysis.runs` must be left at `'all'`. Setting anything else is refused rather than ignored, because a name that selects nothing would otherwise look like it had worked. #### A run selects a directory, not a run @@ -3816,7 +3988,7 @@ RUN SELECTION: here and one analysis of it Nothing is wrong there: `lenient_a` and `lenient_b` differ only in something neither the tables nor the filters depend on, so the pipeline produced one set and both runs own it. Naming both would produce the same single analysis. The report says so every time rather than leaving you to infer it from the directory name. -Which directory a run's results are in is not guessed from directory names — the analysis layer rebuilds the pipeline's own division of the runs and reads the answer off it, so the two can never disagree. +Which directory a run's results are in is not guessed from directory names: the analysis layer rebuilds the pipeline's own division of the runs and reads the answer off it, so the two can never disagree. ## Verification @@ -3825,7 +3997,7 @@ Every module begins by checking the project against the record the pipeline left ### What it refuses -The results on disk were produced under a configuration, and the pipeline recorded that configuration beside them. A module reads the tables and the settings **together** — pool sizes, ploidy and the filter thresholds all decide what a frequency means — so a project that has moved on since is refused, naming what moved: +The results on disk were produced under a configuration, and the pipeline recorded that configuration beside them. A module reads the tables and the settings **together** (pool sizes, ploidy and the filter thresholds all decide what a frequency means), so a project that has moved on since is refused, naming what moved: - **`parameters.config` has changed.** The report lists each parameter, what was recorded and what it says now, and points at `run_parameters.txt` beside your results for the values that produced them. Paths and resources are not compared; they cannot change a number. - **The run table has changed.** Which runs share a directory is decided by that table, so an edit can move a run onto results belonging to somebody else. @@ -3842,14 +4014,14 @@ POOL SIZES: Output POOL SIZES: ploidy 2, 6 pools of 100 individuals - 200 chromosomes, frequencies above 0.0025 ``` -Those are the pipeline's own `poolSize` and `ploidy`, per pool: a pool whose rows set `param_poolSize` holds that many individuals and every other pool holds `poolSize` of them. The chromosome count is `ploidy × poolSize`, and the detection limit is `1 / (2 × ploidy × poolSize)` — the frequency below which step 7's false-positive filter took a call to be error rather than a rare allele, explained in [The Filter Chain](#where-s-comes-from). Pools of one size share a line; a pool of its own size gets one, because its chromosome count and its detection limit both move with it. +Those are the pipeline's own `poolSize` and `ploidy`, per pool: a pool whose rows set `param_poolSize` holds that many individuals and every other pool holds `poolSize` of them. The chromosome count is `ploidy × poolSize`, and the detection limit is `1 / (2 × ploidy × poolSize)`. That is the frequency below which step 7's false-positive filter took a call to be error rather than a rare allele, explained in [The Filter Chain](#where-s-comes-from). Pools of one size share a line; a pool of its own size gets one, because its chromosome count and its detection limit both move with it. **Runs that hold a pool to different sizes never share a results directory.** `poolSize`, `param_poolSize` and `ploidy` are all part of what decides whether two runs share step 7's work ([Multi-run](#multi-run)), so each run's tables are filtered against its own sizes and land under its own name. The analysis layer checks the sizes across a directory's runs regardless, and stops if it ever finds two: a directory whose tables were filtered against two different pool sizes has no answer to what a frequency in it means. ## Output Layout -Everything the analysis layer produces goes under `Analysis/` — on `mainDir` while you are working, and on `storageDir` once [`complete`](#analysis-complete) has moved it, at the same relative path under either. Nothing it does touches `Output/`: +Everything the analysis layer produces goes under `Analysis/`. That is on `mainDir` while you are working, and on `storageDir` once [`complete`](#analysis-complete) has moved it, at the same relative path under either. Nothing it does touches `Output/`: | Path | Holds | `complete` | |---|---|---| @@ -3861,15 +4033,15 @@ Everything the analysis layer produces goes under `Analysis/` — on `mainDir` w `Analysis/Session/` exists so that an analysis run does not overwrite the four session files in `Output/Reports`, which are the record of the pipeline run that produced the results being read. -`Analysis/Main` is where a module puts anything it had to derive — a per-position depth file, a frequency matrix, a callable-sites count. Derivations are shared: a second module wanting the same thing finds it already there rather than building it again. It stays empty until you install a module that derives something, and it can then grow large — a full per-position depth file is measured in gigabytes. +`Analysis/Main` is where a module puts anything it had to derive (a per-position depth file, a frequency matrix, a callable-sites count). Derivations are shared: a second module wanting the same thing finds it already there rather than building it again. It stays empty until you install a module that derives something, and it can then grow large. A full per-position depth file is measured in gigabytes. `Analysis/Results` holds one folder per analysis, named by [`folderName`](#analysis-folder-name). ### The script that produced it { #analysis-script } -**Every published analysis carries the script that produced it.** That is a guarantee of the layer rather than a convention module authors are asked to follow: an analysis handed over without one is refused, nothing is published, and the folder is left exactly as it was. So a result you find in `Analysis/Results` can always be regenerated, and the folder also holds the verification record that cleared it — which names the module, its version, the runs it covered and the configuration it was assembled from. +**Every published analysis carries the script that produced it.** That is a guarantee of the layer rather than a convention module authors are asked to follow: an analysis handed over without one is refused, nothing is published, and the folder is left exactly as it was. So a result you find in `Analysis/Results` can always be regenerated, and the folder also holds the verification record that cleared it, which names the module, its version, the runs it covered and the configuration it was assembled from. -**It carries the libraries it used, folded in.** A module declares the libraries it needs — effective sample size, gene diversity, the harmonic means — and each is installed alongside it; what is published is those functions and the module's own code in one file, headed by the frame version it ran against. A driver that merely `source()`s code the reader does not have would satisfy the letter of the guarantee and none of its point — and a library lives in the module store, which you can uninstall from. The file in your results is the copy that ran, whatever the store holds afterward. +**It carries the libraries it used, folded in.** A module declares the libraries it needs (effective sample size, gene diversity, the harmonic means), and each is installed alongside it; what is published is those functions and the module's own code in one file, headed by the frame version it ran against. A driver that merely `source()`s code the reader does not have would satisfy the letter of the guarantee and none of its point. And a library lives in the module store, which you can uninstall from. The file in your results is the copy that ran, whatever the store holds afterward. **And a `README.md` that says how to read what is in the folder.** { #analysis-readme } Every module declares, for each file it publishes, the section of this manual that explains it; the frame renders one table from those declarations and from what every analysis carries anyway: @@ -3883,17 +4055,17 @@ Everything the analysis layer produces goes under `Analysis/` — on `mainDir` w It links into the copy of this manual installed with the release that produced the folder, so the explanation you follow is the one that was true at the time. The point is the numbers a table cannot explain for itself: a figure that is a bound rather than a measurement, or an estimate carrying an assumption, reads as neither unless something beside it says so. -A module that declares a file and then does not publish it is refused, and a module whose declared section does not exist in the manual is refused before it starts — a link nobody can follow being worse than none. +A module that declares a file and then does not publish it is refused, and a module whose declared section does not exist in the manual is refused before it starts (a link nobody can follow being worse than none). ### One PDF of the whole analysis { #analysis-report } -**Every published folder also carries `report.pdf`: every result in it, in one document, each under the name of the file it came from.** Tables are rendered as tables, figures as figures, and the heading above each is the file name — so a number you want to work with can always be traced back to the file that holds it. +**Every published folder also carries `report.pdf`: every result in it, in one document, each under the name of the file it came from.** Tables are rendered as tables, figures as figures, and the heading above each is the file name, so a number you want to work with can always be traced back to the file that holds it. It is a **capability of the layer, not of any module**. A module already declares what it publishes and what each file is, because the `README.md` is built from those declarations; the report is built from the same ones. So a module gets a report without writing a line of code for it, and so does a module somebody else wrote. What it is not is a substitute for the files. A table longer than forty rows is shown to forty and says so, and the figures are the same images that sit beside it. **Everything in the report is in the folder; the folder is what you compute from.** -**A report that cannot be built does not stop the analysis being published.** The reason is printed and the folder is complete without it — every number was already a file. That is the opposite of how the other publishing checks behave, and deliberately so: a missing script or an undeclared output means the analysis is not what it claims to be, where a missing report means only that a convenience is absent. +**A report that cannot be built does not stop the analysis being published.** The reason is printed and the folder is complete without it; every number was already a file. That is the opposite of how the other publishing checks behave, and deliberately so: a missing script or an undeclared output means the analysis is not what it claims to be, where a missing report means only that a convenience is absent. Building it needs `pandoc` and `typst`, both pinned in the analysis environment. If you are running a module's script by hand rather than through `PoolSeqFlow analysis`, neither will be on your path and no report is written. @@ -3904,7 +4076,7 @@ Building it needs `pandoc` and `typst`, both pinned in the analysis environment. ```groovy params { analysis { - folderName = '' // the module's own name — mds writes to Results/mds + folderName = '' // the module's own name; mds writes to Results/mds // folderName = 'sweep_strict' // a name of your own // folderName = 'MDS/SummerPops' // a path, so related analyses group together } @@ -3913,38 +4085,38 @@ params { Set it in `.config` instead of `analysis.config` to have it apply to one module. -Under `multiRun`, each results directory gets its own folder inside the one you named — `Results/sweep/Shared_1`, `Results/sweep/strict` — using the names the pipeline gave them. A single run has no name anywhere, so its analysis sits in the folder directly. +Under `multiRun`, each results directory gets its own folder inside the one you named (`Results/sweep/Shared_1`, `Results/sweep/strict`), using the names the pipeline gave them. A single run has no name anywhere, so its analysis sits in the folder directly. **Naming the folder is how two settings of one module are told apart.** Nothing is stamped with a date, nothing is versioned, and nothing is overwritten: a folder that already holds an analysis is **refused**, naming what is in it. To run the same module again with different settings, give the new one a folder of its own; to redo an analysis, move the old one out of the way first. -The one thing allowed to be in the folder already is the verification record from a previous attempt. A module that fails after the check leaves the folder holding nothing else, and that retry has to be allowed — otherwise a single failure would make the name unusable. +The one thing allowed to be in the folder already is the verification record from a previous attempt. A module that fails after the check leaves the folder holding nothing else, and that retry has to be allowed; otherwise a single failure would make the name unusable. ### Moving finished analyses to permanent storage { #analysis-complete } -`PoolSeqFlow analysis complete` moves `Analysis/Main` and `Analysis/Results` from your working directory to `storageDir/Analysis/`, keeping the same relative path so that a module can still find an intermediate afterwards — it brings back what it needs on its own, a file at a time, and says so when it does. `Analysis/Logs`, `Analysis/Session` and `Analysis/work` stay where they are: they describe invocations rather than results, and the next analysis rewrites `Session` and appends to `Logs`. +`PoolSeqFlow analysis complete` moves `Analysis/Main` and `Analysis/Results` from your working directory to `storageDir/Analysis/`, keeping the same relative path so that a module can still find an intermediate afterwards; the module brings back what it needs on its own, a file at a time, and says so when it does. `Analysis/Logs`, `Analysis/Session` and `Analysis/work` stay where they are: they describe invocations rather than results, and the next analysis rewrites `Session` and appends to `Logs`. -It asks before it starts, and everything is surveyed before anything moves. **A folder name already taken in permanent storage stops the command with nothing moved** — two different analyses under one name is exactly what naming a folder exists to prevent. Move the older one out of the way, or give the new one a name of its own, and run it again. +It asks before it starts, and everything is surveyed before anything moves. **A folder name already taken in permanent storage stops the command with nothing moved**. Two different analyses under one name is exactly what naming a folder exists to prevent. Move the older one out of the way, or give the new one a name of its own, and run it again. A folder holding nothing but a verification record is passed over and said so, because that is what a failed attempt leaves and archiving it would consume the name the retry needs. -Each item moves on its own, so an interrupted run can simply be run again: what already moved is left alone. There is no command that brings a results folder back to the working directory — move it by hand if you need it there. +Each item moves on its own, so an interrupted run can simply be run again: what already moved is left alone. There is no command that brings a results folder back to the working directory. Move it by hand if you need it there. #### Working, archiving, working again { #analysis-resume } You can keep using a project after `complete` has archived it. Run a module and it finds the intermediates it needs in permanent storage and **copies** them back to your working directory, a named file at a time. They exist in both places from then on, deliberately: the copy in permanent storage is what makes the next `complete` cheap. -That is what the second `complete` is for. An intermediate already in permanent storage is not moved a second time — **the working copy is discarded, and the run says which ones and why.** What decides that is the provenance record beside each intermediate: a few hundred bytes naming the results it was derived from. When the two records agree the two files are the same file, and only one of them needs keeping. +That is what the second `complete` is for. An intermediate already in permanent storage is not moved a second time. **The working copy is discarded, and the run says which ones and why.** What decides that is the provenance record beside each intermediate: a few hundred bytes naming the results it was derived from. When the two records agree the two files are the same file, and only one of them needs keeping. -When they do **not** agree, the command stops and moves nothing. One of the two was derived from results this project no longer holds, and an analysis you have already published may have come from either — so neither is removed for you. Delete the one you do not want and run it again. The same happens if either copy has lost its provenance record: without it there is no way to tell the two apart, and guessing is not an option the command takes. +When they do **not** agree, the command stops and moves nothing. One of the two was derived from results this project no longer holds, and an analysis you have already published may have come from either, so neither is removed for you. Delete the one you do not want and run it again. The same happens if either copy has lost its provenance record: without it there is no way to tell the two apart, and guessing is not an option the command takes. -The cost of a working cycle is therefore one transfer, not two — which matters when `storageDir` is billed per operation, as on S3 or a tape-backed archive. What it costs instead is disk on the working volume, and that is what the discard reclaims. +The cost of a working cycle is therefore one transfer, not two, which matters when `storageDir` is billed per operation, as on S3 or a tape-backed archive. What it costs instead is disk on the working volume, and that is what the discard reclaims. **Do not run it while a module is running.** The two would be moving the same folders in opposite directions, and neither checks for the other. # Modules -**Every module is installed separately** — none comes with the pipeline, and a fresh installation has an empty store. These are the ones published alongside this release, each on its own timetable and at its own version, installed with `PoolSeqFlow analysis modules install ` and documented here so you can read what one computes before deciding to install it. +**Every module is installed separately**: none comes with the pipeline, and a fresh installation has an empty store. These are the ones published alongside this release, each on its own timetable and at its own version, installed with `PoolSeqFlow analysis modules install ` and documented here so you can read what one computes before deciding to install it. | Module | What it computes | Needs | |---|---|---| @@ -3969,7 +4141,7 @@ Each has a page of its own here. What comes first is what they have in common. Every module here reads the same thing: one comma-separated list of read counts per pool per site, in the depth tables. On a genome running to tens of millions of called sites, splitting those strings is most of the time a module spends, and for `basicstats` and `association` it is essentially all of it. So each of the derivations that does per-site work has two implementations, and **the compiled one is the default**. Both produce the same numbers; the choice is only ever about time. -Measured on an Intel i7-7700HQ at 2.8 GHz with R 4.6.1 and GCC 16.2, over a corpus of 3.2 million sites — 92% biallelic, 7% triallelic, 1% tetrallelic, which is what a called cohort looks like. CPU time, scaled to what 100 million called sites would cost in one pass: +Measured on an Intel i7-7700HQ at 2.8 GHz with R 4.6.1 and GCC 16.2, over a corpus of 3.2 million sites (92% biallelic, 7% triallelic, 1% tetrallelic, which is what a called cohort looks like). CPU time, scaled to what 100 million called sites would cost in one pass: | Derivation | Reads | Vectorized R | Compiled | Ratio | |---|---|---|---|---| @@ -3978,9 +4150,9 @@ Measured on an Intel i7-7700HQ at 2.8 GHz with R 4.6.1 and GCC 16.2, over a corp | `allele_frequencies` | six pools | 675 s | 69 s | 10× | | `nei_distance` | six pools | 710 s | 22 s | 32× | -**The last row is a different kind of work and its ratio should not be averaged with the others.** The three parsers split strings, and what limits them is memory traffic; `nei_distance` reads what a parser already produced and does arithmetic on it. Its vectorized form makes one pass per *pair* of pools — fifteen at six pools — so what compiling removes is interpreted call overhead that grows with the square of the pool count, not with the site count. Expect its ratio to climb with more pools where the parsers' will not. It is also the reason `mds` is the one module whose statistic costs about as much as reading the table for it. +**The last row is a different kind of work and its ratio should not be averaged with the others.** The three parsers split strings, and what limits them is memory traffic; `nei_distance` reads what a parser already produced and does arithmetic on it. Its vectorized form makes one pass per *pair* of pools (fifteen at six pools), so what compiling removes is interpreted call overhead that grows with the square of the pool count, not with the site count. Expect its ratio to climb with more pools where the parsers' will not. It is also the reason `mds` is the one module whose statistic costs about as much as reading the table for it. -**For the three parsers, take the ratio as about ten, not as a precise figure.** It is not constant, and it is not constant in a direction worth knowing about: on a corpus small enough to sit in cache the compiled path looks two to three times better than this, and below about ten thousand sites the whole call is shorter than the clock can resolve, so a figure extrapolated from a small test corpus will flatter it by a wide margin. At three million sites one pool's cells are already 48 MB — past any current L3 — and both implementations spend their time waiting on memory rather than on arithmetic. That is the regime a genome is in. A machine with more memory bandwidth than a 2017 laptop does better than this table on both columns. +**For the three parsers, take the ratio as about ten, not as a precise figure.** It is not constant, and it is not constant in a direction worth knowing about: on a corpus small enough to sit in cache the compiled path looks two to three times better than this, and below about ten thousand sites the whole call is shorter than the clock can resolve, so a figure extrapolated from a small test corpus will flatter it by a wide margin. At three million sites one pool's cells are already 48 MB (past any current L3), and both implementations spend their time waiting on memory rather than on arithmetic. That is the regime a genome is in. A machine with more memory bandwidth than a 2017 laptop does better than this table on both columns. What the table is really for is the decision it supports. For `association`, reading a hundred million sites across six pools is **eleven minutes of splitting strings against one**. For `mds`, which pays for the parse and the distance both, it is **twenty-three minutes against a minute and a half**. That is why the compiled path is the default rather than something to ask for, and why a module offering it stops rather than quietly falling back when it cannot build one. @@ -3992,7 +4164,7 @@ Each module publishes its compiled source into the results folder whether or not #### When the compiled path will not build { #analysis-compiled-fails } -Having a compiler and being able to *use* it are two things. `Rcpp::sourceCpp` writes a source file, builds it, and then **executes** the result out of a temporary directory — so a machine that forbids executing from temporary storage fails here even with the toolchain installed. The usual causes: +Having a compiler and being able to *use* it are two things. `Rcpp::sourceCpp` writes a source file, builds it, and then **executes** the result out of a temporary directory, so a machine that forbids executing from temporary storage fails here even with the toolchain installed. The usual causes: | What you see | Why | |---|---| @@ -4000,7 +4172,7 @@ Having a compiler and being able to *use* it are two things. `Rcpp::sourceCpp` w | `cannot open shared object file`, or a build that succeeds and then fails to load | `/tmp` mounted `noexec`, which is common on hardened clusters. Point `TMPDIR` at a filesystem you may execute from | | a permission error while compiling | a read-only home, or a read-only conda environment shared across users | -**The run stops and says so; it never quietly drops to the other path.** A run that silently took a different implementation is a run whose timings mean nothing, and you would have no way to know which produced your numbers. Add `nocpp` after the module name and it will finish — slower, and with identical output. +**The run stops and says so; it never quietly drops to the other path.** A run that silently took a different implementation is a run whose timings mean nothing, and you would have no way to know which produced your numbers. Add `nocpp` after the module name and it will finish, more slowly and with identical output. ## `basicstats` { #basicstats } @@ -4011,39 +4183,39 @@ What is in your published tables, per pool. It is the module to run first: every PoolSeqFlow analysis basicstats ``` -It reads the frequency and depth tables, and it needs your `metadata.csv` — the pool sizes, the ploidy and the experimental design all come from the project as the pipeline recorded it. A project whose metadata file was never copied across is refused rather than described as having no design. +It reads the frequency and depth tables, and it needs your `metadata.csv`: the pool sizes, the ploidy and the experimental design all come from the project as the pipeline recorded it. A project whose metadata file was never copied across is refused rather than described as having no design. -### `design.tsv` — one row per pool { #basicstats-design } +### `design.tsv`: one row per pool { #basicstats-design } | Column | What | |---|---| | `pool` | the `RG_Sample` a column of every published table is named after | -| `libraries` | the `SampleID`s merged into it, `;`-separated — one for most pools, more where reads were merged | +| `libraries` | the `SampleID`s merged into it, `;`-separated (one for most pools, more where reads were merged) | | `n_libraries` | how many, so a merged pool is visible without reading the list | | one per `exp_` column | that pool's value for each of your experimental variables, in the order `metadata.csv` gives them; blank where the cell was | | `pool_size` | individuals in the pool: its `param_poolSize`, or the project's `poolSize` where that cell was blank | | `ploidy` | copies of the genome per individual, from `parameters.config` | | `n_chrom` | `ploidy × pool_size`, the number of chromosomes sampled. **This is the only place ploidy enters any statistic here** | -| `detection_limit` | `1 / (2 × ploidy × pool_size)` — the frequency below which step 7's filter took a call to be error. Not a property of this analysis: it is what already happened to the tables, repeated here so the numbers beside it can be read against it | +| `detection_limit` | `1 / (2 × ploidy × pool_size)`, the frequency below which step 7's filter took a call to be error. Not a property of this analysis: it is what already happened to the tables, repeated here so the numbers beside it can be read against it | -Nothing in this file is estimated. It is the design the results were produced under, printed where an analysis can be read against it — which is the point, because a frequency means nothing without the pool size it was read from. +Nothing in this file is estimated. It is the design the results were produced under, printed where an analysis can be read against it. That is the point, because a frequency means nothing without the pool size it was read from. **A pool of one chromosome is refused**, and it is the only combination that is. A single haploid genome has no segregating sites, and the correction every diversity estimate here applies divides by `n_eff - 1`, which is zero there. Correct `ploidy`, or that pool's `param_poolSize`. -### `sites.tsv` — what the pipeline called { #basicstats-sites } +### `sites.tsv`: what the pipeline called { #basicstats-sites } One row per sequence per kind of variant. It carries no pool, because it is a property of the tables rather than of any one column. | Column | What | |---|---| -| `chrom` | the sequence, **in the order the tables give them** — never sorted, so `chr10` sits where your reference puts it rather than between `chr1` and `chr2` | +| `chrom` | the sequence, **in the order the tables give them** (never sorted, so `chr10` sits where your reference puts it rather than between `chr1` and `chr2`) | | `kind` | `snp` or `indel`. Step 7 publishes the two as separate tables and nothing here pools them | | `sites` | rows of the depth table: **records, not alleles**. A record carrying three alleles is one site | | `alleles` | rows of the *frequency* table: one per allele including the reference, so a triallelic site contributes three | A sequence with no surviving variants has no row here at all, and is indistinguishable from one your reference does not have. That is a limit of reading the published tables: they hold what was called, and nothing records what was looked at and found invariant. -### `depth.tsv` — depth at the called sites { #basicstats-depth } +### `depth.tsv`: depth at the called sites { #basicstats-depth } One row per pool per sequence, over the **SNP** table. @@ -4051,12 +4223,15 @@ One row per pool per sequence, over the **SNP** table. |---|---| | `pool`, `chrom` | which pool, which sequence | | `sites` | SNP sites on that sequence | +| `unmeasured` | how many of them this pool has no reads at, and the three columns below therefore leave out. Zero unless you turned [`vcffilter.dropZeroDepth`](#unmeasured-cells) off | | `depth_mean`, `depth_median` | the ordinary summaries of that pool's depth | | `depth_harmonic` | the harmonic mean, which is the one every effective sample size below is computed from | -**A pool's depth at a site is the sum of its cell in the depth table** — the reads supporting any allele there, after step 7's depth, quality and false-positive filters. It is not coverage, and it is not what `Output/Reports/Depth` measured: the sites here are the ones that survived calling, every one of them carries at least `vcffilter.minDP` reads in **every** sample by construction, and mapping and base quality minima applied to the pileup that they did not. The two numbers are not one quantity measured twice, and this one is always the larger. +**The three depth columns are taken over `sites - unmeasured`, not over `sites`.** A site this pool has no reads at tells you nothing about how deep it was read, so averaging it in as a zero would report a depth nobody measured. The harmonic mean of any set containing a zero is zero, which would take the pool's effective sample size to `NA` on one missed site out of millions. `unmeasured` is published so the denominator is readable rather than assumed. + +**A pool's depth at a site is the sum of its cell in the depth table**: the reads supporting any allele there, after step 7's depth, quality and false-positive filters. It is not coverage, and it is not what `Output/Reports/Depth` measured: the sites here are the ones that survived calling, every one of them carries at least `vcffilter.minDP` reads in **every** sample by construction, and mapping and base quality minima applied to the pileup that they did not. The two numbers are not one quantity measured twice, and this one is always the larger. -### `diversity.tsv` — gene diversity and effective sample size { #basicstats-diversity } +### `diversity.tsv`: gene diversity and effective sample size { #basicstats-diversity } One row per pool, over the called SNP sites of the whole project. @@ -4065,46 +4240,47 @@ One row per pool, over the called SNP sites of the whole project. | `pool` | the `RG_Sample` | | `n_chrom` | `ploidy × pool_size`, as in [`design.tsv`](#basicstats-design) | | `sites` | the called SNP sites | +| `unmeasured` | how many of them this pool has no reads at. Zero unless you turned [`vcffilter.dropZeroDepth`](#unmeasured-cells) off | | `segregating` | how many of them are segregating **for this pool**, by the rule below | -| `depth_harmonic` | the harmonic mean of this pool's depth over those sites | +| `depth_harmonic` | the harmonic mean of this pool's depth over `sites - unmeasured` | | `n_eff_harmonic` | the pool's effective sample size over them, from `depth_harmonic` | | `h_sum` | the **numerator**: the sum over sites of the corrected gene diversity | -| `pi_per_called_site` | `h_sum` divided by the sites that contributed a value to it. That is `sites`, unless a cell of the depth table was missing — which a published table cannot hold, since `vcffilter.minDP` removes a site where any sample falls short | +| `pi_per_called_site` | `h_sum` divided by the sites that contributed a value to it, which is `sites - unmeasured`. A site one pool missed is dropped for that pool and kept for the others, so two pools in one run can rest on different counts | **The diversity is Nei's, over every allele at a site**, corrected for the pool's effective sample size at that site: -$$\hat\pi = \frac{1}{\text{sites}}\sum_{\text{sites}} \frac{n_{\text{eff}}}{n_{\text{eff}} - 1}\left(1 - \sum_j p_j^2\right), \qquad n_{\text{eff}} = \frac{n_{\text{chrom}} \cdot d}{n_{\text{chrom}} + d - 1}$$ +$$\hat\pi = \frac{1}{\text{sites}}\sum_{\text{sites}} \frac{n_{\text{eff}}}{n_{\text{eff}} - 1}\left(1 - \sum_j p_j^2\right), \qquad n_{\text{eff}} = \frac{n_{\text{chrom}} \cdot d}{n_{\text{chrom}} + d − 1}$$ -Not `2p(1-p)`: that form assumes two alleles and a privileged reference, so a triallelic site has to be collapsed or dropped before it can be used. $1 - \sum_j p_j^2$ needs no such choice, is defined for any number of alleles, and reduces to `2p(1-p)` where there are two. See [Nei 1973](#ref-nei1973diversity) for the statistic, [Ferretti et al. 2013](#ref-ferretti2013pool) for why a pooled estimate needs correcting at all, and [Hivert et al. 2018](#ref-hivert2018poolseq) for the effective sample size — including which of the two forms in circulation this is. +Not `2p(1−p)`: that form assumes two alleles and a privileged reference, so a triallelic site has to be collapsed or dropped before it can be used. $1 - \sum_j p_j^2$ needs no such choice, is defined for any number of alleles, and reduces to `2p(1−p)` where there are two. See [Nei 1973](#ref-nei1973diversity) for the statistic, [Ferretti et al. 2013](#ref-ferretti2013pool) for why a pooled estimate needs correcting at all, and [Hivert et al. 2018](#ref-hivert2018poolseq) for the effective sample size, including which of the two forms in circulation this is. -**`h_sum` and `sites` are published, and the ratio is published beside them, because the ratio is the part that is contestable.** Per-site diversity summed over the sites that were *called* is not the same as per-base diversity over the sites that were *callable*, and our tables hold only the former. A project's real π is `h_sum` over the number of positions that could have produced a call, which needs a per-position pass over the ready BAMs that no module does yet. `pi_per_called_site` is therefore an **upper bound** on π and will typically be far above it — invariant sites are missing from the denominator entirely. Quote `h_sum` and the count, not the ratio, unless you mean per called site and say so. +**`h_sum` and `sites` are published, and the ratio is published beside them, because the ratio is the part that is contestable.** Per-site diversity summed over the sites that were *called* is not the same as per-base diversity over the sites that were *callable*, and our tables hold only the former. A project's real π is `h_sum` over the number of positions that could have produced a call, which needs a per-position pass over the ready BAMs that no module does yet. `pi_per_called_site` is therefore an **upper bound** on π and will typically be far above it; invariant sites are missing from the denominator entirely. Quote `h_sum` and the count, not the ratio, unless you mean per called site and say so. **Our number will not match PoPoolation's or poolfstat's on a project with triallelic sites, and the reason is deliberate.** [Kofler et al. 2011a](#ref-kofler2011popoolation) implements estimators defined over two alleles, and poolfstat's reader skips multiallelic records outright; the diversity here sums over every allele the site has. On a strictly biallelic project the two agree. On a viral quasispecies, where the three- and four-allele sites are the signal, discarding them discards the result. **A site is segregating for a pool when an allele other than that pool's own major one reaches `max(detection_limit, minReads / depth)`.** Two limbs, because either alone is wrong: `detection_limit` is `1 / (2 × ploidy × pool_size)`, which at any ordinary depth is below one read and so admits every sequencing error; `minReads / depth` alone stops discriminating once the pool is large enough that one chromosome is rarer than a couple of reads. The crossover is at `depth = minReads / detection_limit`. The pool's **own** major allele and not the cohort's: a pool fixed for whatever the cohort calls alternate is not segregating, and reading the majority off the reference column would report that it is. -### `neff.tsv` — effective sample size, at two levels and from two sources { #basicstats-neff } +### `neff.tsv`: effective sample size, at two levels and from two sources { #basicstats-neff } How many independent chromosomes a frequency read off this data is actually worth. Everything else the analysis layer will ever weight by is this number, so it is reported at every level the data supports rather than collapsed to one. | Column | What | |---|---| -| `level` | `library` — one sequencing run, your `SampleID`; or `pool` — one column of the published tables, your `RG_Sample` | +| `level` | `library` (one sequencing run, your `SampleID`); or `pool` (one column of the published tables, your `RG_Sample`) | | `id` | the library or the pool | | `pool` | the pool it belongs to, so a library row can be read against its pool's | -| `source` | `histogram` — every position the library covered, from step 5; `called` — the sites that survived filtering, from the depth table | +| `source` | `histogram` (every position the library covered, from step 5); `called` (the sites that survived filtering, from the depth table) | | `positions` | how many positions that figure was measured over | | `depth_harmonic` | the harmonic mean depth over them | | `n_chrom`, `n_eff` | the chromosomes sampled, and what they are worth at that depth | -| `estimate` | `exact`, or `lower_bound` — see below | +| `estimate` | `exact`, or `lower_bound` (see below) | -**The two sources are not one quantity measured twice, and `called` is always the larger.** Five things separate them: the histogram is measured before the depth ceiling and the calls after it; `samtools stats` counts every spanning read where the pileup applied a mapping and base quality minimum; a histogram counts positions where the depth table sums `AD`; every called site carries at least `vcffilter.minDP` reads **in every sample** by construction, which censors the low end away; and a site had to be variable to be called at all. Compare a pool's two rows to see how much the filter chain concentrated the data — do not treat their difference as error. +**The two sources are not one quantity measured twice, and `called` is always the larger.** Five things separate them: the histogram is measured before the depth ceiling and the calls after it; `samtools stats` counts every spanning read where the pileup applied a mapping and base quality minimum; a histogram counts positions where the depth table sums `AD`; every called site carries at least `vcffilter.minDP` reads **in every sample** by construction, which censors the low end away; and a site had to be variable to be called at all. Compare a pool's two rows to see how much the filter chain concentrated the data. Do not treat their difference as error. **There is no `library` row from the `called` source, and there cannot be.** The published tables carry one column per `RG_Sample`, so a merged pool's libraries are already summed inside them and nothing can separate them again. Per-library figures come from the histograms or from nowhere. #### A merged pool's genome-wide figure is a lower bound { #basicstats-neff-bound } -When two libraries were merged into one pool, its depth at a position is their depths **added** — and the harmonic mean of that sum cannot be recovered from the two histograms. Adding the histograms would assume the libraries are independent across positions, which is false and errs *optimistic*. So the module adds the parts' harmonic means instead, and marks the row `lower_bound`. +When two libraries were merged into one pool, its depth at a position is their depths **added**, and the harmonic mean of that sum cannot be recovered from the two histograms. Adding the histograms would assume the libraries are independent across positions, which is false and errs *optimistic*. So the module adds the parts' harmonic means instead, and marks the row `lower_bound`. That is sound rather than arbitrary: the harmonic mean is concave and homogeneous of degree one, therefore superadditive, so $H(a+b) \ge H(a) + H(b)$ always. Measured over 100,000 positions: @@ -4114,13 +4290,13 @@ That is sound rather than arbitrary: the harmonic mean is concave and homogeneou | realistically correlated | 29.88 | 17.37 | **47.25** | 47.94 | | independent (unrealistic) | 30.14 | 17.93 | **48.07** | 55.62 | -Exact when the libraries cover positions in proportion to one another, about 1.4% conservative under realistic correlation, and never optimistic. **It understates precision, which is the safe direction** — a test weighted by it is conservative, not anti-conservative. It does assume the libraries covered the same positions, which is why each row carries the count it was measured over. +Exact when the libraries cover positions in proportion to one another, about 1.4% conservative under realistic correlation, and never optimistic. **It understates precision, which is the safe direction**: a test weighted by it is conservative, not anti-conservative. It does assume the libraries covered the same positions, which is why each row carries the count it was measured over. A pool of one library is marked `exact`: there is nothing to add. **A pool with no histogram gets no row from that source, and none is guessed.** `DepthProfile` skips a sample whose ceiling is already decided, so a sound project can lack them; the run names those pools rather than averaging over the libraries that did have one. -### `depth_.png` — depth along a sequence { #basicstats-depth-plots } +### `depth_.png`: depth along a sequence { #basicstats-depth-plots } One file per sequence you name, one panel per pool, a point per called site. @@ -4140,7 +4316,7 @@ params { **A named sequence with no called site is refused**, naming what the tables do have. A sequence on which nothing survived filtering is invisible in the published tables and cannot be told apart from one your reference does not contain, so an empty plot would be a claim the data does not support. -**It is depth, not coverage, and the axis says so.** Every point is a site that was *called*; the gaps between them are sites the pipeline did not call, which is not the same as sites with no reads. A run of thin points is a stretch where calling was sparse — read it against `depth.tsv` for that sequence rather than as a coverage trace. +**It is depth, not coverage, and the axis says so.** Every point is a site that was *called*; the gaps between them are sites the pipeline did not call, which is not the same as sites with no reads. A run of thin points is a stretch where calling was sparse. Read it against `depth.tsv` for that sequence rather than as a coverage trace. ### What it can be set to { #basicstats-settings } @@ -4177,7 +4353,7 @@ For a whole project, `analysis.modules.basicstats.usecpp = false` in `basicstats ### What to cite { #basicstats-citations } -Every folder `basicstats` publishes carries a `CITATIONS.md` and a `references.bib` with these already filled in, and `PoolSeqFlow analysis cite basicstats` prints them without running anything. **Cite the statistics, not only the software** — what a reader needs to check is the estimator, and the estimator is not the program that ran it. +Every folder `basicstats` publishes carries a `CITATIONS.md` and a `references.bib` with these already filled in, and `PoolSeqFlow analysis cite basicstats` prints them without running anything. **Cite the statistics, not only the software**: what a reader needs to check is the estimator, and the estimator is not the program that ran it. | | | |---|---| @@ -4202,11 +4378,11 @@ It reads the depth tables and your `metadata.csv`. The phenotype is a `pt_` colu **This is a model-based test.** It is not an assumption-light one, and there is no assumption-light one available: any test that weights pools by their precision has already conceded that they differ, and a null that then treats them as interchangeable contradicts its own statistic. So the assumptions are stated rather than hidden, and there are three. -1. **How precisely a frequency was measured.** For one allele at one site in one unit, the variance is `p(1−p) × (dispersion + 1/n_eff)` — read sampling and pool sampling through `n_eff`, plus whatever excess uneven pooling left behind. Only the *ratio* between units is ever a claim: multiplying every weight by a constant leaves the result unchanged, so a pool size wrong by the same factor everywhere costs nothing, and one wrong differently per unit costs calibration. +1. **How precisely a frequency was measured.** For one allele at one site in one unit, the variance is `p(1−p) x (dispersion + 1/n_eff)` (read sampling and pool sampling through `n_eff`, plus whatever excess uneven pooling left behind). Only the *ratio* between units is ever a claim: multiplying every weight by a constant leaves the result unchanged, so a pool size wrong by the same factor everywhere costs nothing, and one wrong differently per unit costs calibration. 2. **Units are independent.** A unit is one independent biological entity; two pools are two units unless `analysis.design.technicalRep` says they are one material measured twice. 3. **A site's alleles are several views of one comparison**, which is what the next section is about. -Where those do not describe your experiment, the numbers will still be produced and will still be wrong. [`permutations.tsv`](#association-diagnostics) is what tells you so, and **if your design is one this does not fit, please come and say so** — the roster grows by someone describing a problem this cannot answer, and fitting your data to the nearest available analysis serves nobody. +Where those do not describe your experiment, the numbers will still be produced and will still be wrong. [`permutations.tsv`](#association-diagnostics) is what tells you so, and **if your design is one this does not fit, please come and say so**: the roster grows by someone describing a problem this cannot answer, and fitting your data to the nearest available analysis serves nobody. ### A site with more than two alleles { #association-multiallelic } @@ -4214,17 +4390,17 @@ Where those do not describe your experiment, the numbers will still be produced That is worth unpacking, because it is where this module departs from what most tools do. -**One regression per allele.** A triallelic site gives three fits — reference against the phenotype, first alternate against it, second alternate against it. Each allele has its own frequency and therefore its own variance under assumption 1 above; the weights are the same for all of them, because a weight describes how deeply that unit was read at that site and depth is a property of the site. +**One regression per allele.** A triallelic site gives three fits: reference against the phenotype, first alternate against it, second alternate against it. Each allele has its own frequency and therefore its own variance under assumption 1 above; the weights are the same for all of them, because a weight describes how deeply that unit was read at that site and depth is a property of the site. **The site statistic is the largest |t| among them.** Written `S` in [`association.tsv`](#association-sites). -**Why the reference row is not skipped.** A site's frequencies sum to 1, so the reference carries the negated sum of every alternate. When a phenotype pushes two alternates up together, neither is impressive alone and the reference falls by both — so the reference row is where that signal is, and an analysis that only looks at alternates cannot see it. Measured against a planted signal of exactly that shape, reading the alternates alone recovers **40% fewer** of the sites than reading every allele. +**Why the reference row is not skipped.** A site's frequencies sum to 1, so the reference carries the negated sum of every alternate. When a phenotype pushes two alternates up together, neither is impressive alone and the reference falls by both, so the reference row is where that signal is, and an analysis that only looks at alternates cannot see it. Measured against a planted signal of exactly that shape, reading the alternates alone recovers **40% fewer** of the sites than reading every allele. -`MajorAlleleToRef.py` has already made the reference the cohort's *major* allele, so "reference" here is not a privileged biological state — it is simply the commonest allele, and skipping it would privilege "not the commonest" for no reason. +`MajorAlleleToRef.py` has already made the reference the cohort's *major* allele, so "reference" here is not a privileged biological state but simply the commonest allele, and skipping it would privilege "not the commonest" for no reason. -**Why more alleles is not an unfair advantage.** A site of four alleles offers four chances at a large |t| where a biallelic site offers two, which sounds like it should favor multiallelic sites. It does not, because the p-value is not read off a table: it is read against a null built by rearranging **that same site**, which has the same alleles, the same constraint that they sum to 1, and therefore the same number of chances. The comparison is like for like at every arity, so nothing needs correcting — no Bonferroni, no union bound. +**Why more alleles is not an unfair advantage.** A site of four alleles offers four chances at a large |t| where a biallelic site offers two, which sounds like it should favor multiallelic sites. It does not, because the p-value is not read off a table: it is read against a null built by rearranging **that same site**, which has the same alleles, the same constraint that they sum to 1, and therefore the same number of chances. The comparison is like for like at every arity, so nothing needs correcting. There is no Bonferroni and no union bound. -That is not an argument from first principles alone. Measured on null sites of two, three and four alleles, the false-positive rate is flat across arity, and the sites the module selects have the same allele counts as the genome it selected them from — which is what `arity_mean` and `arity_selected` in [`permutations.tsv`](#association-diagnostics) let you check on your own data. The union-bound alternative — take the smallest allele p and multiply by *k* − 1 — measured **37% above its nominal rate** at triallelic sites, which is why it is not used. +That is not an argument from first principles alone. Measured on null sites of two, three and four alleles, the false-positive rate is flat across arity, and the sites the module selects have the same allele counts as the genome it selected them from, which is what `arity_mean` and `arity_selected` in [`permutations.tsv`](#association-diagnostics) let you check on your own data. The union-bound alternative (take the smallest allele p and multiply by *k* - 1) measured **37% above its nominal rate** at triallelic sites, which is why it is not used. **What it costs.** A site of *k* alleles carries *k* − 1 free tests, not *k*: at a biallelic site the reference and alternate rows are the same test with the sign flipped, and their p-values are identical. So one site is one finding, whatever its arity, and the allele rows tell you which allele carries it rather than giving you several findings. @@ -4243,7 +4419,7 @@ At four units the run says so in a sentence and the table is a **ranking of effe The floor binds hardest when the pools are read to similar depths. Where depths differ a great deal the design carries more information than a bare count of units suggests, and the run can reach further down; `design_floor` and `floor` in [`permutations.tsv`](#association-diagnostics) are those two numbers side by side. -### `association.tsv` — one row per site { #association-sites } +### `association.tsv`: one row per site { #association-sites } Always complete: every site the module could read has a row, whether or not it produced a statistic. @@ -4254,22 +4430,22 @@ Always complete: every site the module could read has a row, whether or not it p | `n_observed`, `n_units` | units carrying data here, and units in the design | | `S` | the site statistic: the largest \|t\| over **every** allele | | `perm_p` | the published p, from the permutation | -| `fdr_p` | `perm_p` corrected across sites, by default Benjamini–Hochberg [Benjamini & Hochberg 1995](#ref-benjamini1995fdr) | +| `fdr_p` | `perm_p` corrected across sites, by default Benjamini-Hochberg [Benjamini & Hochberg 1995](#ref-benjamini1995fdr) | | `mean_weight` | the average effective sample size at this site | | `max_leverage` | how much of the slope one unit is carrying, which at six of them is the question a reader has | | `zero_variance` | 1 where an allele had no residual left; see below | -**A flagged site's `S` is not comparable between implementations and must not be quoted.** Two alleles of a perfectly separated site are algebraically one test, and whether each lands on exactly zero residual or on 1e-32 decides between an infinite *t* and a merely enormous one — we have seen two builds of the same language disagree. `perm_p` is unaffected and remains the answer. +**A flagged site's `S` is not comparable between implementations and must not be quoted.** Two alleles of a perfectly separated site are algebraically one test, and whether each lands on exactly zero residual or on 1e-32 decides between an infinite *t* and a merely enormous one. We have seen two builds of the same language disagree. `perm_p` is unaffected and remains the answer. -### `association_alleles.tsv` — one row per allele { #association-alleles } +### `association_alleles.tsv`: one row per allele { #association-alleles } The effect sizes, for the sites `reportBelow` and `reportTop` select. **This table is a selection and the site table is not**: at a hundred million sites the allele rows are a twenty-gigabyte file, which is a data dump rather than a result. -`b1` is the slope, `se` its standard error, and `t` and `p` the closed-form diagnostics. **`b1 > 0` means the allele is more frequent at the higher phenotype value** — for a categorical scale, at the level [`phenotype.tsv`](#association-phenotype) names as coded 1. +`b1` is the slope, `se` its standard error, and `t` and `p` the closed-form diagnostics. **`b1 > 0` means the allele is more frequent at the higher phenotype value** (for a categorical scale, at the level [`phenotype.tsv`](#association-phenotype) names as coded 1). -The parametric `p` is a diagnostic and not the answer. It is well calibrated on an ordinary site and makes impossible claims on a degenerate one — six reads of evidence will produce a *p* of 1e-9 that no design of six pools can support. +The parametric `p` is a diagnostic and not the answer. It is well calibrated on an ordinary site and makes impossible claims on a degenerate one: six reads of evidence will produce a *p* of 1e-9 that no design of six pools can support. -### `permutations.tsv` — what the run assumed and what it absorbed { #association-diagnostics } +### `permutations.tsv`: what the run assumed and what it absorbed { #association-diagnostics } A guard that passes tells you nothing. These are published so that someone reading your results, without rerunning them, can see what the model was asked to do on your behalf. @@ -4277,22 +4453,22 @@ A guard that passes tells you nothing. These are published so that someone readi |---|---| | `permutations`, `exhaustive` | how many rearrangements, and whether that was all of them | | `floor`, `design_floor` | the smallest p this run could reach, and the smallest the design can | -| `dispersion` | the excess variance uneven pooling left behind, in units of `p(1−p)`. Large means the weights absorbed a lot; it is estimated from your data unless you set it. **A published `0` is a floor, not necessarily a measurement** — the method of moments can return a negative excess when the units scatter less than their own sampling variance predicts, which is noise and not a unit measured better than its depth allows, and it is reported as `0` | +| `dispersion` | the excess variance uneven pooling left behind, in units of `p(1−p)`. Large means the weights absorbed a lot; it is estimated from your data unless you set it. **A published `0` is a floor, not necessarily a measurement**: the method of moments can return a negative excess when the units scatter less than their own sampling variance predicts, which is noise and not a unit measured better than its depth allows, and it is reported as `0` | | `depth_phenotype_cor` | depth lined up with the phenotype. This breaks label-based tests badly and is the reason this module does not use one | | `lambda_gc` | genomic inflation. If it is 3, nothing in the table is a p-value | | `arity_mean`, `arity_selected`, `selected` | the allele counts of the sites selected against all sites. **They should agree**; multiallelic sites at the top of a table are a biological claim, and this is how you check it is one | -### `phenotype.tsv` — the phenotype as it was read { #association-phenotype } +### `phenotype.tsv`: the phenotype as it was read { #association-phenotype } One row per pool: the cell as you wrote it, the number the fit used, the unit it belongs to, and for a categorical scale which level became 1. **Read this first.** A reversed case/control coding has no more of a check than `dd/MM` against `MM/dd` does, and printing the resolved answer is the only place either is caught. -### `qq.png` — the p-values against the uniform they should follow { #association-qq } +### `qq.png`: the p-values against the uniform they should follow { #association-qq } Always drawn. A permutation p is discrete, so the points sit on a ladder rather than a line, and the ladder's ceiling is your design's floor. A curve bending above the diagonal is inflation; the diagonal itself is what a null genome looks like. -### `manhattan_.png` — association along a sequence { #association-manhattan } +### `manhattan_.png`: association along a sequence { #association-manhattan } Drawn only for the sequences you name in `analysis.modules.association.chromosomes`, on the same rule `basicstats` follows: a genome has more sequences than anyone wants plots for. @@ -4302,7 +4478,7 @@ Drawn only for the sequences you name in `analysis.modules.association.chromosom | Setting | Default | What | |---|---|---| -| `phenotypes` | — | which `pt_` column to fit. Required | +| `phenotypes` | no default | which `pt_` column to fit. Required | | `permutations` | 10000 | the budget. The set is enumerated whole while it fits inside this, and sampled above it | | `fdr` | `BH` | anything `p.adjust` takes | | `dispersion` | estimated | the excess variance term. `0` weights by `n_eff` alone | @@ -4341,13 +4517,13 @@ Full entries, with what each is cited for, are in the [Bibliography](#bibliograp Every pool placed as a point, so that pools which resemble each other sit close together. The distance between two pools is Nei's minimum distance, corrected for the fact that neither pool's frequencies were measured exactly, and the picture is a classical multidimensional scaling of that distance matrix. -**This is classical — metric — scaling, which is the same method as principal coordinates analysis (PCoA).** It is not NMDS. Non-metric scaling keeps only the *rank order* of the distances and finds a configuration by iteratively minimising stress; what this does is an eigendecomposition that uses the distances themselves, so the plotted gaps are proportional to the distances rather than merely ordered like them. If you are used to `metaMDS` in `vegan`, this is `cmdscale`/`wcmdscale` — closer to `ape::pcoa` — and the eigenvalue table below is the thing NMDS has no equivalent of. +**This is classical (metric) scaling, which is the same method as principal coordinates analysis (PCoA).** It is not NMDS. Non-metric scaling keeps only the *rank order* of the distances and finds a configuration by iteratively minimising stress; what this does is an eigendecomposition that uses the distances themselves, so the plotted gaps are proportional to the distances rather than merely ordered like them. If you are used to `metaMDS` in `vegan`, this is `cmdscale`/`wcmdscale` (closer to `ape::pcoa`), and the eigenvalue table below is the thing NMDS has no equivalent of. ```bash PoolSeqFlow analysis mds ``` -It reads the depth tables and nothing else. There is nothing to declare and nothing to choose before running it — no phenotype, no time axis, no grouping. That makes it the first thing worth running on a new cohort: it answers "do these samples look like what I think they are" before any model is fitted to them. +It reads the depth tables and nothing else. There is nothing to declare and nothing to choose before running it. No phenotype, no time axis, no grouping. That makes it the first thing worth running on a new cohort: it answers "do these samples look like what I think they are" before any model is fitted to them. ### What it measures, and what the correction is for { #mds-statistic } @@ -4357,13 +4533,13 @@ For two pools A and B at one site, with `J` the probability that two chromosomes distance = (J_A + J_B) / 2 − J_AB ``` -which expands to `½ × Σ (f_A − f_B)²` over **every** allele of the site, the reference included. A triallelic site contributes three terms. No allele is privileged and none is dropped — which matters here, because `MajorAlleleToRef.py` makes the reference the cohort's *major* allele, so dropping it would discard the most informative row rather than a redundant one. The site values are averaged over sites. +which expands to `½ x Σ (f_A − f_B)²` over **every** allele of the site, the reference included. A triallelic site contributes three terms. No allele is privileged and none is dropped, which matters here, because `MajorAlleleToRef.py` makes the reference the cohort's *major* allele, so dropping it would discard the most informative row rather than a redundant one. The site values are averaged over sites. -**The correction is the part that earns its keep.** `J` read off a sample is biased upward: two *reads* from one pool agree more often than two *chromosomes* do, and the shallower the pool the larger that excess. Uncorrected, the excess does not cancel — it depends on `1/n_A + 1/n_B`, so it inflates shallow–shallow pairs more than shallow–deep ones and those more than deep–deep. Simulated on six pools drawn from **one** population differing only in read depth, two at 30× and four at 400×, the two shallow pools come out as the *most distant pair in the matrix* and the leading axes separate the cohort by depth. There is one population. The corrected distance removes that: the same simulation leaves no depth ordering worth reading, with a residual roughly two orders of magnitude below the artefact it replaced. +**The correction is the part that earns its keep.** `J` read off a sample is biased upward: two *reads* from one pool agree more often than two *chromosomes* do, and the shallower the pool the larger that excess. Uncorrected, the excess does not cancel. It depends on `1/n_A + 1/n_B`, so it inflates shallow-shallow pairs more than shallow-deep ones and those more than deep-deep. Simulated on six pools drawn from **one** population differing only in read depth, two at 30× and four at 400×, the two shallow pools come out as the *most distant pair in the matrix* and the leading axes separate the cohort by depth. There is one population. The corrected distance removes that: the same simulation leaves no depth ordering worth reading, with a residual roughly two orders of magnitude below the artefact it replaced. `distance.tsv` publishes the uncorrected sum beside the corrected one, so how much was subtracted is something you can look at rather than take on trust. -The correction subtracts each pool's own diversity at a site, scaled by its effective sample size there, so it inherits everything `n_eff` does — including that ploidy enters exactly once, through `n_chrom = ploidy × poolSize`. Nothing here is limited to diploids. +The correction subtracts each pool's own diversity at a site, scaled by its effective sample size there, so it inherits everything `n_eff` does, including that ploidy enters exactly once, through `n_chrom = ploidy × poolSize`. Nothing here is limited to diploids. ### Distances can come out negative, and that is correct { #mds-negative } @@ -4371,13 +4547,13 @@ The correction subtracts each pool's own diversity at a site, scaled by its effe Read a negative entry as **indistinguishable at this depth**. Its magnitude is noise, not a measurement. -### `mds.tsv` — one row per pool { #mds-coordinates } +### `mds.tsv`: one row per pool { #mds-coordinates } Each pool, the unit it belongs to, its experimental variables, and its coordinate on each axis. -**The ordination places pools, not units.** Nothing is collapsed first, and that is deliberate: two pools of one biological unit landing far apart is precisely what you look at an ordination to find out, and averaging them beforehand would hide it. The `unit` column is there so you can see which points *should* have coincided. This is the one place in the analysis layer where pools rather than units are the unit of analysis, and it is because nothing here is a test — there are no degrees of freedom to get wrong. +**The ordination places pools, not units.** Nothing is collapsed first, and that is deliberate: two pools of one biological unit landing far apart is precisely what you look at an ordination to find out, and averaging them beforehand would hide it. The `unit` column is there so you can see which points *should* have coincided. This is the one place in the analysis layer where pools rather than units are the unit of analysis, and it is because nothing here is a test. There are no degrees of freedom to get wrong. -### `distance.tsv` — one row per pair { #mds-distance } +### `distance.tsv`: one row per pair { #mds-distance } The corrected distance, the uncorrected sum beside it, what the correction removed, and how many sites the pair was averaged over. @@ -4385,7 +4561,7 @@ The corrected distance, the uncorrected sum beside it, what the correction remov Two kinds of site drop out for a pair. One is an ordinary missing cell. The other is subtler: **a site where a pool has exactly one read**. Effective sample size is exactly 1 at depth 1 whatever the pool holds, and one gene copy carries no within-pool diversity for the correction to work from, so such a site is dropped rather than guessed at. -### `eigenvalues.tsv` — how much each axis carries { #mds-eigenvalues } +### `eigenvalues.tsv`: how much each axis carries { #mds-eigenvalues } Every eigenvalue, with its share of the scatter under both denominators, the running totals, and which axes were plotted. @@ -4395,11 +4571,11 @@ Negative eigenvalues carrying a few percent of the total are ordinary for pool-s **The axis signs are pinned.** An eigenvector has a direction but not a sign, so the same data can plot mirrored on two machines and read as though something changed. Each axis is turned so that its largest coordinate is positive. -### `mds.png` — the pools on the leading two axes { #mds-plot } +### `mds.png`: the pools on the leading two axes { #mds-plot } Points labeled by pool, on the leading two axes, with each axis label carrying its share of the absolute eigenvalue sum. -**`colorBy` and `shapeBy` each take an `exp_` column, and they compose.** Coloring by the treatment and shaping by the timepoint puts both factors on one plot, which is usually the question — whether the pools group by the thing you set up, or by when you sampled them: +**`colorBy` and `shapeBy` each take an `exp_` column, and they compose.** Coloring by the treatment and shaping by the timepoint puts both factors on one plot, which is usually the question (whether the pools group by the thing you set up, or by when you sampled them): ```groovy colorBy = 'exp_population' @@ -4408,9 +4584,9 @@ shapeBy = 'exp_time' Either is refused by name if it is not an experimental variable this project declares. -**Shapes are R's plotting symbols and colors are `ggplot2`'s default scale — neither is curated here.** `ggplot2` on its own stops at six shapes and silently assigns none to a seventh, which would drop those pools from the plot with a warning a Nextflow task swallows; naming R's symbols explicitly is what carries a seventh level and past it. The first six are `ggplot2`'s own, so a plot of six or fewer groups is the one it would have drawn anyway. Past 26 there are no more symbols in R and `shapeBy` says so rather than dropping pools — use `colorBy`, whose default scale takes any number of levels. +**Shapes are R's plotting symbols and colors are `ggplot2`'s default scale; neither is curated here.** `ggplot2` on its own stops at six shapes and silently assigns none to a seventh, which would drop those pools from the plot with a warning a Nextflow task swallows; naming R's symbols explicitly is what carries a seventh level and past it. The first six are `ggplot2`'s own, so a plot of six or fewer groups is the one it would have drawn anyway. Past 26 there are no more symbols in R and `shapeBy` says so rather than dropping pools. Use `colorBy`, whose default scale takes any number of levels. -**There is no palette setting, and that is deliberate.** How many levels a variable has, whether it is ordered, and whether a reader needs the categories distinguishable or merely grouped are all properties of your experiment rather than of this module — and every way of getting a palette wrong is quiet. Too few colors and `ggplot2` recycles them; a continuous scale on a discrete variable falls back to something else without stopping. What you get instead is the data: +**There is no palette setting, and that is deliberate.** How many levels a variable has, whether it is ordered, and whether a reader needs the categories distinguishable or merely grouped are all properties of your experiment rather than of this module, and every way of getting a palette wrong is quiet. Too few colors and `ggplot2` recycles them; a continuous scale on a discrete variable falls back to something else without stopping. What you get instead is the data: ```r # Redraw it however you like: mds.tsv is the whole figure, four columns of it @@ -4451,11 +4627,11 @@ analysis { } ``` -`binSize` is a memory knob and changes no result: the distances are sums over sites, so a bin boundary falls between two sites and every bin size adds up the same terms. It exists because the per-allele frequency matrix for a whole genome would be tens of gigabytes, and accumulating a bin at a time means it never has to exist. What it can move is the last bit or two of a published number, because floating-point addition is not associative and a different bin size adds the same terms in a different order — the same reason the compiled path and the vectorized R agree to about fifteen digits rather than to all seventeen. +`binSize` is a memory knob and changes no result: the distances are sums over sites, so a bin boundary falls between two sites and every bin size adds up the same terms. It exists because the per-allele frequency matrix for a whole genome would be tens of gigabytes, and accumulating a bin at a time means it never has to exist. What it can move is the last bit or two of a published number, because floating-point addition is not associative and a different bin size adds the same terms in a different order. That is the same reason the compiled path and the vectorized R agree to about fifteen digits rather than to all seventeen. ### The compiled path { #mds-compiled } -`mds` runs both its per-site parse and its distance accumulation through compiled code by default — `allele_frequencies.cpp` and `nei_distance.cpp`, compiled on your machine: +`mds` runs both its per-site parse and its distance accumulation through compiled code by default (`allele_frequencies.cpp` and `nei_distance.cpp`, compiled on your machine): ```bash PoolSeqFlow analysis mds # compiled, the default @@ -4472,7 +4648,7 @@ Every folder `mds` publishes carries a `CITATIONS.md` and a `references.bib` wit | | | |---|---| -| [Nei 1972](#ref-nei1972distance) | the distance itself — the *minimum* distance of that paper, not the standard distance `D` defined alongside it | +| [Nei 1972](#ref-nei1972distance) | the distance itself: the *minimum* distance of that paper, not the standard distance `D` defined alongside it | | [Gower 1966](#ref-gower1966mds) | the ordination itself: squared distances double centered and decomposed, which is what `cmdscale` implements and where the negative eigenvalues come from | | [Hivert et al. 2018](#ref-hivert2018poolseq) | the effective sample size the sampling correction is scaled by, and which of the two forms in circulation this is | @@ -4481,7 +4657,7 @@ Full entries, with what each is cited for, are in the [Bibliography](#bibliograp # Development History and Principles -> *"A scientific judgment call is not a thing to automate — not by me, and not by anything I hand the work to."* +> *"A scientific judgment call is not a thing to automate, not by me, and not by anything I hand the work to."* Most of this manual is about running PoolSeqFlow. This section is about how it is built, who builds it, and why it takes the form it does. @@ -4489,10 +4665,10 @@ That is in the manual rather than in a README because it is a user's question, n Four pages, and they are meant to be read in order: -- **[History](#how-poolseqflow-was-built)** — who wrote what, and when. It starts here because the commit log gives the wrong answer on its own. -- **[Rationale](#why-a-living-repository)** — why this is a maintained tool rather than a paper, how I decide things, and who the tool is for. -- **[Verification](#how-the-project-is-verified)** — what stops a change being wrong. This is what makes the rest of the section checkable rather than something to take on trust. -- **[Claude Code](#involvement-of-claude-code)** — what was done with an AI agent from v2.0.0 onward, what it got wrong, and what the arrangement demands of the person reviewing it. +- **[History](#how-poolseqflow-was-built)**: who wrote what, and when. It starts here because the commit log gives the wrong answer on its own. +- **[Rationale](#why-a-living-repository)**: why this is a maintained tool rather than a paper, how I decide things, and who the tool is for. +- **[Verification](#how-the-project-is-verified)**: what stops a change being wrong. This is what makes the rest of the section checkable rather than something to take on trust. +- **[Claude Code](#involvement-of-claude-code)**: what was done with an AI agent from v2.0.0 onward, what it got wrong, and what the arrangement demands of the person reviewing it. > *"I am not claiming using an AI agent is a way to build something you do not understand. It is a way to build something you do understand, faster, if you are willing to spend the time saved on reading what's built."* @@ -4505,8 +4681,8 @@ PoolSeqFlow was written by one person over six years, and an AI coding agent has | When | What | Where it lives | |---|---|---| -| 2020–2022 | The original pipeline and the false-positive filtering approach, written in **bash** during my PhD studies. The pool-seq reasoning — what a pool's size does to its detection limit, why a fixed reference is not a privileged one, which filters a real dataset needs — was settled here. | Not in this repository | -| 2025–2026 | The rewrite into **Nextflow**: the step model, the resume logic, the parameter system. Published as **v1.0.0** and **v1.0.1**. The rewrite was needed because I was asked to look at some analysis for some of my collaborators and I decided to redo their variant calling. Without an access to a personal cluster, I needed proper parallelization, for which Nextflow was the perfect vehicle. | This repository, from v1.0.0 | +| 2020-2022 | The original pipeline and the false-positive filtering approach, written in **bash** during my PhD studies. The pool-seq reasoning (what a pool's size does to its detection limit, why a fixed reference is not a privileged one, which filters a real dataset needs) was settled here. | Not in this repository | +| 2025-2026 | The rewrite into **Nextflow**: the step model, the resume logic, the parameter system. Published as **v1.0.0** and **v1.0.1**. The rewrite was needed because I was asked to look at some analysis for some of my collaborators and I decided to redo their variant calling. Without an access to a personal cluster, I needed proper parallelization, for which Nextflow was the perfect vehicle. | This repository, from v1.0.0 | | From v2.0.0, August 2026 | The **v2.x** series and **v3.0.0**, developed with Claude Code: multi-run sharing, storage tiering, the metadata file, the analysis layer and its modules. | This repository | ### The repository is younger than the tool @@ -4515,7 +4691,7 @@ This is worth stating plainly, because the obvious way to judge a project's hist I have done little work after the first work was completed around June 2025. In around March 2026, I decided to package it and put it on my GitHub. That is the reason that the first commit is dated **2026-03-13**. **v1.0.0** was tagged thirteen days later, on **2026-03-26**, with 26 files in the tree. That is pretty much the "end" of the development of the main pipeline. A working pipeline existed before this repository did; what happened in March was that it was put under version control in order to release it. Everything in the first two phases above therefore has no commits behind it at all. -The distribution compounds the impression. **Eight commits reach v1.0.1; every other commit in the repository comes after it**, and the great majority of those fall in a single month, August 2026. The fast development in August was also a necessity rather than just AI enabled development. During August 2026 I spent around 200 hours on the development to help a colleague use this tool and analyze data in September 2026. That is where I made all engineering decisions rather than scientific ones to make the tool accessible by a larger base of scientists. So the log reads as a project built almost entirely in one month — and that is the month the agent arrived. What it actually shows is when the work started being recorded at this granularity, not when it was done. The tree tells the same story from the other side: 26 files at v1.0.1 and many times that today, almost all of the growth being the analysis layer, the test suite and this manual rather than the pipeline itself. +The distribution compounds the impression. **Eight commits reach v1.0.1; every other commit in the repository comes after it**, and the great majority of those fall in a single month, August 2026. The fast development in August was also a necessity rather than just AI enabled development. During August 2026 I spent around 200 hours on the development to help a colleague use this tool and analyze data in September 2026. That is where I made all engineering decisions rather than scientific ones to make the tool accessible by a larger base of scientists. So the log reads as a project built almost entirely in one month, and that is the month the agent arrived. What it actually shows is when the work started being recorded at this granularity, not when it was done. The tree tells the same story from the other side: 26 files at v1.0.1 and many times that today, almost all of the growth being the analysis layer, the test suite and this manual rather than the pipeline itself. ### What the rewrite changed, and what it kept @@ -4531,7 +4707,7 @@ When I decided to analyze new data in 2025 I had more experience in automation, I kept a lot of it as it was. 1. The clipping logic that keeps the A/T and G/C ratios near 1 after trimming, since those can drift once adapters are removed. -2. The alignment and cleaning protocol that prepares BAM files for variant calling — though I replaced all the GATK tools with samtools. +2. The alignment and cleaning protocol that prepares BAM files for variant calling, though I replaced all the GATK tools with samtools. 3. The variant calling protocol. 4. The protocol for obtaining allele frequencies from VCF files. @@ -4552,7 +4728,7 @@ Each release is archived on Zenodo and has a DOI of its own; see [Citation & Lic ### Why an agent was brought in -The decision was about **accessibility**, not about writing code faster. A tool that only its author can install, configure and interpret is not a tool anyone else can use, and most of what v2.x and v3.0 added — the verification step that explains what it is checking, the migration path between releases, the preview, the manual this page is part of — is that problem rather than the science. +The decision was about **accessibility**, not about writing code faster. A tool that only its author can install, configure and interpret is not a tool anyone else can use, and most of what v2.x and v3.0 added (the verification step that explains what it is checking, the migration path between releases, the preview, the manual this page is part of) is that problem rather than the science. It was also a considered decision rather than an experiment. At this point on a completely different side project I was using Claude Code on a larger codebase, which is where I realized that I could implement guardrails to keep pipeline's scientific reasoning and integrity intact. @@ -4562,7 +4738,7 @@ During the time I was working on this side project, I realized that using an age **It does not mark where the assisted work begins.** Two commits carry a `Co-Authored-By` trailer, both from September 2026, because the convention was adopted late rather than at the start. The trailer marks two commits; it does not draw the boundary. The boundary is **v2.0.0**, stated above and in the release table, and [Involvement of Claude Code](#involvement-of-claude-code) says what the work on either side of it actually looked like. -**What the trailer marks is the commit, not the code — which is the opposite of how it reads.** Work is left uncommitted until I have reviewed it in the working tree, and the commits are mine: since v2.0.0 the median commit changes **three files**. Those two commits cover a change of 47 and 22 files, with about 3,300 insertions each and messages of 39 and 41 lines where one line is normal. The changes in them had been reviewed as they landed, like every other change. What was done with the agent was the *packaging* — deciding to separate the additions into two commits rather than one, and writing them out in that much detail. So the trailer records help composing a commit, and a reader who takes it as marking AI-written code has it backwards. +**What the trailer marks is the commit, not the code. That is the opposite of how it reads.** Work is left uncommitted until I have reviewed it in the working tree, and the commits are mine: since v2.0.0 the median commit changes **three files**. Those two commits cover a change of 47 and 22 files, with about 3,300 insertions each and messages of 39 and 41 lines where one line is normal. The changes in them had been reviewed as they landed, like every other change. What was done with the agent was the *packaging*: deciding to separate the additions into two commits rather than one, and writing them out in that much detail. So the trailer records help composing a commit, and a reader who takes it as marking AI-written code has it backwards. ### AI generated text and responsibility @@ -4578,17 +4754,17 @@ I did not write PoolSeqFlow to accompany a paper. The tool, and this manual expl ### A study is finished by design; the methods underneath it are not -When a study goes into a journal it stops. It is correct as of its publication date and it stays that way — the analysis frozen, the software version frozen, the statistics frozen at whatever the field was doing that year. That is fine for a record of what was observed. It is not fine for the methods, because the methods keep moving. I read papers now about how to combine evidence across replicated experiments that describe approaches which simply did not exist when I was learning this work, and any study that used the old approach is still sitting there using it. +When a study goes into a journal it stops. It is correct as of its publication date and it stays that way: the analysis frozen, the software version frozen, the statistics frozen at whatever the field was doing that year. That is fine for a record of what was observed. It is not fine for the methods, because the methods keep moving. I read papers now about how to combine evidence across replicated experiments that describe approaches which simply did not exist when I was learning this work, and any study that used the old approach is still sitting there using it. I want my science to stay alive rather than stop at the moment I published it. A result belongs to when it was produced. A method should not. -The clearest illustration is a tool most people in this field have used. PoPoolation is among the most widely cited pool-seq toolkits there is, and it has been essentially untouched for over a decade. Every project still running it is running that decade's statistics, and there is nothing wrong with the software — it does what it always did. The problem is that it was finished, and the field was not. +The clearest illustration is a tool most people in this field have used. PoPoolation is among the most widely cited pool-seq toolkits there is, and it has been essentially untouched for over a decade. Every project still running it is running that decade's statistics, and there is nothing wrong with the software; it does what it always did. The problem is that it was finished, and the field was not. ### So the tool is the thing I maintain, and the tool is the thing I publish -A repository can do what a paper cannot: it can be corrected. If an estimator here turns out to be the wrong one, or a better one is published next year, the right response is to change it and say what changed — not to leave it in place because the paper describing it is already in print. +A repository can do what a paper cannot: it can be corrected. If an estimator here turns out to be the wrong one, or a better one is published next year, the right response is to change it and say what changed, not to leave it in place because the paper describing it is already in print. -That is not a slogan, it is a design constraint, and it is why the analysis layer is built the way it is. Each analysis is a **module** with its own version, its own citations and its own tests, and modules are published separately from the pipeline. A statistic can be replaced without rewriting anything around it. That seam is the mechanism that makes "maintained" a real claim rather than a promise about my future attention — and because a module is separately versioned and separately published, someone else can ship one without waiting for me. +That is not a slogan, it is a design constraint, and it is why the analysis layer is built the way it is. Each analysis is a **module** with its own version, its own citations and its own tests, and modules are published separately from the pipeline. A statistic can be replaced without rewriting anything around it. That seam is the mechanism that makes "maintained" a real claim rather than a promise about my future attention, and because a module is separately versioned and separately published, someone else can ship one without waiting for me. The obligation that comes with it is that an old result must still be explicable. Every run records the versions it ran under and every published analysis records the module and the settings that produced it, so using a current method and reproducing an old number are not in tension. @@ -4600,19 +4776,19 @@ I am aware that I am in a position to say all these and take a stance, and not e This is not a refusal to publish. I publish research through the ordinary channels when the work calls for it, and a preprint of this may follow once it is more complete rather than in heavy progress. What I am refusing is the idea that the being published in a "respected" journal is what makes it real. -The same reasoning applies to working in the open. The reflex to keep a project hidden until it is finished, in case someone takes the idea, mostly just slows everything down. During my PhD I discussed my experiment freely while it was running — seventy populations, a thousand to two thousand flies in each, carried five generations — because the honest answer was that nobody else was going to run it. **Do something that is hard to do, rather than something you have to hide.** It is not 1920. Either we build a different way of doing this or we miss it entirely. +The same reasoning applies to working in the open. The reflex to keep a project hidden until it is finished, in case someone takes the idea, mostly just slows everything down. During my PhD I discussed my experiment freely while it was running (seventy populations, a thousand to two thousand flies in each, carried five generations), because the honest answer was that nobody else was going to run it. **Do something that is hard to do, rather than something you have to hide.** It is not 1920. Either we build a different way of doing this or we miss it entirely. ### How I decide things -**I do not settle design questions by argument when I can settle them with data.** The depth-cutoff detector in this pipeline had two candidate algorithms and a long discussion behind it, and the discussion was going nowhere. Building twenty synthetic depth distributions — a few clean, the rest deliberately pathological — and looking at what each design did to them ended it in minutes. Two designs died on contact with cases that no amount of reasoning had produced. +**I do not settle design questions by argument when I can settle them with data.** The depth-cutoff detector in this pipeline had two candidate algorithms and a long discussion behind it, and the discussion was going nowhere. Building twenty synthetic depth distributions (a few clean, the rest deliberately pathological) and looking at what each design did to them ended it in minutes. Two designs died on contact with cases that no amount of reasoning had produced. I keep those distributions in the test suite rather than in a notebook, so the evidence stays executable. That is the general form of it: when I decide something on the basis of data, the data becomes part of the repository, and anyone who thinks the decision is wrong can re-run it, and propose a better alternative, if exists. This is a living project and it will only grow with feedback. -The other half is that theory has to survive contact with a real dataset. Pool-seq has plenty of estimators that are correct on paper and weak on data — because coverage is not homogeneous, because pools are not the same size, because a "reference" allele is an accident of which genome got sequenced first. Where this pipeline departs from a published formula, it is usually because the published formula assumes something a real experiment does not provide, and the manual says so at the point where it matters. +The other half is that theory has to survive contact with a real dataset. Pool-seq has plenty of estimators that are correct on paper and weak on data, because coverage is not homogeneous, because pools are not the same size, because a "reference" allele is an accident of which genome got sequenced first. Where this pipeline departs from a published formula, it is usually because the published formula assumes something a real experiment does not provide, and the manual says so at the point where it matters. ### Who this is for -Pool sequencing is unfortunately underused outside a handful of model systems. The method itself does not care whether the organism has a reference-quality genome, a large research community or a commercial kit behind it — it works on populations, which covers a lot of biological studies. What stops people is almost never the biology. It is that the tooling makes assumptions on the data type because it was created to analyze a certain type of organism or expects a bioinformatician on the team. Creating a model-agnostic tool is a challenge and I accept the probability that I might have missed some use cases. If so bring it up, I like a good challenge to make the tool more accessible. +Pool sequencing is unfortunately underused outside a handful of model systems. The method itself does not care whether the organism has a reference-quality genome, a large research community or a commercial kit behind it; it works on populations, which covers a lot of biological studies. What stops people is almost never the biology. It is that the tooling makes assumptions on the data type because it was created to analyze a certain type of organism or expects a bioinformatician on the team. Creating a model-agnostic tool is a challenge and I accept the probability that I might have missed some use cases. If so bring it up, I like a good challenge to make the tool more accessible. So accessibility is not a nice-to-have here, it is the point. A tool only its author can install and interpret is not a tool. That is why the pipeline checks its environment and explains what it is checking, why a configuration can be carried forward across releases instead of rewritten, why there is a preview that shows what a run will produce before it produces it, and why this manual explains what a number means rather than only which flag produces it. @@ -4625,7 +4801,7 @@ So accessibility is not a nice-to-have here, it is the point. A tool only its au ## How the Project Is Verified -Everything on the other pages of this section is a claim about how the project is built. This page is what makes those claims checkable rather than something you have to believe, and it is also the part of this section that has nothing to do with AI — it would be the same discipline, for the same reasons, if I had written every line by hand. +Everything on the other pages of this section is a claim about how the project is built. This page is what makes those claims checkable rather than something you have to believe, and it is also the part of this section that has nothing to do with AI: it would be the same discipline, for the same reasons, if I had written every line by hand. ### Scientific verification @@ -4636,11 +4812,11 @@ A test can confirm that a formula was implemented as written, and that it behave In practice that is a specific and fairly slow kind of work, and it is worth saying what it looks like rather than leaving it as a principle: - **Deriving a quantity rather than porting it.** Where this pipeline computes something with a published name, the constant in front of it is worked out by hand and checked against a case computed by hand, because transcribing a formula from a paper is exactly how a factor of two travels silently from one codebase into the next. -- **Reading the primary source, not the attribution.** Method citations here are checked against the paper before they ship. That is not ceremony: the effective-sample-size correction this pipeline uses was attributed to the wrong source twice before it was right, and the paper it is now cited for never writes the quantity down — it is cited for the result the quantity follows from, with a note saying so. +- **Reading the primary source, not the attribution.** Method citations here are checked against the paper before they ship. That is not ceremony: the effective-sample-size correction this pipeline uses was attributed to the wrong source twice before it was right, and the paper it is now cited for never writes the quantity down; it is cited for the result the quantity follows from, with a note saying so. - **Testing against data whose answer is known by construction**, rather than against what the code currently produces. If the expectation comes from the implementation, the test can only tell me the code has not changed. -- **Asking what the estimator assumes, and whether a real experiment provides it.** Pooled data breaks assumptions that individual genotypes satisfy — unequal pool sizes, non-homogeneous coverage, no privileged reference allele. Where a standard quantity needs adjusting before it means anything here, the manual says so at the point where the number is produced. +- **Asking what the estimator assumes, and whether a real experiment provides it.** Pooled data breaks assumptions that individual genotypes satisfy: unequal pool sizes, non-homogeneous coverage, no privileged reference allele. Where a standard quantity needs adjusting before it means anything here, the manual says so at the point where the number is produced. -None of that is automatable, and it is the part of the work an AI agent is least able to help with — it will produce a correct implementation of the wrong statistic, and produce a simulation showing the implementation is correct. The implementation being correct is not the question. +None of that is automatable, and it is the part of the work an AI agent is least able to help with: it will produce a correct implementation of the wrong statistic, and produce a simulation showing the implementation is correct. The implementation being correct is not the question. So: the suite below tells me I have not broken what I built. It does not tell me I built the right thing. That judgment is mine, and where I have made it the manual says what was chosen and what it assumes, so it is at least visible enough for someone to disagree with. @@ -4648,11 +4824,11 @@ So: the suite below tells me I have not broken what I built. It does not tell me The suites are organized by **seam** rather than by file: the wrapper, configuration migration, parameter resolution, the change guards, the helper programs, the dry run, the analysis frame and the static checks each get one. Most ship with the pipeline, and **each analysis module ships its own**, because a module's tests belong to the module and travel with it when it is published separately. `test/run_tests.sh --list` prints the current set with what each one costs; there is no case count written here, because it moves with every stage and a number in a manual is a number that goes quietly out of date. -Most of them are not testing what people usually mean by a test. The failures this pipeline can produce are rarely crashes — they are a run that completes, reports success, and gives a number that is quietly wrong, because a filter was applied with the wrong pool's threshold or a step reused an artifact that was produced under different settings. So a large share of the suite exists to catch a *plausible* result rather than a broken one. +Most of them are not testing what people usually mean by a test. The failures this pipeline can produce are rarely crashes; they are a run that completes, reports success, and gives a number that is quietly wrong, because a filter was applied with the wrong pool's threshold or a step reused an artifact that was produced under different settings. So a large share of the suite exists to catch a *plausible* result rather than a broken one. ### A suite you will not run is not a suite -The whole thing takes about three quarters of an hour, and that is the real problem to solve. A forty-minute gate between me and a one-line change does not make me careful, it makes me skip the check — so the suite is arranged so that the run I actually do is small. +The whole thing takes about three quarters of an hour, and that is the real problem to solve. A forty-minute gate between me and a one-line change does not make me careful, it makes me skip the check. So the suite is arranged so that the run I actually do is small. Every suite declares what it costs to run, in its own header: @@ -4664,7 +4840,7 @@ Every suite declares what it costs to run, in its own header: The `static` set finishes in seconds on a machine with nothing set up at all, which makes it the loop I develop in. The full run belongs to a release, not to a change. -There is a second axis alongside it. **`--changed` picks the suites for me**, from what each suite declares it covers, expanded through the include graph — so editing a helper program selects the suites that exercise it and nothing else. It deliberately errs wide: touching the test library or the selector itself selects everything. That way a narrow answer is trustworthy and a wide one is merely expensive. +There is a second axis alongside it. **`--changed` picks the suites for me**, from what each suite declares it covers, expanded through the include graph. Editing a helper program therefore selects the suites that exercise it and nothing else. It deliberately errs wide: touching the test library or the selector itself selects everything. That way a narrow answer is trustworthy and a wide one is merely expensive. The rule I hold myself to is that **the cases for a step are written with the step and run on their own**, by name. The full suite is a release gate. Treating it as the per-change gate is how you end up not checking anything. @@ -4672,29 +4848,29 @@ The rule I hold myself to is that **the cases for a step are written with the st Some things cannot be asserted from inside the suite, so they are separate gates and all of them run before a release: -- **`nextflow lint`** over every workflow file, at zero errors *and* zero warnings. The strict parser rejects a good deal of ordinary Groovy, and it reports a parse failure in one file as "not defined" at every call site in *other* files — so a clean lint is worth more than it sounds. +- **`nextflow lint`** over every workflow file, at zero errors *and* zero warnings. The strict parser rejects a good deal of ordinary Groovy, and it reports a parse failure in one file as "not defined" at every call site in *other* files, so a clean lint is worth more than it sounds. - **The citation check.** Every reference is authored once in BibTeX and compiled to the JSON the pipeline reads; the gate regenerates it and fails if the two disagree, so the file a run cites from cannot drift from the file I edit. -- **The manual check.** This manual is one file, and every page of the site is generated from it. The gate re-parses it, resolves every cross-reference, and fails on a link to a heading that no longer exists or two headings that would collide — which is what stops the documentation rotting quietly as things are renamed. -- **The version check**, which fails when the analysis frame, a module or a library has changed without its own version moving, and the **archive check**, which builds the release tarball and asserts that everything a user needs is in it and everything they do not need is out — modules included, since none of them ships. +- **The manual check.** This manual is one file, and every page of the site is generated from it. The gate re-parses it, resolves every cross-reference, and fails on a link to a heading that no longer exists or two headings that would collide. That is what stops the documentation rotting quietly as things are renamed. +- **The version check**, which fails when the analysis frame, a module or a library has changed without its own version moving, and the **archive check**, which builds the release tarball and asserts that everything a user needs is in it and everything they do not need is out (modules included, since none of them ships). ### What this does not do -It does not tell me the science is right — that is [Scientific verification](#scientific-verification) above, and it is the limit that matters most. +It does not tell me the science is right. That is [Scientific verification](#scientific-verification) above, and it is the limit that matters most. -It also does not cover everything. The cross-filesystem paths in the artifact-moving code cannot be reached from the suite, because the sandbox is a single filesystem — the evidence for those is measurement recorded in the development notes, and nothing in the automated run reproduces it. I would rather say that plainly than let a green run imply more than it covers. +It also does not cover everything. The cross-filesystem paths in the artifact-moving code cannot be reached from the suite, because the sandbox is a single filesystem. The evidence for those is measurement recorded in the development notes, and nothing in the automated run reproduces it. I would rather say that plainly than let a green run imply more than it covers. And a suite is only as good as the cases in it. When I fix something, I check that the new case actually fails against the unfixed code before I keep it. A test that passes both ways is worse than no test, because it reads as coverage. ## Involvement of Claude Code -From **v2.0.0** onward this pipeline has been developed with Claude Code, an AI coding agent, working against my review. Everything before v2.0.0 — the pool-seq reasoning, the filtering approach, the Nextflow pipeline itself — I wrote alone. This page says what the arrangement actually is, because "developed with AI" covers a range wide enough to be meaningless, and because I would rather describe it than have it guessed at. +From **v2.0.0** onward this pipeline has been developed with Claude Code, an AI coding agent, working against my review. Everything before v2.0.0 I wrote alone: the pool-seq reasoning, the filtering approach, the Nextflow pipeline itself. This page says what the arrangement actually is, because "developed with AI" covers a range wide enough to be meaningless, and because I would rather describe it than have it guessed at. The short version: **the work got faster and the expertise it demands went up, not down.** ### What the loop looks like -I decide what to build and why. The agent proposes an implementation, usually with an argument for it. I read the code and the argument, and most of the time I send something back — this is wrong, this is slower than it needs to be, this contradicts a decision we already made, this assumption does not hold on real data. Then it changes, and I read it again. Work stays in the working tree until I have reviewed it; nothing is committed that I have not seen. +I decide what to build and why. The agent proposes an implementation, usually with an argument for it. I read the code and the argument, and most of the time I send something back: this is wrong, this is slower than it needs to be, this contradicts a decision we already made, this assumption does not hold on real data. Then it changes, and I read it again. Work stays in the working tree until I have reviewed it; nothing is committed that I have not seen. That is the whole method. It is not novel and it is not automated. What is worth writing down is what it demands from the person doing the reviewing. @@ -4702,33 +4878,33 @@ That is the whole method. It is not novel and it is not automated. What is worth **The errors run in both directions, and the corrections do too.** -The agent has also come back with the opposite of a plausible wrong answer: a confident alarm, graded FATAL, about a result being scientifically incorrect — and been wrong. One review concluded that the pipeline never removes PCR duplicates, which would make every allele frequency it produces suspect. It does: duplicates are marked and removed during cleaning, several steps before anything is called. Another claimed a distance measure "produces negative eigenvalues", stated flatly; measuring it across realistic population structures showed that it usually does not, and only one particular shape produced a single small one. +The agent has also come back with the opposite of a plausible wrong answer: a confident alarm, graded FATAL, about a result being scientifically incorrect. The alarm was wrong. One review concluded that the pipeline never removes PCR duplicates, which would make every allele frequency it produces suspect. It does: duplicates are marked and removed during cleaning, several steps before anything is called. Another claimed a distance measure "produces negative eigenvalues", stated flatly; measuring it across realistic population structures showed that it usually does not, and only one particular shape produced a single small one. -Both were the same mistake in a different direction. Code written to answer a biological question carries context that is not local to the lines being read — the duplicate removal is real but it happens in a different file at a different stage, and whether an eigenvalue is negative depends on data the code does not contain. A model reading the code cannot see either. **On those occasions the scientist has to know, and hold the position** — a FATAL verdict about your own analysis is exactly the moment you are least inclined to argue, and sometimes arguing is correct. +Both were the same mistake in a different direction. Code written to answer a biological question carries context that is not local to the lines being read: the duplicate removal is real but it happens in a different file at a different stage, and whether an eigenvalue is negative depends on data the code does not contain. A model reading the code cannot see either. **On those occasions the scientist has to know, and hold the position**. A FATAL verdict about your own analysis is exactly the moment you are least inclined to argue, and sometimes arguing is correct. None of which means I am the one who is always right. It means the two of us are wrong about **different things**, and it is worth being precise about which. -The science is the part I can stand on. Ten years of working on these questions and six (on and off) on this pipeline is what lets me say a proposal is wrong before I can say why, and that judgment has held up. **Where I have been wrong is the engineering** — repeatedly, and about shape rather than correctness: +The science is the part I can stand on. Ten years of working on these questions and six (on and off) on this pipeline is what lets me say a proposal is wrong before I can say why, and that judgment has held up. **Where I have been wrong is the engineering**, repeatedly, and about shape rather than correctness: - I had a whole "per-window callable-sites" track built into the reporting step, and took it back out the same day. The pipeline was producing an artifact on my guess that somebody would eventually want it, and it was derivable from data already being kept. The measurements behind that are still in the development notes, because they turned out to size a different part of the system. And more importantly it was expensive on the storage. - I gave the analysis layer a wrapper of its own, so that the main command would not have to take an argument. Then I merged it back and amended the contract instead, because once the shared surface was visible the second executable was plainly the more expensive of the two. - I added flag-style subcommands for about an hour before reversing them to bare words, for consistency with the rest of the CLI. -Those are architecture calls, and I make them with much less certainty than I make the scientific ones. Each cost time to build and throw away — and **this is one of the places the agent genuinely earns its keep**, because building the wrong shape fast enough to discover it is wrong is far better than reasoning about the right shape indefinitely. A feature built and reverted in a day is cheap (kind of). The same mistake argued about for a week and then built is not. +Those are architecture calls, and I make them with much less certainty than I make the scientific ones. Each cost time to build and throw away, and **this is one of the places the agent genuinely earns its keep**, because building the wrong shape fast enough to discover it is wrong is far better than reasoning about the right shape indefinitely. A feature built and reverted in a day is cheap (kind of). The same mistake argued about for a week and then built is not. -So: neither my intuition nor the agent's is the authority. **What settles it is the data** — build the case that would distinguish the two claims, run it, and read what comes back. Nearly every argument in this project that took more than an hour ended that way rather than in agreement. +So: neither my intuition nor the agent's is the authority. **What settles it is the data**: build the case that would distinguish the two claims, run it, and read what comes back. Nearly every argument in this project that took more than an hour ended that way rather than in agreement. **A wrong answer that looks wrong costs nothing.** You read it, you reject it, you move on. That is not the failure mode here. -What actually arrives is a locally coherent proposal, often with evidence behind it, that violates something the project settled months earlier. The clearest example: for the phenotype-association module the agent proposed a correction for multiple testing across the alleles at a site, argued it carefully, and backed it with a simulation over two hundred thousand replicates showing the type-I error rate was controlled exactly as claimed. The simulation was correct. The proposal was still wrong, because it worked by discarding the reference allele — and the reference allele in a pool-seq cohort is not a privileged one, it is whichever allele happened to be most common when the call set was polarized. Dropping it privileges "not the most common allele" and throws away the row that carries the signal when two alternates rise together. +What actually arrives is a locally coherent proposal, often with evidence behind it, that violates something the project settled months earlier. The clearest example: for the phenotype-association module the agent proposed a correction for multiple testing across the alleles at a site, argued it carefully, and backed it with a simulation over two hundred thousand replicates showing the type-I error rate was controlled exactly as claimed. The simulation was correct. The proposal was still wrong, because it worked by discarding the reference allele. And the reference allele in a pool-seq cohort is not a privileged one, it is whichever allele happened to be most common when the call set was polarized. Dropping it privileges "not the most common allele" and throws away the row that carries the signal when two alternates rise together. -Nothing in the code or the simulation showed that. It was only visible if you already held the principle. I caught it in three words — *this sounds wrong* — and it took another day's work to establish what the right answer was. +Nothing in the code or the simulation showed that. It was only visible if you already held the principle. I caught it in three words (*this sounds wrong*), and it took another day's work to establish what the right answer was. -That pattern repeated. A claim that a particular distance measure "produces negative eigenvalues" turned out to be too strong once measured: it usually does not, and only one realistic population structure produced one. A performance decision was sized against one to five million sites, when the honest planning figure for the genomes people actually use is a hundred million — the decision inverted once that was said out loud. A citation for an effective-sample-size correction was attributed to the wrong source twice before it was right. A statistic was implemented in a form that did not answer the question it was named for, and both versions were being carried at once. +That pattern repeated. A claim that a particular distance measure "produces negative eigenvalues" turned out to be too strong once measured: it usually does not, and only one realistic population structure produced one. A performance decision was sized against one to five million sites, when the honest planning figure for the genomes people actually use is a hundred million. The decision inverted once that was said out loud. A citation for an effective-sample-size correction was attributed to the wrong source twice before it was right. A statistic was implemented in a form that did not answer the question it was named for, and both versions were being carried at once. Every one of those is a **plausible** wrong answer. Someone without the experience to hold the principles firmly ships all of them, and the results look completely ordinary. -That is the thesis, and it is the opposite of the usual claim made for these tools: generating candidate solutions got cheap, so the entire bottleneck moved onto evaluating them — and judgment does not compress. I spend more of my time thinking about this project than I did when I was writing every line myself. +That is the thesis, and it is the opposite of the usual claim made for these tools: generating candidate solutions got cheap, so the entire bottleneck moved onto evaluating them, and judgment does not compress. I spend more of my time thinking about this project than I did when I was writing every line myself. ### A second, different failure mode @@ -4736,23 +4912,23 @@ The errors above are about analysis. There is another kind that has nothing to d Over the course of this work the agent has: run the full test suite after I explicitly asked it not to, costing the better part of an hour; committed work I had not reviewed, against a standing instruction not to; and reported a test run as passing when twelve cases had failed, because it piped the output through `tail` and the tail of a failure summary is a list of case names that looks exactly like a list of passes. -None of those is a wrong analysis. They are an unreliable process, and no amount of domain expertise on my side prevents them — I can only catch them afterwards. **Anything that has to hold has to be enforced by a check rather than by an instruction.** That is why the rules below are split the way they are: some are for judgment, and those live in prose because judgment cannot be automated; the rest are tests, because prose does not bind. +None of those is a wrong analysis. They are an unreliable process, and no amount of domain expertise on my side prevents them. I can only catch them afterwards. **Anything that has to hold has to be enforced by a check rather than by an instruction.** That is why the rules below are split the way they are: some are for judgment, and those live in prose because judgment cannot be automated; the rest are tests, because prose does not bind. ### The rules this produced Every one of these would improve a project with no AI anywhere near it. What the agent did was make them load-bearing rather than advisable, by supplying a steady stream of the exact mistakes each one prevents. -**A comment says what the code does; a decision goes somewhere else.** A design decision left in a comment reads to whoever comes next as a current constraint, and gets argued from long after it stopped being true — with an agent, that means yesterday's abandoned choice comes back as an objection to what I am asking for today. The opposite mistake is on record too: a file sat in the output directory of every run from 1.0 onward because the reason a deletion had been placed where it was never got written down, and the deletion later moved. So the test is not "why versus what", it is whether the code becomes inexplicable without the line. +**A comment says what the code does; a decision goes somewhere else.** A design decision left in a comment reads to whoever comes next as a current constraint, and gets argued from long after it stopped being true. With an agent, that means yesterday's abandoned choice comes back as an objection to what I am asking for today. The opposite mistake is on record too: a file sat in the output directory of every run from 1.0 onward because the reason a deletion had been placed where it was never got written down, and the deletion later moved. So the test is not "why versus what", it is whether the code becomes inexplicable without the line. -**Two things that must agree are checked, never kept in step by hand.** The wrapper's file list against what the release archive actually ships; the column definitions in the Python parser against the Groovy that renders them; paths computed in two places from the same values. Each pair has a test that re-derives both sides. A pair kept in step by care drifts silently and produces a wrong result rather than a failure — and where two lists are *supposed* to differ, the test says so and says why, so the difference is a decision rather than a discovery. +**Two things that must agree are checked, never kept in step by hand.** The wrapper's file list against what the release archive actually ships; the column definitions in the Python parser against the Groovy that renders them; paths computed in two places from the same values. Each pair has a test that re-derives both sides. A pair kept in step by care drifts silently and produces a wrong result rather than a failure. Where two lists are *supposed* to differ, the test says so and says why, so the difference is a decision rather than a discovery. -**A test's expectations must not come from the code it tests.** The obvious way to write a corpus down is to run the thing and record what it said, which produces a test that cannot fail — a changelog with assertions, locking in whatever the code did that day. The depth-detector corpus derives every bound from how each case was constructed instead, and only the generator is committed, so there is no expectation file anyone can quietly edit to make a failing test pass. +**A test's expectations must not come from the code it tests.** The obvious way to write a corpus down is to run the thing and record what it said, which produces a test that cannot fail: a changelog with assertions, locking in whatever the code did that day. The depth-detector corpus derives every bound from how each case was constructed instead, and only the generator is committed, so there is no expectation file anyone can quietly edit to make a failing test pass. -**Fail loudly, or document — never automate away the decision.** Where the right answer depends on something the pipeline cannot know, it reports and stops. Two depth profiles are deliberately left uncapped because they are indistinguishable from a library that simply ran deep; two runs that disagree about how to build a shared reference are refused by name rather than quietly split. And a computed default never removes the knob: everything derived can be set by hand, and a hand-set value still feeds what is computed from it. +**Fail loudly, or document; never automate away the decision.** Where the right answer depends on something the pipeline cannot know, it reports and stops. Two depth profiles are deliberately left uncapped because they are indistinguishable from a library that simply ran deep; two runs that disagree about how to build a shared reference are refused by name rather than quietly split. And a computed default never removes the knob: everything derived can be set by hand, and a hand-set value still feeds what is computed from it. **Recompute rather than record.** Three times, in unrelated parts of the system, the question was whether to write something down for a later stage to read back. Three times the answer was to recompute it, because a record can be edited after the fact and a derivation keyed on the same identity cannot disagree with the run it describes. -**Commit only what has been reviewed.** Work stays in the working tree until I have read it. The tree is the review surface — one diff, one view — and committing something I have not seen removes my chance to reject it before it becomes history. +**Commit only what has been reviewed.** Work stays in the working tree until I have read it. The tree is the review surface (one diff, one view), and committing something I have not seen removes my chance to reject it before it becomes history. ### What this does not transfer @@ -4766,40 +4942,78 @@ I also cannot tell you how much of the code here the agent wrote, and I am suspi Both of these are in the repository rather than summarized here, because a description of a working method is worth much less than the thing itself. -**`CLAUDE.md`**, at the repository root, is the file the agent reads at the start of every session. It is the rules on this page in the form they are actually given — 108 lines, written as instructions rather than as prose about instructions. It is worth reading against this page: the difference between how a rule is explained and how it has to be worded to hold is most of what I learned. +**`CLAUDE.md`**, at the repository root, is the file the agent reads at the start of every session. It is the rules on this page in the form they are actually given (108 lines, written as instructions rather than as prose about instructions). It is worth reading against this page: the difference between how a rule is explained and how it has to be worded to hold is most of what I learned. **`.claude/development-notes/`** is the record of how the project got here: 21 notes, about 1,900 lines, covering design churn, alternatives tried and dropped, what things used to be, the measurements behind particular choices, and who decided what and when. A whole feature that was built and then reverted is kept there as a patch rather than deleted, because the sizing work behind the decision to revert it is what shaped the analysis layer. -Each note carries the date it was written and the commit it was written against, and is **not** updated to follow the code. They are a record of how the project got here, not a second manual — where a note and this manual disagree, the manual is right and the note is history. The one exception says so at the top: the file of platform traps is appended to as they are found, because a trap does not expire. +Each note carries the date it was written and the commit it was written against, and is **not** updated to follow the code. They are a record of how the project got here, not a second manual. Where a note and this manual disagree, the manual is right and the note is history. The one exception says so at the top: the file of platform traps is appended to as they are found, because a trap does not expire. Both are development material and are kept out of release downloads, so they live in the repository and on the project's page rather than in a tarball. -# Reference - +# Troubleshooting + -## Troubleshooting - +These pages are arranged by what you saw rather than by which part of the pipeline did it: the install refusing, the run not starting, a step failing partway through, results that look wrong, a re-run doing more or less than you expected, or an analysis module misbehaving. Start with whichever one describes your symptom. -### Installation and environment +**If nothing here matches, please open an issue** at [github.com/ozankiratli/PoolSeqFlow/issues](https://github.com/ozankiratli/PoolSeqFlow/issues). A problem that is not on these pages is worth reporting even if you find a way around it, because the fix usually belongs in the pipeline rather than in your project. [Getting help](#getting-help) says what to include, which is what decides whether the first reply answers you or asks you for more detail. + +## Installation and environment + | Problem | Cause and fix | |---|---| | Environment creation fails | `conda update -n base conda`, then retry `./PoolSeqFlow install` | -| `nothing provides __glibc >=N` | Your machine's glibc is older than this release's floor of 2.28. `ldd --version` prints what you have; the floor and why it exists are under [The glibc floor](#glibc-floor). There is no flag for it — use a newer machine | +| `nothing provides __glibc >=N` | Your machine's glibc is older than this release's floor of 2.28. `ldd --version` prints what you have; the floor and why it exists are under [The glibc floor](#glibc-floor). There is no flag for it: use a newer machine | | Missing dependencies after install | Activate it: `conda activate PoolSeqFlow` | -| A tool is found but misbehaves | Check whether `params.software.*` points at a system binary rather than the environment's — version mismatches are not detected | +| A tool is found but misbehaves | Check whether `params.software.*` points at a system binary rather than the environment's; version mismatches are not detected | + +### Why Linux only { #why-not-windows } + +**Windows.** The pipeline moves each file it produces to where it belongs and leaves a symbolic link behind, and it relies on Unix path semantics throughout. Neither behaves correctly on native Windows filesystems, and WSL only works under some filesystem configurations, which is not a guarantee worth documenting. See [Symbolic links and the two storage roots](#symbolic-links-and-the-two-storage-roots). -### The run will not start +**macOS.** Every tool a release runs is pinned to an exact build in `install/environment.yml` and `install/environment-analysis.yml`, and those files are exports from a Linux machine: they name builds that exist for `linux-64` and for no other platform. Conda cannot solve either of them on a Mac, so the install fails before anything else is reached. Supporting macOS means a second set of pinned files exported and verified on a Mac (one for Apple Silicon and one for Intel, since conda treats them as different platforms), and that is a piece of work rather than a flag. Earlier releases listed macOS as supported; that was never true of the pinned environments, and saying so was the error. -#### `null: command not found` +### The glibc floor { #glibc-floor } + +**The floor is glibc 2.28, and it applies to both environments.** In distribution terms that is RHEL and Rocky 8, Debian 10, Ubuntu 18.10, and everything released after them. The one common machine it excludes is CentOS 7, which reached end of life in June 2024. + +`__glibc` is conda's name for the glibc your machine has, and `ldd --version` prints it. Below the floor, conda refuses the solve rather than installing something that would fail later, and the message names a virtual package instead of a distribution: + +```text +LibMambaUnsatisfiableError: Encountered problems while solving: + - nothing provides __glibc >=2.28 needed by rsync-3.4.4-hffd6c76_1 +``` + +**No flag works around it.** Conda offers one that forces the solve, and using it would install binaries the system cannot run, turning a refusal you get in seconds into a crash you get hours into a run. If your machine is below the floor, use a newer one. + +**What sets it is `rsync`, which both environments carry.** Every artifact the pipeline produces is moved to permanent storage by a copy that is verified before the original is removed, and `rsync` is what stages that copy. So the floor is a property of the whole release rather than of the analysis layer: the analysis environment additionally carries a C and C++ compiler, because every module builds its hot path on your machine rather than shipping a binary, but that toolchain reaches back further than `rsync` does and is not what decides this. + +### Coming from 2.2.0 + +3.0 was a structural release rather than a handful of new settings. [Upgrading](#upgrading) is the procedure; this is what changed under it. + +| What changed | What to do | +|---|---| +| **One storage root became two**, and they must be different paths: `mainDir` holds your reads, reference and configuration and is where you run; `storageDir` holds finished results | `migrate_config` prints the `mv` commands for the files that move | +| **`projectDir` is now `storageDir`**, because `projectDir` is a name Nextflow defines for itself | A straight rename; your value is carried over | +| **`RGTags.csv` is replaced by `metadata.csv`.** The old file held raw SAM read-group tags and nothing else; the new one has [seven kinds of column](#kinds-of-metadata-column), carries your pool sizes, and gives the experiment itself (populations, timepoints, replicates) somewhere to live | Not a rename, and not migratable: `rgTagsFile` is reported as no longer used and rewriting the file is yours to do. Start from `$POOLSEQFLOW_HOME/metadata.csv.template`, which documents every column, and read [Metadata](#metadata). **No run starts until `metadata.csv` exists**, so it cannot be deferred | +| **The depth ceiling moved.** `bcftools.maxDepth = 2000` was the only depth control there was, one number for every pileup in the run. A ceiling is now measured for each sample from its own depth histogram and applied to the BAM before calling ([Depth capping](#depth-capping)), and `variantCall.maxDepth` is a second ceiling on top of it, shipping as `0`, which `mpileup` reads as no limit at all | Your old value is deliberately not carried: it would cap you at a number this release never chose, on top of a per-sample cap that cannot see it. It is reported under `Format changed this release`. Automatic capping is `capBAM.maxDepth = -1`, which is what you now have. **To reproduce 2.2.0 results exactly, set `variantCall.maxDepth` back to your old value and `capBAM.maxDepth = 0`** | +| **The installation is separate from your project.** Earlier releases ran from the folder you unpacked, with `parameters.config` beside the pipeline | Install once, and run the installed command from your own project directory. If your project *is* the old unpacked folder, move it out: the pipeline refuses to run inside its own installation | +| **Some parameters are gone**: `params.gff`, `params.dir.scripts`, `params.dir.output.temp`, `rgTagsFile` and `rgTagsPath` | Each is reported under `No longer used`. The `cores` block and the `options` strings look gone and are not: they are computed now and ship commented out, under `Still yours to set` | +| **Multi-run is new**, and off | `multiRun = false` changes nothing about how an existing project behaves | + +## The run will not start + + +### `null: command not found` ```text .command.sh: line 17: null: command not found ``` -Your `parameters.config` predates the installed version. An absent parameter interpolates as the literal string `null`, which is why the error names nothing useful and points at a generated script. Rebuild the file — see [Upgrading](#upgrading). +Your `parameters.config` predates the installed version. An absent parameter interpolates as the literal string `null`, which is why the error names nothing useful and points at a generated script. Rebuild the file (see [Upgrading](#upgrading)). -#### `Process requirement exceeds available CPUs` +### `Process requirement exceeds available CPUs` ```text Process requirement exceeds available CPUs -- req: 12; avail: 8 @@ -4807,158 +5021,234 @@ Process requirement exceeds available CPUs -- req: 12; avail: 8 `threads` is larger than the machine. Tasks reserve what they really use, so an oversized request fails at submission rather than quietly oversubscribing. Set `threads` to the cores you have. [Resources →](#resources) -#### `RUN PARAMETER CHECK` or `METADATA CHANGE CHECK` fails +### `RUN PARAMETERS` names a setting that would produce nothing { #parameter-refusal } + +```text +RUN PARAMETERS: FAIL variantCall.scaleMapQ: DISCARDS EVERY READ (15 is below varQualMin 30) +RUN PARAMETERS: -C caps every read's mapping quality near 15 and -q then +RUN PARAMETERS: rejects anything under 30, so the pileup reaches bcftools +RUN PARAMETERS: call empty and the run produces no variants at all. +RUN PARAMETERS: A setting above would make this run produce nothing. +``` + +Not the change guard: nothing about your existing results is wrong. A setting in `parameters.config` would make the run produce no output at all, and it is being stopped now rather than after the hours of alignment and calling it would waste. The message names the parameter, both values where two interact, and what to change. + +**`WARN` and `NOTE` lines in the same section do not stop anything.** A warning means the setting works and costs more than it looks like it does; a note means it changes what a published number means. Only `FAIL` stops a run. [Every rule, and what each level means ->](#parameter-checks) + +`PoolSeqFlow check project` applies the same rules from the same implementation, so you can see all of this before starting. + +### `RUN PARAMETER CHECK` or `METADATA CHANGE CHECK` fails Working as designed. An analysis-affecting parameter, or the analysis-affecting part of `metadata.csv`, differs from what produced your existing outputs. Because completed steps are skipped by looking for output files, continuing would mix results from two configurations in one folder. -The report names what to delete, and how much that is depends on what you changed — a reordered file invalidates less than an edited tag value, and a changed pool size less again. Deleting what it names clears the check. Or `PoolSeqFlow reset` to discard everything. [Why →](#the-run-refuses-to-mix-settings) +The report names what to delete, and how much that is depends on what you changed. A reordered file invalidates less than an edited tag value, and a changed pool size less again. Deleting what it names clears the check. Or `PoolSeqFlow reset` to discard everything. [Why →](#the-change-guard) -#### `PIPELINE VERSION` fails +### `PIPELINE VERSION` fails -These results were produced by a different release. A project belongs to one release, so this one is absolute: nothing else is compared and there is nothing to delete selectively. Either finish the project under the release that started it — every installed version is on your `PATH` by its own name — or `PoolSeqFlow reset` and start again under this one. [Why →](#a-project-belongs-to-one-release) +These results were produced by a different release. A project belongs to one release, so this one is absolute: nothing else is compared and there is nothing to delete selectively. Either finish the project under the release that started it (every installed version is on your `PATH` by its own name) or `PoolSeqFlow reset` and start again under this one. [Why →](#a-project-belongs-to-one-release) -#### `METADATA CHECK` reports a duplicate `SampleID` +### `METADATA CHECK` reports a duplicate `SampleID` -A `SampleID` appears more than once. A row is looked up by it and only the first match is read, so the duplicate would have silently given a sample the wrong read-group tags — producing a valid BAM that nothing downstream could flag. Every problem in the file is reported at once, with line numbers, so fix the whole list before rerunning. +A `SampleID` appears more than once. A row is looked up by it and only the first match is read, so the duplicate would have silently given a sample the wrong read-group tags, producing a valid BAM that nothing downstream could flag. Every problem in the file is reported at once, with line numbers, so fix the whole list before rerunning. -#### `DIRECTORY CHECK` fails +### A sample `has reads but no row`, and a row has no reads { #reads-but-no-row } -`mainDir` and `storageDir` are the same path, or one of them is the installation. They are two storage tiers and an output moving from one to the other is what marks it finished, which cannot mean anything if they are one place. The installation is a tool that is replaced wholesale on upgrade, so a project inside it would not survive one. [Why →](#symbolic-links-instead-of-copies) +```text +Sample 'A08781_S58' has reads but no row in metadata.csv +NOTE: metadata.csv has a row for 'A08781', which has no reads in Data/ +``` + +**These two lines are one problem, and the answer is the difference between them.** One name has files and no row, the other has a row and no files, and they differ by a suffix your sequencer added. Illumina's demultiplexing appends `_S`, the sample's row number in the sample sheet, so the sample you call `A08781` arrives as `A08781_S58_R1.fq.gz`. -### Failures during the run +`SampleID` is what joins a row to its files, so it has to carry the whole delivered name. **Lengthen `SampleID`, and never rename the files.** Then put the name you want your results under in `RG_Sample`: + +```csv +SampleID,RG_Sample +A08781_S58,A08781 +``` -#### `no usable clip range` +With sixty samples you get sixty of each line. Listing the identifiers from the files themselves is faster than reading them off a sample sheet: + +```bash +find Data -name '*_R1.fq.gz' | sed 's|.*/||; s|_R1\.fq\.gz$||' | sort +``` + +[Sequencer suffixes ->](#when-your-file-names-carry-a-sequencer-suffix) + +### `DIRECTORY CHECK` fails + +`mainDir` and `storageDir` are the same path, or one of them is the installation. They are two storage tiers and an output moving from one to the other is what marks it finished, which cannot mean anything if they are one place. The installation is a tool that is replaced wholesale on upgrade, so a project inside it would not survive one. [Why →](#symbolic-links-and-the-two-storage-roots) + +## Failures during the run + + +### `no usable clip range` ```text CLIPPING READS : ERROR: no usable clip range in CLIPPING READS : exit 3 = unexpected FastQC header; 4 = no cycle within at_gc_error (0.025) ``` -Exit **4** means no read cycle had A/T and G/C ratios inside `at_gc_error`. On a GC-skewed genome this is expected, not a fault — raise `at_gc_error`. +Exit **4** means no read cycle had A/T and G/C ratios inside `at_gc_error`. On a GC-skewed genome this is expected, not a fault. Raise `at_gc_error`. Exit **3** means the FastQC per-base composition table did not have the expected `A`/`T`/`G`/`C` columns, which points at a FastQC version change or a corrupt report. [Trimming & Clipping →](#when-it-refuses-to-run) -#### Symbolic link errors +### Symbolic link errors -Confirm you are on Linux. macOS and Windows — including WSL under some filesystem configurations — are not supported. Also check that `storageDir` is still mounted and was not cleared while the run was in flight. +Confirm you are on Linux. macOS and Windows (including WSL under some filesystem configurations) are not supported. Also check that `storageDir` is still mounted and was not cleared while the run was in flight. -#### A step fails and I cannot tell why +### A step fails and I cannot tell why `.nextflow.log` names the failing process. Each step also mirrors its own `.command.log` and `.command.err` into `Logs//`, which is usually more readable. For a reproducible failure, set `threads = 1`. That removes concurrency as a variable and makes the logs sequential. -### Results are not what I expected +## Results are not what I expected + + +### Fewer sample columns than samples + +Rows in `metadata.csv` sharing an `RG_Sample` are merged into one VCF column and their depths add together. Eight FASTQ pairs with four distinct `RG_Sample` values give four columns (usually intentional, occasionally not). The pooling is printed at the start of every run, before any compute. [Metadata ->](#rg_sample-decides-what-counts-as-a-sample) -#### Fewer sample columns than samples +### A pool is named after my files, or one sample became two pools { #pool-named-after-files } -Rows in `metadata.csv` sharing an `RG_Sample` are merged into one VCF column and their depths add together. Eight FASTQ pairs with four distinct `RG_Sample` values give four columns — usually intentional, occasionally not. The pooling is printed at the start of every run, before any compute. [Metadata →](#rg_sample-decides-what-counts-as-a-sample) +`RG_Sample` is blank, so it was filled from `SampleID`, and `SampleID` carries whatever your sequencer put in the file name. Nothing failed: the run completed and the name simply traveled. `A08781_S58` is now the VCF column, the frequency table header, and the key every analysis result is filed under, with a sample sheet's row number inside it. -#### Sample columns in an unexpected order +The more expensive version of the same cause is a sample sequenced across two lanes. It arrives as `A08781_S58_L001` and `A08781_S58_L002`, and with `RG_Sample` blank those are two different pools rather than one: each carries half the depth it should, and the [cross-sample filter](#6-false-positive-filter-step-7) sees twice as many pools as you have, which changes which alleles survive it. + +**Step 0 prints the pooling it resolved before any compute is spent**, and that line is where to catch this: + +```text +METADATA CHECK: A08781 is one column, pooling A08781_S58_L001, A08781_S58_L002 +``` + +If it names your files rather than your samples, fill in `RG_Sample` and rerun. Doing that changes the pooling, so the run will ask you to clear the outputs produced under the old one, which is correct: those numbers really were computed over the wrong pools. [Sequencer suffixes ->](#when-your-file-names-carry-a-sequencer-suffix) + +### Sample columns in an unexpected order Column order follows `metadata.csv` **row order**. Where rows share an `RG_Sample`, the merged column takes the position of the first of them. [Metadata →](#row-order-decides-column-order) -#### `REF` does not match my reference genome +### `REF` does not match my reference genome Correct. Step 7 re-encodes each site so the most-read allele across the whole cohort becomes `REF`, which is what makes frequencies comparable across samples and runs. If you need the assembly's base, take it from the assembly. [Why →](#major-allele-normalization) -#### A variant I know is real is missing +### A variant I know is real is missing -Work outward through the chain — a read lost at alignment cannot be recovered later. +Work outward through the chain. A read lost at alignment cannot be recovered later. | Check | Parameter | |---|---| | Was it filtered at alignment? | `cleanBAM.mapq` (30 is strict), `cleanBAM.filter` | | Was the depth truncated? | That sample's `Output/Reports/Depth/` report, then `variantCall.maxDepth` | -| Was it seen in too few pools? | `filterFalsePositives.sampleThreshold` — the default discards alleles found in one pool out of eight | +| Was it seen in too few pools? | `filterFalsePositives.sampleThreshold`: the default discards alleles found in one pool out of eight | | Below the frequency floor? | `poolSize`, `ploidy` | | Site removed on quality? | `vcffilter.minQUAL` | [The Filter Chain →](#tuning-the-chain) -#### Much less depth than I sequenced for +### Much less depth than I sequenced for Two usual causes, in order of likelihood: 1. **MAPQ filtering.** At `cleanBAM.mapq = 30`, repetitive genomes lose a lot. Compare read counts in `Output/Aligned/` and `Output/Ready/`. 2. **Duplicate removal.** The step 4 log carries `markdup -s` statistics; a high duplicate rate is a library-prep problem, not a pipeline one. -#### Depth plateaus at one number +### Depth plateaus at one number Something capped it, and the depth report says what. `grep -H 'ceiling applied' Output/Reports/Depth/*_depth_report.txt`. -If the plateau is in **one sample** at an unround number, that is its measured ceiling and it is working as intended — the report gives the reason, and `param_capMaxDepth` overrules it for that sample. If **every sample** plateaus at the same number, the cap is flat rather than measured: either `capBAM.maxDepth` is set to a fixed depth, or `variantCall.maxDepth` is non-zero. [Depth capping →](#depth-capping) +If the plateau is in **one sample** at an unround number, that is its measured ceiling and it is working as intended. The report gives the reason, and `param_capMaxDepth` overrules it for that sample. If **every sample** plateaus at the same number, the cap is flat rather than measured: either `capBAM.maxDepth` is set to a fixed depth, or `variantCall.maxDepth` is non-zero. [Depth capping ->](#depth-capping) + +### The run stopped saying nothing survived a filter { #empty-result-refusals } + +A run that would have produced an empty result is stopped rather than allowed to publish one. There are three of these, each naming the setting responsible, and each stops before anything is written - so no results directory is left holding an empty table for a later run to skip past. + +| What it says | What emptied it | What to look at | +|---|---|---| +| `variant calling produced no records at all` | the pileup reached `bcftools call` with nothing in it | `variantCall.scaleMapQ` against `variantCall.varQualMin` - [below the minimum discards every read](#scalemapq-and-varqualmin) | +| `no site survived the cross-sample filter` | every allele was seen in too few pools | `filterFalsePositives.sampleThreshold`, `poolSize` | +| `no site survived the depth and quality filter` | every site had a sample under the floor, or too low a QUAL | `vcffilter.minDP`, `vcffilter.minQUAL`, `vcffilter.dropZeroDepth` | + +**Why this is a refusal and not a warning.** Every stage downstream succeeds over no records: `bcftools` writes a well-formed header with no rows, `vcftools --recode` exits 0, and the conversion rewrites the header and never enters its body. Before these checks existed, the result of a run that found nothing was a frequency table holding column names and nothing else, from a run that reported success. Measured through all seven stages. + +The depth and quality filter is the most common of the three, and it catches people out because **it is applied to the whole site**: one sample below `minDP` removes that site for every sample, so the shallowest library sets the threshold for the entire cohort. Check `Output/Reports/Coverage/` for your weakest sample before raising it. -#### Genotype-based tools find nothing in my VCFs +### Genotype-based tools find nothing in my VCFs -`FORMAT/GT` is set to `./.` throughout, deliberately — a pool has no genotype, and leaving bcftools' diploid call in place would invite tools to read it as one. Use `AD` and `DP`. +`FORMAT/GT` is set to `./.` throughout, deliberately: a pool has no genotype, and leaving bcftools' diploid call in place would invite tools to read it as one. Use `AD` and `DP`. -#### Annotated VCF contains sites missing from my frequency tables +### Annotated VCF contains sites missing from my frequency tables Step 8 runs on step **6**'s output, in parallel with the frequency branch, so it never sees the step 7 filters. Its allele encoding is also the original reference-based one, not the major-allele normalized one. Join on `CHROM`/`POS` and expect unmatched rows. [Details →](#the-vcf-files) -### Resume behavior +## Resume behavior + -#### A re-run skips too many steps +### A re-run skips too many steps Steps skip themselves when their outputs already exist. Delete the stale outputs, or `PoolSeqFlow reset` to start over. -#### A re-run submits every job anyway +### A re-run submits every job anyway Expected. Step-skipping happens inside each task rather than before it, so a fully resumed run still submits roughly one short job per process per sample. [Resume Logic →](#resume-logic) -#### `-resume` appears to do nothing +### `-resume` appears to do nothing -Correct — PoolSeqFlow does not use Nextflow's `-resume`, and the wrapper never passes it. `PoolSeqFlow run` already resumes. [Resume Logic →](#resume-logic) +Correct. PoolSeqFlow does not use Nextflow's `-resume`, and the wrapper never passes it. `PoolSeqFlow run` already resumes. [Resume Logic ->](#resume-logic) -#### A step reruns after an interrupted job +### A step reruns after an interrupted job If the interruption hit a cross-filesystem move, the partial copy was left under a temporary name rather than under the final one, so the step correctly runs again. That is the atomic move working. -### An analysis module +## An analysis module + -#### A module stops on a C++ compile +### A module stops on a C++ compile -`basicstats` computes its per-site work through a compiled function by default, and stops rather than falling back if it cannot build one. The commonest cause is calling `Rscript` yourself instead of going through `PoolSeqFlow analysis`, which activates the environment the compiler lives in; the next commonest is `/tmp` mounted `noexec`. Add `nocpp` to finish now — the numbers are identical — and see [When the compiled path will not build](#analysis-compiled-fails) for the full list. +`basicstats` computes its per-site work through a compiled function by default, and stops rather than falling back if it cannot build one. The commonest cause is calling `Rscript` yourself instead of going through `PoolSeqFlow analysis`, which activates the environment the compiler lives in; the next commonest is `/tmp` mounted `noexec`. Add `nocpp` to finish now (the numbers are identical) and see [When the compiled path will not build](#analysis-compiled-fails) for the full list. ```bash PoolSeqFlow analysis basicstats nocpp ``` -#### `doFuture is not installed` +### `doFuture is not installed` The module was asked for more than one worker and cannot go parallel. Either install it into the analysis environment, or set `analysis.modules.basicstats.workers = 1`. It does not quietly run on one worker instead: a run that took a different path than the one you asked for is a run whose timings mean nothing. -#### A pool holds one chromosome +### A pool holds one chromosome Refused, and it is the only pool size that is. A single haploid genome has no segregating sites, and the correction every diversity estimate applies divides by `n_eff - 1`, which is zero there. Correct `ploidy`, or that pool's `param_poolSize`. [→](#basicstats-design) -### Getting help +## Getting help + Include the failing step, the relevant `Logs/` excerpt and your `parameters.config` with paths redacted when opening an issue: [github.com/ozankiratli/PoolSeqFlow/issues](https://github.com/ozankiratli/PoolSeqFlow/issues) +# Reference + + ## Bibliography The work PoolSeqFlow is built on, and the work it can be read against. -**This is not the list to cite for your results.** That one is written for you, per run, and names only what the run actually invoked — see [Citing the tools it runs](#citing-the-tools-it-runs). This page is the reading behind the choices: where an estimator comes from, why there is more than one version of it in circulation, and which other software would give a different number from the same reads. +**This is not the list to cite for your results.** That one is written for you, per run, and names only what the run actually invoked (see [Citing the tools it runs](#citing-the-tools-it-runs)). This page is the reading behind the choices: where an estimator comes from, why there is more than one version of it in circulation, and which other software would give a different number from the same reads. -Everything below is compiled from the `references.bib` files in the installation, and holds exactly the entries this manual cites. A citation with no entry, or an entry filed under no heading, fails the build — so the page cannot fall behind the modules as they are added. +Everything below is compiled from the `references.bib` files in the installation, and holds exactly the entries this manual cites. A citation with no entry, or an entry filed under no heading, fails the build, so the page cannot fall behind the modules as they are added. A pooled allele frequency carries two rounds of sampling, and the correction for that is where implementations differ from one another most. !!! warning "There are two effective sample sizes, and they are one apart" - `n·d/(n + d − 1)` and `n·d/(n + d)` are both in circulation for a pool of `n` chromosomes read to depth `d`. **PoolSeqFlow uses the first**, which is the form implied by [Hivert et al. 2018](#ref-hivert2018poolseq)'s `D₂` — their sum of `(d + n − 1)/n` is the sum of `d/n_eff` under that form and no other. + `n·d/(n + d − 1)` and `n·d/(n + d)` are both in circulation for a pool of `n` chromosomes read to depth `d`. **PoolSeqFlow uses the first**, which is the form implied by [Hivert et al. 2018](#ref-hivert2018poolseq)'s `D₂`: their sum of `(d + n - 1)/n` is the sum of `d/n_eff` under that form and no other. They agree closely at high depth and diverge exactly where diversity estimates are most fragile: shallow coverage and small pools. At `n = 2, d = 2` the first gives 1.33 and the second 1.00. **If a number from this pipeline disagrees with one from other software, check which form the other used before concluding the data disagree.** The second form is widely used and hard to attribute: [Kolaczkowski et al. 2011](#ref-kolaczkowski2011clinal) work the pooled sampling through with pool size and depth as separate parameters and define no combined quantity at all, and Gautier et al.'s "effective pool size" is a different measure again. PoolSeqFlow therefore states which form it uses rather than citing one for it. -The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correction it needs is argued in [Ferretti et al. 2013](#ref-ferretti2013pool), and the estimators most likely to be compared against these are [Kofler et al. 2011a](#ref-kofler2011popoolation) and [Kofler et al. 2011b](#ref-kofler2011popoolation2) — built on [Futschik & Schlötterer 2010](#ref-futschik2010markers), and applied at scale in [Fabian et al. 2012](#ref-fabian2012latitudinal). Pooled variant calling as a practice starts with [Koboldt et al. 2009](#ref-koboldt2009varscan). +The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correction it needs is argued in [Ferretti et al. 2013](#ref-ferretti2013pool), and the estimators most likely to be compared against these are [Kofler et al. 2011a](#ref-kofler2011popoolation) and [Kofler et al. 2011b](#ref-kofler2011popoolation2) (built on [Futschik & Schlötterer 2010](#ref-futschik2010markers), and applied at scale in [Fabian et al. 2012](#ref-fabian2012latitudinal)). Pooled variant calling as a practice starts with [Koboldt et al. 2009](#ref-koboldt2009varscan). @@ -4966,22 +5256,22 @@ The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correct #### Gower 1966 { #ref-gower1966mds } -**Gower, J. C.** (1966). Some Distance Properties of Latent Root and Vector Methods Used in Multivariate Analysis. *Biometrika* 53(3/4), 325–338. [10.2307/2333639](https://doi.org/10.2307/2333639) +**Gower, J. C.** (1966). Some Distance Properties of Latent Root and Vector Methods Used in Multivariate Analysis. *Biometrika* 53(3/4), 325-338. [10.2307/2333639](https://doi.org/10.2307/2333639) : The ordination this module draws: a matrix of squared distances double centered into a Gram matrix, whose leading eigenvectors are the coordinates. It is what R's own cmdscale cites and implements, and the source of the negative eigenvalues this module publishes rather than hides - they are what a distance matrix that no flat space holds exactly produces. #### Nei 1972 { #ref-nei1972distance } -**Nei, M.** (1972). Genetic Distance between Populations. *The American Naturalist* 106(949), 283–292. [10.1086/282771](https://doi.org/10.1086/282771) +**Nei, M.** (1972). Genetic Distance between Populations. *The American Naturalist* 106(949), 283-292. [10.1086/282771](https://doi.org/10.1086/282771) : The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because it is the nei_distance library, installed with whatever module declares it. #### Nei 1973 { #ref-nei1973diversity } -**Nei, M.** (1973). Analysis of Gene Diversity in Subdivided Populations. *Proceedings of the National Academy of Sciences* 70(12), 3321–3323. [10.1073/pnas.70.12.3321](https://doi.org/10.1073/pnas.70.12.3321) +**Nei, M.** (1973). Analysis of Gene Diversity in Subdivided Populations. *Proceedings of the National Academy of Sciences* 70(12), 3321-3323. [10.1073/pnas.70.12.3321](https://doi.org/10.1073/pnas.70.12.3321) : The diversity statistic this module reports, H = 1 - sum(p_i^2), summed over every allele at a site rather than over two. It is defined for any number of alleles, which is why a triallelic site needs no collapsing here. #### Benjamini & Hochberg 1995 { #ref-benjamini1995fdr } -**Benjamini, Y. & Hochberg, Y.** (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. *Journal of the Royal Statistical Society: Series B* 57(1), 289–300. [10.1111/j.2517-6161.1995.tb02031.x](https://doi.org/10.1111/j.2517-6161.1995.tb02031.x) +**Benjamini, Y. & Hochberg, Y.** (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. *Journal of the Royal Statistical Society: Series B* 57(1), 289-300. [10.1111/j.2517-6161.1995.tb02031.x](https://doi.org/10.1111/j.2517-6161.1995.tb02031.x) : The default correction across sites, applied to the permutation p and never to the per-allele one. The number of tests passed to it is the count of sites that produced a statistic, so a site whose alleles were all invariant is excluded rather than counted as a test that failed to reject. #### Phipson & Smyth 2010 { #ref-phipson2010permutation } @@ -4991,24 +5281,24 @@ The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correct #### Ferretti et al. 2013 { #ref-ferretti2013pool } -**Ferretti, L., Ramos-Onsins, S. E. & Pérez-Enciso, M.** (2013). Population genomics from pool sequencing. *Molecular Ecology* 22(22), 5561–5576. [10.1111/mec.12522](https://doi.org/10.1111/mec.12522) +**Ferretti, L., Ramos-Onsins, S. E. & Pérez-Enciso, M.** (2013). Population genomics from pool sequencing. *Molecular Ecology* 22(22), 5561-5576. [10.1111/mec.12522](https://doi.org/10.1111/mec.12522) : Why a pooled estimate needs correcting at all: individuals are sampled into the pool and reads are sampled from the pool, so an allele frequency carries both, and the estimator has to account for the smaller of the two. #### Hivert et al. 2018 { #ref-hivert2018poolseq } -**Hivert, V., Leblois, R., Petit, E. J., Gautier, M. & Vitalis, R.** (2018). Measuring Genetic Differentiation from Pool-seq Data. *Genetics* 210(1), 315–330. [10.1534/genetics.118.300900](https://doi.org/10.1534/genetics.118.300900) +**Hivert, V., Leblois, R., Petit, E. J., Gautier, M. & Vitalis, R.** (2018). Measuring Genetic Differentiation from Pool-seq Data. *Genetics* 210(1), 315-330. [10.1534/genetics.118.300900](https://doi.org/10.1534/genetics.118.300900) : The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because it is the n_eff library, installed with whatever module declares it, and every module that weights anything calls it. ### Estimating from pooled reads #### Futschik & Schlötterer 2010 { #ref-futschik2010markers } -**Futschik, A. & Schlötterer, C.** (2010). The Next Generation of Molecular Markers From Massively Parallel Sequencing of Pooled DNA Samples. *Genetics* 186(1), 207–218. [10.1534/genetics.110.114397](https://doi.org/10.1534/genetics.110.114397) +**Futschik, A. & Schlötterer, C.** (2010). The Next Generation of Molecular Markers From Massively Parallel Sequencing of Pooled DNA Samples. *Genetics* 186(1), 207-218. [10.1534/genetics.110.114397](https://doi.org/10.1534/genetics.110.114397) : The estimator theory PoPoolation is built on, and the first treatment of unbiased pi and theta_W from pooled reads. Read it before deciding that a pooled estimate can be computed the way an individually-genotyped one is. #### Kolaczkowski et al. 2011 { #ref-kolaczkowski2011clinal } -**Kolaczkowski, B., Kern, A. D., Holloway, A. K. & Begun, D. J.** (2011). Genomic Differentiation Between Temperate and Tropical Australian Populations of Drosophila melanogaster. *Genetics* 187(1), 245–260. [10.1534/genetics.110.123059](https://doi.org/10.1534/genetics.110.123059) +**Kolaczkowski, B., Kern, A. D., Holloway, A. K. & Begun, D. J.** (2011). Genomic Differentiation Between Temperate and Tropical Australian Populations of Drosophila melanogaster. *Genetics* 187(1), 245-260. [10.1534/genetics.110.123059](https://doi.org/10.1534/genetics.110.123059) : One of the first genome-scale Pool-seq studies, and where its sampling properties are worked through from first principles - n chromosomes drawn from the population and sequenced to depth m, kept as two parameters rather than compressed into one. Worth reading precisely because it does not take the shortcut every effective-sample-size heuristic since has taken. #### Long et al. 2026 { #ref-long2026polygenicity } @@ -5020,7 +5310,7 @@ The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correct #### Koboldt et al. 2009 { #ref-koboldt2009varscan } -**Koboldt, D. C., Chen, K., Wylie, T., Larson, D. E., McLellan, M. D., Mardis, E. R., Weinstock, G. M., Wilson, R. K. & Ding, L.** (2009). VarScan: variant detection in massively parallel sequencing of individual and pooled samples. *Bioinformatics* 25(17), 2283–2285. [10.1093/bioinformatics/btp373](https://doi.org/10.1093/bioinformatics/btp373) +**Koboldt, D. C., Chen, K., Wylie, T., Larson, D. E., McLellan, M. D., Mardis, E. R., Weinstock, G. M., Wilson, R. K. & Ding, L.** (2009). VarScan: variant detection in massively parallel sequencing of individual and pooled samples. *Bioinformatics* 25(17), 2283-2285. [10.1093/bioinformatics/btp373](https://doi.org/10.1093/bioinformatics/btp373) : Pooled variant calling as a thing that can be done at all - SNPs and indels detected from pooled reads at a frequency threshold, which is the premise every tool on this page rests on. PoolSeqFlow calls with BCFtools and applies its own per-pool threshold. #### Kofler et al. 2011a { #ref-kofler2011popoolation } @@ -5030,14 +5320,14 @@ The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correct #### Kofler et al. 2011b { #ref-kofler2011popoolation2 } -**Kofler, R., Pandey, R. V. & Schlötterer, C.** (2011). PoPoolation2: identifying differentiation between populations using sequencing of pooled DNA samples (Pool-Seq). *Bioinformatics* 27(24), 3435–3436. [10.1093/bioinformatics/btr589](https://doi.org/10.1093/bioinformatics/btr589) +**Kofler, R., Pandey, R. V. & Schlötterer, C.** (2011). PoPoolation2: identifying differentiation between populations using sequencing of pooled DNA samples (Pool-Seq). *Bioinformatics* 27(24), 3435-3436. [10.1093/bioinformatics/btr589](https://doi.org/10.1093/bioinformatics/btr589) : F_ST, Fisher's exact test and the Cochran-Mantel-Haenszel test between pools. ### Where these methods have been used #### Fabian et al. 2012 { #ref-fabian2012latitudinal } -**Fabian, D. K., Kapun, M., Nolte, V., Kofler, R., Schmidt, P. S., Schlötterer, C. & Flatt, T.** (2012). Genome-wide patterns of latitudinal differentiation among populations of Drosophila melanogaster from North America. *Molecular Ecology* 21(19), 4748–4769. [10.1111/j.1365-294X.2012.05731.x](https://doi.org/10.1111/j.1365-294X.2012.05731.x) +**Fabian, D. K., Kapun, M., Nolte, V., Kofler, R., Schmidt, P. S., Schlötterer, C. & Flatt, T.** (2012). Genome-wide patterns of latitudinal differentiation among populations of Drosophila melanogaster from North America. *Molecular Ecology* 21(19), 4748-4769. [10.1111/j.1365-294X.2012.05731.x](https://doi.org/10.1111/j.1365-294X.2012.05731.x) : An early large Pool-seq study, and a worked example of what these estimators are for: pi, theta_W and Tajima's D from PoPoolation and F_ST from PoPoolation2, across populations on a cline. Useful for seeing how the numbers are reported and argued from, rather than only how they are computed. @@ -5056,7 +5346,7 @@ The installed copy will print its own citation, with its version filled in: PoolSeqFlow cite ``` -Use that rather than copying from here — it knows which version you have, and this page does not. +Use that rather than copying from here: it knows which version you have, and this page does not. ### Which DOI to use @@ -5064,14 +5354,14 @@ Zenodo issues **two kinds of DOI**, and the difference matters. | DOI | What it identifies | Use it for | |---|---|---| -| [10.5281/zenodo.19245611](https://doi.org/10.5281/zenodo.19245611) | **All versions.** Always resolves to the newest release | Referring to PoolSeqFlow as a piece of software — a related-work mention, a README, a link | +| [10.5281/zenodo.19245611](https://doi.org/10.5281/zenodo.19245611) | **All versions.** Always resolves to the newest release | Referring to PoolSeqFlow as a piece of software: a related-work mention, a README, a link | | A version DOI, one per release | **One specific release**, frozen | **Reporting results.** This is the one a methods section needs | !!! warning "Cite the version you ran, not the newest one" - Results depend on which release produced them. Filters, defaults and parameter names have all changed between versions — `vcffilter.minDP` went from having no effect to removing whole sites, and sample column ordering changed in 2.1.1. A paper citing the current release for numbers produced by an older one is describing a method it did not use. + Results depend on which release produced them. Filters, defaults and parameter names have all changed between versions: `vcffilter.minDP` went from having no effect to removing whole sites, and sample column ordering changed in 2.1.1. A paper citing the current release for numbers produced by an older one is describing a method it did not use. - Find the version that produced a given set of results in that project's `Output/run_parameters.txt`, which lists every release that has run there — `PoolSeqFlow version` tells you only what is installed now, which is not the same thing once you have upgraded. Then open the [all-versions record](https://doi.org/10.5281/zenodo.19245611) and pick that version from the **Versions** list to get its DOI. + Find the version that produced a given set of results in that project's `Output/run_parameters.txt`, which lists every release that has run there. `PoolSeqFlow version` tells you only what is installed now, which is not the same thing once you have upgraded. Then open the [all-versions record](https://doi.org/10.5281/zenodo.19245611) and pick that version from the **Versions** list to get its DOI. If more than one version is listed, the outputs were not all produced by the same release: completed steps are not redone on upgrade. Say so in your methods, or `PoolSeqFlow reset` and re-run under one version. @@ -5106,7 +5396,7 @@ storageDir/Output/ Both are generated from the run that produced them, which makes them accurate in two ways a static list cannot be: - **They carry the versions that actually ran**, asked of each tool at run time rather than read from the environment file. If you repointed a tool at a system installation with `params.software`, the version recorded is the one that did the work. -- **They list only what the run invoked.** A run with `annotate = false` never calls SnpEff, so SnpEff is not in its citations — citing it would be claiming a step that did not happen. +- **They list only what the run invoked.** A run with `annotate = false` never calls SnpEff, so SnpEff is not in its citations. Citing it would be claiming a step that did not happen. An analysis module adds its own: the statistic it implements as well as the packages it computes in, because a diversity estimate a reader cannot trace to a definition is one they cannot check. The reading behind those choices, including work this pipeline does **not** run but can be compared against, is in the [Bibliography](#bibliography). diff --git a/manual/requirements.txt b/manual/requirements.txt index a5df01b..25185c8 100644 --- a/manual/requirements.txt +++ b/manual/requirements.txt @@ -1,6 +1,7 @@ # Build dependencies for the documentation site only. # The pipeline itself needs none of these - see install/environment.yml. # -# pip install -r docs/requirements.txt -# mkdocs serve +# python3 -m venv dev/manual/.venv +# dev/manual/.venv/bin/pip install -r manual/requirements.txt +# dev/scripts/serve_docs.sh mkdocs-material==9.7.7 diff --git a/manual/stylesheets/extra.css b/manual/stylesheets/extra.css index 8acea03..1f8a557 100644 --- a/manual/stylesheets/extra.css +++ b/manual/stylesheets/extra.css @@ -309,7 +309,7 @@ /* -------------------------------------------------------------- tables ---- */ -/* Rules only, no fills — the tables here are dense reference material and read +/* Rules only, no fills -- the tables here are dense reference material and read better without a striped background competing with the code spans inside. */ .md-typeset table:not([class]) { border: none; @@ -477,7 +477,7 @@ margin-bottom: 1.4em; } -/* Badge row — keeps the shields on one line with no bullet artefacts. */ +/* Badge row -- keeps the shields on one line with no bullet artefacts. */ .md-typeset .psf-badges { display: flex; flex-wrap: wrap; diff --git a/metadata.csv.template b/metadata.csv.template index bb2642c..89394f4 100644 --- a/metadata.csv.template +++ b/metadata.csv.template @@ -1,103 +1,38 @@ -# Sample metadata — one row per pair of FASTQ files. This is the file that says what your -# samples ARE. Blank lines and lines starting with # are ignored, so these notes can stay. -# -# The manual explains all of this at length, under "Metadata": -# https://ozankiratli.github.io/PoolSeqFlow/ -# -# --------------------------------------------------------------------------------------- -# SEVEN KINDS OF COLUMN, and the name is what tells them apart. -# -# SampleID Required, and unique. Matched against the sample name your FASTQ file names -# give — with readPattern '*_R{1,2}.fq.gz', Sample1T1Rep1_R1.fq.gz gives the -# sample Sample1T1Rep1 — and it becomes the read group's ID in the BAM. -# -# RG_* A read group tag: RG_Sample, RG_Library, RG_Platform, RG_PlatformUnit, -# RG_Description, RG_Center, RG_Date, RG_FlowOrder. A blank cell omits that -# tag. Nothing else may start with RG_; an unrecognized one is refused. -# -# param_* A parameter from parameters.config, overridden for these samples only: -# param_poolSize, param_capMaxDepth, param_adapter1, param_adapter2. A blank -# cell means "use the global setting". A closed list, like RG_. -# -# exp_* An experimental variable — something you SET. exp_population, exp_treatment, -# exp_replicate; exp_time is time. Any name you like after the prefix: unlike -# RG_ and param_ there is no list to check it against, so a misspelled exp_ -# column is a new variable and not an error. Read by analyses, never by 0-8. -# -# pt_* A phenotype — something you MEASURED on the pool, and the thing an analysis -# tests against. pt_wingspan, pt_resistance. Open, and never read by 0-8. -# -# cov_* A covariate — measured on the pool, but neither set nor the response. -# cov_temperature, cov_altitude, cov_site. Open, and never read by 0-8. -# -# anything Your own — a note, a lab code, the lane, the kit lot. Recorded and never -# else interpreted. Add, remove and edit them freely: step 0's change guard does -# not look at them, so results already produced stay valid. The same is true -# of exp_, pt_ and cov_ columns. -# -# WHY exp_, pt_ AND cov_ ARE THREE PREFIXES AND NOT ONE: only exp_ says what an experiment set -# up. The analysis layer works out which pools are independent of each other — and, where there -# is a time course, which are one thing measured repeatedly — from your exp_ columns, and a -# trait value or a cage temperature differs from pool to pool. So either one written as exp_ -# would make every pool its own unit and leave every series a single timepoint long, quietly, -# because a design with no repeated measurement is a legal design. -# -# --------------------------------------------------------------------------------------- -# RG_Sample IS THE ONE THAT CHANGES THE RESULT. It is the pool a row belongs to, and becomes -# one column in the VCF and one in every frequency table. Rows sharing an RG_Sample are -# MERGED: their reads are pooled and their depths added — which is how you say that two -# sequencing runs, two lanes or two libraries are the same pool of individuals. Leave the -# column out, or a cell blank, and RG_Sample takes the SampleID, so every row is its own pool. -# Step 0 prints the pooling it worked out before any compute is spent. -# -# param_poolSize IS A PROPERTY OF THE POOL: how many individuals went into it. It sets that -# pool's detection limit, sensitivity = 1 / (2 * ploidy * poolSize), so one number for the -# whole run judges a pool of 10 at a pool of 500's resolution. Rows sharing an RG_Sample must -# give the SAME value, and a blank cell counts as a different answer rather than as agreement. -# Leave the column out and every pool uses the global poolSize. -# -# param_capMaxDepth IS A PROPERTY OF THE ROW, not of the pool: it caps one BAM, and each row -# is one BAM. So rows of one pool may carry different values and nothing objects. It takes -1 -# to measure this sample's ceiling from its own depth histogram, 0 to leave it uncapped, or a -# depth to cap it at; anything else is refused by name. Leave it blank to use the global -# capBAM.maxDepth, which ships as -1. -# -# param_adapter1 and param_adapter2 override trim_galore.adapter1 / adapter2 for that row -# only. Give both or neither. A value containing a comma must be quoted: "Pop1, coastal". -# -# --------------------------------------------------------------------------------------- -# exp_, pt_ AND cov_ COLUMNS BELONG TO THE POOL, not to the row — but they differ in what -# happens when the rows of one pool disagree, and the difference is not arbitrary: -# -# exp_ REFUSED by an analysis. You cannot have SET two treatments for one pool. -# pt_ REFUSED by an analysis. One pool has one measured trait value. -# cov_ RECORDED. Two libraries of a pool really can have been reared at two temperatures -# or handled by two people — that is a circumstance, not a contradiction. The pool -# is then reported as having NO single value for it, and nothing is averaged or -# picked for you. -# -# THE PIPELINE ITSELF NEVER REFUSES ANY OF THEM. No step reads these columns, so a run is -# unaffected whatever they say; step 0 reports a disagreement as a note and carries on. -# -# What differs between two rows of one pool and is a property of the ROW — the lane, the run, -# the kit lot — takes no prefix. Not because it does not matter: once two libraries' reads are -# merged into one column, nothing downstream can attribute a read to the row it came from, so -# a row-level fact is unrecoverable rather than unsupported. If every row of a pool DOES share -# one — one technician per pool — then it is a property of the pool after all, and it is cov_. -# -# --------------------------------------------------------------------------------------- -# The table below is an example: eight FASTQ pairs, four pools, each pool sequenced twice, so -# its two rows share an RG_Sample and repeat everything that belongs to the pool. Reading the -# columns left to right: what the sample IS, what the pipeline should do with it, what was SET -# (exp_), what was MEASURED as the response (pt_), what was MEASURED alongside (cov_), and last -# `sequencing_run`, which differs between the two rows of a pool and so carries no prefix. -# Replace all of it with yours. +# Sample metadata: one row per pair of FASTQ files. Blank lines and # lines are ignored. +# The manual explains every column: https://ozankiratli.github.io/PoolSeqFlow/ +# +# COLUMN KINDS, told apart by the prefix: +# SampleID required, unique. THE WHOLE NAME BEFORE THE READ TAG, exactly as your sequencer +# delivered it: with '*_R{1,2}.fq.gz', Sample1T1_S19_L001_R1.fq.gz gives +# Sample1T1_S19_L001. Never shorten it; it is what finds your files. +# RG_* a read group tag: RG_Sample, RG_Library, RG_Platform, RG_PlatformUnit, +# RG_Description, RG_Center, RG_Date, RG_FlowOrder. Closed list. +# param_* overrides a parameters.config setting for these rows: param_poolSize, +# param_capMaxDepth, param_adapter1, param_adapter2. Closed list. +# exp_* what the experiment SET: exp_population, exp_time. Open list. +# pt_* a phenotype MEASURED on the pool: pt_resistance. Open list. +# cov_* a covariate measured alongside: cov_temperature. Open list. +# anything your own. Recorded, never interpreted. +# +# THE TWO THAT CHANGE A RESULT: +# RG_Sample ALWAYS FILL THIS IN. It is the name you want your results under: it names the +# VCF column and every table keyed on it. Rows sharing one are MERGED, reads +# pooled and depths added. Left blank it copies SampleID, sequencer suffix and +# all. Step 0 prints the pooling before any compute is spent. +# param_poolSize individuals in the pool; sets its detection limit. Rows of one pool must +# give the SAME value. Absent: every pool uses the global poolSize. +# +# A value containing a comma must be quoted: "Pop1, coastal". +# +# The table below is an example: eight pairs, four pools, each sequenced on two lanes. Note what +# the two name columns do - SampleID carries the sequencer's own suffix and finds the files, +# RG_Sample is the name the results come out under. Replace it. SampleID,RG_Sample,RG_Library,RG_Platform,RG_PlatformUnit,param_poolSize,exp_population,exp_time,pt_resistance,cov_temperature,sequencing_run -Sample1T1Rep1,Sample1T1,Lib1,ILLUMINA,Unit1,50,Pop1,T1,susceptible,21.5,Run1 -Sample1T1Rep2,Sample1T1,Lib1,ILLUMINA,Unit2,50,Pop1,T1,susceptible,21.5,Run2 -Sample1T2Rep1,Sample1T2,Lib1,ILLUMINA,Unit1,50,Pop1,T2,resistant,22.1,Run1 -Sample1T2Rep2,Sample1T2,Lib1,ILLUMINA,Unit2,50,Pop1,T2,resistant,22.1,Run2 -Sample2T1Rep1,Sample2T1,Lib1,ILLUMINA,Unit1,40,Pop2,T1,susceptible,18.0,Run1 -Sample2T1Rep2,Sample2T1,Lib1,ILLUMINA,Unit2,40,Pop2,T1,susceptible,18.0,Run2 -Sample2T2Rep1,Sample2T2,Lib1,ILLUMINA,Unit1,40,Pop2,T2,resistant,19.4,Run1 -Sample2T2Rep2,Sample2T2,Lib1,ILLUMINA,Unit2,40,Pop2,T2,resistant,19.4,Run2 +Sample1T1_S1_L001,Sample1T1,Lib1,ILLUMINA,Unit1,50,Pop1,T1,susceptible,21.5,Run1 +Sample1T1_S1_L002,Sample1T1,Lib1,ILLUMINA,Unit2,50,Pop1,T1,susceptible,21.5,Run2 +Sample1T2_S2_L001,Sample1T2,Lib1,ILLUMINA,Unit1,50,Pop1,T2,resistant,22.1,Run1 +Sample1T2_S2_L002,Sample1T2,Lib1,ILLUMINA,Unit2,50,Pop1,T2,resistant,22.1,Run2 +Sample2T1_S3_L001,Sample2T1,Lib1,ILLUMINA,Unit1,40,Pop2,T1,susceptible,18.0,Run1 +Sample2T1_S3_L002,Sample2T1,Lib1,ILLUMINA,Unit2,40,Pop2,T1,susceptible,18.0,Run2 +Sample2T2_S4_L001,Sample2T2,Lib1,ILLUMINA,Unit1,40,Pop2,T2,resistant,19.4,Run1 +Sample2T2_S4_L002,Sample2T2,Lib1,ILLUMINA,Unit2,40,Pop2,T2,resistant,19.4,Run2 diff --git a/mkdocs.yml b/mkdocs.yml index f127673..603d72a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -111,7 +111,16 @@ markdown_extensions: - . - docs check_paths: true - - pymdownx.superfences + # The mermaid fence is a custom fence rather than a highlighted code block, so a + # ```mermaid block renders as a diagram. mkdocs-material ships the renderer; nothing + # else is needed. A diagram written this way exists on the site only - the manual also + # ships as one readable file, so anything a reader must have in the tarball stays as + # text or as an svg under manual/assets/. + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true - pymdownx.tilde @@ -136,6 +145,7 @@ nav: - Install: getting-started/install.md - Quick Start: getting-started/quick-start.md - Upgrading: getting-started/upgrading.md + - Uninstalling: getting-started/uninstalling.md - Concepts: - concepts/index.md - Scientific Rationale: concepts/scientific-rationale.md @@ -174,8 +184,16 @@ nav: - Rationale: development/principles.md - Verification: development/verification.md - Claude Code: development/claude-code.md + - Troubleshooting: + - troubleshooting/index.md + - Installation: troubleshooting/installation.md + - Will not start: troubleshooting/startup.md + - During the run: troubleshooting/during-the-run.md + - Unexpected results: troubleshooting/results.md + - Resume: troubleshooting/resume.md + - Analysis modules: troubleshooting/modules.md + - Getting help: troubleshooting/help.md - Reference: - - Troubleshooting: reference/troubleshooting.md - Bibliography: reference/bibliography.md - Changelog: reference/changelog.md - Citation & License: reference/citation.md diff --git a/modules/README.md b/modules/README.md index 18b1117..65dae54 100644 --- a/modules/README.md +++ b/modules/README.md @@ -10,7 +10,7 @@ One directory per module, named exactly as the module is named. Each holds at le A module is a pipeline in its own right. It imports what it wants from `analysis/lib/nf` by relative path and is launched directly; nothing in the frame includes it, so a module that is absent, or one whose directory has been removed, costs the others nothing. -**Import only the workflows you call.** An import you never use still couples you to it — a library workflow that is renamed or removed breaks every module naming it, called or not. +**Import only the workflows you call.** An import you never use still couples you to it -- a library workflow that is renamed or removed breaks every module naming it, called or not. ## What a `main.nf` starts from @@ -40,7 +40,7 @@ workflow { `analysisPlan` takes the module's own name and returns one target per results directory the invocation covers. `PublishResults` takes what you produced for one of them and installs it in `target.results`. The artifact classes it marks required are the `needs` from your `manifest.json`, so you state what you read once and nothing repeats it. It recomputes the pipeline's own partition of the runs, so two runs that produced the same tables are one target and are analyzed once. -A module ships no configuration. `PoolSeqFlow analysis` assembles it — the installation's `analysis/frame.config`, then the project's `analysis.config`, then `.config` — and `frame.config` carries what the pipeline gets from `nextflow.config`, which Nextflow does not read for a module: `bin/` on the task PATH, conda, and the resource ceiling. +A module ships no configuration. `PoolSeqFlow analysis` assembles it -- the installation's `analysis/frame.config`, then the project's `analysis.config`, then `.config` -- and `frame.config` carries what the pipeline gets from `nextflow.config`, which Nextflow does not read for a module: `bin/` on the task PATH, conda, and the resource ceiling. ## Rules @@ -48,15 +48,15 @@ Follow these and a module written by anyone runs beside the ones published here. ### The shape -1. **`main.nf` is required, and the directory name must equal the manifest's `name`.** A directory holding a manifest and no `main.nf` is refused while the DAG is built — before the results folder is cleared — and it stops **every** invocation, not only yours. A directory holding no manifest at all is passed over in silence, so a half-finished install is harmless. -2. **Import the library by a literal relative path**, `'../../lib/nf/plan.nf'`. An interpolated include path is rejected both by `nextflow lint` and at runtime, and the literal resolves because a module RUNS from the store, at `analysis/modules//`, beside the frame it imports. Its source lives at `modules//` here, where that path does not resolve — which is why `00_static` lints a module in an assembled store layout rather than where git keeps it. +1. **`main.nf` is required, and the directory name must equal the manifest's `name`.** A directory holding a manifest and no `main.nf` is refused while the DAG is built -- before the results folder is cleared -- and it stops **every** invocation, not only yours. A directory holding no manifest at all is passed over in silence, so a half-finished install is harmless. +2. **Import the library by a literal relative path**, `'../../lib/nf/plan.nf'`. An interpolated include path is rejected both by `nextflow lint` and at runtime, and the literal resolves because a module RUNS from the store, at `analysis/modules//`, beside the frame it imports. Its source lives at `modules//` here, where that path does not resolve -- which is why `00_static` lints a module in an assembled store layout rather than where git keeps it. 3. **Import only the workflows you call.** An unused import still couples you: a library workflow that is renamed or removed breaks every module naming it, called or not. Import breadth is coupling breadth. -4. **Report your own version.** `manifest.json`'s `version` is the module's, and it moves on the module's timetable — never the pipeline release's. That separation is the reason modules are installed separately at all. The modules published with PoolSeqFlow use **`YYYYMMDD.NNN`** — the day the manifest changed, then a counter from `001` for that day — bumped whenever the manifest is touched and committed with the change. Any version string works as far as the frame is concerned, but `install` picks the newest with `sort -V`, so whatever you choose has to order correctly under it. +4. **Report your own version.** `manifest.json`'s `version` is the module's, and it moves on the module's timetable -- never the pipeline release's. That separation is the reason modules are installed separately at all. The modules published with PoolSeqFlow use **`YYYYMMDD.NNN`** -- the day the manifest changed, then a counter from `001` for that day -- bumped whenever the manifest is touched and committed with the change. Any version string works as far as the frame is concerned, but `install` picks the newest with `sort -V`, so whatever you choose has to order correctly under it. ### Declaring what you read 5. **`needs` in your manifest is the only place you say what you read.** `analysisPlan('yourname')` looks it up there, and so does the verification that runs before you, so the two cannot disagree about which artifacts were checked. -6. **`contract` is the published-table contract you speak** — `freq-1` today. It is bumped only when a column's name or meaning changes, which has not happened since the project's first commit. +6. **`contract` is the published-table contract you speak** -- `freq-1` today. It is bumped only when a column's name or meaning changes, which has not happened since the project's first commit. The classes `needs` may name: @@ -74,13 +74,13 @@ A name no class answers to is refused while the DAG is built. Without that it wo ### Declaring what you run on -6b. **`license` is an SPDX identifier and it is required.** A result your module produces is produced under your terms, not the pipeline's — PoolSeqFlow is Apache-2.0 and imposes nothing on you, but a module that loads a GPL package into its R session is GPL itself. The frame prints it in the verification report so a user can see what an analysis was produced under before it runs, which is the whole reason it is not optional. +6b. **`license` is an SPDX identifier and it is required.** A result your module produces is produced under your terms, not the pipeline's -- PoolSeqFlow is Apache-2.0 and imposes nothing on you, but a module that loads a GPL package into its R session is GPL itself. The frame prints it in the verification report so a user can see what an analysis was produced under before it runs, which is the whole reason it is not optional. -Every module published here is **GPL-3.0-or-later**, and the check that decides it is worth copying: each offers a compiled hot path through `Rcpp`, which is GPL, and each has it on **by default** — `nocpp` turns it off for one run, but the module ships expecting it and stops rather than falling back when `Rcpp` is absent. A dependency that only the default configuration reaches is still a dependency. Run the same test over what you load: `packageDescription("")$License` for each one, and take the strictest. +Every module published here is **GPL-3.0-or-later**, and the check that decides it is worth copying: each offers a compiled hot path through `Rcpp`, which is GPL, and each has it on **by default** -- `nocpp` turns it off for one run, but the module ships expecting it and stops rather than falling back when `Rcpp` is absent. A dependency that only the default configuration reaches is still a dependency. Run the same test over what you load: `packageDescription("")$License` for each one, and take the strictest. -6c. **`frame` is the oldest `analysis/frame.version` your module runs on**, written the same way — `YYYYMMDD.NNN`. It is the axis `contract` does not cover: `contract` is about the shape of the published tables, while `frame` is about the library you import. You call `analysisPlan`, `designJson`, `moduleSettings` and `PublishResults` by name, and a frame that predates one of them is not a table problem. Set it to the frame you developed against and move it when you start calling something newer. +6c. **`frame` is the oldest `analysis/frame.version` your module runs on**, written the same way -- `YYYYMMDD.NNN`. It is the axis `contract` does not cover: `contract` is about the shape of the published tables, while `frame` is about the library you import. You call `analysisPlan`, `designJson`, `moduleSettings` and `PublishResults` by name, and a frame that predates one of them is not a table problem. Set it to the frame you developed against and move it when you start calling something newer. -6d. **`environment` is the oldest PoolSeqFlow release whose analysis environment holds what you need**, written as a release is — `3.0.0`. There is one analysis environment per release and every module shares it, so its package set is a property of the release rather than something negotiated at install time. A module built against a later release's environment is refused here rather than installed and left to fail in R. +6d. **`environment` is the oldest PoolSeqFlow release whose analysis environment holds what you need**, written as a release is -- `3.0.0`. There is one analysis environment per release and every module shares it, so its package set is a property of the release rather than something negotiated at install time. A module built against a later release's environment is refused here rather than installed and left to fail in R. 6e. **`packages` names what has to be added to that environment, pinned.** Each entry is `=` and nothing else: one `=`, an exact version, no build string, no range, no channel prefix. A build string names one platform's build, so a manifest carrying one cannot install anywhere else; the channel is the release's decision and not yours. **Name everything you load, including packages the release's own environment already carries.** The manifest states what YOUR module needs, not what one release happens to provide, and a release that slims its environment must not silently break you. Removing a module never takes a package the release itself is built on: the uninstall subtracts the baseline as well as what other installed modules declare. @@ -88,11 +88,11 @@ Every module published here is **GPL-3.0-or-later**, and the check that decides ### Declaring the libraries you use -6f. **`libraries` names the shared arithmetic your module calls, and each one is installed with you.** A library is the same shape as a module — a folder with its own `manifest.json`, its `.R`, and its `.cpp` if it offers a compiled form — published and versioned on its own timetable and installed into `analysis/modules/lib//`. Declare it and it arrives when you are installed; it leaves when no installed module still declares it, so two modules sharing one never take it from each other. +6f. **`libraries` names the shared arithmetic your module calls, and each one is installed with you.** A library is the same shape as a module -- a folder with its own `manifest.json`, its `.R`, and its `.cpp` if it offers a compiled form -- published and versioned on its own timetable and installed into `analysis/modules/lib//`. Declare it and it arrives when you are installed; it leaves when no installed module still declares it, so two modules sharing one never take it from each other. **You do not list the files.** The frame resolves them: `moduleLibraryFiles('')` returns the `.R` to source, in the order you declared the libraries, and `moduleCompiledFiles('')` returns the `.cpp` beside them. Both read your manifest, so the list exists once. A module that kept its own copy of that list had a second list to keep equal, and a disagreement between them was silent. -**A library declares what it needs too** — its own `packages`, and `libraries` of its own if it calls another. It declares `contract` only if it reads a published table: `allele_frequencies` and `site_diversity` parse depth-table cells and are bound to `freq-1`, while `n_eff` and `chunk_ranges` take numbers and know nothing about the tables. +**A library declares what it needs too** -- its own `packages`, and `libraries` of its own if it calls another. It declares `contract` only if it reads a published table: `allele_frequencies` and `site_diversity` parse depth-table cells and are bound to `freq-1`, while `n_eff` and `chunk_ranges` take numbers and know nothing about the tables. Write a library when more than one module wants the same decision-free arithmetic. A derivation that carries a DECISION does not belong in one: it takes the decision as a required argument with no default, so the module passes the value, declares it in its `gates`, and the report prints it. @@ -105,38 +105,38 @@ Write a library when more than one module wants the same decision-free arithmeti ### Configuration -11. **Every setting you add lives inside the `analysis` scope**, in a sub-scope named after your module. `moduleSettings('yourname', [chromosomes: [], minReads: 2])` takes your whole scope's defaults and returns the whole scope, with a key the defaults do not have refused by name. Ask for the scope once and read the map; there is no per-key call with a fallback, because one could not refuse anything — a misspelling would take the default and your module would produce nothing with nothing said about it. A **top-level** `params` key is caught and named where it is written: it reaches the recorded manifest, and the identity check would otherwise read it as a parameter the results were not produced with and refuse every analysis in the project. +11. **Every setting you add lives inside the `analysis` scope**, in a sub-scope named after your module. `moduleSettings('yourname', [chromosomes: [], minReads: 2])` takes your whole scope's defaults and returns the whole scope, with a key the defaults do not have refused by name. Ask for the scope once and read the map; there is no per-key call with a fallback, because one could not refuse anything -- a misspelling would take the default and your module would produce nothing with nothing said about it. A **top-level** `params` key is caught and named where it is written: it reaches the recorded manifest, and the identity check would otherwise read it as a parameter the results were not produced with and refuse every analysis in the project. 12. **A module ships no configuration file.** `PoolSeqFlow analysis` assembles the three layers; you document the settings your module reads, and the user writes them into `.config`. ### Writing your own results -13. **Publish through `PublishResults`, and never write into `target.results` yourself.** Hand it the target and everything you produced for it; the whole analysis moves into the folder under one rename, so the folder is only ever absent or complete. A module that installs its results one file at a time leaves a half-populated folder when it fails partway, and refuse-if-populated cannot tell that from a collision — so its own folder name becomes unusable for good. +13. **Publish through `PublishResults`, and never write into `target.results` yourself.** Hand it the target and everything you produced for it; the whole analysis moves into the folder under one rename, so the folder is only ever absent or complete. A module that installs its results one file at a time leaves a half-populated folder when it fails partway, and refuse-if-populated cannot tell that from a collision -- so its own folder name becomes unusable for good. 14. **What you hand it must be real files.** `PublishResults` dereferences what Nextflow staged for exactly this reason: `cleanup = true` is set for analysis runs, so the work directory goes on success and a symlinked result becomes a dangling link. Anything you write anywhere else is yours to get right the same way. 15. **Emit the script that produced each result** into the folder beside it, in the same set of files you hand `PublishResults`. A result nobody can regenerate is the reproducibility gap this whole layer exists to close, so this one is **enforced, not asked**: publishing an analysis that carries no `*.R`, `*.r`, `*.Rmd`, `*.rmd`, `*.sh`, `*.py` or `*.jl` fails the run, names what you did produce, and publishes nothing. The folder is left exactly as the verification cleared it, so the retry is not refused. -15b. **Carry your own `citations.json`. It is required, like `manifest.json` and `main.nf`.** The modules published here do not write it by hand: they carry a `references.bib` and compile it with `dev/scripts/bib2citations.py`, so an entry can be pasted from a publisher and the fields that are not bibliographic — `r_package`, `note`, `id` — ride along as extra BibTeX fields. The frame reads only the JSON, so a module built any other way is fine; what it must not be is edited in both places. Every published analysis gets `CITATIONS.md` and `references.bib` written into its folder, in the same rename as the result. The frame contributes PoolSeqFlow, Nextflow and R; **the method your module implements and the R packages it uses are yours to declare**, because you are published separately and the frame cannot hold a list of citations for modules that do not exist yet. At the very least, name the libraries you call — a user cannot credit a package they never learn they depended on. The shape is `analysis/citations.json`'s, one entry per reference, keyed however you like; an entry naming `"r_package": "vegan"` has its version asked of R at run time rather than taken from what the environment pins, so it stays correct if someone repointed it. A directory without the file is refused at DAG-build time, before the verification clears a results folder, exactly as a missing `main.nf` is, and `modules install` refuses an archive that lacks one. +15b. **Carry your own `citations.json`. It is required, like `manifest.json` and `main.nf`.** The modules published here do not write it by hand: they carry a `references.bib` and compile it with `dev/scripts/bib2citations.py`, so an entry can be pasted from a publisher and the fields that are not bibliographic -- `r_package`, `note`, `id` -- ride along as extra BibTeX fields. The frame reads only the JSON, so a module built any other way is fine; what it must not be is edited in both places. Every published analysis gets `CITATIONS.md` and `references.bib` written into its folder, in the same rename as the result. The frame contributes PoolSeqFlow, Nextflow and R; **the method your module implements and the R packages it uses are yours to declare**, because you are published separately and the frame cannot hold a list of citations for modules that do not exist yet. At the very least, name the libraries you call -- a user cannot credit a package they never learn they depended on. The shape is `analysis/citations.json`'s, one entry per reference, keyed however you like; an entry naming `"r_package": "vegan"` has its version asked of R at run time rather than taken from what the environment pins, so it stays correct if someone repointed it. A directory without the file is refused at DAG-build time, before the verification clears a results folder, exactly as a missing `main.nf` is, and `modules install` refuses an archive that lacks one. -15c. **Declare an `outputs` entry for every file you publish, and say where it is explained.** Each is `{"file": "design.tsv", "summary": "one row per pool", "anchor": "the-experimental-design"}` — `file` may be a glob, `summary` is the one-line description the reader gets, and `anchor` names a heading of the manual this release ships. A module published separately has nowhere in that manual to point and gives `"url"` instead, in full. The frame renders one `README.md` from every module's declarations plus what every analysis carries anyway, so a folder found months later says how to read itself and no module invents its own way of saying it. Two things are checked for you: an `anchor` no heading answers to is refused before your module starts, and a `file` you declared and did not publish fails the publish with nothing written. **This is where a number that is not a measurement gets said to be one** — a bound, an estimate with an assumption behind it, a figure whose meaning changes with a setting. A table cell cannot carry that sentence and a `summary` plus a link can. +15c. **Declare an `outputs` entry for every file you publish, and say where it is explained.** Each is `{"file": "design.tsv", "summary": "one row per pool", "anchor": "the-experimental-design"}` -- `file` may be a glob, `summary` is the one-line description the reader gets, and `anchor` names a heading of the manual this release ships. A module published separately has nowhere in that manual to point and gives `"url"` instead, in full. The frame renders one `README.md` from every module's declarations plus what every analysis carries anyway, so a folder found months later says how to read itself and no module invents its own way of saying it. Two things are checked for you: an `anchor` no heading answers to is refused before your module starts, and a `file` you declared and did not publish fails the publish with nothing written. **This is where a number that is not a measurement gets said to be one** -- a bound, an estimate with an assumption behind it, a figure whose meaning changes with a setting. A table cell cannot carry that sentence and a `summary` plus a link can. ### Sharing what you derive 16. **Anything another module could want goes in `Analysis/Main`, through `publishIntermediate`.** It writes the provenance record and then the file, which is the invariant everything else reads: an intermediate that is there is one whose provenance can be read. It is one call rather than two so that the order is not yours to get wrong. -17. **Call `RestoreIntermediates` before you read one.** `PoolSeqFlow analysis complete` moves `Analysis/Main` to permanent storage, and the restore brings back the ones you name, a file at a time. It refuses anything derived from results the project no longer holds — a pipeline re-run leaves the identity check with nothing to object to and every intermediate under it stale, and the record beside each one is what sees that. An intermediate that is nowhere is the ordinary answer on a first run: derive it. +17. **Call `RestoreIntermediates` before you read one.** `PoolSeqFlow analysis complete` moves `Analysis/Main` to permanent storage, and the restore brings back the ones you name, a file at a time. It refuses anything derived from results the project no longer holds -- a pipeline re-run leaves the identity check with nothing to object to and every intermediate under it stale, and the record beside each one is what sees that. An intermediate that is nowhere is the ordinary answer on a first run: derive it. ### Assumptions -17b. **Take the time axis and the series from `analysisPlan`, never from the metadata.** `target.design.time` carries the column, the kind, the unit, and one level per distinct value with an `index` and a `position`; `target.design.series` carries the pools of each series in time order, and `design.roles` says which key columns name a condition, a biological replicate and a technical one. Both are resolved once, by the frame, under settings the user declared — so every module in a project orders time the same way. **`position` is `null` for a categorical axis**, and that is the test: a rate per unit time is meaningless there, so check it rather than assuming an axis you can regress on. Anything worth knowing that is not an error is in `target.design.warnings`, which the verification report and the published README both render; read it if your output should repeat any of it. +17b. **Take the time axis and the series from `analysisPlan`, never from the metadata.** `target.design.time` carries the column, the kind, the unit, and one level per distinct value with an `index` and a `position`; `target.design.series` carries the pools of each series in time order, and `design.roles` says which key columns name a condition, a biological replicate and a technical one. Both are resolved once, by the frame, under settings the user declared -- so every module in a project orders time the same way. **`position` is `null` for a categorical axis**, and that is the test: a rate per unit time is meaningless there, so check it rather than assuming an axis you can regress on. Anything worth knowing that is not an error is in `target.design.warnings`, which the verification report and the published README both render; read it if your output should repeat any of it. -17c. **Degrees of freedom come from `design.units`, never from `design.series`.** A **unit** is one independent biological entity. Two pools are two units unless a column named in `analysis.design.technicalRep` says they are one material measured twice — two lanes of one cage are one unit, three control pools with nothing to tell them apart are three. **`design.series` is empty for a project with no time axis and `design.units` is not**, so a module that reads units works on a one-off comparison and one that reads series does not. Each unit carries its own `pools`, sorted as `design.pools` is, so index the published columns with those and never re-derive the grouping. `design.conditions` is the level above: units with the biological-replicate columns dropped too, each carrying the `units` it holds. +17c. **Degrees of freedom come from `design.units`, never from `design.series`.** A **unit** is one independent biological entity. Two pools are two units unless a column named in `analysis.design.technicalRep` says they are one material measured twice -- two lanes of one cage are one unit, three control pools with nothing to tell them apart are three. **`design.series` is empty for a project with no time axis and `design.units` is not**, so a module that reads units works on a one-off comparison and one that reads series does not. Each unit carries its own `pools`, sorted as `design.pools` is, so index the published columns with those and never re-derive the grouping. `design.conditions` is the level above: units with the biological-replicate columns dropped too, each carrying the `units` it holds. -**Fit the units. Do not take a pool-level statistic and read it against the unit count.** Collapse each unit's pools onto one weighted value first — the weights summed, the value their weighted mean — and fit those. Re-reading a pool-level statistic at fewer degrees of freedom is wrong in both directions: it throws away power when the pools really were independent, and it still fails to control the false-positive rate when they were not. **Anything that resamples resamples units too**, so a permutation moves whole units and never individual pools. None of this needs a branch: where a unit holds one pool the collapse returns that pool unchanged and the unit permutation is the pool permutation, so a design with no replication runs the same code and gets the same answer. +**Fit the units. Do not take a pool-level statistic and read it against the unit count.** Collapse each unit's pools onto one weighted value first -- the weights summed, the value their weighted mean -- and fit those. Re-reading a pool-level statistic at fewer degrees of freedom is wrong in both directions: it throws away power when the pools really were independent, and it still fails to control the false-positive rate when they were not. **Anything that resamples resamples units too**, so a permutation moves whole units and never individual pools. None of this needs a branch: where a unit holds one pool the collapse returns that pool unchanged and the unit permutation is the pool permutation, so a design with no replication runs the same code and gets the same answer. 17d. **An empty `analysis.design.biologicalRep` is not a claim that there are no replicates.** The frame declares nothing it was not told. If your test stratifies on biological replication, refuse by name when that list is empty rather than treating every unit as a stratum of its own. -17e. **Fit only the covariates whose `inDesign` is true.** `design.covariates` carries every declared one, because the report and the published README print them all — a covariate's value is worth having on the record whether or not anything fits it. `inDesign` is the user's answer to which may enter a model, and each one you fit costs a degree of freedom out of `n_units - 2`. Say in your own output how many you fitted and which. +17e. **Fit only the covariates whose `inDesign` is true.** `design.covariates` carries every declared one, because the report and the published README print them all -- a covariate's value is worth having on the record whether or not anything fits it. `inDesign` is the user's answer to which may enter a model, and each one you fit costs a degree of freedom out of `n_units - 2`. Say in your own output how many you fitted and which. -18. **If your statistic assumes something, refuse when it does not hold, and name the assumption.** A gate that silently passes is the failure mode; a gate that stops the run with `ploidy = 3, and this estimator assumes 2` costs a user one minute. Declare it in the manifest's `gates` as well — the frame carries that field, and enforcing it is currently the module's own job. -19. **A decision is yours to state, not the library's to have made.** Where a shared derivation needs a choice — a missing-site policy, say — take it as a required argument with no default, print it in what you write, and **key the intermediate's name on it**: `freqmatrix_pairwise.tsv`, never `freqmatrix.tsv`. Intermediates are reused by name, so a name that omits the decision hands the next module your assumption while its report claims its own. +18. **If your statistic assumes something, refuse when it does not hold, and name the assumption.** A gate that silently passes is the failure mode; a gate that stops the run with `ploidy = 3, and this estimator assumes 2` costs a user one minute. Declare it in the manifest's `gates` as well -- the frame carries that field, and enforcing it is currently the module's own job. +19. **A decision is yours to state, not the library's to have made.** Where a shared derivation needs a choice -- a missing-site policy, say -- take it as a required argument with no default, print it in what you write, and **key the intermediate's name on it**: `freqmatrix_pairwise.tsv`, never `freqmatrix.tsv`. Intermediates are reused by name, so a name that omits the decision hands the next module your assumption while its report claims its own. 20. **Print your assumptions on what you write.** A figure that leaves the project must carry the conditions it was computed under. ### Testing @@ -155,13 +155,13 @@ PoolSeqFlow analysis modules list # what is installed here PoolSeqFlow analysis modules uninstall # remove one, after confirming ``` -`list` and `uninstall` read this directory and nothing else — no environment, no project — so they answer from anywhere. `list` marks a directory that holds a manifest and no `main.nf`, which is the state that stops every analysis run. +`list` and `uninstall` read this directory and nothing else -- no environment, no project -- so they answer from anywhere. `list` marks a directory that holds a manifest and no `main.nf`, which is the state that stops every analysis run. -`available` and `install` read the **catalogue**, over the network, from `modules-repo/index.tsv` on the repository's default branch. **A release carries no copy of it** — it is `export-ignore`d — because a module published after a release still has to be installable into it. `POOLSEQFLOW_MODULE_INDEX` points them at a URL or a local path instead, for a mirror inside an institution or a machine with no network. +`available` and `install` read the **catalogue**, over the network, from `modules-repo/index.tsv` on the repository's default branch. **A release carries no copy of it** -- it is `export-ignore`d -- because a module published after a release still has to be installable into it. `POOLSEQFLOW_MODULE_INDEX` points them at a URL or a local path instead, for a mirror inside an institution or a machine with no network. ## Publishing a module -A published module is a **gzipped tarball unpacking to `/`**, with `manifest.json` and `main.nf` inside it, plus whatever else the module needs — its R, its config, its manual fragment. Then one row in the catalogue: +A published module is a **gzipped tarball unpacking to `/`**, with `manifest.json` and `main.nf` inside it, plus whatever else the module needs -- its R, its config, its manual fragment. Then one row in the catalogue: | Column | What | |---|---| @@ -178,7 +178,7 @@ A published module is a **gzipped tarball unpacking to `/`**, with `manife Several rows may name one module. `install ` takes the newest version whose `contract` this release speaks; naming a version installs exactly that one, which is what a paper's methods section should say. -The catalogue carries two headers of its own, and they are not interchangeable. **`#!index-format`** is the column layout, checked before a single row is read. The columns are matched **by name** out of the header row, so their order does not matter and a column your release has never heard of is ignored rather than misread — which means adding one is additive and needs no bump. Bump it only for a change that cannot be read at all: a required column renamed or removed, or one whose meaning changed under the same name. That stops every already-published release from reading the file, so it is close to a last resort. **`#!index-version`** is what is in the catalogue, in the same `YYYYMMDD.NNN` form, bumped on every publish; nothing is ever refused on it, because a release has to be able to read a catalogue newer than itself. `modules available` prints it, and `install` records it in the `.source` file beside the module — which is the only record afterwards of which catalogue a module came from. +The catalogue carries two headers of its own, and they are not interchangeable. **`#!index-format`** is the column layout, checked before a single row is read. The columns are matched **by name** out of the header row, so their order does not matter and a column your release has never heard of is ignored rather than misread -- which means adding one is additive and needs no bump. Bump it only for a change that cannot be read at all: a required column renamed or removed, or one whose meaning changed under the same name. That stops every already-published release from reading the file, so it is close to a last resort. **`#!index-version`** is what is in the catalogue, in the same `YYYYMMDD.NNN` form, bumped on every publish; nothing is ever refused on it, because a release has to be able to read a catalogue newer than itself. `modules available` prints it, and `install` records it in the `.source` file beside the module -- which is the only record afterwards of which catalogue a module came from. `dev/scripts/bump-analysis-version.sh index` moves it, and `dev/scripts/check-analysis-versions.sh` refuses a change to the rows that left it behind. @@ -188,11 +188,11 @@ For a module in this repository, one command does all of it: dev/scripts/publish-module.sh [ref] ``` -It builds the tarball into `modules-repo/`, reads `contract`, `frame`, `environment` and `summary` out of the module's own manifest at that ref, appends the row, and bumps `#!index-version`. It refuses to overwrite a version that is already published — somebody may have installed it, and its checksum is in the catalogue — so a change means bumping the module's version and publishing that. +It builds the tarball into `modules-repo/`, reads `contract`, `frame`, `environment` and `summary` out of the module's own manifest at that ref, appends the row, and bumps `#!index-version`. It refuses to overwrite a version that is already published -- somebody may have installed it, and its checksum is in the catalogue -- so a change means bumping the module's version and publishing that. -**The tarball is built from the extracted tree rather than piped straight out of `git archive`, and that is not fussiness.** `git archive :modules/` reads a subtree, and `.gitattributes` patterns are anchored at the repository root — so a rule written for `modules/` does not match `test/` inside that subtree, and the module's own cases would ship where a release tarball excludes them. Archiving a tree also stamps `mtime` as *now*, so two builds of one ref would not match. The script drops `test/` and repacks with the commit's timestamp, which makes republishing the same ref produce the same bytes. +**The tarball is built from the extracted tree rather than piped straight out of `git archive`, and that is not fussiness.** `git archive :modules/` reads a subtree, and `.gitattributes` patterns are anchored at the repository root -- so a rule written for `modules/` does not match `test/` inside that subtree, and the module's own cases would ship where a release tarball excludes them. Archiving a tree also stamps `mtime` as *now*, so two builds of one ref would not match. The script drops `test/` and repacks with the commit's timestamp, which makes republishing the same ref produce the same bytes. -The tarballs are committed and served from the site alongside the catalogue, so **a published row and the file it names go out in one commit** — otherwise the row advertises a download that 404s until the next deploy. +The tarballs are committed and served from the site alongside the catalogue, so **a published row and the file it names go out in one commit** -- otherwise the row advertises a download that 404s until the next deploy. Two rules the installer enforces, so build for them: diff --git a/modules/association/manifest.json b/modules/association/manifest.json index 2386a44..0f97d3d 100644 --- a/modules/association/manifest.json +++ b/modules/association/manifest.json @@ -1,6 +1,6 @@ { "name": "association", - "version": "20260910.004", + "version": "20261004.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -74,7 +74,7 @@ "packages": [ "r-dofuture=1.3.0", "r-foreach=1.5.2", - "r-future=1.75.0", + "r-future=1.76.0", "r-ggplot2=4.0.3", "r-jsonlite=2.0.0", "r-rcpp=1.1.2" diff --git a/modules/basicstats/basicstats.R b/modules/basicstats/basicstats.R index e221d19..acf36ad 100644 --- a/modules/basicstats/basicstats.R +++ b/modules/basicstats/basicstats.R @@ -216,22 +216,30 @@ for (name in pool_names) { stats <- column_stats(snp[[name]]) segregating <- is_segregating(snp[[name]], stats$depth, figures$sensitivity, OPTS$minReads) + # Every depth summary below is over the sites this pool was READ at, and `unmeasured` + # counts the rest. A called site holds no reads for a pool only where + # vcffilter.dropZeroDepth is off, which is what makes any of these differ from `sites`. + read_at <- !is.na(stats$depth) & stats$depth > 0 + for (chrom in intersect(chrom_levels, unique(snp$CHROM))) { here <- snp$CHROM == chrom + seen <- stats$depth[here & read_at] depth_rows[[length(depth_rows) + 1]] <- data.frame( pool = name, chrom = chrom, sites = sum(here), - depth_mean = mean(stats$depth[here], na.rm = TRUE), - depth_median = median(stats$depth[here], na.rm = TRUE), - depth_harmonic = harmonic_mean(stats$depth[here]), + unmeasured = sum(here & !read_at), + depth_mean = if (length(seen) > 0) mean(seen) else NA_real_, + depth_median = if (length(seen) > 0) median(seen) else NA_real_, + depth_harmonic = harmonic_mean(seen), stringsAsFactors = FALSE) } size <- n_eff(figures$nChrom, stats$depth) corrected <- stats$h * size / (size - 1) - harmonic_depth <- harmonic_mean(stats$depth) + harmonic_depth <- harmonic_mean(stats$depth[read_at]) diversity_rows[[length(diversity_rows) + 1]] <- data.frame( pool = name, n_chrom = figures$nChrom, sites = length(stats$depth), + unmeasured = sum(!read_at), segregating = sum(segregating, na.rm = TRUE), depth_harmonic = harmonic_depth, n_eff_harmonic = pool_n_eff(figures$nChrom, harmonic_depth), @@ -302,7 +310,7 @@ for (entry in design$pools) { for (row in diversity_rows) { neff_rows[[length(neff_rows) + 1]] <- data.frame( level = "pool", id = row$pool, pool = row$pool, source = "called", - positions = row$sites, depth_harmonic = row$depth_harmonic, + positions = row$sites - row$unmeasured, depth_harmonic = row$depth_harmonic, n_chrom = row$n_chrom, n_eff = row$n_eff_harmonic, estimate = "exact", stringsAsFactors = FALSE) } diff --git a/modules/basicstats/manifest.json b/modules/basicstats/manifest.json index a31616e..23c77bd 100644 --- a/modules/basicstats/manifest.json +++ b/modules/basicstats/manifest.json @@ -1,6 +1,6 @@ { "name": "basicstats", - "version": "20260910.004", + "version": "20261004.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -17,6 +17,7 @@ ], "gates": [ "Depth is the sum of a pool's comma-separated cell in the depth table: reads supporting any allele at that site, after step 7's depth, quality and false-positive filters.", + "Every depth summary is over the sites the pool was READ at, and depth.tsv and diversity.tsv publish the count left out as `unmeasured`. A called site a pool has no reads at is not a depth of zero to be averaged in, and the harmonic mean of any set holding a zero is zero. This differs from `sites` only where vcffilter.dropZeroDepth is off.", "Every statistic is computed over CALLED SITES only. Sites the pipeline did not call are absent from the tables and are not counted as invariant.", "Depth and diversity are computed over the SNP table alone. Indels are counted in sites.tsv and enter nothing else.", "A spanning-deletion allele reaching the SNP table as a * or an X is one allele of that site like any other: it carries reads, so it carries frequency, and dropping it would renormalize the site to something no pool was sequenced at.", @@ -39,7 +40,7 @@ }, { "file": "depth.tsv", - "summary": "each pool's depth at the called SNP sites of each sequence: mean, median and harmonic mean", + "summary": "each pool's depth at the called SNP sites of each sequence it was read at: mean, median and harmonic mean, beside the count of sites it missed", "anchor": "basicstats-depth" }, { @@ -73,7 +74,7 @@ "r-data.table=1.18.6.1", "r-dofuture=1.3.0", "r-foreach=1.5.2", - "r-future=1.75.0", + "r-future=1.76.0", "r-ggplot2=4.0.3", "r-jsonlite=2.0.0", "r-rcpp=1.1.2" diff --git a/modules/basicstats/test/basicstats.sh b/modules/basicstats/test/basicstats.sh index 3f25f68..c9a9ae2 100644 --- a/modules/basicstats/test/basicstats.sh +++ b/modules/basicstats/test/basicstats.sh @@ -256,6 +256,83 @@ test_the_effective_size_is_reported_at_both_levels() { "there is no per-library row from the called sites: the tables are per pool" } +# A POOL WITH NO READS AT A CALLED SITE, which every depth summary here must leave out rather +# than average in as a zero. +# +# Reaching one takes vcffilter.dropZeroDepth off AND vcffilter.minDP at zero, two deliberate +# edits from any default, so the corpus does not carry one and this case makes it: one cell of +# one pool emptied at one site, every other cell of that site left alone. +# +# What it guards, measured before the change on a four-site pool at depth 100: depth_mean fell +# to 75 and read as an ordinary number, and depth_harmonic fell to exactly 0 - harmonic_mean() +# answers 0 to any zero among its values, deliberately and correctly for one position, which +# over a genome says the pool carries no information anywhere. n_eff_harmonic followed it to NA. +# +# The expectations are computed from the table by awk here, not from expected.tsv, because +# freq_corpus.py's arithmetic runs over the corpus as written and this case has changed it. +test_a_pool_that_missed_a_site_is_summarized_over_the_sites_it_saw() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/basicstats-unmeasured") + rm -rf "$sb"; mkdir -p "$sb" + CORPUS_DIR="$sb" + python3 "$REPO_ROOT/test/tools/freq_corpus.py" "$sb" "$sb" + + # TestSample1 is the first sample column, which is column 6. chr1:100 is a biallelic site + # it holds reads at; the zeros are built to the site's own arity rather than assumed. + local table="$sb/Frequencies/Test_snp_depth.tsv" + awk -F'\t' -v OFS='\t' ' + NR > 1 && $1 == "chr1" && $2 == 100 { + n = split($6, counts, ",") + $6 = "0" + for (i = 2; i <= n; i++) $6 = $6 ",0" + } + { print }' "$table" > "$table.zeroed" + mv "$table.zeroed" "$table" + assert_contains "$(awk -F'\t' '$1=="chr1" && $2==100 { print $6 }' "$table")" "0,0" \ + "the fixture should now hold a pool with no reads at chr1:100" + + basicstats_direct "$sb/run" '{"minReads":2,"binSize":100000,"workers":1,"usecpp":false}' "$sb" + assert_file "$sb/run/depth.tsv" "the module should run: $(cat "$sb/run/out.txt" 2>/dev/null)" + + # The mean and the count over TestSample1's chr1 cells that still carry reads. + local want_mean want_seen + want_mean=$(awk -F'\t' 'NR > 1 && $1 == "chr1" { + n = split($6, c, ","); t = 0; for (i = 1; i <= n; i++) t += c[i] + if (t > 0) { sum += t; kept++ } } END { print sum / kept }' "$table") + want_seen=$(awk -F'\t' 'NR > 1 { + n = split($6, c, ","); t = 0; for (i = 1; i <= n; i++) t += c[i] + if (t > 0) kept++ } END { print kept }' "$table") + + assert_eq "1" "$(published_cell2 "$sb/run/depth.tsv" TestSample1 chr1 unmeasured)" \ + "the site it missed is counted and named" + assert_eq "0" "$(published_cell2 "$sb/run/depth.tsv" TestSample2 chr1 unmeasured)" \ + "and no other pool is charged for it" + assert_close "$(published_cell2 "$sb/run/depth.tsv" TestSample1 chr1 depth_mean)" "$want_mean" \ + "the mean is over the sites the pool was read at" + + # THE ONE THAT USED TO COLLAPSE. A single missed site took this to 0 and n_eff with it. + local harmonic + harmonic=$(published_cell2 "$sb/run/depth.tsv" TestSample1 chr1 depth_harmonic) + awk -v h="$harmonic" 'BEGIN { exit !(h > 0) }' \ + || fail_case "one missed site must not take the harmonic depth to $harmonic" + assert_close "$(published_cell "$sb/run/diversity.tsv" TestSample1 unmeasured)" "1" \ + "diversity.tsv counts it too" + # Tested as a string first: awk -v takes NA as a string, and "NA" > 0 compares "NA" with + # "0" character by character and is true, so a numeric test alone passes on the NA this + # case exists to catch. + local neff + neff=$(published_cell "$sb/run/diversity.tsv" TestSample1 n_eff_harmonic) + assert_not_contains "$neff" "NA" "and the effective size must survive it" + awk -v n="$neff" 'BEGIN { exit !(n + 0 > 0) }' \ + || fail_case "and be a positive number, got '$neff'" + + # neff.tsv repeats the called-site figure, so its `positions` is what the figure was taken + # over and not the site count. + assert_eq "$want_seen" \ + "$(awk -F'\t' '$1=="pool" && $2=="TestSample1" && $4=="called" { print $5 }' "$sb/run/neff.tsv")" \ + "the called row counts the positions its depth came from" +} + # A MERGED POOL, where the two levels are genuinely different and the pool figure is a bound. # The corpus's --merged layout is three pools of two libraries: the depth table is named by # RG_Sample and the histograms by SampleID, which is the only shape that separates them. diff --git a/modules/lib/README.md b/modules/lib/README.md index c5a7e3f..ee78e6a 100644 --- a/modules/lib/README.md +++ b/modules/lib/README.md @@ -9,8 +9,8 @@ each module starts for itself. ## What belongs here -Only derivations that carry no decision. A derivation that needs a choice — a missing-site -policy, say — takes it as a required argument with no default, is declared in the calling +Only derivations that carry no decision. A derivation that needs a choice -- a missing-site +policy, say -- takes it as a required argument with no default, is declared in the calling module's `gates`, and keys any intermediate's name on the value. That rule is in `../../modules/README.md` and it is what stops one module handing the next its assumption. @@ -21,13 +21,13 @@ the machine, so they must not need a package the analysis environment pins. A mo may use `data.table` and `ggplot2` freely. A derivation with a compiled form keeps that form in `../cpp/` under the same name, and the R -here stays the reference the compiled one is judged against — never a wrapper that dispatches +here stays the reference the compiled one is judged against -- never a wrapper that dispatches to it. `../cpp/` needs Rcpp and a compiler on the machine, which is why nothing here may reach for it, and a module that offers the compiled path asks for it explicitly and refuses when it cannot be built. **One file per function.** A module declares which it sources, and the publish concatenates -exactly those into one file that ships beside the result — so a published folder carries the +exactly those into one file that ships beside the result -- so a published folder carries the code that computed its numbers, not a driver that refers to code the reader does not have. ## Versioning diff --git a/modules/lib/allele_frequencies/allele_frequencies.R b/modules/lib/allele_frequencies/allele_frequencies.R index ee3db32..bf39194 100644 --- a/modules/lib/allele_frequencies/allele_frequencies.R +++ b/modules/lib/allele_frequencies/allele_frequencies.R @@ -20,13 +20,13 @@ # # `columns` is one column of the depth table per pool: one comma-separated count list per site, # REF first and then each ALT. A site holds any number of alleles, so the frequencies are one -# long vector per pool rather than a rectangle, and `site` is the index that groups them — +# long vector per pool rather than a rectangle, and `site` is the index that groups them -- # `rowsum(x, site)` is a per-site total and `depth[site, ]` is a per-allele depth. # # THE ALT COLUMN IS ONE LIST FOR THE WHOLE COHORT, so every pool's cell at a site holds one # count per allele. A site where they differ is refused, naming the pool and the site. # -# A pool with no reads at a site gets depth 0 and no frequency — there is nothing observed there +# A pool with no reads at a site gets depth 0 and no frequency -- there is nothing observed there # to be a frequency of. bcftools' missing value takes that pool's whole site with it, as it does # in site_diversity(). allele_frequencies <- function(columns) { diff --git a/modules/lib/allele_frequencies/manifest.json b/modules/lib/allele_frequencies/manifest.json index 1b6aba0..be0c027 100644 --- a/modules/lib/allele_frequencies/manifest.json +++ b/modules/lib/allele_frequencies/manifest.json @@ -1,6 +1,6 @@ { "name": "allele_frequencies", - "version": "20260910.002", + "version": "20261004.001", "kind": "library", "contract": "freq-1", "license": "GPL-3.0-or-later", diff --git a/modules/lib/n_eff/manifest.json b/modules/lib/n_eff/manifest.json index 2d8e2e4..f8a66ea 100644 --- a/modules/lib/n_eff/manifest.json +++ b/modules/lib/n_eff/manifest.json @@ -1,6 +1,6 @@ { "name": "n_eff", - "version": "20260910.002", + "version": "20261004.001", "kind": "library", "license": "GPL-3.0-or-later", "frame": "20260910.001", diff --git a/modules/lib/n_eff/pool_n_eff.R b/modules/lib/n_eff/pool_n_eff.R index d53ad14..188d7ea 100644 --- a/modules/lib/n_eff/pool_n_eff.R +++ b/modules/lib/n_eff/pool_n_eff.R @@ -2,7 +2,7 @@ # # 1/n_eff = 1/n_chrom + (1 - 1/n_chrom) * 1/harmonic_depth # -# The harmonic mean is what averages n_eff, because 1/n_eff is linear in 1/depth — so a pool +# The harmonic mean is what averages n_eff, because 1/n_eff is linear in 1/depth -- so a pool # collapses to two numbers rather than one per site, exactly. # # `harmonic_depth` is harmonic_mean() over whatever set of positions is being summarized: the diff --git a/modules/lib/nei_distance/manifest.json b/modules/lib/nei_distance/manifest.json index c4c0de6..3b22155 100644 --- a/modules/lib/nei_distance/manifest.json +++ b/modules/lib/nei_distance/manifest.json @@ -1,6 +1,6 @@ { "name": "nei_distance", - "version": "20260910.002", + "version": "20261004.001", "kind": "library", "contract": "freq-1", "license": "GPL-3.0-or-later", diff --git a/modules/lib/nei_distance/nei_distance.R b/modules/lib/nei_distance/nei_distance.R index 30df7fb..96faf13 100644 --- a/modules/lib/nei_distance/nei_distance.R +++ b/modules/lib/nei_distance/nei_distance.R @@ -11,7 +11,7 @@ # # J_A read off a sample is biased upward by the sampling: two reads from one pool agree more # often than two chromosomes do. The unbiased form subtracts that pool's own diversity at the -# site, scaled by its effective sample size there. J_AB takes no such term — the two pools are +# site, scaled by its effective sample size there. J_AB takes no such term -- the two pools are # sequenced independently, so nothing correlates their draws. # # nei_distance(freq, site, n_eff_site) diff --git a/modules/mds/manifest.json b/modules/mds/manifest.json index 907bfbb..a2c83fe 100644 --- a/modules/mds/manifest.json +++ b/modules/mds/manifest.json @@ -1,6 +1,6 @@ { "name": "mds", - "version": "20260910.003", + "version": "20261004.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -72,7 +72,7 @@ "packages": [ "r-dofuture=1.3.0", "r-foreach=1.5.2", - "r-future=1.75.0", + "r-future=1.76.0", "r-ggplot2=4.0.3", "r-jsonlite=2.0.0", "r-rcpp=1.1.2" diff --git a/multi-run.csv.example b/multi-run.csv.example index 57a1cc1..f4cb358 100644 --- a/multi-run.csv.example +++ b/multi-run.csv.example @@ -1,89 +1,26 @@ -# Multi-run table — EXAMPLE, not a template. -# -# Nothing copies this file. Read it, then write your own table with only the rows and columns -# your comparison actually needs. -# -# To use one: -# 1. write your table, anywhere in your project directory -# 2. in parameters.config, set multiRun = true -# multiRunFile = 'your-table.csv' -# -# The table is read before any compute is spent: an unusable one stops the run outright, and a -# usable one is printed in full - the runs, where each one's results go, and what differs - in -# step 0's report. -# -# --------------------------------------------------------------------------------------- -# THE RULES, ALL FOUR OF THEM -# -# 1. One column must be RunID. It names the run and becomes a directory, so keep it to -# letters, digits, dot, dash and underscore. Results go to storageDir//. -# -# 2. Every other column is a parameter name spelt exactly as parameters.config spells it, -# with NO `params.` prefix. Nested ones are dotted: trim_galore.quality, bcftools.maxDepth. -# ANY parameter may be varied, including the ones the pipeline normally computes for -# itself. A name that is not a parameter is refused rather than ignored. -# -# 3. A BLANK CELL MEANS "take it from parameters.config", so a table is only as long as the -# difference between the runs. There is therefore no way to set a parameter to an empty -# string here. -# -# 4. A value containing a comma must be quoted - "-a AD,DP,SP,INFO/AD". The default -# readPattern, *_R{1,2}.fq.gz, is exactly such a value. -# -# Blank lines and lines starting with # are ignored, so you can keep a run in the file with a -# # in front of it instead of deleting it. -# -# --------------------------------------------------------------------------------------- -# WHAT HAPPENS TO THE VALUES YOU SET -# -# Setting an INPUT to a computed value moves the computed value with it: poolSize re-derives -# filterFalsePositives.sensitivity, trim_galore.quality re-derives trim_galore.options, -# bcftools.maxDepth re-derives bcftools.mpileupOptions, threads re-derives the whole cores -# ladder. You do not have to keep those in step by hand. -# -# Setting a COMPUTED value directly is used exactly as written, and then nothing is derived -# from its inputs any more - pin bcftools.mpileupOptions and bcftools.maxDepth stops meaning -# anything for that run. Step 0 names any column that does this. -# -# --------------------------------------------------------------------------------------- -# FOUR EXAMPLES. Pick the shape you need; do not use all of them at once. - -# --- one: the same reads against two reference genomes --------------------------------- -# The commonest reason to want this. Everything else comes from parameters.config. -# -# RunID,referenceFile,gffFile -# lab_strain,lab_strain.fasta.gz,lab_strain.gff.gz -# field_isolate,field_isolate.fasta.gz,field_isolate.gff.gz - -# --- two: one reference, three trimming stringencies ------------------------------------ -# Note the blank cell on the first row: it takes the configured quality, so the baseline is -# whatever parameters.config already says rather than a number repeated here. -# -# RunID,trim_galore.quality -# trim_default, -# trim_20,20 -# trim_30,30 - -# --- three: a filtering sweep ------------------------------------------------------------ -# These diverge late - nothing before variant calling differs between them. -# -# RunID,vcffilter.minDP,vcffilter.minQUAL,filterFalsePositives.sampleThreshold -# lenient,10,20,0.1 -# standard,20,30,0.2 -# strict,40,50,0.3 - -# --- four: varying pool size, and pinning something outright ----------------------------- -# pool_50 and pool_200 let sensitivity be re-derived from poolSize. pinned_depth does the -# opposite: it writes the mpileup options itself, so bcftools.maxDepth no longer feeds them. -# Step 0 will point that column out. -# -# RunID,poolSize,bcftools.mpileupOptions -# pool_50,50, -# pool_200,200, -# pinned_depth,,"-B -C 50 -q 30 -Q 30 -d 8000 -a AD,DP,SP,INFO/AD -Ou" +# Multi-run table -- EXAMPLE, not a template. Nothing copies this file; write your own. +# The manual explains it: https://ozankiratli.github.io/PoolSeqFlow/ +# +# To use one: in parameters.config set multiRun = true and multiRunFile = 'your-table.csv'. +# Blank lines and # lines are ignored, so a run can be kept commented out instead of deleted. +# +# THE RULES: +# 1. One column must be RunID. It becomes a directory under storageDir, so letters, digits, +# dot, dash and underscore only. +# 2. Every other column is a parameter name spelt as parameters.config spells it, with no +# `params.` prefix. Nested ones are dotted: trim_galore.quality, variantCall.maxDepth. +# Any parameter may be varied. A name that is not one is refused, not ignored. +# 3. A blank cell takes the value from parameters.config, so the table is only as long as +# the difference between the runs. +# 4. Quote a value containing a comma: "-a AD,DP,SP,INFO/AD". The default readPattern, +# *_R{1,2}.fq.gz, is one. +# +# Setting an INPUT re-derives whatever depends on it: poolSize moves sensitivity, threads +# moves the whole cores ladder. Setting a COMPUTED value pins it, and its inputs stop feeding +# it -- pin variantCall.mpileupOptions and variantCall.maxDepth no longer means anything for +# that run. Step 0 names any column that does this. -# --------------------------------------------------------------------------------------- -# A REAL, USABLE TABLE. Two references, and the second trimmed harder. Delete or replace it. +# A real, usable table: two references, the second trimmed harder. Delete or replace it. RunID,referenceFile,gffFile,trim_galore.quality reference_a,reference_a.fasta.gz,reference_a.gff.gz, reference_b,reference_b.fasta.gz,reference_b.gff.gz,30 diff --git a/nextflow.config b/nextflow.config index 3810dbb..c54ba8e 100644 --- a/nextflow.config +++ b/nextflow.config @@ -30,6 +30,16 @@ includeConfig "${launchDir}/parameters.config" // A ternary and not an `if`: the strict parser rejects `if` statements in config. params.cores = params.containsKey('cores') && params.cores != null ? params.cores : [:] +// The value of vcffilter.dropZeroDepth for a project whose config does not set it. Step 7 reads +// it in a ternary and an absent key is null, which a ternary reads as false, so without this a +// config predating the parameter runs as `false` where the template ships `true`. +// +// Below the include and guarded by containsKey, so a project that does set it wins. The map +// guard above it is what keeps a config carrying no vcffilter block at all parsing: the nested +// assignment on a null scope fails the whole file. +params.vcffilter = params.containsKey('vcffilter') && params.vcffilter != null ? params.vcffilter : [:] +params.vcffilter.dropZeroDepth = params.vcffilter.containsKey('dropZeroDepth') ? params.vcffilter.dropZeroDepth : true + // A dry run checks and previews instead of computing. Set by `./PoolSeqFlow dryrun`, which also // runs dryrun.nf: the entry script stops the pipeline, this flag stops step 0 recording anything. params.dryRun = false diff --git a/parameters.config.template b/parameters.config.template index 736c87b..a95dc9a 100644 --- a/parameters.config.template +++ b/parameters.config.template @@ -1,45 +1,31 @@ -// PoolSeqFlow parameters. The manual explains every setting in this file: -// https://ozankiratli.github.io/PoolSeqFlow/ -// -// Configure runs here and nowhere else. Nextflow passes command-line parameters as strings, -// so `--annotate false` sets the string "false", which evaluates as true. +// PoolSeqFlow parameters. The manual explains every setting: https://ozankiratli.github.io/PoolSeqFlow/ params { //main parameters threads = 8 // cores per task memory = '24 GB' // memory ceiling per task - // Your project directory, and the one you launch from. The faster volume. It may be - // neither the installation nor storageDir. + // Your project directory, and the one you launch from. Not the installation, not storageDir. mainDir = "/path/to/project/directory" - // Permanent storage, and a slower volume will do. Called projectDir before 3.0; - // `./PoolSeqFlow migrate_config` carries the old value over. + // Permanent storage; a slower volume will do. Was projectDir before 3.0. storageDir = "/path/to/permanent/storage" - // The files you place, all relative to mainDir: - // mainDir/Data/ the reads, in it or in subfolders of it (dataSource names it) - // mainDir/Reference/ reference + annotation (referenceFile, gffFile) - // mainDir/metadata.csv what your samples ARE (metadataFile) + // Relative to mainDir: Data/ the reads, Reference/ the reference and annotation. dataSource = 'Data' readPattern = "*_R{1,2}.fq.gz" referenceFile = 'reference.fasta.gz' poolSize = 100 ploidy = 2 - // One row per pair of FASTQ files. metadata.csv.template explains the columns; it - // replaces RGTags.csv. + // One row per FASTQ pair; the example file beside it explains the columns. metadataFile = 'metadata.csv' annotate = true gffFile = 'reference.gff.gz' - // Several runs from one invocation - the same reads against two references, say. true - // reads multiRunFile below: a RunID column, plus one column per parameter that differs - // from the values above. See multi-run.csv.example in the installation. + // Several runs from one invocation. true reads multiRunFile; see multi-run.csv.example. multiRun = false multiRunFile = 'runs.csv' referenceFa = "${params.referenceFile.replace('.gz', '')}" - // Core allocation, computed from `threads` and left commented out so that changing - // `threads` moves all of it together. Uncomment any one to pin it; a pinned value still - // feeds the ones computed from it. + // Computed from threads. Uncomment any one to pin it; a pinned value still feeds the rest. cores { // ladder = largest power of two <= threads, capped at 8 // trim = largest N whose N+4 fits in threads: 12+ -> 8, 8+ -> 4, 6+ -> 2, else 1 @@ -66,14 +52,11 @@ params { // trim_galore parameters trim_galore { quality = 25 - // true -> no adapter is passed; Trim Galore auto-detects it (Illumina/Nextera/smallRNA) - // and adapter1/adapter2 below are ignored. - // false -> BOTH adapter1 and adapter2 must be set to your provider's sequences. + // true: Trim Galore auto-detects and ignores adapter1/adapter2. false: both must be set. autodetect = true adapter1 = '' adapter2 = '' - // Built from the values above; uncomment either to take full control of it. - // --cores and --fastqc_args come from task.cpus. + // Built from the values above; uncomment either to take full control. // adapterOptions = autodetect ? '' : "-a ${params.trim_galore.adapter1} -a2 ${params.trim_galore.adapter2}" // options = "--fastqc --paired --retain_unpaired -q ${params.trim_galore.quality} ${params.trim_galore.adapterOptions}" } @@ -83,9 +66,8 @@ params { min_length = 50 // inert on its own - see the options lines below // --cores is supplied by ClipReads from task.cpus options = "" - // To apply min_length, comment the line above and uncomment the one below. ClipReads - // also passes -l, a limit it computes from the FastQC report, and cutadapt applies -m - // AFTER -l: a min_length above that limit discards every read and still exits 0. + // To apply min_length, swap for the line below. Careful: cutadapt applies -m after the + // -l ClipReads computes, so a min_length above that discards every read and exits 0. // options = "-m ${params.cutadapt.min_length}" // discard reads shorter than min_length after clipping } @@ -93,8 +75,7 @@ params { bwa { minScoreOutput = 30 batchSize = 10000000 - // Built from the two values above; uncomment to take full control. -t comes from - // task.cpus. + // Built from the two values above; uncomment to take full control. // options = "-K ${params.bwa.batchSize} -T ${params.bwa.minScoreOutput}" } @@ -105,28 +86,23 @@ params { mapq = 30 } - // The depth ceiling decided at step 5 and applied to each BAM at step 6, before calling. - // -1 measures one per sample from its own depth histogram, a positive number caps every - // sample there, 0 caps nothing. + // Depth ceiling, applied at step 6. -1 measures one per sample, a positive number caps every sample there, + // 0 caps nothing. capBAM { maxDepth = -1 - // How deep step 5's depth histogram looks. A sample with positions deeper than this - // stops the run naming this parameter: a ceiling read from a truncated histogram is - // read from a partial picture. Raising it cannot change a result and is not compared - // against previous runs, so it never costs a reset. + // How deep step 5's histogram looks. Raising it never changes a result, never costs a reset. histogramMax = 100000 } // Step 6: how bcftools calls variants. variantCall { - scaleMapQ = 50 + // -C. At 10 or below it does nothing; below varQualMin it discards every read. + scaleMapQ = 100 baseQualMin = 30 varQualMin = 30 - // mpileup's own ceiling, on top of whatever capBAM applied. Here 0 is no limit, and - // only capBAM.maxDepth takes -1. + // mpileup's own ceiling, on top of capBAM's. 0 is no limit; only capBAM.maxDepth takes -1. maxDepth = 0 - // Built from the four values above; uncomment to take full control of them, maxDepth - // included. -q is mapping quality, -Q is base quality. + // Built from the values above; uncomment below to change it. -q is mapping quality, -Q is base quality. // mpileupOptions = "-B -C ${params.variantCall.scaleMapQ} -q ${params.variantCall.varQualMin} -Q ${params.variantCall.baseQualMin} -d ${params.variantCall.maxDepth} -a AD,DP,SP,INFO/AD -Ou" callOptions = "-m -A -v -Ov" @@ -141,6 +117,8 @@ params { vcffilter { minDP = 20 minQUAL = 30 + // Drop a site where any sample has no reads. Redundant above minDP 1. + dropZeroDepth = true } // Filter False Positives parameters @@ -158,10 +136,7 @@ params { runOptions = "-v" } - // Directories. - // - // Everything the run reads repeatedly is on mainDir; everything it has finished with is - // on storageDir. + // Directories. What the run reads repeatedly is on mainDir, what it has finished with on storageDir. dir { // Placed by you, and never moved or deleted by the pipeline. data = "${params.mainDir}/${params.dataSource}" @@ -177,8 +152,7 @@ params { // /. A single run sits at Output/ directly. allOutputs = params.multiRun ? "${params.storageDir}/Output/All_Runs" : "${params.storageDir}/Output" allLogs = params.multiRun ? "${params.storageDir}/Logs/All_Runs" : "${params.storageDir}/Logs" - // The installation's own, under neither storage root. `projectDir` is Nextflow's - // variable, not a parameter of ours. + // The installation's own. projectDir is Nextflow's variable, not ours. bin = "${projectDir}/bin" // Sourced by another script, never run, so it is not on PATH. lib = "${projectDir}/lib" @@ -223,30 +197,27 @@ params { } } - // Nextflow's own dag, trace, timeline and report. Last in this block, and it has to - // be: interpolation is eager and top to bottom, so a reference to `subpath` from above - // it reads null and the config fails to parse. + // Nextflow's dag, trace, timeline and report. Must stay last: interpolation runs top to + // bottom, so referencing subpath from above it reads null. sessionReports = "${params.dir.allOutputs}/${params.dir.subpath.reports}" } - // Full paths to the files you place. Below the dir block, not beside the file names - // above: interpolation is a single forward pass, and a forward reference fails to parse. + // Full paths to the files you place. Must sit below the dir block: interpolation is a + // single forward pass. referencePath = "${params.dir.references}/${params.referenceFile}" gffPath = "${params.dir.references}/${params.gffFile}" metadataPath = "${params.mainDir}/${params.metadataFile}" multiRunPath = "${params.mainDir}/${params.multiRunFile}" - // The decompressed reference, written by step 1. There is no `gff` beside it: snpEff - // builds its database from the file you placed, gzipped or not. + // The decompressed reference, written by step 1. snpEff reads the GFF you placed as it is. reference = "${params.dir.dictionaries}/${params.referenceFile.replace('.gz', '')}" - // The `**` is what lets the reads sit in subfolders of Data/ as well as directly in it - - // one folder per sample, one per sequencing run, or none at all. It matches across - // directories INCLUDING none, which `**/` does not: `**/` finds only the nested ones. + // The ** lets reads sit in subfolders of Data/ as well as directly in it. Not **/, which + // finds only the nested ones. reads = "${params.dir.data}/**${params.readPattern}" - // How each tool is invoked. Replace a command with a full path to use a system binary - // instead of the conda environment's - at your own risk, as versions may conflict. + // How each tool is invoked. A full path uses a system binary instead of the environment's, + // at your own risk. software { java = 'java' cutadapt = 'cutadapt' @@ -260,13 +231,8 @@ params { snpEff = 'snpEff' unzip = 'unzip' - // Moving a finished artifact into permanent storage. bin/atomic_mv.sh copies it - // with rsync, compares the copy against its source with diff, and only then - // removes the source; find lists what a results folder holds. - // - // These three are verified at the start of a run like the rest, but they are the - // one group a full path above does not repoint: they are called by name from the - // helper scripts in bin/, which read no Nextflow settings. + // Called by name from the helper scripts in bin/, which read no Nextflow settings, so + // a full path here does NOT repoint them. rsync = 'rsync' diff = 'diff' find = 'find' diff --git a/scripts/0_verify_environment.nf b/scripts/0_verify_environment.nf index 25bb938..41524cf 100644 --- a/scripts/0_verify_environment.nf +++ b/scripts/0_verify_environment.nf @@ -1040,6 +1040,27 @@ CURRENT_PARAMS STATUS="PASS" + # The parameter rules, from the same helper `PoolSeqFlow check project` runs, over the same + # resolved set recorded above. One implementation and two callers, so a project cannot be told + # one thing before a run and another during it. Silence means nothing to flag; a FAIL stops the + # run here rather than hours in. + PARAM_RULES=0 + check_parameters.sh < current_params.txt > param_findings.txt || PARAM_RULES=1 + if [ -s param_findings.txt ]; then + while IFS="\$(printf '\\t')" read -r level label headline detail why; do + [ -n "\$level" ] || continue + log_message "RUN PARAMETERS: \$level \$label: \$headline\${detail:+ (\$detail)}" + [ -n "\$why" ] && printf '%s' "\$why" | fold -s -w 62 | while IFS= read -r wrapped; do + log_message "RUN PARAMETERS: \$wrapped" + done + done < param_findings.txt + if [ "\$PARAM_RULES" -ne 0 ]; then + log_message "RUN PARAMETERS: A setting above would make this run produce nothing." + log_message "RUN PARAMETERS: Stopped here rather than after the compute it would waste." + STATUS="FAIL" + fi + fi + # A dry run makes every comparison below and writes none of the files that answer it. DRY_RUN="${params.dryRun}" RECORD="Recording" diff --git a/scripts/1_build_dictionaries.nf b/scripts/1_build_dictionaries.nf index 9ed7d3b..fbff946 100644 --- a/scripts/1_build_dictionaries.nf +++ b/scripts/1_build_dictionaries.nf @@ -72,6 +72,14 @@ process UngzipReference { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/1_BuildDictionary_s1_UngzipReference_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/1_BuildDictionary_s1_UngzipReference_nextflow.log) 2>&1 mkdir -p ${run.dir.dictionaries} @@ -98,12 +106,6 @@ process UngzipReference { echo "UNGZIP ${run.referenceFile}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/1_BuildDictionary_s1_UngzipReference_nextflow.log """ } @@ -120,6 +122,14 @@ process CreateBwaIndex { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/1_BuildDictionary_s2_1_CreateBwaIndex_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/1_BuildDictionary_s2_1_CreateBwaIndex_nextflow.log) 2>&1 echo "BWA INDEX ${run.referenceFile}: Start building BWA index..." # The exact five files the symlinks below point at, one test each. @@ -149,12 +159,6 @@ process CreateBwaIndex { echo "BWA INDEX ${run.referenceFile}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/1_BuildDictionary_s2_1_CreateBwaIndex_nextflow.log """ } @@ -171,6 +175,14 @@ process CreateSamtoolsFaiIndex { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/1_BuildDictionary_s2_2_CreateSamtoolsFaiIndex_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/1_BuildDictionary_s2_2_CreateSamtoolsFaiIndex_nextflow.log) 2>&1 echo "SAMTOOLS INDEX ${run.referenceFile}: Start building samtools fai index..." # The exact file the symlink below points at, not any *.fai in the directory. @@ -191,12 +203,6 @@ process CreateSamtoolsFaiIndex { echo "SAMTOOLS INDEX ${run.referenceFile}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/1_BuildDictionary_s2_2_CreateSamtoolsFaiIndex_nextflow.log """ } @@ -217,6 +223,14 @@ process BuildSnpEffDb { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/1_BuildDictionary_s2_3_BuildSnpEffDb_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/1_BuildDictionary_s2_3_BuildSnpEffDb_nextflow.log) 2>&1 buildSnpEffDb() { echo "SNPEFF DB BUILD: Building SnpEff database..." @@ -298,12 +312,6 @@ process BuildSnpEffDb { ln -s ${build_verify_path} . fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/1_BuildDictionary_s2_3_BuildSnpEffDb_nextflow.log """ } diff --git a/scripts/2_trim_reads.nf b/scripts/2_trim_reads.nf index b4afae8..b5f6c18 100644 --- a/scripts/2_trim_reads.nf +++ b/scripts/2_trim_reads.nf @@ -85,6 +85,14 @@ process TrimReads { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/2_TrimQcClip_s1_TrimReads_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/2_TrimQcClip_s1_TrimReads_${pair_id}_nextflow.log) 2>&1 # Either volume, permanent-first. An absent artifact is find_artifact.sh's ordinary answer, # so emptiness is what the branch tests. @@ -154,12 +162,6 @@ process TrimReads { echo "TRIMMING READS ${pair_id}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/2_TrimQcClip_s1_TrimReads_${pair_id}_nextflow.log """ } @@ -198,6 +200,14 @@ process ClipReads { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/2_TrimQcClip_s2_ClipReads_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/2_TrimQcClip_s2_ClipReads_${pair_id}_nextflow.log) 2>&1 # FastQC is a JVM program; give it the cores this task reserved. export _JAVA_OPTIONS="${run.java.heapSize} -XX:ParallelGCThreads=${task.cpus}" @@ -373,12 +383,6 @@ process ClipReads { fi echo "Clipping completed for ${pair_id}!" - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/2_TrimQcClip_s2_ClipReads_${pair_id}_nextflow.log """ } diff --git a/scripts/3_align.nf b/scripts/3_align.nf index 1dbaf05..e6564de 100644 --- a/scripts/3_align.nf +++ b/scripts/3_align.nf @@ -25,6 +25,14 @@ process Align { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/3_AlignReads_Align_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/3_AlignReads_Align_${pair_id}_nextflow.log) 2>&1 # Either volume, permanent-first. An absent artifact is the ordinary answer, so emptiness is # what the branch tests. @@ -50,12 +58,6 @@ process Align { echo "ALIGNING ${pair_id}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/3_AlignReads_Align_${pair_id}_nextflow.log """ } diff --git a/scripts/4_clean.nf b/scripts/4_clean.nf index fe7c577..cf87098 100644 --- a/scripts/4_clean.nf +++ b/scripts/4_clean.nf @@ -31,6 +31,14 @@ process SortCleanBam { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/4_SortCleanBam_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/4_SortCleanBam_${pair_id}_nextflow.log) 2>&1 # BOTH files, on either volume, looked up independently. The skip branch links the index too # and `ln -s` does not check its target exists, so testing only the BAM leaves a dangling @@ -109,12 +117,6 @@ process SortCleanBam { echo "SORT AND CLEAN BAM ${pair_id}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/4_SortCleanBam_${pair_id}_nextflow.log """ } diff --git a/scripts/5_reports.nf b/scripts/5_reports.nf index 5c451f5..84c6ebe 100644 --- a/scripts/5_reports.nf +++ b/scripts/5_reports.nf @@ -19,6 +19,14 @@ process AlignmentReport { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/5_GenerateReports_s1_AlignmentReport_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/5_GenerateReports_s1_AlignmentReport_${pair_id}_nextflow.log) 2>&1 if [ -f ${target_report} ]; then echo "ALIGNMENT REPORT ${ready_bam}: Found existing alignment report file" echo "ALIGNMENT REPORT ${ready_bam}: Found: ${target_report}" @@ -43,12 +51,6 @@ process AlignmentReport { echo "ALIGNMENT REPORT ${ready_bam}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/5_GenerateReports_s1_AlignmentReport_${pair_id}_nextflow.log """ } @@ -70,6 +72,14 @@ process CoverageReport { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/5_GenerateReports_s2_CoverageReport_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/5_GenerateReports_s2_CoverageReport_${pair_id}_nextflow.log) 2>&1 if [ -f ${target_report} ]; then echo "COVERAGE REPORT ${ready_bam}: Found existing coverage report file" echo "COVERAGE REPORT ${ready_bam}: Found: ${target_report}" @@ -94,12 +104,6 @@ process CoverageReport { echo "COVERAGE REPORT ${ready_bam}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/5_GenerateReports_s2_CoverageReport_${pair_id}_nextflow.log """ } @@ -132,6 +136,14 @@ process DepthProfile { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/5_GenerateReports_s3_DepthProfile_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/5_GenerateReports_s3_DepthProfile_${pair_id}_nextflow.log) 2>&1 if [ -f ${target_folder}/${decision_file} ] && [ -f ${target_folder}/${histogram_file} ] && [ -f ${target_folder}/${report_file} ]; then @@ -199,12 +211,6 @@ process DepthProfile { echo "DEPTH PROFILE ${ready_bam}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/5_GenerateReports_s3_DepthProfile_${pair_id}_nextflow.log """ } diff --git a/scripts/6_variant_call.nf b/scripts/6_variant_call.nf index f233bf9..aaea76f 100644 --- a/scripts/6_variant_call.nf +++ b/scripts/6_variant_call.nf @@ -25,6 +25,14 @@ process CapBAM { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/6_VariantCalling_s1_CapBAM_${pair_id}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/6_VariantCalling_s1_CapBAM_${pair_id}_nextflow.log) 2>&1 # Nothing to do if the VCF this feeds already exists. A header-only BAM satisfies the output # declaration; VariantCall finds the VCF and never opens it. @@ -42,12 +50,6 @@ process CapBAM { echo "CAP BAM ${pair_id}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/6_VariantCalling_s1_CapBAM_${pair_id}_nextflow.log """ } @@ -74,6 +76,14 @@ process VariantCall { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/6_VariantCall_${run.vcf.fileName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/6_VariantCall_${run.vcf.fileName}_nextflow.log) 2>&1 # Either volume: still here while step 7 or 8 may read it, promoted once both are done. vcf_at=\$(find_artifact.sh "${rel_vcf}/${vcf_file}" ${search_roots} || true) @@ -92,6 +102,22 @@ process VariantCall { ${run.software.bcftools} call ${run.variantCall.callOptions} \ -o ${vcf_file} + # bcftools writes a well-formed header and no records when the pileup reaches it empty, + # and every later step succeeds over nothing, so this is the only place a call set that + # found nothing can still be told apart from one that has not been made yet. + called=\$(grep -vc '^#' ${vcf_file} || true) + if [ "\$called" -eq 0 ]; then + echo "VARIANT CALL ${vcf_file}: ERROR: variant calling produced no records at all." + echo "VARIANT CALL ${vcf_file}: The pileup reached bcftools call with nothing in it." + echo "VARIANT CALL ${vcf_file}: variantCall.scaleMapQ is the setting that does this:" + echo "VARIANT CALL ${vcf_file}: above 10 and below variantCall.varQualMin it caps every" + echo "VARIANT CALL ${vcf_file}: read's mapping quality under the minimum that then" + echo "VARIANT CALL ${vcf_file}: rejects it, and mpileup emits nothing without failing." + echo "VARIANT CALL ${vcf_file}: The pileup ran as: ${run.variantCall.mpileupOptions}" + echo "VARIANT CALL ${vcf_file}: Nothing was published, so this costs nothing already made." + exit 1 + fi + echo "VARIANT CALL ${vcf_file}: Fixing minor header issue..." sed -i 's/##INFO=> ${dir_log}/6_VariantCall_${run.vcf.fileName}_nextflow.log """ } diff --git a/scripts/7_vcf2freq.nf b/scripts/7_vcf2freq.nf index e05d30d..6efb465 100644 --- a/scripts/7_vcf2freq.nf +++ b/scripts/7_vcf2freq.nf @@ -54,6 +54,14 @@ process SortRefAltByFrequency { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/7_s1_SortRefAltByFrequency_${vcf.baseName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/7_s1_SortRefAltByFrequency_${vcf.baseName}_nextflow.log) 2>&1 echo "SORT ALLELES BY FREQ ${vcf}: Sorting alleles by frequency..." if [ -f ${target_snp_freq_tsv} ] || [ -f ${target_indel_freq_tsv} ]; then @@ -97,12 +105,6 @@ process SortRefAltByFrequency { echo "SORT ALLELES BY FREQ ${vcf}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/7_s1_SortRefAltByFrequency_${vcf.baseName}_nextflow.log """ } @@ -144,7 +146,6 @@ process FilterPotentialFalsePositives { indel_freq_tsv = "${indel_freq_base}.tsv" target_indel_freq_tsv = "${target_folder_freq}/${indel_freq_tsv}" - sensitivity = run.filterFalsePositives.sensitivity threshold = run.filterFalsePositives.sampleThreshold @@ -156,6 +157,15 @@ process FilterPotentialFalsePositives { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/7_s2_FilterFalsePositives_${vcf.baseName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/7_s2_FilterFalsePositives_${vcf.baseName}_nextflow.log) 2>&1 + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: Filtering possible false positives..." if [ -f ${target_snp_freq_tsv} ] || [ -f ${target_indel_freq_tsv} ]; then echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: Found at least one of the existing freq files" @@ -194,6 +204,20 @@ process FilterPotentialFalsePositives { echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: Order might change after filtering, reordering alleles again..." MajorAlleleToRef.py "\$TMP_FILE" "${filterfp_vcf}" + # In the else branch only: the skip branches above write a zero-byte placeholder on + # purpose, and a size test would refuse those instead. + surviving=\$(grep -vc '^#' ${filterfp_vcf} || true) + if [ "\$surviving" -eq 0 ]; then + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: ERROR: no site survived the cross-sample filter." + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: Every called site was removed, so there is" + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: nothing left to publish a frequency for." + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: This run required an allele in ${threshold} of the" + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: samples at a frequency above ${sensitivity}." + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: filterFalsePositives.sampleThreshold and poolSize" + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: are what move those. Nothing was published." + exit 1 + fi + echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: Moving ${filterfp_vcf} to ${target_folder_vcf}..." atomic_mv.sh ${filterfp_vcf} ${target_filterfp_vcf} echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: Creating symbolic link..." @@ -206,12 +230,6 @@ process FilterPotentialFalsePositives { echo "FILTER POTENTIAL FALSE POSITIVES ${vcf}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/7_s2_FilterFalsePositives_${vcf.baseName}_nextflow.log """ } @@ -233,6 +251,13 @@ process DepthAndQualityFilter { // "*_dq.vcf" output glob cannot pick it up. filterdp_vcf = "${vcf.baseName}_dp.vcf" + // bcftools excludes the whole SITE when the expression holds for any one sample, so this + // is an intersection across samples: every published site has every sample at or above + // minDP. The zero term is redundant whenever minDP is 1 or more. + depth_exclude = run.vcffilter.dropZeroDepth + ? "FMT/DP<${run.vcffilter.minDP} || FMT/DP==0" + : "FMT/DP<${run.vcffilter.minDP}" + filterdq_base = "${vcf.baseName}_dq" filterdq_vcf = "${filterdq_base}.vcf" filterdq_recode_vcf = "${filterdq_base}.recode.vcf" @@ -258,6 +283,15 @@ process DepthAndQualityFilter { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/7_s3_DepthAndQualityFilter_${vcf.baseName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/7_s3_DepthAndQualityFilter_${vcf.baseName}_nextflow.log) 2>&1 + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Filtering VCF for depth and quality ${vcf.baseName}..." if [ -f ${target_snp_freq_tsv} ] || [ -f ${target_indel_freq_tsv} ]; then echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Found at least one of the existing freq files" @@ -278,14 +312,29 @@ process DepthAndQualityFilter { ln -s ${target_filterdq_vcf} . echo "DEPTH AND QUALITY FILTER VCF ${vcf}: COMPLETED" else - echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Depth Filtering VCF..." - ${run.software.bcftools} view -e "FMT/DP<${run.vcffilter.minDP}" -Ov -o ${filterdp_vcf} ${vcf} + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Depth Filtering VCF, excluding ${depth_exclude}" + ${run.software.bcftools} view -e "${depth_exclude}" -Ov -o ${filterdp_vcf} ${vcf} echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Quality Filtering VCF..." ${run.software.vcftools} --vcf ${filterdp_vcf} \ --minQ ${run.vcffilter.minQUAL} \ --recode --recode-INFO-all \ --out ${filterdq_base} + # In the else branch only: the skip branches above write a zero-byte placeholder on + # purpose, and a size test would refuse those instead. + surviving=\$(grep -vc '^#' ${filterdq_recode_vcf} || true) + if [ "\$surviving" -eq 0 ]; then + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: ERROR: no site survived the depth and quality filter." + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Both are applied to the whole SITE: one sample under" + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: the depth removes it for every sample, so the" + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: shallowest library sets the threshold for the cohort." + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: This run used minDP ${run.vcffilter.minDP}, minQUAL ${run.vcffilter.minQUAL}," + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: dropZeroDepth ${run.vcffilter.dropZeroDepth}. Check the weakest sample in" + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Output/Reports/Coverage before raising any of them." + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Nothing was published, so this costs nothing already made." + exit 1 + fi + echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Renaming ${filterdq_recode_vcf} as ${filterdq_vcf} and moving to ${target_folder_vcf}" atomic_mv.sh ${filterdq_recode_vcf} ${target_filterdq_vcf} echo "DEPTH AND QUALITY FILTER VCF ${vcf}: Creating symbolic link..." @@ -298,12 +347,6 @@ process DepthAndQualityFilter { echo "DEPTH AND QUALITY FILTER VCF ${vcf}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/7_s3_DepthAndQualityFilter_${vcf.baseName}_nextflow.log """ } @@ -344,6 +387,14 @@ process SplitSNPsAndINDELs { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/7_s4_SplitSNPsAndINDELs_${vcf.baseName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/7_s4_SplitSNPsAndINDELs_${vcf.baseName}_nextflow.log) 2>&1 echo "SPLIT SNPS AND INDELS ${vcf}: Splitting ${vcf.baseName} to SNP and INDEL VCFs..." # AND, where the three processes above use OR: this one has TWO outputs, and calculating a # missing table needs a real split VCF, not the dummy this branch emits. @@ -406,12 +457,6 @@ process SplitSNPsAndINDELs { rm \$(realpath ${vcf}) fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/7_s4_SplitSNPsAndINDELs_${vcf.baseName}_nextflow.log """ } @@ -437,6 +482,14 @@ process CalculateFrequencies { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/7_s5_CalculateFrequencies_${vcf.baseName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/7_s5_CalculateFrequencies_${vcf.baseName}_nextflow.log) 2>&1 echo "CALCULATE FREQUENCIES ${vcf}: Calculating Frequencies" if [ -f ${target_freq_file} ] && [ -f ${target_depth_file} ]; then @@ -463,12 +516,6 @@ process CalculateFrequencies { echo "CALCULATE FREQUENCIES ${vcf}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/7_s5_CalculateFrequencies_${vcf.baseName}_nextflow.log """ } diff --git a/scripts/8_annotate_variants.nf b/scripts/8_annotate_variants.nf index df5135e..0c7c6ac 100644 --- a/scripts/8_annotate_variants.nf +++ b/scripts/8_annotate_variants.nf @@ -26,6 +26,14 @@ process AnnotateVariants { """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/8_AnnotateVariants_${vcf.baseName}_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/8_AnnotateVariants_${vcf.baseName}_nextflow.log) 2>&1 export _JAVA_OPTIONS="${run.java.heapSize} -XX:ParallelGCThreads=${task.cpus}" @@ -74,12 +82,6 @@ process AnnotateVariants { echo "ANNOTATING VCF ${vcf}: COMPLETED" fi - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/8_AnnotateVariants_${vcf.baseName}_nextflow.log """ } diff --git a/scripts/9_completion.nf b/scripts/9_completion.nf index e1582c1..a82f5ac 100644 --- a/scripts/9_completion.nf +++ b/scripts/9_completion.nf @@ -85,16 +85,18 @@ process PromoteArtifacts { """ echo "PROMOTING ${label}: finished; nothing is promoted for this stage yet" - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${log_file} """ else """ set -eo pipefail + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${log_file} + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${log_file}) 2>&1 echo "PROMOTING ${label}: ${src}" echo "PROMOTING ${label}: -> ${dst}" @@ -149,12 +151,6 @@ process PromoteArtifacts { # Only if it is genuinely empty. rmdir "${src}" 2>/dev/null || true - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" - cat .command.log - } >> ${log_file} """ } diff --git a/scripts/citations.nf b/scripts/citations.nf index c084720..6be2ab3 100644 --- a/scripts/citations.nf +++ b/scripts/citations.nf @@ -17,6 +17,14 @@ process WriteCitations { probes = (run.software.collect { name, cmd -> "${name}=${cmd}" } + ["nextflow=nextflow", "python=python3"]).join(' ') """ + mkdir -p ${dir_log} + { + echo "" + echo "===== run=${workflow.runName} | session=${workflow.sessionId} | attempt=${task.attempt} | \$(date -Is) =====" + } >> ${dir_log}/Citations_WriteCitations_nextflow.log + # Written as it happens rather than copied at the end: a task killed by a signal leaves what + # it had reached. stdbuf keeps tee line-buffered, so the last line is not held in a buffer. + exec > >(stdbuf -oL tee -a ${dir_log}/Citations_WriteCitations_nextflow.log) 2>&1 . ${run.dir.lib}/tool_version.sh versions="" @@ -41,12 +49,6 @@ process WriteCitations { # Not piped into tee, which would make the exit status tee's. cat citations.txt - mkdir -p ${dir_log} - { - echo "" - echo "===== run=${workflow.runName} | session=${workflow.sessionId} | \$(date -Is) =====" - cat .command.log - } >> ${dir_log}/Citations_WriteCitations_nextflow.log """ } diff --git a/scripts/variants.nf b/scripts/variants.nf index 8921335..6e932b6 100644 --- a/scripts/variants.nf +++ b/scripts/variants.nf @@ -41,7 +41,7 @@ def stepParameterMap() { // threshold from them, and a blank param_poolSize cell takes the run's poolSize. The // sensitivity derived from them cannot stand in for them - pinned by hand, it lets two // runs of different pool sizes share one results directory. - 7: [ artifact: ['vcffilter.minDP', 'vcffilter.minQUAL', + 7: [ artifact: ['vcffilter.minDP', 'vcffilter.minQUAL', 'vcffilter.dropZeroDepth', 'filterFalsePositives.sensitivity', 'filterFalsePositives.sampleThreshold', 'poolSize', 'ploidy', diff --git a/test/README.md b/test/README.md index 15e3d78..c14773c 100644 --- a/test/README.md +++ b/test/README.md @@ -12,7 +12,7 @@ test/run_tests.sh --keep leave the working directories behind for inspe ``` **Run `--fast` constantly, and the full suite only for a release or the end of a major -feature.** In between, run the suite that covers what you changed — `--suite ` takes a +feature.** In between, run the suite that covers what you changed -- `--suite ` takes a substring and may be repeated, and the mapping from what you touched to which suite to run is the table in `CLAUDE.md` at the repository root. @@ -34,7 +34,7 @@ the same arrangement `dev/` uses. | `lib/harness.sh` | Assertions and result accounting | | `lib/sandbox.sh` | Throwaway project directories, and the stub conda | | `lib/analysis.sh` | The analysis layer's shared fixtures: baselines, planted results, the corpus | -| `tools/make_fixture.py` | Generates the fixture. A development tool — its output is committed | +| `tools/make_fixture.py` | Generates the fixture. A development tool -- its output is committed | | `tools/freq_corpus.py` | The analytic corpus a planted result carries, and what a module must compute from it | | `data/base/` | The committed fixture: 6 samples, 20 kb genome, 3 genes, ~80x | | `suites/00_static.sh` | Syntax, release packaging, version consistency. No data needed | @@ -45,12 +45,12 @@ the same arrangement `dev/` uses. | `suites/03_helpers.sh` | Unit coverage for `bin/`, called directly. No conda, no fixture | | `suites/06_dryrun.sh` | The layout preview, and what `dryclean` will and will not delete | -The analysis layer is nine suites rather than one. It runs **no** pipeline — artifacts are planted — but most of it still starts a Nextflow run per case, which is what makes running only the seam you touched worth doing. +The analysis layer is nine suites rather than one. It runs **no** pipeline -- artifacts are planted -- but most of it still starts a Nextflow run per case, which is what makes running only the seam you touched worth doing. | Path | What it is | |---|---| | `suites/08_analysis_frame.sh` | What the frame is, what it reads, and what keeps it optional | -| `suites/07_analysis_rlib.sh` | The shared R library, called directly. **No Nextflow at all** — seconds, not minutes | +| `suites/07_analysis_rlib.sh` | The shared R library, called directly. **No Nextflow at all** -- seconds, not minutes | | `suites/09_analysis_modules.sh` | The module store, and what a module gets from the frame: its library, settings and artifact classes | | `suites/10_analysis_verify.sh` | What the frame checks before a module reads anything | | `suites/11_analysis_plan.sh` | Which results an invocation covers, and where each lands | @@ -59,7 +59,7 @@ The analysis layer is nine suites rather than one. It runs **no** pipeline — a | `suites/14_analysis_series.sh` | Series, units and conditions | | `suites/15_analysis_results.sh` | What a module writes, what it carries, and completion | -**A module ships its own cases**, discovered from `analysis/modules/*/test/*.sh` and run under a suite named after the module. A module is a pipeline published on its own timetable, so what judges it travels with it — which is also what lets a module somebody else wrote be tested the way ours are. The harness, the fixtures and the assertions stay shared; only the cases are the module's. They are `export-ignore`d from a release for the same reason `test/` is. +**A module ships its own cases**, discovered from `analysis/modules/*/test/*.sh` and run under a suite named after the module. A module is a pipeline published on its own timetable, so what judges it travels with it -- which is also what lets a module somebody else wrote be tested the way ours are. The harness, the fixtures and the assertions stay shared; only the cases are the module's. They are `export-ignore`d from a release for the same reason `test/` is. **Every suite declares its cost** in a `# cost:` line, one of three words, and `--cost` selects on it: @@ -69,12 +69,12 @@ The analysis layer is nine suites rather than one. It runs **no** pipeline — a | `jvm` | starts Nextflow per case, against planted artifacts | `05_guards` `06_dryrun`, the other analysis suites, and a module's own | | `pipeline` | runs the pipeline itself against the fixture | `04_pipeline` | -The line between `static` and `jvm` is whether a case **builds** something. Asking `have_tools` and skipping is static — that is how `00_static` holds its `nextflow lint` case. Calling for a baseline or a pipeline run is not, and `00_static` refuses a suite that does both. +The line between `static` and `jvm` is whether a case **builds** something. Asking `have_tools` and skipping is static -- that is how `00_static` holds its `nextflow lint` case. Calling for a baseline or a pipeline run is not, and `00_static` refuses a suite that does both. An undeclared suite reads as `pipeline`, so it can never slip into a cheap run by accident, and a static one that reaches for a builder is refused by name. **`--changed` works it out for you.** `dev/scripts/select-tests.py` reads what each suite -declares it *runs* — a `# covers:` line in its header — and expands that through the include +declares it *runs* -- a `# covers:` line in its header -- and expands that through the include graph into what the suite therefore depends on. A change to a file in that set selects the suite. Run the script directly to see the reasoning before committing to it: @@ -83,8 +83,8 @@ $ dev/scripts/select-tests.py analysis/lib/nf/plan.nf $ dev/scripts/select-tests.py bin/depth_cutoff.py # 04_pipeline 05_guards 03_helpers ``` -**Two halves, and only one of them is maintained by hand.** The graph is derived — `include {} -from`, a shell `source`, a bare call to a helper in `bin/` — so a new import needs no +**Two halves, and only one of them is maintained by hand.** The graph is derived -- `include {} +from`, a shell `source`, a bare call to a helper in `bin/` -- so a new import needs no bookkeeping. What a suite *runs* has to be declared, because no parser can see it, and `00_static` refuses a source file that no suite reaches as well as a claimed path that does not exist. @@ -93,12 +93,12 @@ exist. direction that answers the question: reading upwards alone misses a suite that goes through the same entry script, and reading both ways connects the whole repository to itself through `bin/atomic_mv.sh`, which twelve `.nf` files name. A change to the harness, or to the selector -itself, selects everything — a selection that is too big costs minutes, one that is too small +itself, selects everything -- a selection that is too big costs minutes, one that is too small costs a bug nobody was looking for. ## What to run, and when -Counts move with every stage, so read them off the run rather than from here — +Counts move with every stage, so read them off the run rather than from here -- `grep -hc '^test_' test/suites/*.sh analysis/modules/*/test/*.sh` is the check, and it has to name the module suites too or it undercounts. At the time of writing: @@ -106,7 +106,7 @@ name the module suites too or it undercounts. At the time of writing: |---|---|---|---| | `--cost static` | 236 | ~40 s | Needs nothing installed. The first thing to reach for | | `--fast` | 274 of 493 | ~2 min | The above, plus every case elsewhere that builds nothing | -| `--suite ` | varies | 3 s – 5 min | What you changed. `analysis_rlib` is 9 cases in three seconds | +| `--suite ` | varies | 3 s - 5 min | What you changed. `analysis_rlib` is 9 cases in three seconds | | everything | 493 | ~50 min | A release, or the end of a major feature | The two slow suites are slow for one reason: a Nextflow run costs about **21 seconds of @@ -115,12 +115,12 @@ suite runtime is essentially a count of `nextflow run` invocations. Two conseque knowing before adding a case: - Prefer a **unit test in `03_helpers.sh`** over an end-to-end one. `bin/classify_manifest.sh` - exists as a separate script for exactly this reason — its edge cases (a value containing + exists as a separate script for exactly this reason -- its edge cases (a value containing `=`, an empty value, no trailing newline, an unparseable line) are milliseconds there and a JVM start each through a pipeline run. When new guard logic is worth testing thoroughly, extract it to `bin/` first. - Never give a case its own setup run. `05_guards.sh` builds one verified project and each - case works on a copy — 22ms against 21s. Doing it per case was most of that suite's + case works on a copy -- 22ms against 21s. Doing it per case was most of that suite's runtime and tested nothing. ## Two rules the suite is built around @@ -158,7 +158,7 @@ Two helpers in `lib/sandbox.sh` are worth knowing about before writing a new cas `task_count ` reads the Nextflow trace and returns how many tasks that process ran. Use it whenever a change touches channel wiring. Every singleton -artifact here — the verify token, the reference, both indexes, the snpEff marker — rides a +artifact here -- the verify token, the reference, both indexes, the snpEff marker -- rides a value channel, which is what lets one index broadcast against N samples. An operator inserted into such a path turns it into a queue channel, and the run then still reports SUCCESS while doing the work once instead of N times. `test_each_step_runs_once_per_sample` @@ -166,8 +166,8 @@ is the standing guard; extend it rather than trusting an exit status. `run_dictionaries_only ` and `run_verify_only ` run step 1 and step 0 by themselves, for questions that would otherwise cost a full end-to-end run. `-entry` does not -work under the strict parser, so each generates a small include file into the sandbox — -generated rather than committed, because the include path has to be `./scripts/…` to resolve +work under the strict parser, so each generates a small include file into the sandbox -- +generated rather than committed, because the include path has to be `./scripts/...` to resolve where it runs, and a committed copy carrying that path fails `nextflow lint .`. ## The fixture, and what it can and cannot tell you @@ -181,7 +181,7 @@ python3 test/tools/make_fixture.py test/data/base ``` `planted.tsv` records **what went in**. It is not an answer key, and asserting output -values against it directly does not work — trimming, the FastQC-driven clip, the +values against it directly does not work -- trimming, the FastQC-driven clip, the proper-pair filter and the caller all sit in between. An earlier version of the fixture tried to be exactly predictable by giving every fragment the same length; that left bwa with a zero-variance insert size distribution, so every indel-bearing pair was flagged @@ -197,18 +197,31 @@ Two details worth knowing when writing assertions against the frequency tables: - `MajorAlleleToRef.py` re-polarizes every site to the cohort major allele, so a planted ALT routinely becomes the REF. Match rows on the allele **base**, never on "the row where REF differs from ALLELE". -- Sites where no sample varies are absent by design — fixed-for-the-same-allele everywhere +- Sites where no sample varies are absent by design -- fixed-for-the-same-allele everywhere is dropped by the false-positive filter, and never-varying sites are never called. Both are symmetric and intended; see the fixed-site matrix in the project notes. ## Known gaps -- **`depth2freq.awk` and `MajorAlleleToRef.py` still have no unit coverage** and are exercised - end to end only. Most of this gap has closed since it was written — `03_helpers.sh` now covers - `classify_manifest.sh`, `find_artifact.sh`, `depth_cutoff.py`, `filterFalsePositives.sh`, both - parsers and `atomic_mv.sh`, and `config_migrate.sh` has a suite of its own — but those two are - the ones left. -- No fault injection: the failure paths hardened during the audit — a mid-pipe tool death, - an interrupted decompress, a failed database copy — have no cases. +- **Every `bin/` helper now has unit coverage.** `03_helpers.sh` covers `classify_manifest.sh`, + `find_artifact.sh`, `depth_cutoff.py`, `filterFalsePositives.sh`, both parsers, `atomic_mv.sh`, + `cap_depth.awk` and `MajorAlleleToRef.py`, and `config_migrate.sh` has a suite of its own. + `depth2freq.awk` has cases + there too, but only for what a zero-depth cell publishes and for the row it must not carry + forward: its shape -- one row per allele, `ALT` becoming `ALLELE`, the column count held -- + is still asserted only by `04_pipeline`, through a whole run. +- **The fixture cannot hold a zero-depth cell**, so `vcffilter.dropZeroDepth` has no behavior case + and the two settings are compared by the expression each builds instead. Every sample carries + ~75x across the whole reference and the called VCF has no `DP=0` anywhere. Thinning one sample + was measured and does not work: at 250 pairs bwa logs `skip orientation FR as there are not + enough pairs`, estimates no insert size, nothing is flagged properly paired, and step 4's `0x2` + filter discards every read -- the trap `test/tools/make_fixture.py`'s docstring warns about. The + requirements conflict, because bwa needs thousands of pairs and a zero-depth cell needs almost + none locally. **A sample with a regional coverage gap satisfies both**: an option in + `make_fixture.py` restricting one sample's fragments to part of the genome, and a committed + fixture of its own beside `base/`, about 2 MB. Not built; the toggle is redundant above + `minDP 1`, so nothing a default run publishes depends on it. +- No fault injection: the failure paths hardened during the audit -- a mid-pipe tool death, + an interrupted decompress, a failed database copy -- have no cases. - The `-m` parsing in `ClipReads` is only covered end to end. Finer cases would want that logic moved into a `bin/` helper where it can be called directly. diff --git a/test/data/vcf/README.md b/test/data/vcf/README.md index c0ecea9..e1b68be 100644 --- a/test/data/vcf/README.md +++ b/test/data/vcf/README.md @@ -2,12 +2,12 @@ `called.vcf` is genuine `bcftools call` output, not a hand-written file, and that distinction has already cost this project one false bug report. `bcftools call -m` emits an overlapping SNP -and indel as **two records at the same position**, never as one mixed multiallelic record — a +and indel as **two records at the same position**, never as one mixed multiallelic record -- a structure that is easy to get wrong by hand and that makes `norm -m -` / `norm -m+` look broken when it is not. Anything testing `bin/filterFalsePositives.sh`, `bin/createDepthFile.sh`, `bin/depth2freq.awk` or `bin/MajorAlleleToRef.py` should start from this file. -**Where it came from.** A full pipeline run over `test/data/base/` — the same six samples, the +**Where it came from.** A full pipeline run over `test/data/base/` -- the same six samples, the same 20 kb synthetic reference, aligned with bwa and called with the pipeline's own `variantCall.mpileupOptions` / `callOptions`. It is the `Test.vcf` that run produced, with the absolute paths in `##bcftoolsCommand` and `##reference` rewritten to relative ones so the @@ -17,17 +17,17 @@ committed file says nothing about the machine that made it. Nothing else was tou | | | |---|---| -| 135 records, 6 samples | sample columns are named `TestSample1`…`TestSample6` | +| 135 records, 6 samples | sample columns are named `TestSample1`...`TestSample6` | | 16 indels, 119 SNPs | both branches of step 7's SNP/indel split have input | | no multiallelic records | `ALT` never holds a comma, so `norm -m -` is a no-op on count | | no sample has `DP=0` | a division-by-zero case has to be constructed, not found here | | lowest non-zero ALT fraction is **0.012048** | this is the number that decides whether a sensitivity bites | That last row is the one to keep in mind. Sensitivity is `1 / (2 * ploidy * poolSize)`, so -the default `poolSize = 100` gives `0.0025` — below every non-zero fraction in the file, which +the default `poolSize = 100` gives `0.0025` -- below every non-zero fraction in the file, which is why the default filter drops records on the sample-count clause rather than on sensitivity. A test that wants a per-pool threshold to actually change the outcome needs a pool small enough -that `s` clears 0.012 — `poolSize = 10` gives `0.025`, `poolSize = 1` gives `0.25`. +that `s` clears 0.012 -- `poolSize = 10` gives `0.025`, `poolSize = 1` gives `0.25`. **Do not hand-edit it.** Regenerate it from a real run if it ever needs to change, and say so here. A record edited by hand to make a test pass stops being evidence of anything. diff --git a/test/lib/harness.sh b/test/lib/harness.sh index 2c1b754..0d2adc4 100644 --- a/test/lib/harness.sh +++ b/test/lib/harness.sh @@ -14,6 +14,7 @@ CASE_FAILED=0 CASE_SKIPPED=0 CASE_MESSAGES=() FAILED_CASES=() +SKIPPED_CASES=() if [ -t 1 ]; then C_PASS=$'\033[32m'; C_FAIL=$'\033[31m'; C_SKIP=$'\033[33m' @@ -92,9 +93,14 @@ run_case() { "$fn" if [ "$CASE_SKIPPED" -eq 1 ]; then + local why="" TESTS_SKIPPED=$((TESTS_SKIPPED + 1)) + if [ ${#CASE_MESSAGES[@]} -gt 0 ]; then + why=" (${CASE_MESSAGES[0]})" + fi + SKIPPED_CASES+=("$CURRENT_SUITE / $label$why") printf ' %sSKIP%s %s' "$C_SKIP" "$C_OFF" "$label" - [ ${#CASE_MESSAGES[@]} -gt 0 ] && printf ' %s(%s)%s' "$C_DIM" "${CASE_MESSAGES[0]}" "$C_OFF" + [ -n "$why" ] && printf ' %s%s%s' "$C_DIM" "${why# }" "$C_OFF" printf '\n' return 0 fi @@ -123,11 +129,19 @@ print_summary() { fi [ "$TESTS_SKIPPED" -gt 0 ] && printf ', %d skipped' "$TESTS_SKIPPED" printf '\n' + local c if [ "$TESTS_FAILED" -gt 0 ]; then printf '\nFailed:\n' - local c for c in "${FAILED_CASES[@]}"; do printf ' %s\n' "$c" done fi + # Listed for the same reason failures are: a skip is a case that did not run, and the count + # alone does not say which. + if [ "$TESTS_SKIPPED" -gt 0 ]; then + printf '\nSkipped:\n' + for c in "${SKIPPED_CASES[@]}"; do + printf ' %s\n' "$c" + done + fi } diff --git a/test/run_tests.sh b/test/run_tests.sh index 355ce56..40ca3d1 100755 --- a/test/run_tests.sh +++ b/test/run_tests.sh @@ -22,6 +22,18 @@ # Exit status is 0 only when every case that ran passed. Skips do not fail the run: a # machine without the conda environment can still check everything that does not need it. # +# A FULL RUN - no filters and no --fast - looks for no installed environment at all. It solves +# install/environment.yml and install/environment-analysis.yml into a scratch pair, runs against +# those, and removes them on the way out: those files are what the release ships, and the suite +# writes to an environment, so a release gate must not touch the one the current release depends +# on. That costs minutes, and it is the run that decides whether to publish. +# +# A filtered run resolves this tree's exact version and skips what is not installed, so --fast +# and --suite stay cheap. It never borrows another release's environment either. +# +# TEST_CONDA_ENV and TEST_ANALYSIS_ENV name an environment to use in place of either, which is +# how dev/scripts/prep-version.sh passes in the pair it has already built. +# # Deliberately not `set -e`. A failing assertion has to be recorded and reported, not abort # the whole run - a suite that stops at its first problem hides the rest of them. set -uo pipefail @@ -96,40 +108,39 @@ conda_env_dirs() { # The installed environment for this tree's version. is "" for the pipeline environment # and "-analysis" for the other; is the binary that proves it is usable. # -# THIS TREE'S VERSION FIRST, and another only with a warning on stderr. A bare PoolSeqFlow-* -# glob takes whatever sorts first, which is the oldest environment on the machine, so a tree at -# 3.1.2 with 3.1.1 still installed measured 3.1.1 and said nothing. The same glob had already -# been caught doing the same thing in prep-version.sh. +# THIS TREE'S VERSION, BY EXACT NAME, AND NOTHING ELSE. # -# The analysis environment is excluded by name rather than by probe, because it carries nextflow -# too - measured, all four installed environments do - so `PoolSeqFlow-*` probed for bin/nextflow -# returns PoolSeqFlow--analysis as the pipeline environment. +# It used to fall back to any PoolSeqFlow-* environment that had the probe, warning on +# stderr. That is the wrong shape: a tree at 3.3.0 with 3.1.1 still installed measured 3.1.1's +# packages, and a number computed against versions no user receives is worse than no number, +# because the run is green and describes other software. Z, 2026-10-04: *"Again this is the wrong +# shape. It should create a new scratch environment to test."* So an absent environment is absent, +# and a full run builds a scratch one below rather than borrowing a release's. +# +# The old glob needed to exclude *-analysis by name, because that environment carries nextflow +# too - measured, all four installed environments do - so a probe for bin/nextflow returned the +# analysis environment as the pipeline one. An exact name cannot make that mistake. find_release_env() { - local suffix="$1" probe="$2" dir candidate other="" + local suffix="$1" probe="$2" dir candidate while IFS= read -r dir; do candidate="$dir/PoolSeqFlow-${TREE_VERSION}${suffix}" if [ -x "$candidate/bin/$probe" ]; then printf '%s' "$candidate" return 0 fi - for candidate in "$dir"/PoolSeqFlow-*"$suffix"; do - [ -z "$suffix" ] && case "$candidate" in *-analysis) continue ;; esac - [ -x "$candidate/bin/$probe" ] && [ -z "$other" ] && other="$candidate" - done done < <(conda_env_dirs) - if [ -n "$other" ]; then - printf 'WARNING: PoolSeqFlow-%s%s is not installed; testing %s instead\n' \ - "$TREE_VERSION" "$suffix" "$(basename "$other")" >&2 - printf '%s' "$other" - fi return 0 } # The conda environment supplying nextflow, bwa, samtools and the rest. Point # TEST_CONDA_ENV at another one to test against it; suites that need tools skip without it. -if [ -z "${TEST_CONDA_ENV:-}" ]; then - TEST_CONDA_ENV=$(find_release_env "" nextflow) -fi +# +# NEITHER ENVIRONMENT IS RESOLVED HERE. Which one a run should use depends on whether it is a +# full run, and that is not known until the suites have been selected - so both are settled in +# one place below, after the scope is decided. Only whether the caller named one is recorded now, +# because the assignment that follows would otherwise erase the answer. +TEST_CONDA_ENV_GIVEN=0 +[ -n "${TEST_CONDA_ENV:-}" ] && TEST_CONDA_ENV_GIVEN=1 TEST_CONDA_ENV="${TEST_CONDA_ENV:-}" export TEST_CONDA_ENV @@ -158,34 +169,11 @@ export -f have_analysis_r # below describes - the fix landed on the activation and never reached the search that decides # whether there is anything to activate. So the hook was resolved correctly from a variable that # was always empty. -if [ -z "${TEST_ANALYSIS_ENV:-}" ]; then - TEST_ANALYSIS_ENV=$(find_release_env -analysis Rscript) -fi +TEST_ANALYSIS_ENV_GIVEN=0 +[ -n "${TEST_ANALYSIS_ENV:-}" ] && TEST_ANALYSIS_ENV_GIVEN=1 TEST_ANALYSIS_ENV="${TEST_ANALYSIS_ENV:-}" export TEST_ANALYSIS_ENV -# THE conda SHELL FUNCTION, DEFINED BUT NOT USED YET. Sourcing the hook is what makes -# `conda activate` exist in a script at all; the activation itself happens further down, around -# the block of suites that declare `# env: analysis`, so nothing else in the run has an -# environment on its PATH. -# -# It used to activate here, for the whole process. That put the analysis environment behind -# every one of the 500-odd cases that do not want it, and made the pipeline suites unable to -# notice a missing tool: both environments carry nextflow, samtools, bcftools and rsync, so -# anything dropped from the pipeline environment was quietly answered by the analysis one. -# -# THE HOOK IS FOUND FROM THE ENVIRONMENT'S OWN PATH, NOT FROM `conda info --base`. That command -# prints a plugin's load error onto stdout alongside the answer - anaconda-anon-usage does it on -# every invocation - so the variable holds an error message with a path stuck on the end, the -# `-f` test fails against nonsense, and the activation is skipped in silence. An environment -# lives at /envs/, so the base is two directories up and needs nothing to say so. -if [ -n "$TEST_ANALYSIS_ENV" ]; then - _conda_hook="$(dirname "$(dirname "$TEST_ANALYSIS_ENV")")/etc/profile.d/conda.sh" - # shellcheck disable=SC1091 - [ -f "$_conda_hook" ] && . "$_conda_hook" - unset _conda_hook -fi - # The analysis environment's Rscript, empty when there is none. A module's own R may use the # packages that environment pins, and some of its paths need one the system R does not carry; # the shared library stays base R and is tested against whatever Rscript is on PATH. @@ -326,13 +314,27 @@ for candidate in "${POOLSEQFLOW_TEST_XDEV:-}" /dev/shm /var/tmp; do done export TEST_XDEV_TMPDIR +# Conda environments this run built, and only those. Named before the trap is set, because a +# conda environment outlives the process and a build interrupted part way still leaves one. +SCRATCH_ENVS="" + cleanup() { + local env if [ "$KEEP" -eq 1 ]; then printf '\nworking directory kept at %s\n' "$TEST_TMPDIR" [ -n "$TEST_XDEV_TMPDIR" ] && printf 'second filesystem kept at %s\n' "$TEST_XDEV_TMPDIR" + for env in ${SCRATCH_ENVS:-}; do + printf 'scratch environment kept: %s\n' "$env" + printf ' remove it with: conda env remove -n %s -y\n' "$env" + done else rm -rf "$TEST_TMPDIR" [ -n "$TEST_XDEV_TMPDIR" ] && rm -rf "$TEST_XDEV_TMPDIR" + for env in ${SCRATCH_ENVS:-}; do + printf '\nremoving the scratch environment %s\n' "$env" + conda env remove --name "$env" --yes > /dev/null 2>&1 \ + || printf 'WARNING: could not remove %s\n' "$env" >&2 + done fi } trap cleanup EXIT @@ -384,20 +386,6 @@ fi printf '%sPoolSeqFlow test suite%s\n' "$C_HEAD" "$C_OFF" -# BOTH ENVIRONMENTS, NAMED. Only the pipeline one was printed, so an analysis environment that -# was never found looked exactly like a run that did not need one - and the skips it caused read -# as a machine without R rather than as a suite that had failed to look in the right place. -if have_tools; then - printf '%stools: %s%s\n' "$C_DIM" "$TEST_CONDA_ENV" "$C_OFF" -else - printf '%stools: none found - suites needing the pipeline will skip%s\n' "$C_DIM" "$C_OFF" -fi -if have_analysis_r; then - printf '%sanalysis: %s%s\n' "$C_DIM" "$TEST_ANALYSIS_ENV" "$C_OFF" -else - printf '%sanalysis: none found - suites needing R will skip%s\n' "$C_DIM" "$C_OFF" -fi - # WHAT THIS RUN COVERS, before it covers it. A filtered run and a full one are told apart by # this line and by nothing else in the output, and a green result over a subset nobody chose # reads exactly like a green result over everything. @@ -407,6 +395,14 @@ for suite in "${SUITES[@]}"; do matches_any "$(suite_cost "$suite")" "${COST_FILTERS[@]+"${COST_FILTERS[@]}"}" || continue SELECTED+=("$(basename "$suite" .sh)") done +# A FULL RUN IS EVERY SUITE WITH EVERY CASE: no filters, and not --fast, which skips the cases +# that run the pipeline. It is the only run that builds an environment it cannot find, so what +# counts as one is decided here, beside the line that reports it. +FULL_RUN=0 +if [ "${#SUITE_FILTERS[@]}" -eq 0 ] && [ "${#CASE_FILTERS[@]}" -eq 0 ] \ + && [ "${#COST_FILTERS[@]}" -eq 0 ] && [ "$FAST" -eq 0 ]; then + FULL_RUN=1 +fi if [ "${#SUITE_FILTERS[@]}" -eq 0 ] && [ "${#CASE_FILTERS[@]}" -eq 0 ] \ && [ "${#COST_FILTERS[@]}" -eq 0 ]; then printf '%sscope: every suite (%d)%s\n' "$C_DIM" "${#SELECTED[@]}" "$C_OFF" @@ -422,6 +418,114 @@ if [ "${#SELECTED[@]}" -eq 0 ]; then exit 2 fi +# A FULL RUN BUILDS BOTH FROM THE SHIPPED FILES AND LOOKS FOR NOTHING INSTALLED. +# +# Two reasons it must not reach for PoolSeqFlow-, even when one is sitting there: +# +# THE SUITE WRITES TO AN ENVIRONMENT. 02_launcher installs and uninstalls, and installing a +# module puts that module's own packages into the analysis environment. Running the release +# gate against the environment the current release depends on can change it, mid-cycle, under +# the maintainer. Z, 2026-10-04: *"We're trying to avoid changing the existing environment +# during the release cycle."* +# +# AND IT IS THE WRONG TOOL SET. install/environment.yml is what this release will ship, and in +# a cycle it has just been re-exported with packages moved; the installed environment belongs +# to the release being replaced. Green against that describes software nobody receives. +# +# TEST_CONDA_ENV and TEST_ANALYSIS_ENV still win, which is how dev/scripts/prep-version.sh hands +# over the pair it has already built instead of paying for a second one. +# +# A FILTERED RUN resolves this tree's own version by exact name and skips what is not there. +# --fast, --cost static and --suite are what a change is checked with, several times an +# hour, and a solve inside one of those would stop them being used. +# +# THE SCRATCH NAME BELONGS TO NO RELEASE. Building PoolSeqFlow- instead would leave +# an environment solved from this tree's files standing under a version those files have not +# shipped as - the error dev/scripts/check-exported-floor.sh exists to avoid. +build_scratch_env() { + local label="$1" file="$2" suffix="$3" probe="$4" scratch prefix + scratch="PoolSeqFlow-suite-$$$suffix" + if ! command -v conda > /dev/null 2>&1; then + printf '%s%s: no conda, so no scratch environment can be built%s\n' \ + "$C_DIM" "$label" "$C_OFF" >&2 + return 1 + fi + if [ ! -f "$REPO_ROOT/$file" ]; then + printf '%s%s: %s is missing, so there is nothing to build from%s\n' \ + "$C_DIM" "$label" "$file" "$C_OFF" >&2 + return 1 + fi + printf '%s%s: solving %s into %s. This takes minutes.%s\n' \ + "$C_DIM" "$label" "$file" "$scratch" "$C_OFF" >&2 + # Recorded before the status is judged: a solve that fails part way still leaves one behind. + SCRATCH_ENVS="$SCRATCH_ENVS $scratch" + if ! conda env create --name "$scratch" --file "$REPO_ROOT/$file" --yes \ + > "$TEST_TMPDIR/solve-$label.log" 2>&1; then + printf 'ERROR: %s did not solve:\n' "$file" >&2 + tail -n 15 "$TEST_TMPDIR/solve-$label.log" >&2 + return 1 + fi + prefix=$(conda env list | awk -v n="$scratch" '$1 == n {print $NF}') + if [ -z "$prefix" ] || [ ! -x "$prefix/bin/$probe" ]; then + printf 'ERROR: %s was created but has no bin/%s\n' "$scratch" "$probe" >&2 + return 1 + fi + printf '%s' "$prefix" +} + +if [ "$FULL_RUN" -eq 1 ]; then + [ "$TEST_CONDA_ENV_GIVEN" -eq 1 ] \ + || TEST_CONDA_ENV=$(build_scratch_env tools install/environment.yml "" nextflow) || true + [ "$TEST_ANALYSIS_ENV_GIVEN" -eq 1 ] \ + || TEST_ANALYSIS_ENV=$(build_scratch_env analysis install/environment-analysis.yml \ + -analysis Rscript) || true +else + [ "$TEST_CONDA_ENV_GIVEN" -eq 1 ] \ + || TEST_CONDA_ENV=$(find_release_env "" nextflow) + [ "$TEST_ANALYSIS_ENV_GIVEN" -eq 1 ] \ + || TEST_ANALYSIS_ENV=$(find_release_env -analysis Rscript) +fi +export TEST_CONDA_ENV TEST_ANALYSIS_ENV + +# THE conda SHELL FUNCTION, DEFINED BUT NOT USED YET. Sourcing the hook is what makes +# `conda activate` exist in a script at all; the activation itself happens further down, around +# the block of suites that declare `# env: analysis`, so nothing else in the run has an +# environment on its PATH. +# +# It used to activate here, for the whole process. That put the analysis environment behind +# every one of the 500-odd cases that do not want it, and made the pipeline suites unable to +# notice a missing tool: both environments carry nextflow, samtools, bcftools and rsync, so +# anything dropped from the pipeline environment was quietly answered by the analysis one. +# +# THE HOOK IS FOUND FROM THE ENVIRONMENT'S OWN PATH, NOT FROM `conda info --base`. That command +# prints a plugin's load error onto stdout alongside the answer - anaconda-anon-usage does it on +# every invocation - so the variable holds an error message with a path stuck on the end, the +# `-f` test fails against nonsense, and the activation is skipped in silence. An environment +# lives at /envs/, so the base is two directories up and needs nothing to say so. +if [ -n "$TEST_ANALYSIS_ENV" ]; then + _conda_hook="$(dirname "$(dirname "$TEST_ANALYSIS_ENV")")/etc/profile.d/conda.sh" + # shellcheck disable=SC1091 + [ -f "$_conda_hook" ] && . "$_conda_hook" + unset _conda_hook +fi + +# BOTH ENVIRONMENTS, NAMED, and after any build so this says what was really used. Only the +# pipeline one was printed once, so an analysis environment that was never found looked exactly +# like a run that did not need one - and the skips it caused read as a machine without R rather +# than as a suite that had failed to look in the right place. +if have_tools; then + printf '%stools: %s%s\n' "$C_DIM" "$TEST_CONDA_ENV" "$C_OFF" +else + printf '%stools: none for %s - suites needing the pipeline will skip%s\n' \ + "$C_DIM" "$TREE_VERSION" "$C_OFF" +fi +if have_analysis_r; then + printf '%sanalysis: %s%s\n' "$C_DIM" "$TEST_ANALYSIS_ENV" "$C_OFF" +else + printf '%sanalysis: none for %s - suites needing R will skip%s\n' \ + "$C_DIM" "$TREE_VERSION" "$C_OFF" +fi + SUITES_RUN=0 CURRENT_SUITE="" ACTIVE_ENV="none" diff --git a/test/suites/00_static.sh b/test/suites/00_static.sh index 91cf9bd..eb336b0 100644 --- a/test/suites/00_static.sh +++ b/test/suites/00_static.sh @@ -1,9 +1,10 @@ #!/bin/bash # Checks that need no data: syntax, release packaging, version consistency. # cost: static -# covers: PoolSeqFlow install/ dev/scripts/ modules/repo/index.tsv .gitattributes +# covers: PoolSeqFlow install/ lib/ dev/scripts/ modules/repo/index.tsv .gitattributes # covers: analysis/citations.json citations/citations.json citations/references.bib # covers: analysis/references.bib manual/references.bib +# covers: parameters.config.template metadata.csv.template multi-run.csv.example # `nextflow lint` was brought to zero warnings during the post-2.2.0 audit. Held there # deliberately: once the count is zero a new warning is a signal rather than noise. @@ -682,6 +683,52 @@ test_check_install_hint_uses_the_versioned_environment() { # A malformed version has to be refused before anything is cloned, updated or exported. # This runs no conda commands - the check is ahead of them. +# prep-version.sh sources lib/wrapper_lib.sh and calls into it at its step 2, which sits on the +# far side of an hour of solving and updating. It names what it needs in WRAPPER_LIB_NEEDS and +# refuses at once when one of them is absent, so a function that moves costs a second instead of +# an hour. That guard is only worth having while the list matches the calls under it, which is +# what this checks - in both directions, because a list that has drifted is a guard covering +# nothing while reading as one that covers everything. +# +# The malformed-version case below cannot catch this: it exits at the version regex, which is +# twenty lines before the source. +test_prep_version_declares_what_it_takes_from_wrapper_lib() { + local out + out=$(cd "$REPO_ROOT" && python3 - <<'PY' +import pathlib, re + +DEF = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)\(\) \{", re.M) +script = pathlib.Path("dev/scripts/prep-version.sh").read_text(encoding="utf-8") +lib = pathlib.Path("lib/wrapper_lib.sh").read_text(encoding="utf-8") + +defined = set(DEF.findall(lib)) +own = set(DEF.findall(script)) + +found = re.search(r'^WRAPPER_LIB_NEEDS="([^"]*)"', script, re.M) +if not found: + print("prep-version.sh declares no WRAPPER_LIB_NEEDS, so nothing holds the list honest") + raise SystemExit +declared = found.group(1).split() + +for name in declared: + if name not in defined: + print("declares %s(), which lib/wrapper_lib.sh does not define" % name) + +# Comments go, and the declaration itself with them: the names in it are a list, not calls. Only +# names that ARE functions in wrapper_lib are looked for, so a bare word cannot cry wolf. +body = re.sub(r"^\s*#.*$", "", script, flags=re.M) +body = re.sub(r"^WRAPPER_LIB_NEEDS=.*$", "", body, flags=re.M) +for name in sorted(defined): + if name in own or name in declared: + continue + if re.search(r"(?/dev/null | head -1) [ -n "$file" ] || { fail_case "no source file for step $step"; continue; } - # The step's entry, taken by matching brackets rather than by line shape: three of the - # seven entries fit on one line, and a line-based reader silently ran two of them - # together - which made the check pass because one step's declarations covered the - # next one's reads. - declared=$(STEP="$step" python3 -c ' -import os, re, sys -text = open(sys.argv[1]).read() -step = os.environ["STEP"] -start = re.search(r"^ %s: \[" % step, text, re.M) -if not start: - sys.exit("no map entry for step " + step) -i, depth = start.end() - 1, 0 -while i < len(text): - if text[i] == "[": depth += 1 - elif text[i] == "]": - depth -= 1 - if depth == 0: break - i += 1 -body = re.sub(r"//[^\n]*", "", text[start.end() - 1:i]) -print("\n".join(sorted(set(re.findall(r"'"'"'([^'"'"']*)'"'"'", body))))) -' "$variants") + # stderr folded in, so a refusal from the helper becomes the reason this case gives rather + # than a line on the terminal and an empty list here. The old inline version did not read + # the status at all: a missing entry left `declared` empty and every one of the step's + # reads was reported undeclared, which named ten problems for one cause. + declared=$(step_declared_parameters "$variants" "$step" 2>&1) \ + || { fail_case "$declared"; continue; } reads=$(step_parameter_reads "$file" \ | grep -Ev "$excluded" | grep -Ev "$indirect" | grep -Ev "$through") @@ -1469,8 +1550,19 @@ test_the_completion_offers_every_verb_the_wrapper_takes() { printf "%s\n" "${COMPREPLY[@]}"' _ "$REPO_ROOT" | sort -u) [ -n "$dispatched" ] || { fail_case "no verbs were extracted from the wrapper's case"; return; } - [ -n "$offered" ] || { fail_case "the completion offered nothing at all"; return; } - assert_eq "$dispatched" "$offered" "the completion and the wrapper must agree on the verbs" + [ -n "$offered" ] || { fail_case "the bash completion offered nothing at all"; return; } + assert_eq "$dispatched" "$offered" "the bash completion and the wrapper must agree" + + # THE ZSH COMPLETION IS A SECOND LIST AND HAS TO AGREE TOO. It is native rather than a + # bash completion run through bashcompinit, because that emulates `compgen` and the + # emulation ignores the `--` prefix argument - so a shared file offers every candidate + # whatever has been typed. Two files is the cost of that, and this is what keeps them + # from drifting apart or from the wrapper. + local zsh_offered + zsh_offered=$(sed -n "/^ verbs=(/,/^ )/p" "$REPO_ROOT/lib/_PoolSeqFlow" \ + | sed -n "s/^ '\([a-z_]*\):.*/\1/p" | sort -u) + [ -n "$zsh_offered" ] || { fail_case "the zsh completion offered nothing at all"; return; } + assert_eq "$dispatched" "$zsh_offered" "the zsh completion and the wrapper must agree" } # The second level, for the two subcommands that have a fixed set. `analysis` also offers the @@ -1482,11 +1574,15 @@ test_the_completion_offers_the_subcommands_each_verb_takes() { reply() { COMP_WORDS=("${@:2}" ""); COMP_CWORD=$1; _poolseqflow; printf "%s\n" "${COMPREPLY[@]}"; } printf "check: %s\n" "$(reply 2 PoolSeqFlow check | sort | tr "\n" " ")" printf "modules: %s\n" "$(reply 3 PoolSeqFlow analysis modules | sort | tr "\n" " ")" + printf "init: %s\n" "$(reply 2 PoolSeqFlow init | sort | tr "\n" " ")" + printf "uninstall: %s\n" "$(reply 2 PoolSeqFlow uninstall | sort | tr "\n" " ")" ' _ "$REPO_ROOT") assert_contains "$out" "check: install project " \ "check takes the two targets its usage names" assert_contains "$out" "modules: available install list uninstall " \ "analysis modules takes the four verbs its usage names" + assert_contains "$out" "init: multi " "init takes the one word its usage names" + assert_contains "$out" "uninstall: all " "uninstall takes the one word its usage names" } diff --git a/test/suites/01_migrate.sh b/test/suites/01_migrate.sh index 06382d9..cc3bd5f 100644 --- a/test/suites/01_migrate.sh +++ b/test/suites/01_migrate.sh @@ -251,3 +251,26 @@ test_the_sample_table_note_is_absent_for_a_current_config() { assert_not_contains "$MIGRATE_OUTPUT" "IS REPLACED BY metadata.csv" \ "a config already using metadata.csv should not see the note" } + +# vcffilter.dropZeroDepth is the first parameter ADDED since the storageDir marker, so it is the +# first whose arrival a migration has to explain rather than carry. A config that never set it +# now holds a default nobody chose, and the report line says only `NEW ... true`. +# +# THE SHAPE IS THE TEMPLATE FOR EVERY LATER ADDITION: state the default behavior, and say what to +# write for the other one. It does not describe what the user had, which is what keeps it true for +# every project rather than only the ones a change of behavior reaches. +test_a_new_parameter_states_its_default() { + migrate_config_with -e '/dropZeroDepth/d' + assert_contains "$MIGRATE_OUTPUT" "vcffilter.dropZeroDepth IS NEW" \ + "the arrival should be called out" + assert_contains "$MIGRATE_OUTPUT" "The default is true" "stating the default behavior" + assert_contains "$MIGRATE_OUTPUT" "set it to false" "and what to write for the other one" +} + +# A config that already sets it chose its own value, so there is no default to announce. +test_the_new_parameter_note_is_absent_when_it_was_set() { + migrate_config_with -e 's|^ dropZeroDepth = true| dropZeroDepth = false|' + assert_not_contains "$MIGRATE_OUTPUT" "vcffilter.dropZeroDepth IS NEW" \ + "a config that already sets it saw no arrival" + assert_eq "false" "$(migrated_value dropZeroDepth)" "and its own value is carried" +} diff --git a/test/suites/02_launcher.sh b/test/suites/02_launcher.sh index 1f72312..20d911d 100644 --- a/test/suites/02_launcher.sh +++ b/test/suites/02_launcher.sh @@ -2,7 +2,8 @@ # The ./PoolSeqFlow wrapper's environment handling, against a stub conda. # cost: static # covers: PoolSeqFlow lib/wrapper_lib.sh lib/tool_version.sh bin/check_install.sh -# covers: bin/check_analysis_install.sh bin/check_project.sh +# covers: bin/check_analysis_install.sh bin/check_project.sh bin/check_parameters.sh +# covers: parameters.config.template metadata.csv.template # # These run entirely against the fake conda in lib/sandbox.sh. Nothing here creates, # activates or removes a real environment: a test suite that could delete an operator's @@ -160,6 +161,145 @@ test_check_project_runs_in_the_project_with_the_environment_active() { assert_not_contains "$LAUNCHER_OUTPUT" "STUB check_install ran" "not the installation one" } +# THE REAL bin/check_project.sh, not the stub the cases above use: what is under test here is the +# script's own verdict rather than the wrapper's dispatch to it. +# +# PATH is cut back so that nextflow cannot be found, which keeps these cases free of a JVM and +# asserts the property that makes them possible: the byte-order-mark verdict is printed BEFORE +# the parse it protects, so it stands whether or not nextflow is there. A check that could only +# speak after a successful parse would be silent in the one case it exists for. +# +# WHY IT EXISTS, measured 2026-09-26: a UTF-8 mark makes Nextflow refuse the config, and the +# refusal it writes is `Unexpected character: ''` - the character rendered as nothing, against a +# line the user can see is correct. Windows editors add the mark when they save as UTF-8. +check_project_on() { + CP_OUT=$(cd "$1" && env PATH="/usr/bin:/bin" bash "$REPO_ROOT/bin/check_project.sh" 2>&1) + CP_STATUS=$? +} + +test_check_project_passes_a_config_with_no_byte_order_mark() { + local proj; proj=$(guard_path "$TEST_TMPDIR/bom-clean") + rm -rf "$proj"; mkdir -p "$proj" + cp "$REPO_ROOT/parameters.config.template" "$proj/parameters.config" + check_project_on "$proj" + assert_contains "$CP_OUT" "NO BYTE-ORDER MARK" "a plain file should be said to be plain" + assert_not_contains "$CP_OUT" "dos2unix" "and no fix should be offered for it" +} + +test_check_project_names_a_utf8_byte_order_mark_in_the_config() { + local proj; proj=$(guard_path "$TEST_TMPDIR/bom-utf8") + rm -rf "$proj"; mkdir -p "$proj" + { printf '\xef\xbb\xbf'; cat "$REPO_ROOT/parameters.config.template"; } > "$proj/parameters.config" + check_project_on "$proj" + assert_status 1 "$CP_STATUS" "a config Nextflow cannot parse is a failure, not a note" + assert_contains "$CP_OUT" "STARTS WITH A BYTE-ORDER MARK" "the verdict should name the cause" + assert_contains "$CP_OUT" "utf-8" "and which mark it is" + assert_contains "$CP_OUT" "dos2unix $proj/parameters.config" \ + "and give a command that can be pasted, with the file in it" +} + +# The utf-16 branch, so the check is not a test for one three-byte prefix. Excel's "Unicode Text" +# and a PowerShell redirect both write this, and dos2unix converts it to UTF-8 - measured. +test_check_project_names_a_utf16_byte_order_mark_in_the_config() { + local proj; proj=$(guard_path "$TEST_TMPDIR/bom-utf16") + rm -rf "$proj"; mkdir -p "$proj" + python3 -c "import sys; open(sys.argv[1],'wb').write(open(sys.argv[2],'rb').read().decode().encode('utf-16'))" \ + "$proj/parameters.config" "$REPO_ROOT/parameters.config.template" + check_project_on "$proj" + assert_status 1 "$CP_STATUS" "a UTF-16 config cannot be parsed either" + assert_contains "$CP_OUT" "STARTS WITH A BYTE-ORDER MARK" "the verdict should name the cause" + assert_contains "$CP_OUT" "utf-16" "and say it is a UTF-16 one, not a UTF-8 one" +} + +# THE PARAMETER RULES ARE bin/check_parameters.sh's, and its own cases are in 03_helpers where +# they cost nothing. What these assert is the WIRING: that check_project.sh calls it, renders each +# level the way this script renders everything else, folds the explanation, and lets a FAIL decide +# its own exit status. One rule is used as the vehicle and it does not matter which. +# +# A STUBBED nextflow, so these stay in a suite that starts no JVM. It answers the two forms the +# script uses: a bare `config` for the parse verdict, and `config -flat` for the values read after +# it. STATUS IS NOT ASSERTED except where noted: PATH is cut back so no tool resolves, so the +# tools section fails in every one of these. +check_project_with_pileup() { # mpileup-option-string + local opts="$1" proj stub + proj=$(guard_path "$TEST_TMPDIR/pileup-check") + rm -rf "$proj"; mkdir -p "$proj" + cp "$REPO_ROOT/parameters.config.template" "$proj/parameters.config" + cp "$REPO_ROOT/metadata.csv.template" "$proj/metadata.csv" + stub=$(guard_path "$TEST_TMPDIR/pileup-check-stub") + rm -rf "$stub"; mkdir -p "$stub" + cat > "$stub/nextflow" <&1) + CP_STATUS=$? +} + +# SILENCE IS THE ANSWER FOR A SOUND PARAMETER SET. The helper prints a finding per rule that has +# something to say and nothing otherwise, so a healthy project gets one line rather than nine. +test_check_project_says_nothing_to_flag_for_the_shipped_defaults() { + check_project_with_pileup "-B -C 100 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou" + assert_contains "$CP_OUT" "NOTHING TO FLAG" "the template's own values should raise nothing" + assert_not_contains "$CP_OUT" "DISCARDS EVERY READ" "and certainly not a failure" +} + +test_check_project_renders_a_parameter_failure() { + check_project_with_pileup "-B -C 15 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou" + assert_contains "$CP_OUT" "DISCARDS EVERY READ" "the verdict should be rendered" + assert_contains "$CP_OUT" "variantCall.scaleMapQ" "against the parameter it belongs to" + assert_contains "$CP_OUT" "15 is below varQualMin 30" "with the detail beside it" + # The explanation is folded under the row rather than run off the terminal. + assert_contains "$CP_OUT" "Raise scaleMapQ above varQualMin" "and the explanation printed" + assert_status 1 "$CP_STATUS" "and a parameter failure should fail the check" +} + +# THE BOUNDARY MOVES WITH varQualMin, so a check written against a constant would be wrong. The +# same -C 15 that fails above is sound here. +test_check_project_accepts_a_scale_that_clears_a_lower_minimum() { + check_project_with_pileup "-B -C 15 -q 5 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou" + assert_contains "$CP_OUT" "NOTHING TO FLAG" "15 clears a minimum of 5" + assert_not_contains "$CP_OUT" "DISCARDS EVERY READ" "and must not be refused" +} + +# A NOTE IS NOT A WARNING AND NOT A FAILURE, and the three are rendered differently. An option +# string pinned by hand with no -C in it cannot be judged, and saying so is the answer. +test_check_project_reports_an_unjudgeable_option_string() { + check_project_with_pileup "-B -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou" + assert_contains "$CP_OUT" "NOT CHECKED" "an option string without -C is unjudged" + assert_contains "$CP_OUT" "no -C or -q in mpileupOptions" "and says why" + assert_not_contains "$CP_OUT" "DISCARDS EVERY READ" "never guessed at" +} + +# The helper is part of the installation, so its absence is an incomplete install rather than a +# clean bill of health. Reported, not passed over. +test_check_project_reports_a_missing_parameter_checker() { + local saved + saved=$(guard_path "$TEST_TMPDIR/checker-saved") + cp "$REPO_ROOT/bin/check_parameters.sh" "$saved" + chmod -x "$REPO_ROOT/bin/check_parameters.sh" + check_project_with_pileup "-B -C 100 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou" + chmod +x "$REPO_ROOT/bin/check_parameters.sh" + assert_contains "$CP_OUT" "NOT CHECKED" "a missing checker should be reported" + assert_contains "$CP_OUT" "check_parameters.sh is missing" "and named" +} + + # `conda env create` takes its name from environment.yml unless -n overrides it. Without the # override every release lands in one environment again, which is the bug being fixed. test_install_creates_the_versioned_name_explicitly() { @@ -321,7 +461,7 @@ test_install_reports_environments_left_from_other_versions() { run_launcher_with_envs "base PoolSeqFlow PoolSeqFlow-0.1.0" install assert_contains "$LAUNCHER_OUTPUT" "Other PoolSeqFlow environments" "should report what else is installed" assert_contains "$LAUNCHER_OUTPUT" "unversioned" "the legacy env should be labeled, not called a version" - assert_contains "$LAUNCHER_OUTPUT" "uninstall_all" "should offer the bulk removal command" + assert_contains "$LAUNCHER_OUTPUT" "uninstall all" "should offer the bulk removal command" } test_list_marks_the_current_version() { @@ -350,7 +490,7 @@ test_uninstall_explains_itself_when_the_environment_is_absent() { test_uninstall_all_removes_every_poolseqflow_environment() { local log - run_launcher_with_envs "base PoolSeqFlow PoolSeqFlow-0.1.0 $VERSIONED_ENV" uninstall_all <<< "y" + run_launcher_with_envs "base PoolSeqFlow PoolSeqFlow-0.1.0 $VERSIONED_ENV" uninstall all <<< "y" log=$(cat "$LAUNCHER_CONDA_LOG") assert_contains "$log" "env remove -n PoolSeqFlow " "should remove the legacy environment" assert_contains "$log" "env remove -n PoolSeqFlow-0.1.0" "should remove other versions" @@ -358,7 +498,7 @@ test_uninstall_all_removes_every_poolseqflow_environment() { } test_uninstall_all_aborts_on_a_negative_answer() { - run_launcher_with_envs "base PoolSeqFlow PoolSeqFlow-0.1.0" uninstall_all <<< "n" + run_launcher_with_envs "base PoolSeqFlow PoolSeqFlow-0.1.0" uninstall all <<< "n" assert_status 1 "$LAUNCHER_STATUS" "declining should exit non-zero" assert_contains "$LAUNCHER_OUTPUT" "Aborted" "should say it aborted" assert_not_contains "$(cat "$LAUNCHER_CONDA_LOG")" "env remove" "must remove nothing" @@ -367,22 +507,40 @@ test_uninstall_all_aborts_on_a_negative_answer() { # Piped into a script or run from CI there is no one to answer, and silence must not be # taken for consent. test_uninstall_all_aborts_without_a_terminal() { - run_launcher_with_envs "base PoolSeqFlow" uninstall_all < /dev/null + run_launcher_with_envs "base PoolSeqFlow" uninstall all < /dev/null assert_status 1 "$LAUNCHER_STATUS" "no confirmation should exit non-zero" assert_contains "$LAUNCHER_OUTPUT" "no confirmation received" "should say why it stopped" assert_not_contains "$(cat "$LAUNCHER_CONDA_LOG")" "env remove" "must remove nothing" } -# The wrapper takes exactly one subcommand; parameters.config.template documents that. +# Four subcommands take a word of their own -- `analysis`, `check`, `init multi`, `uninstall all`. +# Every other takes none, and the ones that do take exactly the words they name. test_wrapper_rejects_extra_arguments() { - run_launcher_with_envs "base" uninstall all - assert_status 1 "$LAUNCHER_STATUS" "two arguments should be refused" + run_launcher_with_envs "base" list extra + assert_status 1 "$LAUNCHER_STATUS" "a verb that takes no word should refuse one" + run_launcher_with_envs "base" init bogus + assert_status 1 "$LAUNCHER_STATUS" "init should refuse a word that is not multi" + run_launcher_with_envs "base" uninstall bogus + assert_status 1 "$LAUNCHER_STATUS" "uninstall should refuse a word that is not all" + run_launcher_with_envs "base" init multi extra + assert_status 1 "$LAUNCHER_STATUS" "init should refuse a second word" run_launcher_with_envs "base" assert_status 1 "$LAUNCHER_STATUS" "no argument should be refused" run_launcher_with_envs "base" nonsense_command assert_status 1 "$LAUNCHER_STATUS" "an unknown subcommand should be refused" } +# The old single-token names shipped in 3.2.0, so they say where they went rather than reading as +# an unknown subcommand. +test_the_renamed_subcommands_say_what_they_are_now() { + run_launcher_with_envs "base" init_multi + assert_status 1 "$LAUNCHER_STATUS" "init_multi should be refused" + assert_contains "$LAUNCHER_OUTPUT" "'init multi'" "and should name what replaced it" + run_launcher_with_envs "base" uninstall_all + assert_status 1 "$LAUNCHER_STATUS" "uninstall_all should be refused" + assert_contains "$LAUNCHER_OUTPUT" "'uninstall all'" "and should name what replaced it" +} + # Everything `init` writes is something you then edit, so a second run must leave it alone. test_init_populates_a_project_without_overwriting() { local proj out @@ -413,13 +571,13 @@ test_init_multi_switches_multirun_on_without_inventing_a_table() { proj=$(guard_path "$TEST_TMPDIR/init-multi-project") rm -rf "$proj"; mkdir -p "$proj" - out=$(cd "$proj" && POOLSEQFLOW_HOME="$REPO_ROOT" bash "$REPO_ROOT/PoolSeqFlow" init_multi 2>&1) + out=$(cd "$proj" && POOLSEQFLOW_HOME="$REPO_ROOT" bash "$REPO_ROOT/PoolSeqFlow" init multi 2>&1) assert_contains "$(grep -E '^[[:space:]]*multiRun[[:space:]]*=' "$proj/parameters.config")" \ - "true" "init_multi should switch multiRun on" + "true" "init multi should switch multiRun on" assert_contains "$out" "multi-run.csv.example" "should point at the rules for writing a table" - [ -f "$proj/multi-run.csv.example" ] || fail_case "init_multi should leave the example beside you" + [ -f "$proj/multi-run.csv.example" ] || fail_case "init multi should leave the example beside you" if [ -e "$proj/runs.csv" ]; then - fail_case "init_multi must not invent a run table" + fail_case "init multi must not invent a run table" fi } @@ -1535,7 +1693,7 @@ test_uninstall_aborts_on_a_negative_answer() { assert_dir "$LAUNCHER_PREFIX/opt/PoolSeqFlow-$PSF_VERSION" "and leave the pipeline in place" } -# The same shape as uninstall_all: no terminal means no consent, so nothing goes. +# The same shape as `uninstall all`: no terminal means no consent, so nothing goes. test_uninstall_aborts_without_a_terminal() { run_launcher_with_envs "base $VERSIONED_ENV" install local sb; sb=$(dirname "$LAUNCHER_PREFIX") @@ -1613,7 +1771,30 @@ test_install_puts_the_tab_completion_where_bash_finds_it() { assert_file "$file" "the completion should be installed under XDG_DATA_HOME" assert_contains "$(cat "$file" 2>/dev/null)" "complete -F _poolseqflow" \ "and it should be the real completion, not an empty placeholder" - assert_contains "$LAUNCHER_OUTPUT" "bashcompinit" "the message should tell a zsh user what to add" +} + +# WHAT TO DO NEXT DEPENDS ON THE SHELL, and the message says only the half that applies. +# +# Printing both cases buried the half with work in it: Z installed 3.2.0, read a message +# explaining bash and zsh, and did not add the two zsh lines - so completion did not work for +# the person who had just written it. $SHELL is set here rather than inherited, or the case +# would assert whatever shell the operator happens to use. +test_the_completion_advice_matches_the_users_shell() { + SHELL=/bin/zsh run_launcher_with_envs "base $VERSIONED_ENV" install + assert_status 0 "$LAUNCHER_STATUS" "install should succeed under zsh" + # The ACTIONABLE part, not the greeting: zsh cannot find a completion until its directory + # is on $fpath, so the line that puts it there is what the message owes a zsh user. The + # wording around it is free to change without failing this. + assert_contains "$LAUNCHER_OUTPUT" "fpath=(" "zsh needs the fpath line" + assert_contains "$LAUNCHER_OUTPUT" "zsh/site-functions" "naming the directory it just wrote" + + SHELL=/bin/bash run_launcher_with_envs "base $VERSIONED_ENV" install + assert_status 0 "$LAUNCHER_STATUS" "install should succeed under bash" + # `fpath=(` and not `~/.zshrc`: the PATH advice names ~/.zshrc too, for a prefix that is + # not on PATH - which it never is in a sandbox - so that needle matches a message this case + # is not about. + assert_not_contains "$LAUNCHER_OUTPUT" "fpath=(" \ + "and must not hand a bash user the zsh incantation" } # UNINSTALLING THE LAST VERSION TAKES IT WITH IT. A completion left behind completes a command diff --git a/test/suites/03_helpers.sh b/test/suites/03_helpers.sh index 838ce7a..8c4bdf4 100644 --- a/test/suites/03_helpers.sh +++ b/test/suites/03_helpers.sh @@ -833,6 +833,169 @@ A1,12 assert_contains "$PM_ERR" "pt_wingspan" "and showing the shape of one" } +# ------------------------------------------------- files a person actually saved -- +# +# Every metadata and multi-run file in this suite until now was written BY this suite: ASCII, +# LF, commas, exact spelling. None of them was ever the file a collaborator sends back after +# opening it in Excel, so this whole class of defect was invisible by construction. +# +# FOUND ON A COLLABORATOR'S MACHINE, 2026-09-25. A correct metadata.csv was refused with +# "line 1: no 'SampleID' column", on a file whose first word is plainly SampleID. Measured +# afterwards: SEVEN different file shapes produce that identical message - a UTF-8 byte-order +# mark, a semicolon delimiter, a tab delimiter, 'Sample ID', 'sampleid', a zero-width space, and +# curly quotes - and `dos2unix` fixes exactly one of them. +# +# The mark had a second form that is worse than the refusal. With SampleID in any column but the +# first, the file was ACCEPTED: the leading column's name carried the mark, so it matched +# nothing, RG_Sample was then defaulted from SampleID, and every pool that should have merged +# became its own pool. Exit 0, no message, different numbers. +# +# So the assertions below are not only that each shape is handled. Where a file is genuinely +# unusable they assert the MESSAGE NAMES WHAT WAS READ, because the defect that cost a day was a +# message that reported what we wanted and never what we found. + +# Write metadata.csv from a printf FORMAT rather than a literal, so a case can place bytes that +# no editor displays. pm() takes a literal and cannot express one in an ASCII source file. +pm_bytes() { + helpers_sandbox + # shellcheck disable=SC2059 + printf "$1" > "$HELPERS_DIR/metadata.csv" + PM_STATUS=0 + python3 "$REPO_ROOT/bin/parse_metadata.py" "$HELPERS_DIR/metadata.csv" \ + > "$HELPERS_DIR/metadata.json" 2>"$HELPERS_DIR/pm.err" || PM_STATUS=$? + PM_ERR=$(cat "$HELPERS_DIR/pm.err") +} + +test_metadata_accepts_a_byte_order_mark() { + pm_bytes '\xef\xbb\xbfSampleID,RG_Sample,param_poolSize\nA1,PoolA,50\n' + assert_status 0 "$PM_STATUS" "Excel's CSV UTF-8 is how a collaborator saves: $PM_ERR" + assert_contains "$(cat "$HELPERS_DIR/metadata.json")" '"SampleID": "A1"' \ + "and the column should be SampleID, not a name carrying the mark" +} + +# THE SILENT FORM, and the reason the fix is the codec in rows_of() rather than a check on the +# header. Before it, this file was accepted with PoolA orphaned under a key nothing reads and +# RG_Sample defaulted from SampleID - two lanes of one pool becoming two pools, in a run that +# reported success and published a VCF with an extra column. +test_metadata_does_not_lose_the_first_column_to_a_byte_order_mark() { + pm_bytes '\xef\xbb\xbfRG_Sample,SampleID\nPoolA,A1\nPoolA,A2\n' + assert_status 0 "$PM_STATUS" "the file is well formed: $PM_ERR" + local json; json=$(cat "$HELPERS_DIR/metadata.json") + assert_contains "$json" '"RG_Sample": "PoolA"' \ + "both rows belong to PoolA; a mark on that column splits the pool" + assert_not_contains "$json" '"RG_Sample": "A1"' \ + "RG_Sample must not be defaulted from SampleID when the user set it" +} + +# A SEMICOLON FILE is what Excel writes wherever the comma is the decimal mark, and dos2unix does +# nothing for it. The whole line arrives as one column, so listing what was read says "your +# delimiter is wrong" where "no SampleID column" says something the user can see is false. +test_metadata_names_the_columns_it_read_when_sampleid_is_missing() { + pm 'SampleID;RG_Sample;param_poolSize +A1;PoolA;50 +' + assert_status 1 "$PM_STATUS" "a semicolon file has no SampleID column" + assert_contains "$PM_ERR" "names read from that line" "the message should say what it read" + assert_contains "$PM_ERR" "'SampleID;RG_Sample;param_poolSize'" \ + "showing the whole line as the single column it parsed" +} + +# A ZERO-WIDTH SPACE is not whitespace, so no strip() removes it. It is the shape where the +# listing has to print an ESCAPE rather than the character, or the message shows 'SampleID' +# spelled correctly and claims it is missing - which is where this started. +test_metadata_shows_an_invisible_character_as_an_escape() { + pm_bytes '\xe2\x80\x8bSampleID,RG_Sample\nA1,PoolA\n' + assert_status 1 "$PM_STATUS" "a zero-width space is not stripped and does not match" + # The needle is assembled from a lone backslash on purpose. Written out as the escape + # sequence it stands for, it becomes the character itself in too many editors - which is + # exactly the confusion this case is about, and it would make the assertion pass on a + # message that printed the character invisibly. + local esc='\' + assert_contains "$PM_ERR" "${esc}u200b" "the listing must escape it, not print it invisibly" +} + +# Excel's "Unicode Text" and a PowerShell redirect both write UTF-16. UnicodeDecodeError is a +# ValueError, so it fell past the OSError and csv.Error handlers and reached the user as a raw +# traceback, which names no file and suggests nothing. +test_metadata_reports_a_utf16_file_rather_than_crashing() { + helpers_sandbox + python3 -c \ + "import sys; open(sys.argv[1],'wb').write('SampleID,RG_Sample\nA1,PoolA\n'.encode('utf-16'))" \ + "$HELPERS_DIR/metadata.csv" + PM_STATUS=0 + python3 "$REPO_ROOT/bin/parse_metadata.py" "$HELPERS_DIR/metadata.csv" \ + > "$HELPERS_DIR/metadata.json" 2>"$HELPERS_DIR/pm.err" || PM_STATUS=$? + PM_ERR=$(cat "$HELPERS_DIR/pm.err") + assert_status 1 "$PM_STATUS" "a UTF-16 file cannot be read as CSV" + assert_contains "$PM_ERR" "not UTF-8 text" "and the message should say why" + assert_contains "$PM_ERR" "CSV UTF-8" "and what to save it as instead" + assert_not_contains "$PM_ERR" "Traceback" "rather than a traceback" +} + +# The multi-run table is edited in the same spreadsheet as the metadata and had the identical +# line. Its two messages contradicted each other on screen, both blaming an invisible mark: +# "'RunID' is not a parameter name" directly above "no 'RunID' column". +mr_bytes() { + helpers_sandbox + # shellcheck disable=SC2059 + printf "$1" > "$HELPERS_DIR/runs.csv" + MR_OUT=$(python3 "$REPO_ROOT/bin/parse_multirun.py" "$HELPERS_DIR/runs.csv" 2>"$HELPERS_DIR/stderr") + MR_STATUS=$? + MR_ERR=$(cat "$HELPERS_DIR/stderr") +} + +test_multirun_accepts_a_byte_order_mark() { + mr_bytes '\xef\xbb\xbfRunID,referenceFile\nrefA,a.fasta.gz\n' + assert_status 0 "$MR_STATUS" "the multi-run table is written in Excel too: $MR_ERR" + assert_contains "$MR_OUT" '"RunID": "refA"' "and RunID should be RunID" +} + +test_multirun_names_the_columns_it_read_when_runid_is_missing() { + mr 'RunID;referenceFile +refA;a.fasta.gz +' + assert_status 1 "$MR_STATUS" "a semicolon file has no RunID column" + assert_contains "$MR_ERR" "names read from that line" "the message should say what it read" + assert_contains "$MR_ERR" "'RunID;referenceFile'" "as the single column it parsed" +} + +# A NOTE AND NOT A REFUSAL, and the note is about the FILE rather than any row: the parsers read +# past a mark, so nothing computed from this table is wrong. It is said because the editor that +# added one adds it to everything it saves, and in parameters.config a mark stops the run +# outright - measured against this release. This is the cheap place to find that out. +test_metadata_notes_a_byte_order_mark_it_read_past() { + pm_bytes '\xef\xbb\xbfSampleID,RG_Sample\nA1,PoolA\n' + assert_status 0 "$PM_STATUS" "the file is usable" + assert_contains "$PM_ERR" "usable, with notes" "and reported as usable" + assert_contains "$PM_ERR" "byte-order mark" "with the mark named" + assert_contains "$PM_ERR" "parameters.config" "and the file where it would be fatal" + assert_contains "$PM_ERR" "dos2unix" "and what removes it" +} + +test_metadata_says_nothing_about_a_file_with_no_mark() { + pm 'SampleID,RG_Sample +A1,PoolA +' + assert_status 0 "$PM_STATUS" "an ordinary file is ordinary" + assert_not_contains "$PM_ERR" "byte-order mark" "and gets no note about one" +} + +test_multirun_notes_a_byte_order_mark_it_read_past() { + mr_bytes '\xef\xbb\xbfRunID,referenceFile\nrefA,a.fasta.gz\n' + assert_status 0 "$MR_STATUS" "the file is usable" + assert_contains "$MR_ERR" "usable, with notes" "and reported as usable" + assert_contains "$MR_ERR" "byte-order mark" "with the mark named" + assert_contains "$MR_ERR" "dos2unix" "and what removes it" +} + +test_multirun_says_nothing_about_a_file_with_no_mark() { + mr 'RunID,referenceFile +refA,a.fasta.gz +' + assert_status 0 "$MR_STATUS" "an ordinary file is ordinary" + assert_not_contains "$MR_ERR" "byte-order mark" "and gets no note about one" +} + # ---------------------------------------------------------------- citations -- # Writes both citation files into a scratch directory and echoes it. Runs against the real @@ -1414,3 +1577,625 @@ test_atomic_mv_leaves_nothing_behind_when_it_fails() { assert_no_file "$HELPERS_DIR/dst/x" "and write nothing" assert_count 0 "$(find "$HELPERS_DIR/dst" -name '.atomic_mv.*' | wc -l)" "and stage nothing" } + +# depth2freq.awk: the depth table's counts as one row per allele. +# +# THE ZERO-DEPTH CELL IS WHAT THESE EXIST FOR. Until vcffilter.dropZeroDepth was added, a +# published table could not hold one - vcffilter.minDP removes a site where any sample falls +# short, and the smallest useful minDP is 1 - so the converter's total == 0 branch was +# unreachable from a real run and wrote 0. With dropZeroDepth = false and minDP = 0 it is +# reachable, and a 0 there is the same character a pool fixed for the other allele publishes. +# +# The input shapes below are real bcftools output, measured rather than assumed: a sample with +# no reads gets GT ./. and AD 0,0, so the cell is zeros and not a missing value. + +# Convert a depth table written inline, and leave the result in D2F_OUT. +d2f() { + printf '%s' "$1" > "$HELPERS_DIR/depth.tsv" + D2F_OUT=$(awk -f "$REPO_ROOT/bin/depth2freq.awk" < "$HELPERS_DIR/depth.tsv") +} + +# One cell of that result, by data row and by column, both 1-based. +d2f_cell() { # row column + printf '%s' "$D2F_OUT" | awk -F'\t' -v r="$2" 'NR == '"$(( $1 + 1 ))"' { print $r }' +} + +test_depth2freq_writes_na_for_a_sample_with_no_reads() { + helpers_sandbox + # SampleB saw nothing here; SampleA is fixed for the alternate. + d2f 'CHROM POS REF ALT TOTAL_AD SampleA SampleB +chrT 30 A C 0,20 0,20 0,0 +' + assert_eq "NA" "$(d2f_cell 1 7)" "a pool with no reads has no frequency on the REF row" + assert_eq "NA" "$(d2f_cell 2 7)" "nor on any other allele of that site" +} + +# THE DISCRIMINATION THE CHANGE IS FOR. A pool that was measured and carries none of an allele +# publishes 0, and that is a real observation. If this case and the one above ever agree, the +# table has stopped saying which of the two a reader is looking at. +# +# It is the control and it passes on both sides of the NA change, which is its job: the pair +# is what has meaning, and a zero turning into NA here would be the regression. +test_depth2freq_keeps_zero_for_an_allele_a_sample_was_measured_without() { + helpers_sandbox + d2f 'CHROM POS REF ALT TOTAL_AD SampleA SampleB +chrT 30 A C 0,20 0,20 0,0 +' + assert_eq "0" "$(d2f_cell 1 6)" "20 reads and none of them REF is a frequency of 0" + assert_eq "1" "$(d2f_cell 2 6)" "and the allele they all carry is 1" +} + +# NA is per cell, not per column: the pools that did see reads are published as they were. +test_depth2freq_leaves_the_other_pools_of_the_site_alone() { + helpers_sandbox + d2f 'CHROM POS REF ALT TOTAL_AD S1 S2 S3 +chr1 100 A G 60,40 30,10 0,0 20,20 +' + assert_eq "0.75" "$(d2f_cell 1 6)" "the first pool is untouched" + assert_eq "NA" "$(d2f_cell 1 7)" "the empty one between them is NA" + assert_eq "0.5" "$(d2f_cell 1 8)" "and so is the third" +} + +# And per site, not per pool: a pool empty at one site still publishes the sites it was read at. +test_depth2freq_na_does_not_spread_down_a_column() { + helpers_sandbox + d2f 'CHROM POS REF ALT TOTAL_AD S1 S2 +chr1 100 A G 60,40 30,10 0,0 +chr1 200 A G 60,40 30,10 30,10 +' + assert_eq "NA" "$(d2f_cell 1 7)" "empty at the first site" + assert_eq "0.75" "$(d2f_cell 3 7)" "and read at the second" +} + +# TOTAL_AD is converted by the same loop as the sample columns and takes the same rule. +test_depth2freq_applies_the_rule_to_total_ad() { + helpers_sandbox + d2f 'CHROM POS REF ALT TOTAL_AD S1 +chr1 100 A G 0,0 0,0 +' + assert_eq "NA" "$(d2f_cell 1 5)" "a cohort total of no reads is NA as well" +} + +# parsed_vals is global and indexed by column and allele, and the write loop runs over the +# CELL's counts while the print loop runs over the ALT column's alleles. Where a cell holds +# fewer counts than its row declares alleles, the print loop reads whatever an earlier row +# left at that index - and publishes a frequency measured at a different site. +# +# It cannot reach here from a run: AD is Number=R, so bcftools writes one count per allele, +# and MajorAlleleToRef.py stops on a ragged cell before this ever sees it. The case is written +# against the converter's own contract rather than against what currently feeds it. Measured: +# with the clear removed, chr1:200's third allele publishes 0.25, carried from chr1:100. +# +# The row is still ragged afterwards and the cell comes out empty. Refusing it outright is a +# different question and is not what this asserts. +test_depth2freq_does_not_carry_a_count_from_an_earlier_row() { + helpers_sandbox + d2f 'CHROM POS REF ALT TOTAL_AD S1 +chr1 100 A G,T 2,1,1 2,1,1 +chr1 200 A G,T 1,1 1,1 +' + assert_eq "0.25" "$(d2f_cell 3 6)" "the complete row is converted as it always was" + assert_not_contains "$(d2f_cell 6 6)" "0.25" \ + "and the short cell below it must not republish that number" +} + +# cap_depth.awk: truncate a coordinate-sorted SAM so no reference position is covered more +# than `cap` times. +# +# THE CONTRACT, NOT THE IMPLEMENTATION. The first case below recomputes per-position depth +# from the OUTPUT and asserts no position exceeds the cap. That is what the helper promises +# and it holds whatever the inside looks like -- which matters here, because the inside was a +# per-position array scanned twice per read and is now a difference array read once, and the +# case had to survive that without being rewritten. +# +# Written after the fact: this helper shipped from 1.0 with no unit coverage at all, and the +# rewrite was verified against three real BAMs before any of these existed. + +# The fixture is APPENDED TO A FILE rather than composed in a variable. `$(...)` strips +# trailing newlines, so building the SAM by concatenating command substitutions silently glues +# the last record of one stack onto the first of the next -- which produced a malformed +# fixture that one of the cases below passed against anyway. +cap_begin() { # reference-sequence names + : > "$HELPERS_DIR/in.sam" + printf '@HD\tVN:1.6\tSO:coordinate\n' >> "$HELPERS_DIR/in.sam" + local c + for c in "$@"; do printf '@SQ\tSN:%s\tLN:100000\n' "$c" >> "$HELPERS_DIR/in.sam"; done +} + +# A stack of identical reads at one position. SEQ and QUAL are `*`: nothing here reads them. +cap_stack() { # chrom start count cigar + local i + for i in $(seq 1 "$3"); do + printf 'r%s_%s_%s\t0\t%s\t%s\t60\t%s\t*\t0\t0\t*\t*\n' \ + "$1" "$2" "$i" "$1" "$2" "$4" >> "$HELPERS_DIR/in.sam" + done +} + +cap_record() { printf '%s\n' "$1" >> "$HELPERS_DIR/in.sam"; } + +cap_run() { # cap-value + CAP_OUT=$(awk -f "$REPO_ROOT/bin/cap_depth.awk" -v cap="$1" < "$HELPERS_DIR/in.sam" \ + 2> "$HELPERS_DIR/cap.err") + CAP_STATUS=$? + CAP_ERR=$(cat "$HELPERS_DIR/cap.err") +} + +# The deepest any reference position is covered in a SAM stream, computed here rather than +# taken from the helper: walk each record's CIGAR, count the positions it consumes, take the +# maximum over all of them. +deepest() { + printf '%s\n' "$1" | awk ' + !/^@/ && $4 > 0 { + n = ""; span = 0 + for (i = 1; i <= length($6); i++) { + c = substr($6, i, 1) + if (c >= "0" && c <= "9") { n = n c; continue } + if (c ~ /[MDN=X]/) span += n + 0 + n = "" + } + for (p = $4; p < $4 + span; p++) d[$3 "\t" p]++ + } + END { m = 0; for (k in d) if (d[k] > m) m = d[k]; print m }' +} + +test_cap_depth_never_leaves_a_position_over_the_cap() { + helpers_sandbox + local c + for c in 1 5 17 30 61 200; do + # Overlapping stacks at 1, 40 and 80: a read starting at 40 spans into the one at 80, + # so what it must be judged against is not the depth at its own start alone. + cap_begin chr1 + cap_stack chr1 1 30 100M + cap_stack chr1 40 30 100M + cap_stack chr1 80 30 100M + cap_run "$c" + assert_status 0 "$CAP_STATUS" "capping at $c should succeed" + local got; got=$(deepest "$CAP_OUT") + [ "$got" -le "$c" ] \ + || fail_case "cap $c: a position is covered $got times in the output" + done +} + +# Nothing is dropped that did not have to be: at a cap at or above the deepest position the +# stream passes through untouched. +test_cap_depth_keeps_everything_under_the_cap() { + helpers_sandbox + cap_begin chr1 + cap_stack chr1 1 12 100M + cap_run 12 + assert_eq "12" "$(printf '%s' "$CAP_OUT" | grep -c '^r')" \ + "a cap equal to the depth should drop nothing" + assert_contains "$CAP_ERR" "dropped 0" "and say so" +} + +# THE HEADER IS NOT A RECORD. It passes through ahead of everything and is not counted. +test_cap_depth_passes_the_header_through() { + helpers_sandbox + cap_begin chr1 + cap_stack chr1 1 5 100M + cap_run 2 + assert_eq "@HD VN:1.6 SO:coordinate" "$(printf '%s' "$CAP_OUT" | sed -n '1p')" \ + "the header leads the output" + assert_contains "$CAP_ERR" "kept 2, dropped 3" "and is not in the tally" +} + +# A NEW REFERENCE SEQUENCE STARTS FROM NOTHING. Carrying state across would cap the second one +# against depth accumulated on the first, and positions repeat between them. +test_cap_depth_resets_at_a_new_reference_sequence() { + helpers_sandbox + cap_begin chr1 chr2 + cap_stack chr1 1 10 100M + cap_stack chr2 1 10 100M + cap_run 10 + assert_eq "10" "$(printf '%s' "$CAP_OUT" | grep -c ' chr1 ')" "chr1 keeps its ten" + assert_eq "10" "$(printf '%s' "$CAP_OUT" | grep -c ' chr2 ')" "and chr2 its own ten" + assert_contains "$CAP_ERR" "dropped 0" "neither counted against the other" +} + +# A RECORD THAT CONSUMES NO REFERENCE IS NOT CAPPED. A CIGAR of only soft clips covers no +# position, so there is nothing to count it against and it survives however deep the stack at +# its own coordinate is. +# +# ORDER IS THE WHOLE CASE. The soft-clipped read sits DIRECTLY after the stack, on the same +# reference sequence, at a coordinate already at the cap -- that is the only arrangement in +# which the guard does anything. Written first with an unmapped record in between, where the +# `*` RNAME forces a new-sequence reset that clears the depth before the soft-clipped read is +# judged: the case passed with the guard deleted outright. The unmapped record is now last, +# where a sorted BAM puts it anyway. +test_cap_depth_keeps_records_that_cover_no_position() { + helpers_sandbox + cap_begin chr1 + cap_stack chr1 1 4 100M + cap_record 's1 0 chr1 1 60 100S * 0 0 * *' + cap_record 'u1 4 * 0 0 * * 0 0 * *' + cap_run 1 + assert_contains "$CAP_OUT" "s1 0" "a read consuming no reference survives a capped position" + assert_contains "$CAP_OUT" "u1 4" "and so does an unmapped one" + assert_eq "1" "$(printf '%s' "$CAP_OUT" | grep -c ' 100M ')" "while the stack is still capped" + assert_contains "$CAP_ERR" "kept 3" "all three counted as kept, with the one capped read" +} + +# ONLY M, D, N, = AND X CONSUME THE REFERENCE. A deletion makes a read reach FURTHER than its +# sequence length, and a read counted short would leave the positions past its sequence +# uncapped. 50M20D50M spans 120 where a plain 100M spans 100, so the stack at 110 sits inside +# the first read's reach and outside the second's -- which is what makes this discriminating +# rather than merely passing. +test_cap_depth_measures_the_span_from_the_cigar() { + helpers_sandbox + cap_begin chr1 + cap_stack chr1 1 3 50M20D50M + cap_stack chr1 110 3 10M + cap_run 3 + assert_contains "$CAP_ERR" "dropped 3" \ + "the stack at 110 is already at the cap, reached by the deletion-spanning reads" + local got; got=$(deepest "$CAP_OUT") + [ "$got" -le 3 ] || fail_case "a position is covered $got times at cap 3" +} + +test_cap_depth_refuses_a_cap_that_is_not_a_positive_depth() { + helpers_sandbox + local v + for v in 0 -1 abc ''; do + cap_begin chr1 + cap_stack chr1 1 2 100M + cap_run "$v" + assert_status 2 "$CAP_STATUS" "a cap of '$v' should be refused" + assert_contains "$CAP_ERR" "must be a positive depth" "saying what a cap is" + assert_not_contains "$CAP_ERR" "kept" "and no tally after a usage error" + done +} + +# MajorAlleleToRef.py: re-polarize a VCF so REF is the allele the whole cohort read most. +# +# It runs TWICE in step 7 -- once before the false-positive filter and again after it, because +# removing alleles can change which one is major -- so anything it gets wrong is applied to every +# published frequency, twice. +# +# THE INVARIANTS RUN OVER THE REAL VCF. test/data/vcf/called.vcf is genuine bcftools output and +# its README says anything testing this script should start there. What it does NOT contain is a +# multiallelic record or a spanning deletion, so the ordering cases below are hand-written: the +# point of those is an exact permutation, which needs exact input. +# +# Written after the fact. This was the last bin/ helper with no unit coverage. + +M2R_OUT="" +M2R_STATUS=0 +M2R_LOG="" + +# Run the script over an inline VCF body, with a two-sample header unless one is given. +m2r() { # body [format] [sample-columns] + local body="$1" fmt="${2:-GT:DP:AD}" cols="${3:-S1 S2}" + { + printf '##fileformat=VCFv4.2\n' + printf '##INFO=\n' + printf '#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\t%s\n' "$cols" + printf '%s\n' "$body" + } > "$HELPERS_DIR/in.vcf" + python3 "$REPO_ROOT/bin/MajorAlleleToRef.py" \ + "$HELPERS_DIR/in.vcf" "$HELPERS_DIR/out.vcf" > "$HELPERS_DIR/m2r.log" 2>&1 + M2R_STATUS=$? + M2R_LOG=$(cat "$HELPERS_DIR/m2r.log") + M2R_OUT=$(grep -v '^#' "$HELPERS_DIR/out.vcf" 2>/dev/null || true) +} + +# One field of one data row of the result, both 1-based. +m2r_field() { printf '%s\n' "$M2R_OUT" | sed -n "${1}p" | cut -f"$2"; } + +test_major_allele_to_ref_flips_when_the_alternate_is_more_read() { + helpers_sandbox + # Cohort AD is 15,135: the alternate is read nine times as often, so it becomes REF. + m2r 'chr1 100 . A G 50 . DP=150;AD=15,135 GT:DP:AD 0/1:100:10,90 0/1:50:5,45' + assert_status 0 "$M2R_STATUS" "the script should succeed" + assert_eq "G" "$(m2r_field 1 4)" "the most-read allele becomes REF" + assert_eq "A" "$(m2r_field 1 5)" "and the former reference becomes the alternate" + assert_contains "$(m2r_field 1 8)" "AD=135,15" "INFO/AD follows the same order" +} + +test_major_allele_to_ref_leaves_an_already_major_reference_alone() { + helpers_sandbox + m2r 'chr1 100 . A G 50 . DP=150;AD=135,15 GT:DP:AD 0/1:100:90,10 0/1:50:45,5' + assert_eq "A" "$(m2r_field 1 4)" "a reference already most-read stays" + assert_eq "G" "$(m2r_field 1 5)" "and so does the alternate" + assert_contains "$(m2r_field 1 8)" "AD=135,15" "with the counts untouched" +} + +# AN EXACT TIE KEEPS THE REFERENCE, which rests on Python's sort being stable rather than on +# anything the script says. If that ever changes, REF flips on every tied site in a run and +# nothing else would report it. +test_major_allele_to_ref_keeps_the_reference_on_an_exact_tie() { + helpers_sandbox + m2r 'chr1 100 . A G 50 . DP=100;AD=50,50 GT:DP:AD 0/1:100:50,50' 'GT:DP:AD' 'S1' + assert_eq "A" "$(m2r_field 1 4)" "a tie must not flip the reference" + assert_eq "G" "$(m2r_field 1 5)" "nor reorder the alternate" +} + +# ONE ORDERING FOR THE WHOLE SITE. The permutation comes from INFO/AD and no sample chooses its +# own, so a sample whose own counts disagree with the cohort is still reordered the cohort's way. +# Getting this wrong would mis-assign every frequency at the site while looking entirely normal. +test_major_allele_to_ref_reorders_every_sample_by_the_same_permutation() { + helpers_sandbox + # S2's own majority is the REFERENCE (30 against 10), against the cohort's alternate (80 + # against 60). It must still be reordered the cohort's way. + m2r 'chr1 100 . A G 50 . DP=140;AD=60,80 GT:DP:AD 0/1:100:10,90 0/1:40:30,10' + assert_eq "G" "$(m2r_field 1 4)" "the cohort decides REF" + assert_eq "90,10" "$(printf '%s' "$(m2r_field 1 10)" | cut -d: -f3)" \ + "the sample agreeing with the cohort is reordered" + assert_eq "10,30" "$(printf '%s' "$(m2r_field 1 11)" | cut -d: -f3)" \ + "and so is the sample that disagrees, the same way" +} + +# DEPTH IS REDEFINED AS THE SUM OF THE REORDERED COUNTS, in INFO and in every sample. The rest of +# the chain depends on it: vcffilter.minDP filters on FORMAT/DP, and the depth table sums the same +# AD. +# +# THE TWO HALVES CATCH DIFFERENT THINGS, measured by removing each recompute in turn. +# +# INFO: the real VCF is enough. mpileup's INFO/DP is raw depth including reads assigned to no +# allele, so it already disagrees with sum(INFO/AD) on 35 of the 135 records, and removing that +# recompute fails the sweep below by exactly that count. +# +# FORMAT: the real VCF proves nothing. bcftools writes each sample's DP equal to sum(AD) already, +# so the invariant holds whether or not this script touches it -- measured: deleting the +# FORMAT/DP assignment left the sweep passing. What makes that recompute load-bearing is the +# SECOND invocation, after filterFalsePositives.sh has removed alleles and left DP larger than the +# counts that remain, so the record below is shaped like that one. +test_major_allele_to_ref_rewrites_depth_as_the_sum_of_the_counts() { + helpers_sandbox + # DP says 200 and 200 while the counts total 150 and 100: what an allele removal leaves behind. + m2r 'chr1 100 . A G 50 . DP=200;AD=15,135 GT:DP:AD 0/1:200:10,90 0/1:80:5,45' + assert_contains "$(m2r_field 1 8)" "DP=150" "INFO/DP is recomputed from the counts that remain" + assert_eq "100" "$(printf '%s' "$(m2r_field 1 10)" | cut -d: -f2)" \ + "and so is each sample's own DP" + assert_eq "50" "$(printf '%s' "$(m2r_field 1 11)" | cut -d: -f2)" "for every sample" + + python3 "$REPO_ROOT/bin/MajorAlleleToRef.py" \ + "$REPO_ROOT/test/data/vcf/called.vcf" "$HELPERS_DIR/real.vcf" > /dev/null 2>&1 + assert_status 0 "$?" "the real VCF should convert" + local bad + bad=$(grep -v '^#' "$HELPERS_DIR/real.vcf" | awk -F'\t' ' + { split($9, f, ":"); for (i in f) if (f[i] == "AD") ad = i; else if (f[i] == "DP") dp = i + for (s = 10; s <= NF; s++) { + split($s, g, ":"); n = split(g[ad], c, ","); t = 0 + for (j = 1; j <= n; j++) t += c[j] + if (t != g[dp] + 0) bad++ + } + split($8, info, ";"); isum = 0 + for (i in info) if (info[i] ~ /^AD=/) { n = split(substr(info[i], 4), a, ","); for (j = 1; j <= n; j++) isum += a[j] } + for (i in info) if (info[i] ~ /^DP=/) if (substr(info[i], 4) + 0 != isum) bad++ + } END { print bad + 0 }') + assert_eq "0" "$bad" "every DP should equal the sum of its own AD, in INFO and every sample" +} + +# EVERY GENOTYPE IS BLANKED. Re-polarizing invalidates the caller's calls and a pool has no +# genotype to replace them with, so leaving them would invite a tool to read one. +test_major_allele_to_ref_blanks_every_genotype() { + helpers_sandbox + python3 "$REPO_ROOT/bin/MajorAlleleToRef.py" \ + "$REPO_ROOT/test/data/vcf/called.vcf" "$HELPERS_DIR/real.vcf" > /dev/null 2>&1 + local called + called=$(grep -v '^#' "$HELPERS_DIR/real.vcf" | awk -F'\t' ' + { split($9, f, ":"); for (i in f) if (f[i] == "GT") gt = i + for (s = 10; s <= NF; s++) { split($s, g, ":"); if (g[gt] != "./.") n++ } } END { print n + 0 }') + assert_eq "0" "$called" "no sample should keep a genotype" +} + +test_major_allele_to_ref_orders_a_multiallelic_site_by_count() { + helpers_sandbox + # Counts 10 / 50 / 30 for A / G / T, so the order becomes G, T, A. + m2r 'chr1 100 . A G,T 50 . DP=90;AD=10,50,30 GT:DP:AD 0/1:90:10,50,30' 'GT:DP:AD' 'S1' + assert_eq "G" "$(m2r_field 1 4)" "the most-read allele leads" + assert_eq "T,A" "$(m2r_field 1 5)" "and the rest follow in count order" + assert_contains "$(m2r_field 1 8)" "AD=50,30,10" "INFO/AD takes the same order" + assert_eq "50,30,10" "$(printf '%s' "$(m2r_field 1 10)" | cut -d: -f3)" "and so does the sample" +} + +# A SPANNING DELETION IS AN ALLELE LIKE ANY OTHER. It carries reads, and the analysis layer +# declares that it reaches the SNP table as one, so it must order by count and not be special. +test_major_allele_to_ref_treats_a_spanning_deletion_as_an_allele() { + helpers_sandbox + m2r 'chr1 100 . A G,* 50 . DP=90;AD=10,50,30 GT:DP:AD 0/1:90:10,50,30' 'GT:DP:AD' 'S1' + assert_eq "G" "$(m2r_field 1 4)" "the most-read allele still leads" + assert_eq "*,A" "$(m2r_field 1 5)" "and the star sorts on its count like the rest" +} + +test_major_allele_to_ref_copies_the_header_through() { + helpers_sandbox + m2r 'chr1 100 . A G 50 . DP=150;AD=15,135 GT:DP:AD 0/1:100:10,90 0/1:50:5,45' + assert_eq "$(grep -c '^#' "$HELPERS_DIR/in.vcf")" "$(grep -c '^#' "$HELPERS_DIR/out.vcf")" \ + "every header line should survive" + assert_contains "$(head -1 "$HELPERS_DIR/out.vcf")" "##fileformat=VCFv4.2" "in order" +} + +# The one guard it has. Without FORMAT/AD there is nothing to reorder, and carrying on would +# publish frequencies from counts it never read. +test_major_allele_to_ref_refuses_a_vcf_without_format_ad() { + helpers_sandbox + m2r 'chr1 100 . A G 50 . DP=150;AD=15,135 GT:DP 0/1:100 0/1:50' 'GT:DP' 'S1 S2' + assert_status 1 "$M2R_STATUS" "a VCF with no FORMAT/AD should be refused" + assert_contains "$M2R_LOG" "No FORMAT/AD field" "saying what is missing" +} + +# check_parameters.sh: judge a resolved parameter set, one finding per line. +# +# THE RULES LIVE HERE AND NOWHERE ELSE. Both bin/check_project.sh and step 0 call this, so a +# project cannot be told one thing before a run and another during it. 02_launcher asserts the +# wiring through check_project.sh; these are the rules themselves, at static cost. +# +# WHAT BELONGS IN IT: a setting that makes the run produce NOTHING, or that silently changes what +# a published number means. minDP 20 against minDP 5 is a scientific choice and is not its +# business. Every threshold below was measured, not chosen. + +# The shipped defaults, which every case starts from and mutates. +cp_defaults() { + cat <<'FLAT' +params.variantCall.mpileupOptions = '-B -C 100 -q 30 -Q 30 -d 0 -a AD,DP,SP,INFO/AD -Ou' +params.filterFalsePositives.sampleThreshold = 0.2 +params.vcffilter.minDP = 20 +params.vcffilter.dropZeroDepth = true +params.ploidy = 2 +params.poolSize = 100 +params.capBAM.maxDepth = -1 +params.variantCall.maxDepth = 0 +params.fastqc.memory = 2048 +FLAT +} + +# Run the helper over the defaults with the given assignments replacing theirs. +CP_FIND="" +CP_RC=0 +cp_check() { # params.key=value ... + local a k + cp_defaults > "$HELPERS_DIR/flat.txt" + for a in "$@"; do + k=${a%%=*} + grep -v "^${k} = " "$HELPERS_DIR/flat.txt" > "$HELPERS_DIR/flat.tmp" || true + mv "$HELPERS_DIR/flat.tmp" "$HELPERS_DIR/flat.txt" + printf '%s = %s\n' "${a%%=*}" "${a#*=}" >> "$HELPERS_DIR/flat.txt" + done + CP_FIND=$(bash "$REPO_ROOT/bin/check_parameters.sh" < "$HELPERS_DIR/flat.txt") + CP_RC=$? +} + +# The level and verdict for one parameter, or empty when it said nothing. +cp_verdict() { printf '%s\n' "$CP_FIND" | awk -F'\t' -v k="$1" '$2 == k { print $1, $3 }'; } + +# SILENCE IS THE ANSWER FOR A SOUND SET, and it is the case most likely to rot: a rule added with +# a wrong threshold makes the shipped template noisy, and a user who sees a warning on a default +# stops reading warnings. +test_check_parameters_says_nothing_about_the_shipped_defaults() { + helpers_sandbox + cp_check + assert_eq "" "$CP_FIND" "the template's own values should produce no finding at all" + assert_status 0 "$CP_RC" "and exit clean" +} + +# -C caps every read's mapping quality near its own value and -q rejects anything below the +# minimum, so a -C under -q leaves the pileup empty. Measured on real pools at six values of -q +# (12, 15, 20, 30, 40, 50): one below returned zero sites every time. +test_check_parameters_refuses_a_scale_below_the_quality_minimum() { + helpers_sandbox + cp_check "params.variantCall.mpileupOptions='-B -C 15 -q 30 -Q 30'" + assert_eq "FAIL DISCARDS EVERY READ" "$(cp_verdict variantCall.scaleMapQ)" \ + "a scale under the minimum should be refused" + assert_status 1 "$CP_RC" "and set a failing status" +} + +# THE BOUNDARY IS THE RELATIONSHIP, NOT A RANGE. The same 15 is sound against a lower minimum, +# and a rule written against a constant would call this broken. +test_check_parameters_accepts_a_scale_that_clears_a_lower_minimum() { + helpers_sandbox + cp_check "params.variantCall.mpileupOptions='-B -C 15 -q 5 -Q 30'" + assert_eq "" "$(cp_verdict variantCall.scaleMapQ)" "15 clears a minimum of 5" + assert_status 0 "$CP_RC" "and is not a failure" +} + +test_check_parameters_separates_an_inert_scale_from_a_deliberate_zero() { + helpers_sandbox + cp_check "params.variantCall.mpileupOptions='-B -C 5 -q 30 -Q 30'" + assert_eq "WARN INERT AT 5" "$(cp_verdict variantCall.scaleMapQ)" \ + "below 11 the adjustment does nothing, whatever was written" + cp_check "params.variantCall.mpileupOptions='-B -C 0 -q 30 -Q 30'" + assert_eq "NOTE OFF" "$(cp_verdict variantCall.scaleMapQ)" \ + "zero is a deliberate off, not an accident" +} + +# Clearing the minimum is not being safe: at -C equal to -q only the best-placed reads survive, +# measured at 6% of the sites an unadjusted run called. +test_check_parameters_warns_when_the_scale_barely_clears_the_minimum() { + helpers_sandbox + cp_check "params.variantCall.mpileupOptions='-B -C 35 -q 30 -Q 30'" + assert_eq "WARN SEVERE AT 35" "$(cp_verdict variantCall.scaleMapQ)" \ + "just above the minimum is still severe" + assert_status 0 "$CP_RC" "but not a failure: it does produce records" +} + +# An option string pinned by hand may carry neither flag. Saying so beats guessing from settings +# it was not built from, which is the trap the whole composed-value approach exists to avoid. +test_check_parameters_declines_to_judge_an_option_string_without_the_pair() { + helpers_sandbox + cp_check "params.variantCall.mpileupOptions='-B -q 30 -Q 30'" + assert_eq "NOTE NOT CHECKED" "$(cp_verdict variantCall.scaleMapQ)" "unjudged, and said so" + assert_status 0 "$CP_RC" "and never a failure" +} + +# sampleThreshold is a fraction of the samples. Above 1 it asks for more samples than exist: +# measured at 1.5 against the fixture VCF, 0 of 135 sites survived. +test_check_parameters_refuses_a_sample_threshold_above_one() { + helpers_sandbox + cp_check "params.filterFalsePositives.sampleThreshold=1.5" + assert_eq "FAIL REMOVES EVERY SITE" "$(cp_verdict filterFalsePositives.sampleThreshold)" \ + "a fraction above 1 cannot be satisfied" + cp_check "params.filterFalsePositives.sampleThreshold=0" + assert_eq "WARN INERT AT 0" "$(cp_verdict filterFalsePositives.sampleThreshold)" \ + "and at 0 it asks for nothing" +} + +# A MISSING KEY MUST NOT CRASH AND MUST NOT INVENT A FINDING. Every rule declines to judge a value +# it does not have, because after resolution an absent key means the project's config never defined +# it and there is nothing to compare. The one thing that is never acceptable is a spurious verdict +# about a parameter nobody set. +test_check_parameters_survives_a_config_with_nothing_in_it() { + helpers_sandbox + printf 'params.mainDir = /somewhere\n' > "$HELPERS_DIR/flat.txt" + CP_FIND=$(bash "$REPO_ROOT/bin/check_parameters.sh" < "$HELPERS_DIR/flat.txt") + CP_RC=$? + assert_status 0 "$CP_RC" "an empty parameter set is unjudgeable, not a failure" + # The pileup pair is the only rule that speaks, and it speaks to say it cannot judge. + assert_eq "NOTE NOT CHECKED" "$(cp_verdict variantCall.scaleMapQ)" "and says so" + assert_eq "" "$(cp_verdict poolSize)" "no verdict on a poolSize nobody set" + assert_eq "" "$(cp_verdict ploidy)" "nor on ploidy" + assert_eq "" "$(cp_verdict fastqc.memory)" "nor on fastqc.memory" + assert_eq "" "$(cp_verdict filterFalsePositives.sampleThreshold)" "nor on the threshold" +} + +# n_chrom is ploidy times poolSize. At 1 the unbiased diversity correction n_eff/(n_eff - 1) is +# infinite -- measured, n_eff(1, 50) is exactly 1 -- so any diversity computed over such a pool is +# meaningless. A WARNING and not a failure, because the PIPELINE is unaffected: it publishes +# frequencies perfectly well and only an analysis degrades. Refusing a run here would be stricter +# than the thing being protected. +test_check_parameters_warns_about_a_pool_of_one_chromosome() { + helpers_sandbox + cp_check "params.poolSize=1" "params.ploidy=1" + assert_eq "WARN ONE CHROMOSOME" "$(cp_verdict poolSize)" "one chromosome cannot carry diversity" + assert_status 0 "$CP_RC" "but the pipeline runs, so it must not fail the check" + cp_check "params.ploidy=0" + assert_eq "FAIL NOT A PLOIDY" "$(cp_verdict ploidy)" "and a ploidy below 1 is not a ploidy" + # A haploid pool of two is legitimate and must not be caught by the same rule. + cp_check "params.poolSize=2" "params.ploidy=1" + assert_eq "" "$(cp_verdict poolSize)" "two haploid individuals are a pool" +} + +# capBAM.maxDepth -1 asks step 5 to measure a ceiling per sample. A positive variantCall.maxDepth +# then caps every sample flat at pileup time as well, so the smaller wins and the measured +# ceilings stop deciding. +test_check_parameters_notes_a_flat_ceiling_over_measured_ones() { + helpers_sandbox + cp_check "params.variantCall.maxDepth=500" + assert_eq "NOTE OVERRIDES THE MEASURED CEILINGS" "$(cp_verdict variantCall.maxDepth)" \ + "a flat ceiling undercuts the per-sample ones" + # Not a conflict when nothing is being measured. + cp_check "params.variantCall.maxDepth=500" "params.capBAM.maxDepth=0" + assert_eq "" "$(cp_verdict variantCall.maxDepth)" \ + "with capBAM off there is no measured ceiling to override" +} + +test_check_parameters_refuses_a_memory_size_carrying_a_unit() { + helpers_sandbox + cp_check "params.fastqc.memory=2G" + assert_eq "FAIL NOT A PLAIN NUMBER" "$(cp_verdict fastqc.memory)" "FastQC rejects a unit" + assert_status 1 "$CP_RC" "so the run would fail later; better here" +} + +# A FINDING IS ONE LINE OF FIVE TAB-SEPARATED FIELDS, because both callers read it with `read`. +# An explanation running onto a second line, or a stray tab inside one, would silently shift every +# field after it. +test_check_parameters_emits_one_line_of_five_fields_per_finding() { + helpers_sandbox + cp_check "params.variantCall.mpileupOptions='-B -C 15 -q 30 -Q 30'" \ + "params.filterFalsePositives.sampleThreshold=1.5" "params.fastqc.memory=2G" + assert_eq "3" "$(printf '%s\n' "$CP_FIND" | wc -l)" "three findings, three lines" + assert_eq "" "$(printf '%s\n' "$CP_FIND" | awk -F'\t' 'NF != 5 { print NR }')" \ + "every line should hold exactly five fields" + assert_eq "" "$(printf '%s\n' "$CP_FIND" | awk -F'\t' '$1 !~ /^(FAIL|WARN|NOTE)$/ { print NR }')" \ + "and every level should be one of the three" +} diff --git a/test/suites/04_pipeline.sh b/test/suites/04_pipeline.sh index cb275e4..4cd2855 100644 --- a/test/suites/04_pipeline.sh +++ b/test/suites/04_pipeline.sh @@ -3,6 +3,7 @@ # cost: pipeline # env: pipeline # covers: poolseqflow.nf scripts/ nextflow.config bin/ +# covers: parameters.config.template # # One full run is shared by the cases that only inspect its results, because a run costs # about a minute and there is no reason to pay for it more than once. Cases that need a @@ -880,10 +881,10 @@ multirun_run() { # guard. Its own sandbox, never a copy of the shared one. write_sandbox_config "$MULTIRUN_SB" "s|^ multiRun .*| multiRun = true|" cat > "$MULTIRUN_SB/main/runs.csv" <<'CSV' -RunID,vcffilter.minQUAL,poolSize,annotate -inherit,,, -filter,1000,, -plain,,25,false +RunID,vcffilter.minQUAL,poolSize,annotate,vcffilter.dropZeroDepth +inherit,,,, +filter,1000,,, +plain,,25,false,false CSV MULTIRUN_STATUS=$(run_pipeline "$MULTIRUN_SB") [ "$MULTIRUN_STATUS" = "0" ] @@ -1300,134 +1301,121 @@ test_a_truncated_depth_histogram_names_the_parameter() { "and never send the user into the installation" } -# THE COUNTS BEHIND EVERY PUBLISHED FREQUENCY. Until E4a the depth table existed only inside a -# pipe and was thrown away, so nothing anywhere said how many reads a frequency was computed -# from. The analysis layer needs it for n_eff, and a reader needs it to judge a frequency at all. -test_the_depth_table_is_published_beside_the_frequencies() { - needs_run || return - local f="$PIPELINE_SB/store/Output/Frequencies" - assert_file "$f/Test_snp_depth.tsv" "the SNP depth table should be published" - assert_file "$f/Test_indel_depth.tsv" "the INDEL depth table should be published" - - # THE TWO TABLES ARE NOT THE SAME SHAPE, and reading them as if they were is the mistake - # this case exists to stop. The depth table is one row per SITE, each cell holding - # comma-separated read counts in REF-then-ALT order; depth2freq.awk expands that into one - # row per ALLELE. Same columns, different granularity. - assert_eq "$(head -1 "$f/Test_snp_freq.tsv" | tr '\t' '\n' | wc -l)" \ - "$(head -1 "$f/Test_snp_depth.tsv" | tr '\t' '\n' | wc -l)" \ - "depth and frequency tables should have the same column count" - assert_contains "$(head -1 "$f/Test_snp_depth.tsv")" "TOTAL_AD" "and the same header" - - # Cells are counts, not frequencies: the first sample column of the first row is a list of - # whole numbers. A frequency here would mean the tee landed after the conversion. - local cell; cell=$(sed -n '2p' "$f/Test_snp_depth.tsv" | cut -f6) - printf '%s' "$cell" | grep -qE '^[0-9]+(,[0-9]+)*$' \ - || fail_case "depth cells should be comma-separated read counts, got '$cell'" - - # Every allele of every site reached the frequency table: one frequency row per allele, - # where a site's allele count is REF plus its comma-separated ALTs. - local alleles freq_rows - alleles=$(awk -F'\t' 'NR > 1 { n = split($4, a, ","); total += n + 1 } END { print total }' \ - "$f/Test_snp_depth.tsv") - freq_rows=$(( $(wc -l < "$f/Test_snp_freq.tsv") - 1 )) - assert_eq "$alleles" "$freq_rows" \ - "the two tables should describe the same sites and the same alleles" -} - -# The depth table rides along with the frequency table rather than costing a task of its own. -test_the_depth_table_costs_no_extra_task() { - needs_run || return - # Two tables, SNP and INDEL, from the two CalculateFrequencies tasks. - assert_eq "2" "$(task_count "$PIPELINE_SB" "VCF2Frequencies:CalculateFrequencies")" \ - "one task per split file, writing both its depth table and its frequency table" -} - -# READS MAY SIT IN SUBFOLDERS OF Data/, or directly in it, or both at once. +# AN EMPTY RESULT USED TO BE A SUCCESSFUL RUN. Every stage from bcftools call to the published +# table exits 0 over no records -- measured, all seven of them, including `vcftools --recode`, +# which was the one link nobody had checked. bcftools writes a well-formed header and no rows; +# depth2freq.awk rewrites the header and its body loop never runs. What the user got was a +# frequency table holding column names and nothing else, from a run that reported success. # -# THIS RUNS STEP 2, NOT STEP 0, because step 0 cannot answer it. Step 0 finds reads with -# `find -name`, which has always recursed, so a nested layout passed its sample match long -# before the reads could actually be staged - the divergence this change closes. What had to -# move is `params.reads`, which readPairChannel globs, and only a step that consumes that -# channel exercises it. Written against step 0 first, where reverting the glob left the case -# passing. -test_reads_are_found_in_subfolders_of_data() { - have_tools || { skip_case "no conda environment"; return; } - [ "${TEST_FAST:-0}" = "1" ] && { skip_case "--fast"; return; } - local sb status s - sb=$(make_pipeline_sandbox "nested-reads") - write_sandbox_config "$sb" - - # One sample in a folder of its own, one two levels down, the rest left flat. - mkdir -p "$sb/main/Data/TestSample1" "$sb/main/Data/batch2/TestSample2" - mv "$sb/main/Data/TestSample1_R"*.fq.gz "$sb/main/Data/TestSample1/" - mv "$sb/main/Data/TestSample2_R"*.fq.gz "$sb/main/Data/batch2/TestSample2/" - - status=$(run_trim_only "$sb") - assert_status 0 "$status" "step 2 should complete over a mixed layout; see $sb/run.out" - - # Every sample trimmed, wherever its reads were: the nested two are the point, and the - # flat ones prove `**` still matches at depth zero. - for n in 1 2 3 4 5 6; do - s="TestSample$n" - assert_count 2 "$(find "$sb/main/Utilized" -name "${s}_R[12]_clipped.fq.gz" 2>/dev/null | wc -l)" \ - "$s should have been trimmed whatever folder its reads were in" - done -} +# THE THREE CASES BELOW ENTER THAT STATE BY THE THREE DIFFERENT DOORS, because the guards are +# three separate checks and one case would only prove one of them. Each asserts the parameter +# that caused it is named, which is the whole point of refusing rather than publishing. +# +# A fresh sandbox each time, never the shared run: every stage of step 7 skips when a later +# artifact already exists, so a project that has published tables never re-enters the chain. -# HIDDEN FOLDERS ARE EXCLUDED FROM THE READ CHANNEL, not only from step 0's search. +# A PILEUP THAT REACHES bcftools call WITH NOTHING IN IT. varQualMin 500 is the door: no read +# carries a mapping quality anywhere near it, so -q 500 skips every one and mpileup emits no +# records at all while exiting 0. Measured on the fixture's own ready BAMs: 0 records against 123 +# at the default. # -# THIS RUNS STEP 2, because step 0 cannot answer it. The two prune independently - step 0 with -# `find -prune`, the channel with a filter in readPairChannel - and a case that runs only step 0 -# passes with the channel's filter deleted outright. Measured: it did. +# WHY BOTH SETTINGS MOVE. scaleMapQ has to stay ABOVE varQualMin or bin/check_parameters.sh refuses +# the pair at step 0 and the run never reaches step 6. 600 against 500 satisfies that rule (it +# warns, which is right: it does emit records for ordinary values) while still killing every read, +# because -C only ever downgrades a mapping quality and never raises one. So this case needs the +# two layers to be independent, and fails if either stops working. # -# `.snapshot` is NetApp's, exposed read-only inside every directory on much HPC storage and -# holding a copy of every file per snapshot. Unfiltered, the channel emits the sample twice and -# trims it twice, with both runs writing the outputs named after it. -test_hidden_folders_are_excluded_from_the_read_channel() { - have_tools || { skip_case "no conda environment"; return; } - [ "${TEST_FAST:-0}" = "1" ] && { skip_case "--fast"; return; } - local sb status samples - sb=$(make_pipeline_sandbox "hidden-channel") - write_sandbox_config "$sb" - samples=$(find "$sb/main/Data" -name '*_R1.fq.gz' | wc -l) - - mkdir -p "$sb/main/Data/.snapshot/nightly" - cp "$sb/main/Data/TestSample1_R1.fq.gz" "$sb/main/Data/.snapshot/nightly/" - cp "$sb/main/Data/TestSample1_R2.fq.gz" "$sb/main/Data/.snapshot/nightly/" - - status=$(run_trim_only "$sb") - assert_status 0 "$status" "the snapshot copy must not disturb the run; see $sb/run.out" +# TWO EARLIER ROUTES DID NOT WORK, recorded so they are not retried. scaleMapQ = 15 is the same +# failure by a shorter path and step 0 now refuses it. baseQualMin = 60 does nothing whatsoever: +# measured, 123 records at -Q 30 and 123 at -Q 60, indels included. +test_a_call_set_with_no_records_stops_the_run() { + if ! have_tools; then skip_case "no conda environment"; return; fi + if [ "${TEST_FAST:-0}" = "1" ]; then skip_case "--fast"; return; fi + local sb status out + sb=$(make_pipeline_sandbox "empty-callset") + write_sandbox_config "$sb" \ + 's|^ scaleMapQ .*| scaleMapQ = 600|' \ + 's|^ varQualMin .*| varQualMin = 500|' + status=$(run_pipeline "$sb") + assert_status 1 "$status" "an empty call set must stop the run; see $sb/run.out" + out=$(cat "$sb/run.out") + assert_contains "$out" "produced no records" "the refusal should say what happened" + assert_contains "$out" "The pileup ran as:" "and show the options it ran with" + assert_not_contains "$out" "in scripts/6_variant_call.nf" \ + "and never send the user into the installation" + # NOTHING PUBLISHED. The point of refusing here is that a later run with a sane value finds + # no empty artifact to skip past. + assert_no_file "$sb/store/Output/Frequencies/Test_snp_freq.tsv" \ + "no frequency table should have been written" + # AND IT REACHES Logs/. The refusal exits before the end of the process script, where the + # log is normally copied out of the task directory, so without archive_log the one run that + # most needs explaining is the one that leaves Logs/ empty for that step. + assert_contains "$(cat "$sb/store/Logs/6_variant_call/"*VariantCall*.log 2>/dev/null)" \ + "produced no records" "the refusal must reach Logs/, not only the terminal" +} - # One trim per sample, and the count is what shows the copy was excluded: unfiltered, the - # channel emits TestSample1 twice and this reads one higher than the sample count. - assert_count "$samples" "$(task_count "$sb" TrimQcClip:TrimReads)" \ - "each sample should be trimmed once, the hidden copy not at all" +# The depth and quality filter is applied to the whole SITE, so one sample under the floor +# removes it for every sample. A minDP above every sample's depth removes all of them. +test_a_depth_filter_that_removes_every_site_stops_the_run() { + if ! have_tools; then skip_case "no conda environment"; return; fi + if [ "${TEST_FAST:-0}" = "1" ]; then skip_case "--fast"; return; fi + local sb status out + sb=$(make_pipeline_sandbox "empty-after-depth") + write_sandbox_config "$sb" 's|^ minDP .*| minDP = 100000|' + status=$(run_pipeline "$sb") + assert_status 1 "$status" "a total depth wipeout must stop the run; see $sb/run.out" + out=$(cat "$sb/run.out") + assert_contains "$out" "no site survived the depth and quality filter" "saying what happened" + assert_contains "$out" "minDP 100000" "and reporting the value it used" + assert_no_file "$sb/store/Output/Frequencies/Test_snp_freq.tsv" \ + "no frequency table should have been written" + assert_contains "$(cat "$sb/store/Logs/7_vcf2freq/"*DepthAndQuality*.log 2>/dev/null)" \ + "no site survived" "the refusal must reach Logs/, not only the terminal" } -# `reads` IN parameters.config IS NOT CONSULTED, and this pins that. +# THE CROSS-SAMPLE GUARD HAS NO CASE, and this says why so nobody writes one that cannot work. +# It refuses a run where no site survives filterFalsePositives.sh, and the only configuration that +# reached it was sampleThreshold above 1 -- which bin/check_parameters.sh now refuses at step 0, +# before the pipeline starts. Measured while looking for another door: at sampleThreshold 1.0 with +# poolSize 2, 39 of 135 sites still survive, and sensitivity is not the binding clause at fixture +# depth, so no value step 0 permits empties that filter. # -# deriveRunPaths() assigns p.reads directly - not through fill(), which is what respects a user -# setting - so the value a run globs is always the one it computes, into each variant map. The -# copy in parameters.config exists so a reader can see how the path is built. +# The guard stays. What it now protects against is DATA rather than configuration: a cohort whose +# variants are shared by too few pools. Nothing in a fixture-driven suite can produce that without +# a fixture built for it, and it is not worth one for a guard whose message is three lines. + +# vcffilter.dropZeroDepth, which had no coverage at any level. 00_static proves it is DECLARED in +# stepParameterMap() -- delete it from scripts/variants.nf and a static case fails in seconds -- +# and 05_guards proves flipping it invalidates a finished project. Neither watches the expression +# it actually builds, and that ternary in scripts/7_vcf2freq.nf is the thing that can invert. # -# Worth a case because the asymmetry is invisible: the knobs beside it in the same file ARE -# respected, since deriveInto() fills them only when absent. And because it cost real time - -# the `**` was added to parameters.config.template first and nothing changed, which reads as -# the feature not working rather than as the line not being read. -test_the_reads_setting_in_the_config_is_not_consulted() { - have_tools || { skip_case "no conda environment"; return; } - [ "${TEST_FAST:-0}" = "1" ] && { skip_case "--fast"; return; } - local sb status samples - sb=$(make_pipeline_sandbox "reads-override") - samples=$(find "$sb/main/Data" -name '*_R1.fq.gz' | wc -l) - # A path that matches nothing anywhere. If it were read, the run would find no reads at all. - write_sandbox_config "$sb" \ - 's|^ reads .*| reads = "/nonexistent/nothing/**_R{1,2}.fq.gz"|' - assert_contains "$(cat "$sb/main/parameters.config")" "/nonexistent/nothing/" \ - "the sandbox config should carry the bogus path, or this case proves nothing" - - status=$(run_trim_only "$sb") - assert_status 0 "$status" "the derived value should be used regardless; see $sb/run.out" - assert_count "$samples" "$(task_count "$sb" TrimQcClip:TrimReads)" \ - "every sample should still be trimmed, from the path the resolver computed" +# WHY THIS IS NOT A BEHAVIOR CASE, measured rather than assumed. Observing the toggle needs a +# site where one sample has no reads, and this fixture cannot hold one: every sample carries ~75x +# over the whole reference, and test/data/vcf/README.md says the same of the called VCF. Thinning +# one sample to 250 pairs was tried and produces NO coverage at all, not low coverage -- +# `[M::mem_pestat] skip orientation FR as there are not enough pairs`, so bwa estimates no insert +# size, nothing is flagged properly paired, and step 4's 0x2 filter discards the lot. That is the +# trap make_fixture.py's own docstring warns about. The two requirements are in direct conflict: +# bwa needs thousands of pairs to pair-estimate, and a zero-depth cell needs almost none locally. +# A regional-gap sample satisfies both and is new committed fixture data; see test/README.md. +# +# So this rides the multi-run instead, where `plain` already diverges at step 7, and asserts the +# expression each run was handed. It costs no extra pipeline run. +test_dropzerodepth_decides_the_depth_expression() { + needs_multirun || return + local on off + on=$(cat "$MULTIRUN_SB/store/Logs/inherit/7_vcf2freq/7_s3_DepthAndQualityFilter_"*.log) + off=$(cat "$MULTIRUN_SB/store/Logs/plain/7_vcf2freq/7_s3_DepthAndQualityFilter_"*.log) + + assert_contains "$on" "FMT/DP<20" "the inheriting run filters on the configured depth" + assert_contains "$on" "FMT/DP==0" "and dropZeroDepth=true adds the zero term" + assert_contains "$off" "FMT/DP<20" "the diverging run keeps the same depth floor" + assert_not_contains "$off" "FMT/DP==0" "and dropZeroDepth=false must not add the zero term" + + # Redundant above minDP 1, which is why this is safe to set on a run whose numbers other + # cases assert: the site sets are identical and only the expression differs. + local on_rows off_rows + on_rows=$(awk 'END{print NR}' "$MULTIRUN_SB/store/Output/inherit/Frequencies/Test_snp_freq.tsv") + off_rows=$(awk 'END{print NR}' "$MULTIRUN_SB/store/Output/plain/Frequencies/Test_snp_freq.tsv") + [ "$on_rows" -gt 1 ] && [ "$off_rows" -gt 1 ] || fail_case "both runs should publish sites (got $on_rows and $off_rows rows)" } diff --git a/test/suites/05_guards.sh b/test/suites/05_guards.sh index f07ea22..69547a2 100644 --- a/test/suites/05_guards.sh +++ b/test/suites/05_guards.sh @@ -6,6 +6,7 @@ # covers: scripts/metadata.nf bin/parse_metadata.py bin/parse_multirun.py # covers: bin/classify_manifest.sh # covers: poolseqflow.nf +# covers: parameters.config.template nextflow.config # # These run step 0 alone rather than the whole pipeline, so they are cheap. What they are # about is the distinction the parameter check draws between a value the user changed and a @@ -98,6 +99,34 @@ test_a_changed_parameter_value_fails_the_run() { assert_contains "$report" "STATUS=FAIL" "the stage should record a failure" } +# vcffilter.dropZeroDepth is the FIRST parameter added since v3.0.0, so it is the first whose +# tracking had never been exercised. What this asserts is that it is ANALYSIS-AFFECTING: it +# reaches the recorded manifest and the comparison names it, so a project that flips it is +# stopped rather than quietly mixing results produced under the other answer. +# +# WHAT IT DOES NOT ASSERT, checked rather than assumed: this says nothing about step 7's artifact +# identity. analysisParams() in 0_verify_environment.nf is an EXCLUSION list -- anything not named +# in skipKey or skipPrefix counts -- so the guard fires whether or not the parameter appears in +# stepParameterMap(). Measured: deleting it from scripts/variants.nf:44 leaves this case passing. +# The declaration is 00_static's business; what the toggle DOES is the expression case in +# 04_pipeline. +# +# So the thing that would break this is an edit to that exclusion list -- `vcffilter.` added to +# skipPrefix, or this key added to skipKey. capBAM.histogramMax is already excluded on exactly +# that reasoning, which is why the list is a plausible place for a wrong entry. +test_flipping_dropzerodepth_fails_the_run() { + guards_ready || return + write_sandbox_config "$GUARD_SB" 's|^ dropZeroDepth .*| dropZeroDepth = false|' + local status report + status=$(run_verify_only "$GUARD_SB") + report=$(guard_report) + assert_status 1 "$status" "flipping dropZeroDepth should fail the run" + assert_contains "$report" "parameters.config has CHANGED" "should name the file that moved" + assert_contains "$report" "dropZeroDepth" "should name the parameter" + assert_contains "$report" "was true" "should show the recorded value" + assert_contains "$report" "now false" "should show the new value" +} + # THE EXECUTION DEFAULTS A PROJECT MAY REPLACE, and until E6g it could replace none of them. # # nextflow.config's includeConfig sat ABOVE its own assignments and Nextflow is later-wins, so @@ -147,6 +176,33 @@ OVERRIDE "a project must not be able to move the work directory out of mainDir" } +# A DEFAULT HAS TO SURVIVE THE PARAMETER BEING ABSENT, or it is not a default. Step 7 reads +# vcffilter.dropZeroDepth in a ternary and Nextflow resolves an absent key to null, which a +# ternary reads as false -- so a config written before the parameter existed ran as `false` +# where the template ships `true`, with nothing saying so. Measured through the real engine +# before nextflow.config resolved it: RAW=[null], and the expression came out without the zero +# term. +# +# The second half is what keeps the fix from being worse than the hole: the resolution sits +# below the include and tests containsKey, so a project that says false still gets false. A +# plain assignment there would win silently over every project on the machine. +test_an_absent_parameter_resolves_to_its_default() { + if [ "${TEST_FAST:-0}" = "1" ]; then skip_case "--fast"; return; fi + local sb flat + sb=$(make_pipeline_sandbox "absent-default") + + write_sandbox_config "$sb" '/dropZeroDepth/d' + flat=$(sandbox_config_flat "$sb") + [ -n "$flat" ] || { fail_case "nextflow config produced nothing for $sb"; return; } + assert_contains "$flat" "params.vcffilter.dropZeroDepth = true" \ + "a config that never sets it must resolve to the shipped default" + + write_sandbox_config "$sb" 's|^ dropZeroDepth .*| dropZeroDepth = false|' + flat=$(sandbox_config_flat "$sb") + assert_contains "$flat" "params.vcffilter.dropZeroDepth = false" \ + "and a project that sets it must still win" +} + # THE HELPERS' OWN DEPENDENCIES ARE VERIFIED LIKE ANY OTHER TOOL. bin/atomic_mv.sh copies an # artifact with rsync, compares the copy against its source with diff, and only then removes the # source; find lists what a results folder holds. All three are called by bare name from bin/, @@ -696,7 +752,11 @@ TABLE assert_contains "$out" "RUN trim trim_galore.quality=30" "the row's own value" assert_contains "$out" "RUN trim trim_galore.options=--fastqc --paired --retain_unpaired -q 30 " \ "trim_galore.quality must re-derive options" - assert_contains "$out" "RUN depth variantCall.mpileupOptions=-B -C 50 -q 30 -Q 30 -d 4000 -a AD,DP,SP,INFO/AD -Ou" \ + # -C is the TEMPLATE'S scaleMapQ, which this row does not set: it moved from 50 to 100 when + # the default was measured, and this expectation did not, so the case failed from then until + # the suite was next run in full. The `pinned` row below keeps 50 deliberately, as a value + # that differs from the default and so cannot pass by coincidence. + assert_contains "$out" "RUN depth variantCall.mpileupOptions=-B -C 100 -q 30 -Q 30 -d 4000 -a AD,DP,SP,INFO/AD -Ou" \ "variantCall.maxDepth must re-derive mpileupOptions" # A row setting a DERIVED value directly wins, even against its own input in the same row. @@ -1323,3 +1383,68 @@ test_a_read_pattern_matching_nothing_is_refused() { assert_contains "$out" "No FASTQ files found" "saying nothing matched" assert_contains "$out" "Expected pattern" "and what it was looking for" } + +# THE PARAMETER RULES RUN IN STEP 0, not only in `PoolSeqFlow check project`. They are +# bin/check_parameters.sh's, with their own cases in 03_helpers; what these two assert is that step +# 0 calls it, renders what it says, and lets the LEVEL decide whether the run continues. +# +# A FRESH sandbox, never the baseline one: changing a parameter in the baseline would also trip the +# change guard, and then there would be two reasons for one failure and no way to tell them apart. +# On a first run there is nothing recorded yet, so the parameter rules speak alone. +verify_fresh_config() { # name sed-expression + local sb + sb=$(make_pipeline_sandbox "$1") + write_sandbox_config "$sb" "$2" + VF_STATUS=$(run_verify_only "$sb") + VF_REPORT=$(cat "$sb/store/Output/Reports/0_verify_environment.txt" 2>/dev/null) + VF_SB="$sb" +} + +# scaleMapQ under varQualMin makes the pileup emit nothing. Caught here, it costs a step-0 run; +# uncaught, it costs alignment and calling first and then fails in step 6. +test_a_parameter_that_would_produce_nothing_stops_the_run() { + if ! have_tools; then skip_case "no conda environment"; return; fi + if [ "${TEST_FAST:-0}" = "1" ]; then skip_case "--fast"; return; fi + verify_fresh_config "params-fail" 's|^ scaleMapQ .*| scaleMapQ = 15|' + assert_status 1 "$VF_STATUS" "a parameter that produces nothing must stop the run; see $VF_SB/run.out" + assert_contains "$VF_REPORT" "DISCARDS EVERY READ" "the report should carry the verdict" + assert_contains "$VF_REPORT" "variantCall.scaleMapQ" "against the parameter responsible" + assert_contains "$VF_REPORT" "15 is below varQualMin 30" "with both numbers" + # THE EXPLANATION IS ASSERTED BY ITS INDENT, not by its words. It is wrapped with `fold -s`, + # so no multi-word phrase survives reliably: the first attempt looked for "Raise scaleMapQ + # above varQualMin" and the fold landed between "Raise" and "scaleMapQ". The deeper indent is + # what says a wrapped explanation was printed at all, and it holds however the prose changes. + assert_contains "$VF_REPORT" "RUN PARAMETERS: " \ + "the explanation should be folded in under the verdict" + assert_contains "$VF_REPORT" "rather than after the compute it would waste" \ + "and say why it stopped here" + # NOTHING ALIGNED. The point of catching it at step 0 is that no compute is spent. + assert_no_file "$VF_SB/store/Output/VCF/Test.vcf" "no VCF should have been produced" +} + +# A WARNING IS NOT A FAILURE, and the difference has to survive the trip through step 0. scaleMapQ +# 35 against varQualMin 30 is severe and measured at 6% of the sites, but it does produce records, +# so the run is the user's to make. +test_a_parameter_warning_does_not_stop_the_run() { + if ! have_tools; then skip_case "no conda environment"; return; fi + if [ "${TEST_FAST:-0}" = "1" ]; then skip_case "--fast"; return; fi + verify_fresh_config "params-warn" 's|^ scaleMapQ .*| scaleMapQ = 35|' + assert_status 0 "$VF_STATUS" "a warning must not stop the run; see $VF_SB/run.out" + assert_contains "$VF_REPORT" "SEVERE AT 35" "but it should still be said" + assert_not_contains "$VF_REPORT" "DISCARDS EVERY READ" "and not reported as the other thing" +} + +# SILENCE FOR THE SHIPPED DEFAULTS. The case most likely to rot: a rule added with a wrong +# threshold makes every ordinary run noisy, and a user who sees a warning on a default stops +# reading warnings at all. +test_the_shipped_defaults_raise_no_parameter_finding() { + if ! have_tools; then skip_case "no conda environment"; return; fi + if [ "${TEST_FAST:-0}" = "1" ]; then skip_case "--fast"; return; fi + verify_fresh_config "params-clean" 's|^ poolSize .*| poolSize = 100|' + assert_status 0 "$VF_STATUS" "the defaults should verify; see $VF_SB/run.out" + local level + for level in FAIL WARN NOTE; do + assert_not_contains "$VF_REPORT" "RUN PARAMETERS: $level" \ + "the template's own values should raise no $level" + done +} diff --git a/test/suites/06_dryrun.sh b/test/suites/06_dryrun.sh index 469ae7e..48327ce 100644 --- a/test/suites/06_dryrun.sh +++ b/test/suites/06_dryrun.sh @@ -3,6 +3,7 @@ # cost: jvm # env: pipeline # covers: dryrun.nf scripts/variants.nf scripts/resolve_parameters.nf +# covers: parameters.config.template # # What these are about is the promise the subcommand makes. A preview verifies the project and # builds the directory tree it would fill, and it leaves the project able to say "this has never diff --git a/test/suites/10_analysis_verify.sh b/test/suites/10_analysis_verify.sh index c9d1068..b14084c 100644 --- a/test/suites/10_analysis_verify.sh +++ b/test/suites/10_analysis_verify.sh @@ -4,6 +4,7 @@ # env: analysis # covers: analysis/0_verify_analysis.nf analysis/lib/nf/citations.nf # covers: analysis.nf +# covers: parameters.config.template # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # diff --git a/test/tools/freq_corpus.py b/test/tools/freq_corpus.py index 3745c9c..5291b68 100644 --- a/test/tools/freq_corpus.py +++ b/test/tools/freq_corpus.py @@ -62,9 +62,11 @@ THE INDEL TABLE is separate because the site counts report SNPs and indels separately, and because diversity is computed over the SNP table alone - a gate the module states. -NO CELL IS ZERO-DEPTH. vcffilter.minDP removes a site where ANY sample falls below the depth, -so a published table cannot hold one; the NA path in n_eff() is exercised by the library's own -unit tests instead. +NO CELL IS ZERO-DEPTH. Reaching one takes both vcffilter.minDP at zero and +vcffilter.dropZeroDepth off, which is two deliberate edits away from any default, so this +corpus does not carry one and the NA paths in n_eff(), site_diversity() and +allele_frequencies() are exercised by the libraries' own unit tests instead. What a published +table does with such a cell is bin/depth2freq.awk's answer and is covered in 03_helpers. """ import itertools