From d8b7ceee27fc561fc0d9067aca7da1d25d63dc90 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Wed, 7 Oct 2026 09:18:16 -0400 Subject: [PATCH 1/6] Version bump reverted --- .../changelog-3.3.0-prose.md | 51 +++++++++++++++ CHANGELOG.md | 65 ------------------- PoolSeqFlow | 4 +- modules/association/manifest.json | 2 +- modules/basicstats/manifest.json | 2 +- modules/lib/allele_frequencies/manifest.json | 2 +- modules/lib/n_eff/manifest.json | 2 +- modules/lib/nei_distance/manifest.json | 2 +- modules/mds/manifest.json | 2 +- nextflow.config | 2 +- 10 files changed, 60 insertions(+), 74 deletions(-) create mode 100644 .claude/development-notes/changelog-3.3.0-prose.md diff --git a/.claude/development-notes/changelog-3.3.0-prose.md b/.claude/development-notes/changelog-3.3.0-prose.md new file mode 100644 index 0000000..1901f78 --- /dev/null +++ b/.claude/development-notes/changelog-3.3.0-prose.md @@ -0,0 +1,51 @@ +# The 3.3.0 release notes, kept while the bump is reverted + +**Written 2026-10-07 against `main` at `6550cb5`.** The prose of the `## [3.3.0]` section of `CHANGELOG.md` as Z left it, from its heading down to `### Commits`, copied verbatim below. + +## Why it is here + +v3.3.0 was tagged at `babb475` on 2026-10-07 and never released. `release.yml` stopped at its analysis version gate: `raise-module-floors.sh` had moved the `environment` floor of six manifests (association, basicstats, mds, allele_frequencies, n_eff, nei_distance) from 3.0.0 to 3.3.0, and those edits landed in the version-bump commit, which moved none of their versions. No GitHub release was created, and the tag was deleted locally and on origin. + +A review of the new `publish-module.sh --all-pending` then found defects, and Z decided they are to be fixed before releasing: "This is more serious." So the bump is reverted on `dev`, and `bump-version.sh --revert` deletes this section along with it. + +The revert does not fix what stopped the release. Raising the floors at the next bump puts the same edit in the same kind of commit, and the gate fails the same way until `raise-module-floors.sh` or the gate changes. + +## What to check when it goes back + +The next bump writes its own heading, date and commit list, and this prose goes back above `### Commits`. + +- The module versions under Analysis modules are the ones prepared for this cycle. A module bumped again before the release carries a new version, and its line has to name that one. +- Anything that lands on `dev` before the next bump and changes what a user sees needs a line of its own. `publish-module.sh --all-pending` is a maintainer tool and does not. + +## The section + +```markdown +## [3.3.0] - 2026-10-06 + +**The shallowest library no longer has to decide which sites the whole cohort keeps.** Until now one pool below `vcffilter.minDP` removed a site for every pool. `vcffilter.keepLowDepthAsZero` keeps the site instead and writes that pool's reads there as unread. It is off by default, so nothing changes until you turn it on. + +**Two things to read before upgrading**, both under Changed: the default of `variantCall.scaleMapQ` moved from 50 to 100, and a migrated `parameters.config` keeps its 50; and `init_multi` and `uninstall_all` are now two words. + +### Added + +- **`vcffilter.keepLowDepthAsZero` and `vcffilter.minSamples`.** With the switch on, every cell below `minDP` (one pool's reads at one site) is written as unread, which the depth table carries as zeros and the frequency table as `NA`, and a site is kept when at least `minSamples` of its cells reach `minDP`. `minSamples` defaults to 2 and runs from 1 to the number of pools with reads; anything else is refused before the run starts. Every read still counts toward deciding that an allele exists, because the false-positive filter runs first; only the cells at the floor count toward measuring its frequency. The manual's depth and quality filter section has the details, including what it does to `TOTAL_AD`. +- **Settings that would make a run produce nothing are refused at step 0**, and `PoolSeqFlow check project` reports the same findings before you run. A `scaleMapQ` below `varQualMin` discards every read, a `sampleThreshold` above 1 removes every site, a `ploidy` or `poolSize` below 1 breaks every detection limit, and a `fastqc.memory` with a unit is refused by FastQC. Settings that work but cost more than they look, or change what a number means, are warned about and the run continues. The manual lists them all. +- **Files saved on Windows.** A `metadata.csv` or `runs.csv` that Excel saved as "CSV UTF-8" is read as it is, and a UTF-16 one is refused with what to save it as instead. A `parameters.config` that begins with a byte-order mark is named by `check project`, with the command that removes it; Nextflow on its own refuses the file over a character nothing displays. +- **Native zsh completion**, with a description beside each command. The installer prints the one line `~/.zshrc` needs. + +### Changed + +- **`variantCall.scaleMapQ` defaults to 100, where it was 50.** It is the `-C` that lowers the mapping quality of reads with many mismatches. Measured on real pools, 100 removes the sites that are artifacts (Ti/Tv 0.848, nearly four times the soft-clip bias of the sites kept), while 50 removed five times as many and took real variants with them. 50 also lowered the frequencies it kept, by 0.035 on average against 0.008 at 100, and by 0.081 in pools between 0.50 and 0.75. **A migrated `parameters.config` keeps 50**: `migrate_config` carries your value across, as it does every setting that still exists, and reports it under `Kept your value`. Set it to 100 before the first run under 3.3.0. The manual has the full measurements. +- **`init_multi` is `init multi`, and `uninstall_all` is `uninstall all`.** The old spellings stop with a message giving the new one. +- **A pool with no reads at a site is `NA` in the frequency table**, where it was 0, because a frequency over zero reads does not exist. By default no such cell reaches a table: the depth filter now also drops a site with an unread pool at `minDP` 0, which 3.2.0 kept and published as 0. With `keepLowDepthAsZero` on, `NA` is what an unread cell reads. +- **Every step writes its log as it runs.** A step that fails, or is killed, leaves what it had reached in `Logs/`. Before this, a step copied its log there only at the end, so a failure lost exactly the log that would have explained it. Each attempt starts with a line naming the run, the session and the time. +- **Capping is about seven times faster on real data**, with identical output. +- **A call set or filter that leaves nothing stops the run** and says which settings to look at, instead of finishing as though it had succeeded. Nothing is published from it. +- **Environments.** FastQC 0.12.1 to 0.13.0, Perl 5.32.1 to 5.44.0, pandoc 3.11 to 3.12, R's future and doFuture to their next minor versions, and patch updates beneath them. bwa, samtools, bcftools, cutadapt, Trim Galore and snpEff did not move. + +### Analysis modules + +- **association `20261006.001`.** Two ways it published `perm_p` 0, which no permutation p can be. A site where any unit had no reads came out 0, and `fdr_p` with it; in 3.2.0 that took `minDP` 0. And the strongest sites in a table, far beyond any rearrangement, failed to count themselves and came out 0 too. Each site is now tested over the units it was read in, and not tested when fewer than three were read. `design_floor` is 1/n!, where it said 2/n!, so four units can reach 0.042. +- **mds `20261006.002`.** A pair of pools averaged over fewer than 30 shared sites is flagged, in `distance.tsv`, on the console and under the plot. In simulation, a distance near 0.02 measured over 30 shared sites is off by about half itself, and real sites are noisier. +- **basicstats `20261006.001`.** A pool's depth summaries cover only the sites it was read at, and `depth.tsv` and `diversity.tsv` count the rest in a new `unmeasured` column. A site with no reads for a pool was averaged in as a depth of 0, and one such site was enough to turn that pool's harmonic depth into 0 and its effective sample size into NA; in 3.2.0 that took `minDP` 0. +``` diff --git a/CHANGELOG.md b/CHANGELOG.md index 0407370..b2c67c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,70 +10,6 @@ In practice a version number says what upgrading will cost you: a third number i --- -## [3.3.0] - 2026-10-06 - -**The shallowest library no longer has to decide which sites the whole cohort keeps.** Until now one pool below `vcffilter.minDP` removed a site for every pool. `vcffilter.keepLowDepthAsZero` keeps the site instead and writes that pool's reads there as unread. It is off by default, so nothing changes until you turn it on. - -**Two things to read before upgrading**, both under Changed: the default of `variantCall.scaleMapQ` moved from 50 to 100, and a migrated `parameters.config` keeps its 50; and `init_multi` and `uninstall_all` are now two words. - -### Added - -- **`vcffilter.keepLowDepthAsZero` and `vcffilter.minSamples`.** With the switch on, every cell below `minDP` (one pool's reads at one site) is written as unread, which the depth table carries as zeros and the frequency table as `NA`, and a site is kept when at least `minSamples` of its cells reach `minDP`. `minSamples` defaults to 2 and runs from 1 to the number of pools with reads; anything else is refused before the run starts. Every read still counts toward deciding that an allele exists, because the false-positive filter runs first; only the cells at the floor count toward measuring its frequency. The manual's depth and quality filter section has the details, including what it does to `TOTAL_AD`. -- **Settings that would make a run produce nothing are refused at step 0**, and `PoolSeqFlow check project` reports the same findings before you run. A `scaleMapQ` below `varQualMin` discards every read, a `sampleThreshold` above 1 removes every site, a `ploidy` or `poolSize` below 1 breaks every detection limit, and a `fastqc.memory` with a unit is refused by FastQC. Settings that work but cost more than they look, or change what a number means, are warned about and the run continues. The manual lists them all. -- **Files saved on Windows.** A `metadata.csv` or `runs.csv` that Excel saved as "CSV UTF-8" is read as it is, and a UTF-16 one is refused with what to save it as instead. A `parameters.config` that begins with a byte-order mark is named by `check project`, with the command that removes it; Nextflow on its own refuses the file over a character nothing displays. -- **Native zsh completion**, with a description beside each command. The installer prints the one line `~/.zshrc` needs. - -### Changed - -- **`variantCall.scaleMapQ` defaults to 100, where it was 50.** It is the `-C` that lowers the mapping quality of reads with many mismatches. Measured on real pools, 100 removes the sites that are artifacts (Ti/Tv 0.848, nearly four times the soft-clip bias of the sites kept), while 50 removed five times as many and took real variants with them. 50 also lowered the frequencies it kept, by 0.035 on average against 0.008 at 100, and by 0.081 in pools between 0.50 and 0.75. **A migrated `parameters.config` keeps 50**: `migrate_config` carries your value across, as it does every setting that still exists, and reports it under `Kept your value`. Set it to 100 before the first run under 3.3.0. The manual has the full measurements. -- **`init_multi` is `init multi`, and `uninstall_all` is `uninstall all`.** The old spellings stop with a message giving the new one. -- **A pool with no reads at a site is `NA` in the frequency table**, where it was 0, because a frequency over zero reads does not exist. By default no such cell reaches a table: the depth filter now also drops a site with an unread pool at `minDP` 0, which 3.2.0 kept and published as 0. With `keepLowDepthAsZero` on, `NA` is what an unread cell reads. -- **Every step writes its log as it runs.** A step that fails, or is killed, leaves what it had reached in `Logs/`. Before this, a step copied its log there only at the end, so a failure lost exactly the log that would have explained it. Each attempt starts with a line naming the run, the session and the time. -- **Capping is about seven times faster on real data**, with identical output. -- **A call set or filter that leaves nothing stops the run** and says which settings to look at, instead of finishing as though it had succeeded. Nothing is published from it. -- **Environments.** FastQC 0.12.1 to 0.13.0, Perl 5.32.1 to 5.44.0, pandoc 3.11 to 3.12, R's future and doFuture to their next minor versions, and patch updates beneath them. bwa, samtools, bcftools, cutadapt, Trim Galore and snpEff did not move. - -### Analysis modules - -- **association `20261006.001`.** Two ways it published `perm_p` 0, which no permutation p can be. A site where any unit had no reads came out 0, and `fdr_p` with it; in 3.2.0 that took `minDP` 0. And the strongest sites in a table, far beyond any rearrangement, failed to count themselves and came out 0 too. Each site is now tested over the units it was read in, and not tested when fewer than three were read. `design_floor` is 1/n!, where it said 2/n!, so four units can reach 0.042. -- **mds `20261006.002`.** A pair of pools averaged over fewer than 30 shared sites is flagged, in `distance.tsv`, on the console and under the plot. In simulation, a distance near 0.02 measured over 30 shared sites is off by about half itself, and real sites are noisier. -- **basicstats `20261006.001`.** A pool's depth summaries cover only the sites it was read at, and `depth.tsv` and `diversity.tsv` count the rest in a new `unmeasured` column. A site with no reads for a pool was averaged in as a depth of 0, and one such site was enough to turn that pool's harmonic depth into 0 and its effective sample size into NA; in 3.2.0 that took `minDP` 0. - -### Commits - -- (2a15aea) small change in releasing, test for linting -- (b96bfca) zsh completion added -- (c5bacb4) dos2unix for parameters, metadata, and runs files -- (51b649c) manual and documents passes -- (8d0447c) dropZeroDept toggle added -- (b6e0ce6) Comments shortened -- (ebd69af) Comments shortened -- (eedf598) scaleMapQ decisions are recorded -- (772db1f) Analysis modules are updated to be compatible with the zero depth sites change -- (85eb380) cap_depth is fixed for performance -- (bffc6a0) Empty vcf, depth, and freq files now produce error -- (ad85fb1) Logging reworked -- (aaeab55) Test suite for dropZeroDepth added -- (6746280) major allele to ref tests -- (f54a5ae) SampleID RG_Sample clarifications -- (8e8a078) parameter checks added to step0 -- (9b80793) config migrate is updated after dropZeroDepth -- (6127b53) Full release cycle rework -- (b96b818) wrapper tidiness -- (2b50c1c) Releasing improvements -- (87edc97) releasing cycle streamlined further -- (f2c8187) Manual rewrite -- (998c307) prep version is rescoped -- (1224341) Minor fix on how prep version works -- (d76ba2f) Prep for v3.3.0 -- (7419281) module publishing in release cycle is handled with a script now -- (1e36baf) mds site diversity limit established -- (e87e742) factorial calculation fixed -- (e04215b) dropZeroDepth replaced with keepLowDepthAsZero, and minSamples is added to vcfFilter -- (30aa234) Prep for v3.3.0 - ---- - ## [3.2.0] - 2026-09-24 **Your raw reads can stay where they were handed to you.** `Data/` no longer has to be one flat folder (one folder per sample, one per sequencing run, nested as deep as you like, or all together as before). Nothing you already have needs moving, not one pinned tool changed version, and no result is affected. As with every release, the new installation's module store starts empty and your modules are installed into it again. @@ -603,7 +539,6 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba --- -[3.3.0]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.3.0 [3.2.0]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.2.0 [3.1.2]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.1.2 [3.1.1]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.1.1 diff --git a/PoolSeqFlow b/PoolSeqFlow index 3987dbe..32286b4 100755 --- a/PoolSeqFlow +++ b/PoolSeqFlow @@ -1,6 +1,6 @@ #!/bin/bash # -# Version: 3.3.0 +# Version: 3.2.0 # Author : Ozan L. Z. Kiratli # email : ozankiratli@protonmail.com # @@ -43,7 +43,7 @@ set -e eval "$(conda shell.bash hook)" -VERSION="3.3.0" +VERSION="3.2.0" usage() { echo "Usage: $0 {install|init [multi]|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall [all]}" diff --git a/modules/association/manifest.json b/modules/association/manifest.json index ea96975..323f231 100644 --- a/modules/association/manifest.json +++ b/modules/association/manifest.json @@ -4,7 +4,7 @@ "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", - "environment": "3.3.0", + "environment": "3.0.0", "summary": "each allele's frequency regressed on a phenotype measured per pool, with a permutation p", "needs": [ "depths" diff --git a/modules/basicstats/manifest.json b/modules/basicstats/manifest.json index 11b42d9..52e6b29 100644 --- a/modules/basicstats/manifest.json +++ b/modules/basicstats/manifest.json @@ -4,7 +4,7 @@ "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", - "environment": "3.3.0", + "environment": "3.0.0", "summary": "site counts, depth, effective pool size and gene diversity, per pool", "needs": [ "frequencies", diff --git a/modules/lib/allele_frequencies/manifest.json b/modules/lib/allele_frequencies/manifest.json index cb9a283..be0c027 100644 --- a/modules/lib/allele_frequencies/manifest.json +++ b/modules/lib/allele_frequencies/manifest.json @@ -5,7 +5,7 @@ "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", - "environment": "3.3.0", + "environment": "3.0.0", "summary": "per-site allele frequencies parsed from the depth table, vectorized R with a compiled path", "libraries": [], "packages": [ diff --git a/modules/lib/n_eff/manifest.json b/modules/lib/n_eff/manifest.json index 988de3e..f8a66ea 100644 --- a/modules/lib/n_eff/manifest.json +++ b/modules/lib/n_eff/manifest.json @@ -4,7 +4,7 @@ "kind": "library", "license": "GPL-3.0-or-later", "frame": "20260910.001", - "environment": "3.3.0", + "environment": "3.0.0", "summary": "effective pool size at a site and over a pool, and the harmonic mean it needs", "libraries": [], "packages": [] diff --git a/modules/lib/nei_distance/manifest.json b/modules/lib/nei_distance/manifest.json index 20c87b3..3b22155 100644 --- a/modules/lib/nei_distance/manifest.json +++ b/modules/lib/nei_distance/manifest.json @@ -5,7 +5,7 @@ "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", - "environment": "3.3.0", + "environment": "3.0.0", "summary": "Nei's minimum distance accumulated over sites, combined and averaged, with a compiled path", "libraries": [], "packages": [ diff --git a/modules/mds/manifest.json b/modules/mds/manifest.json index 473def0..7f076dd 100644 --- a/modules/mds/manifest.json +++ b/modules/mds/manifest.json @@ -4,7 +4,7 @@ "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", - "environment": "3.3.0", + "environment": "3.0.0", "summary": "the pools placed by Nei's minimum distance, on a classical MDS (principal coordinates analysis)", "needs": [ "depths" diff --git a/nextflow.config b/nextflow.config index 94038f3..ce62f6f 100644 --- a/nextflow.config +++ b/nextflow.config @@ -60,7 +60,7 @@ manifest { mainScript = 'poolseqflow.nf' license = 'Apache-2.0' doi = '10.5281/zenodo.19245611' - version = '3.3.0' + version = '3.2.0' // Advisory: Nextflow warns and carries on. Prefixing with `!` would make it abort. nextflowVersion = '>=26.04.0' } From 1b04cb10ef17d2e68c8b8f19ca78fee64e64028e Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Thu, 8 Oct 2026 10:13:57 -0400 Subject: [PATCH 2/6] module updates, association analysis improved, report.pdf now shows the reports --- modules/README.md | 25 +- modules/association/association.R | 72 ++- modules/association/main.nf | 1 + modules/association/manifest.json | 12 +- modules/association/report.Rmd | 211 +++++++++ modules/association/test/association.sh | 557 ++++++++++++++++++++++-- modules/basicstats/basicstats.R | 4 +- modules/basicstats/manifest.json | 5 +- modules/basicstats/report.Rmd | 282 ++++++++++++ modules/basicstats/test/basicstats.sh | 104 +++++ modules/mds/citations.json | 13 +- modules/mds/main.nf | 2 + modules/mds/manifest.json | 6 +- modules/mds/mds.R | 27 +- modules/mds/references.bib | 13 +- modules/mds/report.Rmd | 191 ++++++++ modules/mds/test/mds.sh | 206 ++++++++- 17 files changed, 1662 insertions(+), 69 deletions(-) create mode 100644 modules/association/report.Rmd create mode 100644 modules/basicstats/report.Rmd create mode 100644 modules/mds/report.Rmd diff --git a/modules/README.md b/modules/README.md index dbd8103..8be1367 100644 --- a/modules/README.md +++ b/modules/README.md @@ -4,7 +4,7 @@ One directory per module, named exactly as the module is named. Each holds at le | File | What | |---|---| -| `manifest.json` | `name`, `version`, `contract`, `summary`, `license`, `frame`, `environment`, and optionally `needs`, `gates`, `outputs` and `packages` | +| `manifest.json` | `name`, `version`, `contract`, `summary`, `license`, `frame`, `environment`, and optionally `needs`, `gates`, `outputs`, `packages` and `report` | | `main.nf` | the module's own Nextflow pipeline | | `citations.json` | the methods and packages your module should be cited with | @@ -118,6 +118,22 @@ Write a library when more than one module wants the same decision-free arithmeti 15c. **Declare an `outputs` entry for every file you publish, and say where it is explained.** Each is `{"file": "design.tsv", "summary": "one row per pool", "anchor": "the-experimental-design"}` -- `file` may be a glob, `summary` is the one-line description the reader gets, and `anchor` names a heading of the manual this release ships. A module published separately has nowhere in that manual to point and gives `"url"` instead, in full. The frame renders one `README.md` from every module's declarations plus what every analysis carries anyway, so a folder found months later says how to read itself and no module invents its own way of saying it. Two things are checked for you: an `anchor` no heading answers to is refused before your module starts, and a `file` you declared and did not publish fails the publish with nothing written. **This is where a number that is not a measurement gets said to be one** -- a bound, an estimate with an assumption behind it, a figure whose meaning changes with a setting. A table cell cannot carry that sentence and a `summary` plus a link can. +15d. **Lay out your own report by naming an `.Rmd` in your directory as `report`.** Every published folder carries `report.pdf`. Without a `report`, it holds each `.tsv` table and `.png` figure you declared, under its file's name, and leaves any other file out. With one, the frame knits your file under its own header, and what follows the header is yours: tables grouped by what they answer and figures drawn from them, the way `basicstats/report.Rmd`, `association/report.Rmd` and `mds/report.Rmd` do. Your chunks run in the task's directory, in a UTF-8 locale, with the functions of `analysis/lib/rmd/report.R` already loaded, and they read only what you published: + +| Function | What it does | +|---|---| +| `report_read(name)` | one of your published tables, every column as text, or `NULL` when you did not publish it | +| `report_files(pattern)` | the published files a glob matches, in name order | +| `report_number(x, digits, decimals)` | numbers as a table prints them: counts whole with thousands separated, anything else to `digits` significant digits or `decimals` places, a magnitude below 1e-4 in scientific notation, and `NA`, `NaN`, `Inf` and `-Inf` as themselves | +| `report_table(table, caption, align, groups, size, max_rows, source, total)` | a table that runs across pages with its header repeated. Typst measures every cell and sets it at the largest of 10 to 7 points at which it fits; past that the headers wrap, and the size goes down to 4 points. A newline in a cell sets it on two lines and nothing else breaks a cell; `groups` prints a label once over a run of columns | +| `report_text(x)` | a name as markdown that reads exactly as written; put every name you print outside a table through it, since a `*` or an `@` in one is otherwise emphasis or a citation | +| `report_image(path, caption, width)` | a figure already on disk; at the full width a PNG is kept to 7.5 in tall | +| `report_plot(plot, name, width, height, caption)` | a ggplot drawn for the report and placed like `report_image` | +| `report_columns(items, most)` | items split into as few groups of at most `most` as hold them, sizes differing by at most one, for a table with a column per pool | +| `report_palette(pools)` | one color per pool, in the order given | + +Call `report_table`, `report_image` and `report_plot` from a chunk with `results = "asis"`. A table of one row per site can run to millions of rows, which `report_read` would hold whole as text; read such a table with `data.table::fread` and only the columns you show, as `association/report.Rmd` does, and declare `r-data.table` in your `packages`. A chunk that fails stops the report rather than printing its error into it, and the analysis is published without a report and with the reason. **Say in each caption which file the numbers come from**, so a reader can always get from the report back to the file. A frame older than `20261008.001` ignores the key and lays out what you declared, file by file, so a module that adds a report still installs and runs on it. + ### Sharing what you derive 16. **Anything another module could want goes in `Analysis/Main`, through `publishIntermediate`.** It writes the provenance record and then the file, which is the invariant everything else reads: an intermediate that is there is one whose provenance can be read. It is one call rather than two so that the order is not yours to get wrong. @@ -151,6 +167,7 @@ Write a library when more than one module wants the same decision-free arithmeti PoolSeqFlow analysis modules available # what is published PoolSeqFlow analysis modules install # newest version this release can read PoolSeqFlow analysis modules install 1.2.0 # or exactly that one +PoolSeqFlow analysis modules install all # every one this release can run PoolSeqFlow analysis modules list # what is installed here PoolSeqFlow analysis modules uninstall # remove one, after confirming ``` @@ -188,14 +205,16 @@ For a module in this repository, one command does all of it: dev/scripts/publish-module.sh [ref] ``` -It builds the tarball into `modules-repo/`, reads `contract`, `frame`, `environment` and `summary` out of the module's own manifest at that ref, appends the row, and bumps `#!index-version`. It refuses to overwrite a version that is already published -- somebody may have installed it, and its checksum is in the catalogue -- so a change means bumping the module's version and publishing that. +It builds the tarball into `modules/repo/`, reads `contract`, `frame`, `environment` and `summary` out of the module's own manifest at that ref, appends the row, and bumps `#!index-version`. It refuses to overwrite a version that is already published -- somebody may have installed it, and its checksum is in the catalogue -- so a change means bumping the module's version and publishing that. -`dev/scripts/publish-module.sh --list` shows every module and library whose version the catalogue does not carry yet, and `--all-pending` publishes all of them from `HEAD`, one at a time through the same path. It refuses to start while any of them differs from `HEAD`. +`dev/scripts/publish-module.sh --list` shows every module and library in the tree and whether the catalogue carries its version, and `--all-pending` publishes every one it does not, from `HEAD`, one at a time through the same path. Before writing anything it refuses a pending module that differs from `HEAD`, a manifest it cannot read, and a name two directories share. **The tarball is built from the extracted tree rather than piped straight out of `git archive`, and that is not fussiness.** `git archive :modules/` reads a subtree, and `.gitattributes` patterns are anchored at the repository root -- so a rule written for `modules/` does not match `test/` inside that subtree, and the module's own cases would ship where a release tarball excludes them. Archiving a tree also stamps `mtime` as *now*, so two builds of one ref would not match. The script drops `test/` and repacks with the commit's timestamp, which makes republishing the same ref produce the same bytes. The tarballs are committed and served from the site alongside the catalogue, so **a published row and the file it names go out in one commit** -- otherwise the row advertises a download that 404s until the next deploy. +The site deploys after a release, and otherwise only when the Documentation workflow is run by hand on `main` (Actions, Documentation, Run workflow), which is how a module published between releases goes live. A push to `main` deploys nothing. Either way it refuses while `main` declares a version that has no published release. + Two rules the installer enforces, so build for them: 22. **The checksum must match before anything is unpacked.** An archive becomes code that runs on someone's machine, so a mismatch stops the install with nothing written. A row whose `sha256` is out of date makes the module uninstallable, not silently different. diff --git a/modules/association/association.R b/modules/association/association.R index f3fb250..baefe9b 100644 --- a/modules/association/association.R +++ b/modules/association/association.R @@ -313,6 +313,17 @@ if (length(unknown) > 0) { stop("association.R: no phenotype called ", paste(unknown, collapse = ", "), ". This project declares ", paste(declared, collapse = ", "), ".") } +# Each phenotype is published as association_.tsv and alleles_.tsv, the stem its name +# with every character outside [A-Za-z0-9._-] made _. +fitted_phenotypes <- declared[declared %in% chosen] +stems <- gsub("[^A-Za-z0-9._-]", "_", fitted_phenotypes) +if (anyDuplicated(stems)) { + shared <- stems[duplicated(stems)][1] + stop("association.R: the phenotypes ", + paste0("'", fitted_phenotypes[stems == shared], "'", collapse = " and "), + " would be published under one name, association_", shared, ".tsv. Rename one of ", + "them in the metadata.") +} # One row per unit, in design order, so a unit's pools index the published columns rather than # being re-derived from the metadata. @@ -439,11 +450,12 @@ for (entry in design$phenotypes[declared %in% chosen]) { if (nrow(table) == 0) next parsed <- parse_columns(as.list(table[pool_order])) - # Named: a unit holds the NAMES of its pools and indexes the columns with them. - weight <- vapply(seq_along(pool_order), - function(i) n_eff(n_chrom[i], parsed$depth[, i]), - numeric(nrow(parsed$depth))) - colnames(weight) <- pool_order + # Named: a unit holds the NAMES of its pools and indexes the columns with them. A matrix + # whatever the site count: vapply returns a bare vector for a table of one site. + weight <- matrix(vapply(seq_along(pool_order), + function(i) n_eff(n_chrom[i], parsed$depth[, i]), + numeric(nrow(parsed$depth))), + nrow = nrow(parsed$depth), dimnames = list(NULL, pool_order)) held <- roll_up(parsed$freq, weight, parsed$site, unit_pools) sites <- nrow(held$weight) @@ -530,7 +542,9 @@ for (entry in design$phenotypes[declared %in% chosen]) { message("association.R: ", short, " tested site(s) of ", entry$column, " (", kind, ") were read in too few units to reach 0.05. A site read in m ", "units is rearranged among those m alone and has the floor of an m-unit ", - "design, 1/m!; n_observed in association.tsv says which sites these are. ", + "design, 1/m!; n_observed in association_", + gsub("[^A-Za-z0-9._-]", "_", entry$column), ".tsv says which sites ", + "these are. ", "Their p ranks them and cannot make them significant.") } } @@ -559,24 +573,36 @@ for (entry in design$phenotypes[declared %in% chosen]) { sites_table <- do.call(rbind, site_rows) alleles_table <- do.call(rbind, allele_rows) -write.table(sites_table, file.path(out, "association.tsv"), sep = "\t", quote = FALSE, - row.names = FALSE, na = "NA") - -# The allele table is a SELECTION and the site table is not. What selected it is printed in the -# published header, so a filtered table cannot be read as the whole one. +# Two files per phenotype: association_.tsv, every site of the sequences fitted, and +# alleles_.tsv, the alleles of a SELECTION of them. The selection is printed in the published +# header: the sites at perm_p reportBelow or under, and the first reportTop by perm_p, ties to the +# larger S with every flagged site first, which is the order the report ranks them in. below <- if (is.null(OPTS$reportBelow)) 0.05 else as.numeric(OPTS$reportBelow) top <- if (is.null(OPTS$reportTop)) 1000L else as.integer(OPTS$reportTop) - -key <- paste(sites_table$phenotype, sites_table$kind, sites_table$chrom, sites_table$pos) -ranked <- order(sites_table$perm_p, na.last = NA) -keep <- unique(c(key[!is.na(sites_table$perm_p) & sites_table$perm_p <= below], - key[head(ranked, top)])) -selected <- paste(alleles_table$phenotype, alleles_table$kind, - alleles_table$chrom, alleles_table$pos) %in% keep - -write.table(alleles_table[selected, , drop = FALSE], - file.path(out, "association_alleles.tsv"), sep = "\t", quote = FALSE, - row.names = FALSE, na = "NA") +site_columns <- c("kind", "chrom", "pos", "k", "n_observed", "n_units", "S", "perm_p", "fdr_p", + "mean_weight", "max_leverage", "zero_variance") +allele_columns <- c("kind", "chrom", "pos", "allele", "b1", "se", "t", "p") +# A phenotype with no site read still gets both files, as a header. +rows_of <- function(table, phenotype, columns) { + if (is.null(table)) { + return(stats::setNames(data.frame(matrix(nrow = 0, ncol = length(columns))), columns)) + } + table[table$phenotype == phenotype, columns, drop = FALSE] +} +for (i in seq_along(fitted_phenotypes)) { + sites <- rows_of(sites_table, fitted_phenotypes[i], site_columns) + alleles <- rows_of(alleles_table, fitted_phenotypes[i], allele_columns) + tested <- which(!is.na(sites$perm_p)) + flagged <- !is.na(sites$zero_variance[tested]) & sites$zero_variance[tested] == 1 + ranked <- tested[order(sites$perm_p[tested], -ifelse(flagged, Inf, abs(sites$S[tested])))] + key <- paste(sites$kind, sites$chrom, sites$pos, sep = "\r") + keep <- unique(c(key[tested[sites$perm_p[tested] <= below]], key[head(ranked, top)])) + selected <- paste(alleles$kind, alleles$chrom, alleles$pos, sep = "\r") %in% keep + write.table(sites, file.path(out, paste0("association_", stems[i], ".tsv")), sep = "\t", + quote = FALSE, row.names = FALSE, na = "NA") + write.table(alleles[selected, , drop = FALSE], file.path(out, paste0("alleles_", stems[i], ".tsv")), + sep = "\t", quote = FALSE, row.names = FALSE, na = "NA") +} write.table(do.call(rbind, report), file.path(out, "permutations.tsv"), sep = "\t", quote = FALSE, row.names = FALSE, na = "NA") @@ -610,7 +636,7 @@ write.table(do.call(rbind, lapply(design$phenotypes[declared %in% chosen], pheno drawable <- requireNamespace("ggplot2", quietly = TRUE) if (!drawable) { message("association.R: ggplot2 is not installed, so no figures were drawn. Every number ", - "they would have shown is in association.tsv and permutations.tsv.") + "they would have shown is in the association_ tables and permutations.tsv.") } # Drawn when no site was tested as well: qq.png is a declared output, and the frame refuses a diff --git a/modules/association/main.nf b/modules/association/main.nf index 95907ba..283217f 100644 --- a/modules/association/main.nf +++ b/modules/association/main.nf @@ -96,6 +96,7 @@ process Analyze { "# ${useCompiled(settings) ? 'allele_frequencies.cpp, compiled at run time' : 'allele_frequencies(), vectorized R'}" + ", in bins of ${settings.binSize} sites over ${workers} worker(s)", "# phenotype ${settings.phenotypes.join(', ')}, up to ${settings.permutations} rearrangements, ${settings.fdr} across sites", + "# alleles_.tsv: the sites at perm_p ${settings.reportBelow} or under and the first ${settings.reportTop} by perm_p, for each phenotype", "# beta > 0 means the allele is more frequent at the higher phenotype value"].join('\n').replace("'", "'\\''") """ diff --git a/modules/association/manifest.json b/modules/association/manifest.json index 323f231..a94def9 100644 --- a/modules/association/manifest.json +++ b/modules/association/manifest.json @@ -1,6 +1,6 @@ { "name": "association", - "version": "20261006.001", + "version": "20261008.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -28,15 +28,16 @@ "bin/cap_depth.awk keeps reads in coordinate order until the cap rather than subsampling, so at a capped position the retained reads are biased toward those mapping earliest. That is an allele-frequency bias and not only a depth truncation.", "filterFalsePositives.sh admits an allele that clears half of one chromosome's frequency in a fraction of the samples, and MajorAlleleToRef.py re-polarizes on cohort totals. Both depend on which pools were in the run, so a result from six pools is not the same quantity as one from twelve and the two are not comparable across targets." ], + "report": "report.Rmd", "outputs": [ { - "file": "association.tsv", - "summary": "one row per site, always complete: how many alleles and units it held, the site statistic, its permutation p, the adjusted p, and the leverage of the unit carrying most of the slope", + "file": "association_*.tsv", + "summary": "one per phenotype, named after it, with a row for every site of the sequences fitted: how many alleles and units it held, the site statistic, its permutation p, the adjusted p, and the leverage of the unit carrying most of the slope", "anchor": "association-sites" }, { - "file": "association_alleles.tsv", - "summary": "one row per allele for the sites reportBelow and reportTop select: the slope, its standard error, and the parametric t and p as diagnostics", + "file": "alleles_*.tsv", + "summary": "one per phenotype, with a row per allele of the sites reportBelow and reportTop select from that phenotype's: the slope, its standard error, and the parametric t and p as diagnostics", "anchor": "association-alleles" }, { @@ -72,6 +73,7 @@ } ], "packages": [ + "r-data.table=1.18.6.1", "r-dofuture=1.4.0", "r-foreach=1.5.2", "r-future=1.76.0", diff --git a/modules/association/report.Rmd b/modules/association/report.Rmd new file mode 100644 index 0000000..2e92a00 --- /dev/null +++ b/modules/association/report.Rmd @@ -0,0 +1,211 @@ +```{r association-tables} +# association's own report, knitted by the frame's report.Rmd with the functions of +# analysis/lib/rmd/report.R. It reads the tables this module published and nothing else. +# +# Each phenotype's two tables are association_.tsv and alleles_.tsv, the stem its name +# with every character outside [A-Za-z0-9._-] made _. They are read into one table each with the +# phenotype as a column, the site tables by data.table and for the columns shown. Phenotypes keep +# permutations.tsv's order. +phenotype_rows <- report_read("phenotype.tsv") +runs <- report_read("permutations.tsv") +num <- function(x) suppressWarnings(as.numeric(x)) + +phenotypes <- unique(runs$phenotype) +stem <- function(p) gsub("[^A-Za-z0-9._-]", "_", p) +site_rows <- as.data.frame(data.table::rbindlist(lapply(phenotypes, function(p) { + sites <- data.table::fread( + file.path(report_folder(), paste0("association_", stem(p), ".tsv")), sep = "\t", + na.strings = "NA", + select = c("kind", "chrom", "pos", "k", "n_observed", "S", "perm_p", "fdr_p", + "max_leverage", "zero_variance"), + colClasses = list(character = c("kind", "chrom")), data.table = FALSE) + sites$phenotype <- rep(p, nrow(sites)) + sites +}))) +allele_rows <- do.call(rbind, lapply(phenotypes, function(p) { + alleles <- report_read(paste0("alleles_", stem(p), ".tsv")) + if (is.null(alleles) || nrow(alleles) == 0) return(NULL) + cbind(phenotype = p, alleles, stringsAsFactors = FALSE) +})) +if (is.null(allele_rows)) { + allele_rows <- data.frame(phenotype = character(0), kind = character(0), + chrom = character(0), pos = character(0), allele = character(0), + b1 = character(0), t = character(0), stringsAsFactors = FALSE) +} +units <- max(num(runs$units)) +SHOWN <- 20 + +# The site each allele row belongs to, and per site the allele with the largest |t|, or with the +# steepest slope at a flagged site, compared at six significant digits, where a biallelic site's +# two |t| differ only by rounding. Between alleles that tie, the one whose slope is positive. +site_key <- function(table) paste(table$phenotype, table$kind, table$chrom, table$pos, sep = "\r") +flagged_sites <- site_key(site_rows)[!is.na(site_rows$zero_variance) & site_rows$zero_variance == 1] +allele_key <- site_key(allele_rows) +size <- ifelse(allele_key %in% flagged_sites, abs(num(allele_rows$b1)), abs(num(allele_rows$t))) +strongest <- allele_rows[order(-signif(size, 6), num(allele_rows$b1) <= 0), , drop = FALSE] +strongest <- strongest[!duplicated(site_key(strongest)), , drop = FALSE] + +# An ordinal phenotype's levels in the order its numbers give them, as phenotype.tsv wrote them. +level_order <- function(p) { + rows <- phenotype_rows[phenotype_rows$phenotype == p & !is.na(num(phenotype_rows$value)), , + drop = FALSE] + paste(unique(rows$shown[order(num(rows$value))]), collapse = " < ") +} +# An allele longer than 20 bases, as its first 12 and its length. +allele_text <- function(a) { + ifelse(is.na(a), "NA", + ifelse(nchar(a) > 20, paste0(substr(a, 1, 12), "... (", nchar(a), " bases)"), a)) +} +``` + +`r length(phenotypes)` `r if (length(phenotypes) == 1) "phenotype" else "phenotypes"` fitted over `r units` units: `r paste(report_text(phenotypes), collapse = ", ")`. + +## The phenotype as it was read + +```{r association-phenotype, results = "asis"} +for (p in phenotypes) { + rows <- phenotype_rows[phenotype_rows$phenotype == p, , drop = FALSE] + coded <- unique(rows$coded_one[!is.na(rows$coded_one) & nzchar(rows$coded_one)]) + table <- data.frame(Unit = rows$unit, Pool = rows$pool, "As written" = rows$shown, + "Read as" = rows$value, Fitted = rows$fitted, + check.names = FALSE, stringsAsFactors = FALSE) + scale <- if (identical(rows$kind[1], "binary") && length(coded) > 0) { + paste0(". The level coded 1 is ", coded[1]) + } else if (identical(rows$kind[1], "ordinal")) { + paste0(". Each level is read as its place in the order ", level_order(p), ", from 0") + } else "" + report_table(table, caption = paste0( + p, ", ", rows$kind[1], ", as each pool's cell was written and the number the fit used ", + "(phenotype.tsv)", scale, ".")) +} +``` + +## What the design can show + +```{r association-design, results = "asis"} +# Every fit in a run has the one design floor its units allow. +limit <- suppressWarnings(min(num(runs$design_floor), na.rm = TRUE)) +if (is.finite(limit) && limit > 0.05) { + cat("**With ", units, " units, the smallest p any site can reach is ", + report_number(limit, digits = 3), ", above 0.05.** The sites below are a ranking of ", + "effect sizes and not a test.\n\n", sep = "") +} +labels <- paste0(runs$phenotype, ", ", ifelse(runs$kind == "snp", "SNPs", + ifelse(runs$kind == "indel", "indels", runs$kind))) +measures <- list( + "Units" = report_number(runs$units), + "Rearrangements" = paste0(report_number(runs$permutations), + ifelse(runs$exhaustive == "TRUE", ", every one", ", sampled")), + "Smallest p the design allows" = report_number(runs$design_floor, digits = 3), + "Smallest p this run could report" = report_number(runs$floor, digits = 3), + "Sites read" = report_number(runs$sites), + "Sites tested" = report_number(runs$tested), + "Selected, fdr_p \u2264 0.05" = report_number(runs$selected), + "dispersion" = report_number(runs$dispersion, digits = 3), + "depth_phenotype_cor" = report_number(runs$depth_phenotype_cor, digits = 3), + "lambda_gc" = report_number(runs$lambda_gc, digits = 3), + "Alleles a site, tested" = report_number(runs$arity_mean, digits = 3), + "Alleles a site, selected" = report_number(runs$arity_selected, digits = 3)) +table <- data.frame(Measure = names(measures), check.names = FALSE, stringsAsFactors = FALSE) +for (i in seq_along(labels)) table[[labels[i]]] <- vapply(measures, function(m) m[i], "") +report_table(table, caption = paste0( + "What each fit assumed and absorbed (permutations.tsv). lambda_gc reads above 1 on a clean ", + "table: compare it with the units and alleles a site was read in, not with 1.")) +``` + +## The strongest sites + +```{r association-sites, results = "asis"} +for (p in phenotypes) { + rows <- site_rows[site_rows$phenotype == p & !is.na(site_rows$perm_p), , drop = FALSE] + cat("\n\n### ", report_text(p), "\n\n", sep = "") + if (nrow(rows) == 0) { + cat("No site of ", report_text(p), " was tested: none was read in three units that ", + "differ in it.\n\n", sep = "") + next + } + # Ties in p go to the larger S, and every flagged site ahead of the rest in file order: its S + # is not comparable between implementations. + flagged_row <- !is.na(rows$zero_variance) & rows$zero_variance == 1 + rows <- rows[order(rows$perm_p, -ifelse(flagged_row, Inf, abs(rows$S))), , drop = FALSE] + floor_p <- min(num(runs$floor[runs$phenotype == p])) + # At the floor: the next p a run can report is twice it. + at_floor <- sum(rows$perm_p <= 1.5 * floor_p) + shown <- utils::head(rows, SHOWN) + best <- strongest[match(site_key(shown), site_key(strongest)), , drop = FALSE] + flagged <- !is.na(shown$zero_variance) & shown$zero_variance == 1 + kind <- phenotype_rows$kind[phenotype_rows$phenotype == p][1] + coded <- unique(phenotype_rows$coded_one[phenotype_rows$phenotype == p & + !is.na(phenotype_rows$coded_one) & + nzchar(phenotype_rows$coded_one)]) + direction <- if (identical(kind, "binary") && length(coded) > 0) { + paste0("in the units coded ", coded[1]) + } else if (identical(kind, "ordinal")) { + paste0("further along ", level_order(p)) + } else "at the higher value" + lead <- if (nrow(rows) == 1) paste0("The one tested site of ", p) + else if (nrow(rows) <= SHOWN) paste0("All ", nrow(rows), " tested sites of ", p, + ", smallest permutation p first, ties broken by ", + "the larger S with a flagged site first") + else paste0("The ", SHOWN, " sites of ", p, " with the smallest permutation p, ties ", + "broken by the larger S with a flagged site first") + reach <- if (nrow(rows) == 1) { + if (at_floor == 1) "It reaches" else "It does not reach" + } else { + paste(report_number(at_floor), "of the", report_number(nrow(rows)), "tested sites", + if (at_floor == 1) "reaches" else "reach") + } + table <- data.frame(Sequence = shown$chrom, Position = report_number(shown$pos), + Kind = shown$kind, Alleles = report_number(shown$k), + "Units read" = report_number(shown$n_observed), + S = ifelse(flagged, "flagged", report_number(shown$S, digits = 3)), + "perm_p" = report_number(shown$perm_p, digits = 3), + "fdr_p" = report_number(shown$fdr_p, digits = 3), + "Strongest allele" = allele_text(best$allele), + "Its slope" = report_number(best$b1, digits = 3), + "Leverage" = report_number(shown$max_leverage, digits = 2), + check.names = FALSE, stringsAsFactors = FALSE) + report_table(table, align = c("l", "r", "l", "r", "r", "r", "r", "r", "l", "r", "r"), + caption = paste0( + lead, " (association_", stem(p), ".tsv), with the allele of the largest |t| at each, or ", + "of the steepest slope at a flagged site, and its slope (alleles_", stem(p), ".tsv). A ", + "positive slope means the allele is more frequent ", direction, ". ", reach, " the ", + "smallest p this run could report.", + if (any(flagged)) { + " A flagged site's S is not comparable between implementations and is not printed." + }, + if (anyNA(best$allele)) { + paste0(" NA is a site alleles_", stem(p), ".tsv holds no row for: reportBelow and ", + "reportTop choose the sites it holds.") + })) +} +``` + +## The p-values against the uniform + +```{r association-qq, results = "asis"} +for (path in report_files("qq.png")) { + report_image(path, caption = paste0("Every tested site's permutation p against the uniform, ", + "a panel per phenotype (qq.png, drawn from the ", + "association\\_ tables). The points sit on a ladder whose ", + "ceiling is the design's floor.")) +} +``` + +```{r association-manhattan, results = "asis"} +# The Manhattan plots in the order of the sequences they draw, each under the sequence's own +# name. A plot's file name carries the name with every character outside [A-Za-z0-9._-] made _. +plots <- report_files("manhattan_*.png") +if (length(plots) > 0) { + cat("\n\n## Association along a sequence\n\n") + sequences <- unique(site_rows$chrom) + drawn <- sub("^manhattan_(.*)[.]png$", "\\1", basename(plots)) + which_sequence <- match(drawn, gsub("[^A-Za-z0-9._-]", "_", sequences)) + for (i in order(which_sequence, drawn)) { + name <- if (is.na(which_sequence[i])) drawn[i] else sequences[which_sequence[i]] + report_image(plots[i], caption = paste0( + "Association along ", report_text(name), " (", report_text(basename(plots[i])), + "). A flat ceiling is the design's floor and not the data.")) + } +} +``` diff --git a/modules/association/test/association.sh b/modules/association/test/association.sh index 602e85c..69270cc 100644 --- a/modules/association/test/association.sh +++ b/modules/association/test/association.sh @@ -5,6 +5,7 @@ # covers: modules/association/ modules/lib/ # covers: test/tools/freq_corpus.py # covers: analysis.nf modules/association/main.nf +# covers: analysis/lib/rmd/ # covers: dev/validation/agree.R dev/validation/lib.R modules/mds/mds.R bin/mask_depth.awk # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. @@ -25,14 +26,15 @@ association_corpus() { python3 "$REPO_ROOT/test/tools/freq_corpus.py" "$CORPUS_DIR" "$CORPUS_DIR" } -# Run the module's R directly over the corpus, under one set of options, into $1. +# Run the module's R directly over the corpus, under one set of options, into $1. $4 names the +# depth tables, comma-separated, and is the SNP table alone unless given. # # Every library's .R rather than the list the manifest names: they are standalone function # definitions, so a superset is harmless, and the case then cannot go stale when that list # changes. The Nextflow case is what proves main.nf assembles the same thing, and 00_static is # what proves the manifest declares exactly what the module calls. association_direct() { - local dest="$1" options="$2" design="${3:-}" + local dest="$1" options="$2" design="${3:-}" depths="${4:-Test_snp_depth.tsv}" mkdir -p "$dest" [ -n "$design" ] || design="$CORPUS_DIR/design.json" cat "$REPO_ROOT"/modules/lib/*/*.R \ @@ -42,7 +44,7 @@ association_direct() { --design "$design" --pools "$CORPUS_DIR/pools.json" \ --options "$dest/options.json" \ --cpp "$REPO_ROOT/modules/lib/allele_frequencies/allele_frequencies.cpp" \ - --depths 'Test_snp_depth.tsv' --out "$dest" ) > "$dest/out.txt" 2>&1 + --depths "$depths" --out "$dest" ) > "$dest/out.txt" 2>&1 } # Empty one pool's cell at one site to the site's own arity, every count of it set to 0: a pool @@ -108,12 +110,12 @@ agrees_with_oracle() { local run="$1" oracle="$2" chrom pos column; shift 2 while IFS=$'\t' read -r chrom pos; do for column in "$@"; do - assert_close "$(site_cell "$run/association.tsv" "$chrom" "$pos" "$column")" \ + assert_close "$(site_cell "$run/association_pt_wingspan.tsv" "$chrom" "$pos" "$column")" \ "$(oracle_value "$oracle" "assoc.$chrom.$pos.$column")" \ "$chrom:$pos: $column against the oracle" done done < <(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } - { print $(h["chrom"]) "\t" $(h["pos"]) }' "$run/association.tsv") + { print $(h["chrom"]) "\t" $(h["pos"]) }' "$run/association_pt_wingspan.tsv") } # The options a case uses unless it is testing one of them. @@ -143,7 +145,7 @@ test_association_computes_what_the_corpus_says() { prefix="${phenotype##*:}" association_direct "$sb/${prefix}" \ "${ASSOCIATION_OPTIONS/pt_wingspan/${phenotype%%:*}}" - sites="$sb/${prefix}/association.tsv" + sites="$sb/${prefix}/association_${phenotype%%:*}.tsv" if [ ! -s "$sites" ]; then fail_case "$prefix: nothing published"$'\n'"$(cat "$sb/${prefix}/out.txt")" return @@ -183,7 +185,7 @@ test_the_permutation_moves_residuals_and_not_labels() { local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-scheme") association_corpus "$sb" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" - assert_close "$(site_cell "$sb/run/association.tsv" chr1 700 perm_p)" \ + assert_close "$(site_cell "$sb/run/association_pt_wingspan.tsv" chr1 700 perm_p)" \ "$(corpus_expects "assoc.chr1.700.perm_p")" \ "chr1:700 under residual permutation" } @@ -333,7 +335,7 @@ test_a_site_is_tested_over_the_units_it_was_read_in() { python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" - if [ ! -s "$sb/run/association.tsv" ]; then + if [ ! -s "$sb/run/association_pt_wingspan.tsv" ]; then fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" return fi @@ -342,11 +344,11 @@ test_a_site_is_tested_over_the_units_it_was_read_in() { for site in chr1:100:6 chr1:250:5 chr2:250:5 chr1:700:4 chr2:100:3 chr10:500:2 chr1:400:1 \ chr2:400:1; do IFS=: read -r chrom pos read <<< "$site" - assert_eq "$read" "$(site_cell "$sb/run/association.tsv" "$chrom" "$pos" n_observed)" \ + assert_eq "$read" "$(site_cell "$sb/run/association_pt_wingspan.tsv" "$chrom" "$pos" n_observed)" \ "$chrom:$pos is read in $read units" done agrees_with_oracle "$sb/run" "$sb/oracle.tsv" S perm_p fdr_p mean_weight max_leverage - assert_eq "NA" "$(site_cell "$sb/run/association.tsv" chr10 500 fdr_p)" \ + assert_eq "NA" "$(site_cell "$sb/run/association_pt_wingspan.tsv" chr10 500 fdr_p)" \ "a site read in 2 units is left out of the adjustment as well as the test" assert_close "$(published_cell "$sb/run/permutations.tsv" pt_wingspan depth_phenotype_cor)" \ "$(oracle_value "$sb/oracle.tsv" assoc.depth_phenotype_cor)" \ @@ -361,7 +363,7 @@ test_a_site_is_tested_over_the_units_it_was_read_in() { # identity is one of the rearrangements counted. assert_eq "" "$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } $(h["perm_p"]) == "0" { print $(h["chrom"]) ":" $(h["pos"]) }' \ - "$sb/run/association.tsv")" \ + "$sb/run/association_pt_wingspan.tsv")" \ "no site may publish a permutation p of 0" } @@ -388,15 +390,15 @@ test_a_unit_keeps_the_pools_of_it_that_were_read() { > "$sb/oracle.tsv" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" "$sb/paired.json" - if [ ! -s "$sb/run/association.tsv" ]; then + if [ ! -s "$sb/run/association_pt_wingspan.tsv" ]; then fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" return fi - assert_eq "3" "$(site_cell "$sb/run/association.tsv" chr1 700 n_observed)" \ + assert_eq "3" "$(site_cell "$sb/run/association_pt_wingspan.tsv" chr1 700 n_observed)" \ "U1 is still read at chr1:700 through TestSample1" - assert_eq "2" "$(site_cell "$sb/run/association.tsv" chr2 550 n_observed)" \ + assert_eq "2" "$(site_cell "$sb/run/association_pt_wingspan.tsv" chr2 550 n_observed)" \ "U2 is not read at chr2:550, where both of its pools were emptied" - local stat; stat=$(site_cell "$sb/run/association.tsv" chr1 700 S) + local stat; stat=$(site_cell "$sb/run/association_pt_wingspan.tsv" chr1 700 S) case "$stat" in ""|NA) fail_case "chr1:700 must be tested over its three units, and S is '$stat'" ;; esac @@ -502,7 +504,7 @@ PY local run column for run in flat alone; do - if [ ! -s "$sb/$run/association.tsv" ]; then + if [ ! -s "$sb/$run/association_pt_wingspan.tsv" ]; then fail_case "the $run run published nothing"$'\n'"$(cat "$sb/$run/out.txt")" return fi @@ -513,13 +515,13 @@ PY || fail_case "the $run run must say that it tested nothing" done while IFS=$'\t' read -r chrom pos; do - assert_eq "3" "$(site_cell "$sb/flat/association.tsv" "$chrom" "$pos" n_observed)" \ + assert_eq "3" "$(site_cell "$sb/flat/association_pt_wingspan.tsv" "$chrom" "$pos" n_observed)" \ "$chrom:$pos is read in three units that share a wingspan" for column in S perm_p fdr_p max_leverage zero_variance; do - assert_eq "NA" "$(site_cell "$sb/flat/association.tsv" "$chrom" "$pos" "$column")" \ + assert_eq "NA" "$(site_cell "$sb/flat/association_pt_wingspan.tsv" "$chrom" "$pos" "$column")" \ "$chrom:$pos has no spread in the phenotype, so no $column" done - assert_eq "1" "$(site_cell "$sb/alone/association.tsv" "$chrom" "$pos" n_observed)" \ + assert_eq "1" "$(site_cell "$sb/alone/association_pt_wingspan.tsv" "$chrom" "$pos" n_observed)" \ "$chrom:$pos is read in TestSample1 alone" done <<< "$sites" assert_eq "NA" "$(published_cell "$sb/alone/permutations.tsv" pt_wingspan dispersion)" \ @@ -527,6 +529,48 @@ PY agrees_with_oracle "$sb/alone" "$sb/oracle.tsv" mean_weight } +# A DEPTH TABLE OF ONE SITE IS FITTED LIKE ANY OTHER. The weights are built by vapply, which +# returns a bare vector rather than a matrix for one site, and the run stopped on "attempt to set +# 'colnames' on an object with less than two dimensions" - an indel table holding a single indel +# took the whole run with it, SNPs included. Found by a review on 2026-10-05 and left; fixed on +# 2026-10-08, when a cohort built to exercise the report reached it. +test_a_depth_table_of_one_site_is_fitted() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-one-site") + association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_indel_depth.tsv" + awk 'NR <= 2' "$table" > "$table.new" && mv "$table.new" "$table" + local chrom pos + IFS=$'\t' read -r chrom pos < <(awk -F'\t' 'NR == 2 { print $1 "\t" $2 }' "$table") + python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" + association_direct "$sb/run" "$ASSOCIATION_OPTIONS" "" "Test_snp_depth.tsv,Test_indel_depth.tsv" + if [ ! -s "$sb/run/association_pt_wingspan.tsv" ]; then + fail_case "nothing published"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + + local column + for column in S perm_p fdr_p n_observed; do + assert_close "$(site_cell "$sb/run/association_pt_wingspan.tsv" "$chrom" "$pos" "$column")" \ + "$(oracle_value "$sb/oracle.tsv" "assoc.$chrom.$pos.$column")" \ + "the one indel, $chrom:$pos: $column against the oracle" + done + assert_eq "14" "$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["kind"]) == "snp"' "$sb/run/association_pt_wingspan.tsv" \ + | wc -l | tr -d ' ')" \ + "and every SNP is published beside it" + + # The indel alone is a phenotype with one tested site, which the report names in the singular. + association_direct "$sb/alone" "$ASSOCIATION_OPTIONS" "" "Test_indel_depth.tsv" + analysis_render_report "$sb/alone" "$sb/report" "$REPO_ROOT/modules/association/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/report/report_knit.log")"; return; } + local md; md=$(cat "$sb/report/report.md") + assert_contains "$md" "The one tested site of pt_wingspan (association_pt_wingspan.tsv)" \ + "one tested site is the one, not all 1" + assert_contains "$md" "It does not reach the smallest p this run could report." \ + "and whether it reaches the floor is said of it alone" +} + # THE STRONGEST SITE STILL COUNTS ITSELF. A rearrangement reaches the observed statistic to within # rounding, and the rounding grows with it: rebuilt, the identity lands up to 4e-14 of S away # from S. Counted against an absolute 1e-12 it went uncounted above S of about 100, nothing else @@ -545,12 +589,12 @@ test_the_strongest_site_still_counts_itself() { >> "$table" python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" - if [ ! -s "$sb/run/association.tsv" ]; then + if [ ! -s "$sb/run/association_pt_wingspan.tsv" ]; then fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" return fi for pos in 213 341 399; do - assert_close "$(site_cell "$sb/run/association.tsv" chr3 "$pos" perm_p)" \ + assert_close "$(site_cell "$sb/run/association_pt_wingspan.tsv" chr3 "$pos" perm_p)" \ "0.00138888888888889" "chr3:$pos beats every rearrangement but its own" done agrees_with_oracle "$sb/run" "$sb/oracle.tsv" S perm_p fdr_p @@ -564,7 +608,7 @@ test_a_phenotype_that_disagrees_within_a_unit_refuses() { local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-disagree") association_corpus "$sb" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" "$CORPUS_DIR/design_timed.json" - [ -s "$sb/run/association.tsv" ] \ + [ -s "$sb/run/association_pt_wingspan.tsv" ] \ && fail_case "a unit carrying two phenotype values must refuse, and it published instead" grep -q "values of pt_wingspan" "$sb/run/out.txt" \ || fail_case "the refusal must name the column and the values"$'\n'"$(cat "$sb/run/out.txt")" @@ -586,13 +630,13 @@ test_both_paths_through_the_parse_agree() { # A FAILURE, NOT A SKIP. The analysis environment pins gcc_linux-64 and gxx_linux-64, so a # compiled path that does not build is a broken release rather than a machine without a # compiler. As a skip this read as "no compiler" and the run still exited 0. - if [ ! -s "$sb/compiled/association.tsv" ]; then + if [ ! -s "$sb/compiled/association_pt_wingspan.tsv" ]; then fail_case "the compiled path did not build: $(tail -3 "$sb/compiled/out.txt")" return fi - diff -q "$sb/plain/association.tsv" "$sb/compiled/association.tsv" >/dev/null \ + diff -q "$sb/plain/association_pt_wingspan.tsv" "$sb/compiled/association_pt_wingspan.tsv" >/dev/null \ || fail_case "the compiled parse and the R parse published different site tables" - diff -q "$sb/plain/association_alleles.tsv" "$sb/compiled/association_alleles.tsv" \ + diff -q "$sb/plain/alleles_pt_wingspan.tsv" "$sb/compiled/alleles_pt_wingspan.tsv" \ >/dev/null \ || fail_case "the compiled parse and the R parse published different allele tables" } @@ -640,6 +684,445 @@ test_association_cites_the_statistics_it_computes() { && fail_case "association must not redefine hivert2018: a BibTeX key is defined once" } +# NAMING SEQUENCES RESTRICTS THE FIT TO THEM, and each one named is drawn. Z, 2026-10-08: "User +# should be able to pick chromosomes ... Otherwise with scaffolds etc it is messy." The manual had +# said the setting only chose which sequences got a Manhattan plot. With the dispersion pinned a +# site's statistic does not move; its fdr_p does, adjusted across the named sequences alone. +test_naming_sequences_restricts_the_fit_to_them() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-chromosomes") + association_corpus "$sb" + local options + options=$(printf '%s' "$ASSOCIATION_OPTIONS" | sed 's/"chromosomes":\[\]/"chromosomes":["chr10"]/') + assert_contains "$options" '"chromosomes":["chr10"]' "the options name chr10" + association_direct "$sb/all" "$ASSOCIATION_OPTIONS" + association_direct "$sb/chr10" "$options" + if [ ! -s "$sb/chr10/association_pt_wingspan.tsv" ]; then + fail_case "nothing published"$'\n'"$(cat "$sb/chr10/out.txt")" + return + fi + + local sites + sites=$(awk -F'\t' 'NR > 1 && $1 == "chr10"' "$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" | wc -l) + assert_eq "$((sites + 1))" "$(wc -l < "$sb/chr10/association_pt_wingspan.tsv" | tr -d ' ')" \ + "a header and a row for each of chr10's $sites sites" + assert_eq "" "$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["chrom"]) != "chr10"' "$sb/chr10/association_pt_wingspan.tsv")" \ + "and no row off chr10" + assert_eq "manhattan_chr10.png" "$(cd "$sb/chr10" && ls manhattan_*.png 2>/dev/null | tr '\n' ' ' | sed 's/ $//')" \ + "chr10 is drawn, and nothing else" + assert_eq "$(site_cell "$sb/all/association_pt_wingspan.tsv" chr10 1700 S)" \ + "$(site_cell "$sb/chr10/association_pt_wingspan.tsv" chr10 1700 S)" \ + "the dispersion pinned, chr10:1700's statistic is the same either way" + [ "$(site_cell "$sb/all/association_pt_wingspan.tsv" chr10 1700 fdr_p)" \ + != "$(site_cell "$sb/chr10/association_pt_wingspan.tsv" chr10 1700 fdr_p)" ] \ + || fail_case "chr10:1700's fdr_p must be adjusted across chr10 alone" +} + +# EACH PHENOTYPE IS PUBLISHED IN FILES OF ITS OWN. Z, 2026-10-08: "Let's do one file per +# phenotype ... Joining files for comparing sounds good." The site table is the phenotype's whole +# table with no phenotype column, and its allele table holds the sites its own selection names: in +# one table for every phenotype the first reportTop slots went to whichever came first, so a second +# phenotype could get none. +# +# THREE UNITS OF TWO put most sites at the floor, 1/6, which is where the tie-break decides the +# selection. For pt_resistant the tied site first in file order is chr2:400 and the one with the +# largest S chr10:500, so reportTop 1 tells the report's order from the file's. +test_each_phenotype_is_published_in_files_of_its_own() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-per-phenotype") + association_corpus "$sb" + paired_design "$CORPUS_DIR/design.json" "$sb/three.json" + association_direct "$sb/run" \ + '{"phenotypes":["pt_wingspan","pt_resistant"],"permutations":5000,"dispersion":0,"fdr":"BH","reportBelow":0,"reportTop":1,"chromosomes":[],"binSize":100000,"workers":1,"usecpp":false}' \ + "$sb/three.json" + + local phenotype header sites + for phenotype in pt_wingspan pt_resistant; do + if [ ! -s "$sb/run/association_$phenotype.tsv" ]; then + fail_case "$phenotype has no site table"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + assert_file "$sb/run/alleles_$phenotype.tsv" "$phenotype has an allele table of its own" + header=$(head -1 "$sb/run/association_$phenotype.tsv") + assert_not_contains "$header" "phenotype" "the file names the phenotype, so no column does" + sites=$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + !seen[$(h["chrom"]) "\t" $(h["pos"])]++ { print $(h["chrom"]) "\t" $(h["pos"]) }' \ + "$sb/run/alleles_$phenotype.tsv") + assert_eq "$(report_site_order "$sb/run" "$phenotype" | head -1)" "$sites" \ + "$phenotype's allele table holds its own first site, in the report's order" + done + assert_eq "chr10 500" "$(report_site_order "$sb/run" pt_resistant | head -1)" \ + "the corpus's tie at 1/6 for pt_resistant goes to chr10:500, by S" + assert_no_file "$sb/run/association.tsv" "and no one table holds every phenotype" +} + +# TWO PHENOTYPES THAT WOULD SHARE A FILE NAME ARE REFUSED before any work, naming both. A name's +# characters outside [A-Za-z0-9._-] become _ in its file name, so pt_wing span and pt_wing_span +# would both be association_pt_wing_span.tsv. +test_two_phenotypes_under_one_file_name_refuse() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-file-names") + association_corpus "$sb" + python3 - "$CORPUS_DIR/design.json" "$sb/clash.json" <<'PY' +import copy, json, sys +design = json.load(open(sys.argv[1])) +wingspan = next(entry for entry in design["phenotypes"] if entry["column"] == "pt_wingspan") +for name in ("pt_wing span", "pt_wing_span"): + design["phenotypes"].append(dict(copy.deepcopy(wingspan), column=name)) +json.dump(design, open(sys.argv[2], "w")) +PY + association_direct "$sb/run" \ + "${ASSOCIATION_OPTIONS/\"pt_wingspan\"/\"pt_wing span\",\"pt_wing_span\"}" "$sb/clash.json" + assert_contains "$(cat "$sb/run/out.txt")" \ + "the phenotypes 'pt_wing span' and 'pt_wing_span' would be published under one name, association_pt_wing_span.tsv" \ + "the refusal names both and the file they would share" + assert_no_file "$sb/run/permutations.tsv" "and nothing is published" +} + +# The row the report's site table gives one site of the phenotype $2, out of its two published +# tables in $1: every number as REPORT_NUMBER_AWK prints it, the leverage to two digits and the +# rest to three, and a flagged site's S as the word. The allele is the one of the largest |t|, or +# of the steepest slope where the site is flagged, at six significant digits, between two that tie +# the one whose slope is positive, and past 20 bases its first 12 and its length. A site the allele +# table holds no row for shows NA for both. +# +# awk reads Inf as 0 (see assert_close), so nothing here converts a flagged site's S or t. +report_site_row() { + local run="$1" phenotype="$2" chrom="$3" pos="$4" + awk -F'\t' -v c="$chrom" -v s="$pos" "$REPORT_NUMBER_AWK"' + function cell(x, d) { return "\"" rnum(x, d) "\"" } + function allele_text(a) { + return length(a) > 20 ? substr(a, 1, 12) "... (" length(a) " bases)" : a + } + FNR == 1 { split("", h); for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["chrom"]) != c || $(h["pos"]) != s { next } + FILENAME ~ /\/association_[^\/]*[.]tsv$/ { + flagged = $(h["zero_variance"]) == 1 + row = " \"" c "\", \"" commas(s) "\", \"" $(h["kind"]) "\", " cell($(h["k"]), 3) \ + ", " cell($(h["n_observed"]), 3) ", " \ + (flagged ? "\"flagged\"" : cell($(h["S"]), 3)) ", " \ + cell($(h["perm_p"]), 3) ", " cell($(h["fdr_p"]), 3) + leverage = cell($(h["max_leverage"]), 2) + next + } + { + size = flagged ? $(h["b1"]) : $(h["t"]) + if (size < 0) size = -size + size = sprintf("%.6g", size) + 0 + slope = $(h["b1"]) + 0 + if (!seen++ || size > best || (size == best && slope > 0 && chosen <= 0)) { + best = size; allele = $(h["allele"]); chosen = slope + } + } + END { + if (row == "") exit + print row ", " (seen ? "\"" allele_text(allele) "\", " cell(chosen, 3) \ + : "\"NA\", \"NA\"") ", " leverage "," + }' "$run/association_$phenotype.tsv" "$run/alleles_$phenotype.tsv" +} + +# The sites of the phenotype $2 in the order its table gives them, out of its site table in $1: the +# smallest perm_p first, a flagged site ahead of the others it ties with, then the larger |S|, then +# file order. One "chrompos" a line. +report_site_order() { + awk -F'\t' -v OFS='\t' ' + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["perm_p"]) == "NA" { next } + { + flagged = $(h["zero_variance"]) == 1 + size = $(h["S"]) + 0 + if (size < 0) size = -size + print $(h["perm_p"]), (flagged ? 0 : 1), (flagged ? 0 : size), NR, $(h["chrom"]), $(h["pos"]) + }' "$1/association_$2.tsv" \ + | LC_ALL=C sort -t "$(printf '\t')" -k1,1g -k2,2n -k3,3gr -k4,4n | cut -f5,6 +} + +# The first two cells of each row of the table in the report.md text in $1 whose caption holds $2, +# as "chrompos" with the position's thousands marks taken out. +report_table_sites() { + printf '%s\n' "$1" | awk -v want="$2" ' + /^#layout\(region/ { n = 0 } + /^ let flat = \($/ { inside = 1; next } + inside && /^ \)$/ { inside = 0; next } + inside { rows[++n] = $0; next } + index($0, "caption: \"") && index($0, want) { + for (i = 1; i <= n; i++) { + split(rows[i], cell, "\", \"") + sub(/^ *"/, "", cell[1]) + gsub(",", "", cell[2]) + print cell[1] "\t" cell[2] + } + exit + }' +} + +# The rows of the report's table of what each fit assumed, out of permutations.tsv in $1, as +# report.md holds them. +report_measures_rows() { + awk -F'\t' "$REPORT_NUMBER_AWK"' + function add(label, value) { + if (!(label in line)) order[++m] = label + line[label] = line[label] ", \"" value "\"" + } + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + { + add("Units", rnum($(h["units"]), 3)) + add("Rearrangements", rnum($(h["permutations"]), 3) \ + ($(h["exhaustive"]) == "TRUE" ? ", every one" : ", sampled")) + add("Smallest p the design allows", rnum($(h["design_floor"]), 3)) + add("Smallest p this run could report", rnum($(h["floor"]), 3)) + add("Sites read", rnum($(h["sites"]), 3)) + add("Sites tested", rnum($(h["tested"]), 3)) + add("Selected, fdr_p \342\211\244 0.05", rnum($(h["selected"]), 3)) + add("dispersion", rnum($(h["dispersion"]), 3)) + add("depth_phenotype_cor", rnum($(h["depth_phenotype_cor"]), 3)) + add("lambda_gc", rnum($(h["lambda_gc"]), 3)) + add("Alleles a site, tested", rnum($(h["arity_mean"]), 3)) + add("Alleles a site, selected", rnum($(h["arity_selected"]), 3)) + } + END { for (i = 1; i <= m; i++) print " \"" order[i] "\"" line[order[i]] "," }' \ + "$1/permutations.tsv" +} + +# The rows of the report's phenotype table for $2, out of phenotype.tsv in $1, each cell as written. +report_phenotype_rows() { + awk -F'\t' -v p="$2" ' + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["phenotype"]) == p { + print " \"" $(h["unit"]) "\", \"" $(h["pool"]) "\", \"" $(h["shown"]) "\", \"" \ + $(h["value"]) "\", \"" $(h["fitted"]) "\"," + }' "$1/phenotype.tsv" +} + +# The sentence the report's site table gives one phenotype's count at the floor, out of the +# published tables in $1: the tested sites whose perm_p equals the run's smallest floor to within +# printing. +report_floor_count() { + awk -F'\t' -v p="$2" ' + FNR == 1 { split("", h); for (i = 1; i <= NF; i++) h[$i] = i; next } + FILENAME ~ /permutations[.]tsv$/ { + if ($(h["phenotype"]) != p) next + if (least == "" || $(h["floor"]) + 0 < least) least = $(h["floor"]) + 0 + next + } + $(h["perm_p"]) != "NA" { tested++; if ($(h["perm_p"]) + 0 <= least * (1 + 1e-6)) at++ } + END { printf "%d of the %d tested sites %s the smallest p this run could report", + at, tested, (at == 1 ? "reaches" : "reach") }' \ + "$1/permutations.tsv" "$1/association_$2.tsv" +} + +# The caption of one phenotype's site table in the report.md in $1. Each phenotype's caption +# carries a count that can be the other's word for word, so a count is looked for in its own. +report_site_caption() { + printf '%s\n' "$1" | grep -F 'figure(kind: table, caption: ' | grep -E "sites of $2[, ]" +} + +# THE REPORT'S TABLES, read out of the typst they are drawn with: no Nextflow and no PDF. +# +# Both phenotypes over both depth tables, and every expectation is read out of the published +# tables here by awk rather than from the R the report runs. chr1:700 is biallelic, so its two +# alleles carry one |t| with opposite slopes and the report names the one whose slope is +# positive: for pt_wingspan that is the ALT, which is the case that tells the rule from taking the +# first row. At chr1:400 the two |t| differ in the fifteenth digit, which is what tells a +# comparison at six digits from one at full precision. pt_resistant's chr10:1600 and chr10:1800 +# are flagged. +test_association_report_lays_out_the_sites() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-report-md") + association_corpus "$sb" + local options + options=$(printf '%s' "$ASSOCIATION_OPTIONS" \ + | sed 's/"phenotypes":\["pt_wingspan"\]/"phenotypes":["pt_wingspan","pt_resistant"]/') + assert_contains "$options" '"pt_resistant"' "the options ask for both phenotypes" + association_direct "$sb/run" "$options" "" "Test_snp_depth.tsv,Test_indel_depth.tsv" + if [ ! -s "$sb/run/association_pt_wingspan.tsv" ]; then + fail_case "nothing published"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + analysis_render_report "$sb/run" "$sb/report" "$REPO_ROOT/modules/association/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/report/report_knit.log")"; return; } + local md; md=$(cat "$sb/report/report.md") + + local expected + while IFS= read -r expected; do + assert_contains "$md" "$expected" "the design table holds permutations.tsv: $expected" + done < <(report_measures_rows "$sb/run") + + local phenotype reach + for phenotype in pt_wingspan pt_resistant; do + while IFS= read -r expected; do + assert_contains "$md" "$expected" "$phenotype as phenotype.tsv holds it: $expected" + done < <(report_phenotype_rows "$sb/run" "$phenotype") + assert_eq "$(report_site_order "$sb/run" "$phenotype")" \ + "$(report_table_sites "$md" "sites of $phenotype,")" \ + "$phenotype: every tested site, in the order the caption gives" + reach=$(report_floor_count "$sb/run" "$phenotype") + assert_contains "$(report_site_caption "$md" "$phenotype")" "$reach" "$phenotype: $reach" + done + + local site row chrom pos + for site in pt_wingspan:chr1:700 pt_wingspan:chr2:550 pt_wingspan:chr1:400 \ + pt_resistant:chr1:700 pt_resistant:chr1:400 \ + pt_resistant:chr10:1600 pt_resistant:chr10:1800; do + IFS=: read -r phenotype chrom pos <<< "$site" + row=$(report_site_row "$sb/run" "$phenotype" "$chrom" "$pos") + [ -n "$row" ] || { fail_case "$site: no row to expect in $sb/run"; continue; } + assert_contains "$md" "$row" "$site is laid out from its tables: $row" + done + assert_contains "$(report_site_row "$sb/run" pt_wingspan chr1 700)" '"T", "0.0587",' \ + "the corpus gives pt_wingspan's chr1:700 the ALT, T, as the allele that rises" + assert_contains "$(report_site_row "$sb/run" pt_resistant chr10 1600)" '"flagged"' \ + "and pt_resistant's chr10:1600 is flagged" + + local coded + coded=$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["phenotype"]) == "pt_resistant" { print $(h["coded_one"]); exit }' \ + "$sb/run/phenotype.tsv") + assert_eq "present" "$coded" "the corpus codes pt_resistant's present as 1" + assert_contains "$md" "more frequent in the units coded $coded" \ + "a binary phenotype's slope reads by the level coded 1" + assert_contains "$md" "more frequent at the higher value" "and a quantitative one's by its value" + assert_not_contains "$md" "can reach is" "six units put the floor below 0.05" + assert_not_contains "$md" "NA is a site" "every site is in the allele table at reportTop 100000" + + # Two states no corpus site holds, planted in the published tables. pt_wingspan's chr1:700 one + # step above the floor, 2/720: it is not at the floor, however the two were printed. And a + # third allele at the flagged chr10:1600 of pt_resistant whose |t| is the largest and whose + # slope is not the steepest: a flagged site's |t| is not comparable, so its slope decides. + awk -F'\t' -v OFS='\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; print; next } + $(h["chrom"]) == "chr1" && $(h["pos"]) == 700 { + $(h["perm_p"]) = sprintf("%.15g", 2 / 720) + } + { print }' "$sb/run/association_pt_wingspan.tsv" > "$sb/stepped.tsv" \ + && mv "$sb/stepped.tsv" "$sb/run/association_pt_wingspan.tsv" + awk -F'\t' -v OFS='\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; print; next } + $(h["chrom"]) == "chr10" && $(h["pos"]) == 1600 && $(h["b1"]) > 0 { + $(h["t"]) = "1e16" + } + { print } + END { print "snp", "chr10", "1600", "C", "0.5", "0.1", "Inf", "0" }' \ + "$sb/run/alleles_pt_resistant.tsv" > "$sb/planted.tsv" \ + && mv "$sb/planted.tsv" "$sb/run/alleles_pt_resistant.tsv" + analysis_render_report "$sb/run" "$sb/planted" "$REPO_ROOT/modules/association/report.Rmd" md \ + || { fail_case "the planted report should knit: $(cat "$sb/planted/report_knit.log")"; return; } + md=$(cat "$sb/planted/report.md") + assert_eq "0.00277777777777778" "$(site_cell "$sb/run/association_pt_wingspan.tsv" chr1 700 perm_p)" \ + "chr1:700 now sits one step above the floor" + reach=$(report_floor_count "$sb/run" pt_wingspan) + assert_contains "$(report_site_caption "$md" pt_wingspan)" "$reach" \ + "a site one step above the floor is not counted at it: $reach" + row=$(report_site_row "$sb/run" pt_resistant chr10 1600) + assert_contains "$row" '"G", "1",' "the planted C has the largest |t| and G the steepest slope" + assert_contains "$md" "$row" "and the report names the steepest: $row" +} + +# A TABLE CUT AT TWENTY, a sampled run, an allele table that holds three sites, and every sequence +# named for a Manhattan plot: the paths a real run takes that the corpus does not. +# +# The corpus's SNP table twice over, the copy 5,000 bases on, gives 28 sites and 26 tested: chr10:1500 +# and its copy are flagged untested. A budget of 100 rearrangements cannot enumerate 720, so the +# run's floor, 1/101, parts from the design's, 1/720. reportBelow 0 and reportTop 3 leave 23 of +# the 26 with no allele rows. +test_the_report_takes_the_paths_a_real_run_takes() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-report-paths") + association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" + awk -F'\t' -v OFS='\t' 'NR == 1 { print; next } + { print; $2 = $2 + 5000; copy[++n] = $0 } + END { for (i = 1; i <= n; i++) print copy[i] }' "$table" > "$table.new" \ + && mv "$table.new" "$table" + association_direct "$sb/run" \ + '{"phenotypes":["pt_wingspan"],"permutations":100,"dispersion":0,"fdr":"BH","reportBelow":0,"reportTop":3,"chromosomes":["chr1","chr2","chr10"],"binSize":100000,"workers":1,"usecpp":false}' + if [ ! -s "$sb/run/association_pt_wingspan.tsv" ]; then + fail_case "nothing published"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + analysis_render_report "$sb/run" "$sb/report" "$REPO_ROOT/modules/association/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/report/report_knit.log")"; return; } + local md; md=$(cat "$sb/report/report.md") + + assert_eq "26" "$(report_site_order "$sb/run" pt_wingspan | grep -c .)" "26 tested sites" + assert_contains "$md" "The 20 sites of pt_wingspan with the smallest permutation p" \ + "the table says it shows twenty" + assert_eq "$(report_site_order "$sb/run" pt_wingspan | head -20)" \ + "$(report_table_sites "$md" "sites of pt_wingspan with")" \ + "and they are the first twenty in the order its caption gives" + + local expected + while IFS= read -r expected; do + assert_contains "$md" "$expected" "the design table holds permutations.tsv: $expected" + done < <(report_measures_rows "$sb/run") + assert_contains "$(report_measures_rows "$sb/run")" '"100, sampled"' "the run sampled" + assert_eq "0.00990" "$(awk -F'\t' "$REPORT_NUMBER_AWK"' + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + { print rnum($(h["floor"]), 3); exit }' "$sb/run/permutations.tsv")" \ + "and could report 1/101 where the design allows 1/720" + + assert_contains "$md" '"NA", "NA",' "a site the allele table does not hold shows NA" + assert_contains "$md" "NA is a site alleles_pt_wingspan.tsv holds no row for" \ + "and the caption says what NA means" + + assert_contains "$md" "## Association along a sequence" "every named sequence is drawn" + local at1 at2 at10 + at1=$(grep -n -F 'Association along chr1 (manhattan\_chr1\.png)' <<< "$md" | cut -d: -f1) + at2=$(grep -n -F 'Association along chr2 (manhattan\_chr2\.png)' <<< "$md" | cut -d: -f1) + at10=$(grep -n -F 'Association along chr10 (manhattan\_chr10\.png)' <<< "$md" | cut -d: -f1) + if [ -z "$at1" ] || [ -z "$at2" ] || [ -z "$at10" ]; then + fail_case "a Manhattan plot is missing its caption, the file name escaped: $at1 $at2 $at10" + elif [ "$at1" -gt "$at2" ] || [ "$at2" -gt "$at10" ]; then + fail_case "the Manhattan plots are in file-name order, not the tables' chr1, chr2, chr10" + fi + assert_eq "4" "$(grep -c '^!\[' <<< "$md")" "qq.png and three Manhattan plots are placed" +} + +# THE DESIGN AND THE SCALE ARE READ BACK IN WORDS. Three units of two put the smallest p any site +# can reach at 1/6, which the report says before any site; every fit in a run shares that floor, +# as it is set by the units alone. An ordinal phenotype is read as each level's place in its order, +# so a positive slope means further along that order, not toward any one level. +test_the_report_says_what_the_design_and_the_scale_allow() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-report-scale") + association_corpus "$sb" + paired_design "$CORPUS_DIR/design.json" "$sb/three.json" + association_direct "$sb/three" "$ASSOCIATION_OPTIONS" "$sb/three.json" + analysis_render_report "$sb/three" "$sb/three-report" \ + "$REPO_ROOT/modules/association/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/three-report/report_knit.log")"; return; } + assert_contains "$(cat "$sb/three-report/report.md")" \ + "**With 3 units, the smallest p any site can reach is 0.167, above 0.05.** The sites below are a ranking of effect sizes and not a test." \ + "three units cannot reach 0.05, and the report says so first" + + python3 - "$CORPUS_DIR/design.json" "$sb/ordinal.json" <<'PY' +import json, sys +design = json.load(open(sys.argv[1])) +levels = ["low", "mid", "high"] +stage = {"TestSample1": 0, "TestSample2": 1, "TestSample3": 0, + "TestSample4": 2, "TestSample5": 1, "TestSample6": 2} +design["phenotypes"].append({ + "column": "pt_stage", "kind": "ordinal", "levels": levels, + "values": [{"pool": entry["pool"], "shown": levels[stage[entry["pool"]]], + "group": stage[entry["pool"]], "value": float(stage[entry["pool"]])} + for entry in design["pools"]]}) +json.dump(design, open(sys.argv[2], "w")) +PY + association_direct "$sb/ordinal" "${ASSOCIATION_OPTIONS/pt_wingspan/pt_stage}" "$sb/ordinal.json" + if [ ! -s "$sb/ordinal/association_pt_stage.tsv" ]; then + fail_case "the ordinal run published nothing"$'\n'"$(cat "$sb/ordinal/out.txt")" + return + fi + analysis_render_report "$sb/ordinal" "$sb/ordinal-report" \ + "$REPO_ROOT/modules/association/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/ordinal-report/report_knit.log")"; return; } + local md; md=$(cat "$sb/ordinal-report/report.md") + assert_contains "$md" "Each level is read as its place in the order low < mid < high, from 0" \ + "the phenotype table says how the levels were read" + assert_contains "$md" "more frequent further along low < mid < high" \ + "and a positive slope reads along the order" + assert_not_contains "$md" "units coded mid" "never toward the level coded 1" +} + # ONE CASE THROUGH NEXTFLOW, and it is what every other case here cannot do. The rest call the # module's R directly, which proves the arithmetic and says nothing about main.nf - so a fault # in how the PROCESS assembles its command survives every one of them. @@ -694,7 +1177,8 @@ CFG assert_status 0 "$status" "association should run; see $ANALYSIS_SB/run.out" local dir="$ANALYSIS_SB/main/Analysis/Results/association" - assert_file "$dir/association.tsv" "the site table" + assert_file "$dir/association_pt_wingspan.tsv" "the site table, named after its phenotype" + assert_file "$dir/alleles_pt_wingspan.tsv" "and its allele table" assert_file "$dir/permutations.tsv" "the diagnostics that say what it assumed" assert_file "$dir/phenotype.tsv" "the phenotype as it was fitted" assert_file "$dir/association.R" "the script that produced them" @@ -703,4 +1187,25 @@ CFG assert_file "$dir/allele_frequencies.cpp" "the compiled parse, published either way" assert_contains "$(cat "$dir/association.R")" "allele_frequencies <- function" \ "the libraries it declares must be folded into the published script" + + # association lays out its own report, and the frame knits it under its header. + if ! have_report_tools; then + skip_case "the analysis environment has no pandoc and typst"; return + fi + if ! command -v pdftotext > /dev/null 2>&1 || ! command -v pdfimages > /dev/null 2>&1; then + skip_case "no pdftotext and pdfimages to read the report back"; return + fi + assert_file "$dir/report.pdf" "the report is built; see $ANALYSIS_SB/run.out" + local text section; text=$(pdf_text "$dir/report.pdf") + for section in "The phenotype as it was read" "What the design can show" "The strongest sites" \ + "The p-values against the uniform"; do + assert_contains "$text" "$section" "the report has its '$section' section" + done + local floor; floor=$(published_cell "$dir/permutations.tsv" pt_wingspan design_floor) + assert_contains "$text" "$(awk -v x="$floor" "$REPORT_NUMBER_AWK"'BEGIN { print rnum(x, 3) }')" \ + "the design's floor, $floor, is printed" + assert_eq "1" "$(pdf_figures "$dir/report.pdf")" "qq.png is placed, and no sequence was named" + assert_eq "" "$(pdf_overlapping_words "$dir/report.pdf")" "and no word is printed over another" + assert_eq "" "$(pdf_missing_words "$dir/report.pdf" Measure Leverage Sequence Position)" \ + "or into its neighbor" } diff --git a/modules/basicstats/basicstats.R b/modules/basicstats/basicstats.R index b2e871a..27f6d28 100644 --- a/modules/basicstats/basicstats.R +++ b/modules/basicstats/basicstats.R @@ -188,9 +188,11 @@ kinds <- vapply(depth_files, kind_of, "") # sort() would put chr10 before chr2 in every table below. chrom_levels <- unique(unlist(lapply(tables, function(dt) dt$CHROM), use.names = FALSE)) -# Site counts, per chromosome and per kind. A property of the tables, so they carry no pool. +# Site counts, per chromosome and per kind. A property of the tables, so they carry no pool. A table +# with no rows, as step 7 writes when no indel survived, adds none. site_rows <- do.call(rbind, lapply(seq_along(tables), function(i) { dt <- tables[[i]] + if (nrow(dt) == 0) return(NULL) alleles <- 1 + lengths(strsplit(dt$ALT, ",", fixed = TRUE)) here <- intersect(chrom_levels, unique(dt$CHROM)) data.frame(chrom = here, kind = unname(kinds[i]), diff --git a/modules/basicstats/manifest.json b/modules/basicstats/manifest.json index 52e6b29..c93addf 100644 --- a/modules/basicstats/manifest.json +++ b/modules/basicstats/manifest.json @@ -1,6 +1,6 @@ { "name": "basicstats", - "version": "20261006.001", + "version": "20261008.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -25,8 +25,9 @@ "A site is segregating for a pool when an allele other than that pool's OWN major one reaches max(detection limit, minReads / depth). The cohort's major allele is not used: a pool fixed for what the cohort calls alternate is not segregating.", "A pool of one chromosome is refused: the unbiased correction n_eff/(n_eff - 1) divides by zero there, and a single haploid genome has no segregating sites to measure.", "Pool sizes and ploidy come from the project's metadata as the frame resolved them, not from anything this module infers.", - "Nothing is plotted until a sequence is named. A genome has more sequences than anyone wants plots for, and which of them is worth looking at is not something this module guesses." + "No depth plot along a sequence is drawn until a sequence is named. A genome has more sequences than anyone wants plots for, and which of them is worth looking at is not something this module guesses." ], + "report": "report.Rmd", "outputs": [ { "file": "design.tsv", diff --git a/modules/basicstats/report.Rmd b/modules/basicstats/report.Rmd new file mode 100644 index 0000000..84b6a82 --- /dev/null +++ b/modules/basicstats/report.Rmd @@ -0,0 +1,282 @@ +```{r basicstats-tables} +# basicstats' own report, knitted by the frame's report.Rmd with the functions of +# analysis/lib/rmd/report.R. It reads the tables this module published and nothing else. +# +# Pools keep design.tsv's order and sequences the order sites.tsv gives them. +design <- report_read("design.tsv") +diversity <- report_read("diversity.tsv") +neff <- report_read("neff.tsv") +depth <- report_read("depth.tsv") +sites <- report_read("sites.tsv") +num <- function(x) suppressWarnings(as.numeric(x)) + +pools <- design$pool +fixed <- c("pool", "libraries", "n_libraries", "pool_size", "ploidy", "n_chrom", + "detection_limit") +variables <- setdiff(names(design), fixed) +div <- diversity[match(pools, diversity$pool), ] +genome <- neff[neff$level == "pool" & neff$source == "histogram", ] +genome <- genome[match(pools, genome$pool), ] + +sequences <- unique(sites$chrom) +of_kind <- function(kind, column) { + rows <- sites$kind == kind + totals <- tapply(num(sites[[column]][rows]), factor(sites$chrom[rows], levels = sequences), sum) + counts <- stats::setNames(as.numeric(totals), sequences) + counts[is.na(counts)] <- 0 + counts +} +snp_sites <- of_kind("snp", "sites") +snp_alleles <- of_kind("snp", "alleles") +indel_sites <- of_kind("indel", "sites") +indel_alleles <- of_kind("indel", "alleles") + +# Named in the tables below: every sequence when there are few, otherwise the SHOWN with the most +# called SNP sites, still in the tables' order. The rest are added up in one row. +SHOWN <- 20 +named <- sequences +if (length(sequences) > SHOWN + 1) { + named <- sequences[sequences %in% sequences[order(-snp_sites)][seq_len(SHOWN)]] +} +rest <- setdiff(sequences, named) +rest_label <- sprintf("%d other sequences", length(rest)) + +# One pool's column of depth.tsv, for the named sequences in their order. +depth_key <- paste(depth$pool, depth$chrom, sep = "\r") +depth_at <- function(pool, column) { + num(depth[[column]][match(paste(pool, named, sep = "\r"), depth_key)]) +} + +# Each pool's figures over the sequences added up in one row: its unmeasured sites summed, and its +# harmonic mean depth recombined from each sequence's harmonic mean and the sites it was taken +# over, n / sum(n_i / H_i). +in_rest <- depth$chrom %in% rest +read_n <- num(depth$sites) - num(depth$unmeasured) +harmonic <- num(depth$depth_harmonic) +counted <- in_rest & read_n > 0 & !is.na(harmonic) & harmonic > 0 +by_pool <- function(values, rows) { + totals <- tapply(values[rows], factor(depth$pool[rows], levels = pools), sum) + stats::setNames(as.numeric(totals), pools) +} +rest_harmonic <- by_pool(read_n, counted) / by_pool(read_n / harmonic, counted) +rest_unmeasured <- by_pool(num(depth$unmeasured), in_rest) +rest_unmeasured[is.na(rest_unmeasured)] <- 0 +``` + +`r length(pools)` `r if (length(pools) == 1) "pool" else "pools"`, `r report_number(sum(snp_sites))` called SNP sites and `r report_number(sum(indel_sites))` indel sites, on `r length(sequences)` `r if (length(sequences) == 1) "sequence" else "sequences"`. + +## Gene diversity + +```{r basicstats-diversity, results = "asis"} +table <- data.frame(Pool = pools, check.names = FALSE, stringsAsFactors = FALSE) +for (v in variables) table[[v]] <- design[[v]] +table[["Chromosomes"]] <- report_number(div$n_chrom) +table[["Sites read"]] <- report_number(num(div$sites) - num(div$unmeasured)) +table[["Segregating"]] <- report_number(div$segregating) +table[["h_sum"]] <- report_number(div$h_sum, decimals = 2) +table[["\u03c0 per called site"]] <- report_number(div$pi_per_called_site, digits = 4) +report_table(table, caption = paste0( + "Gene diversity over the ", report_number(sum(snp_sites)), " called SNP sites, each pool over ", + "the sites it was read at (diversity.tsv). \u03c0 per called site leaves invariant sites out ", + "of its denominator and is an upper bound on \u03c0.")) +``` + +## Depth and effective sample size + +```{r basicstats-neff, results = "asis"} +bound <- !is.na(genome$estimate) & genome$estimate == "lower_bound" +table <- data.frame(Pool = pools, check.names = FALSE, stringsAsFactors = FALSE) +table[["Unmeasured"]] <- report_number(div$unmeasured) +table[["Depth"]] <- report_number(div$depth_harmonic, decimals = 1) +table[["n_eff"]] <- report_number(div$n_eff_harmonic, decimals = 1) +table[["Positions"]] <- report_number(genome$positions) +table[["Depth "]] <- paste0(ifelse(bound, "\u2265 ", ""), report_number(genome$depth_harmonic, decimals = 1)) +table[["n_eff "]] <- paste0(ifelse(bound, "\u2265 ", ""), report_number(genome$n_eff, decimals = 1)) +report_table(table, + groups = c("", "At the called SNP sites", "At the called SNP sites", "At the called SNP sites", + "Genome-wide", "Genome-wide", "Genome-wide"), + caption = paste0( + "Depth is the harmonic mean, which every n_eff is computed from. At the called SNP sites: ", + "over the sites each pool was read at (diversity.tsv). Genome-wide: over every position ", + "its libraries covered, from the depth histograms (neff.tsv); NA where a pool has none, ", + "and \u2265 where a merged pool's figure is a lower bound.")) + +libraries <- neff[neff$level == "library", ] +if (any(num(design$n_libraries) > 1) && nrow(libraries) > 0) { + table <- data.frame(Library = libraries$id, Pool = libraries$pool, + Positions = report_number(libraries$positions), + Depth = report_number(libraries$depth_harmonic, decimals = 1), + n_eff = report_number(libraries$n_eff, decimals = 1), + check.names = FALSE, stringsAsFactors = FALSE) + report_table(table, caption = "Each library on its own, genome-wide (neff.tsv).") +} +``` + +```{r basicstats-pools-figure, results = "asis"} +# A merged pool's genome-wide figures are lower bounds and are drawn as their own series. +genome_series <- ifelse(!is.na(genome$estimate) & genome$estimate == "lower_bound", + "genome-wide, a lower bound", "genome-wide") +panel <- function(metric, values, series) { + data.frame(metric = metric, pool = pools, value = num(values), series = series, + stringsAsFactors = FALSE) +} +long <- rbind( + panel("\u03c0 per called site", div$pi_per_called_site, "called SNP sites"), + panel("Segregating sites", div$segregating, "called SNP sites"), + panel("Depth (harmonic)", div$depth_harmonic, "called SNP sites"), + panel("Depth (harmonic)", genome$depth_harmonic, genome_series), + panel("n_eff", div$n_eff_harmonic, "called SNP sites"), + panel("n_eff", genome$n_eff, genome_series), + if (any(num(div$unmeasured) > 0)) panel("Unmeasured sites", div$unmeasured, "called SNP sites")) +long <- long[!is.na(long$value), ] +long$metric <- factor(long$metric, levels = unique(long$metric)) +long$pool <- factor(long$pool, levels = rev(pools)) +figure <- ggplot2::ggplot(long, ggplot2::aes(x = value, y = pool, shape = series)) + + ggplot2::geom_point(size = 2) + + ggplot2::facet_wrap(~ metric, scales = "free_x", nrow = 1) + + ggplot2::scale_shape_manual(values = c("called SNP sites" = 16, "genome-wide" = 1, + "genome-wide, a lower bound" = 2)) + + ggplot2::scale_x_continuous(n.breaks = 3) + + ggplot2::labs(x = NULL, y = NULL, shape = NULL) + + ggplot2::theme_bw(base_size = 9) + + ggplot2::theme(legend.position = "bottom", panel.spacing.x = ggplot2::unit(14, "pt")) +report_plot(figure, "pools", width = 7.5, height = min(8.5, 0.22 * length(pools) + 1.6), + caption = "The pools side by side, from diversity.tsv and neff.tsv as the two tables above give them.") +``` + +## Depth by sequence + +```{r basicstats-depth, results = "asis"} +# The pools a table holds side by side, at most eight to a table. +pool_groups <- report_columns(pools) +ends <- cumsum(lengths(pool_groups)) +for (k in seq_along(pool_groups)) { + group <- pool_groups[[k]] + table <- data.frame(Sequence = named, check.names = FALSE, stringsAsFactors = FALSE) + table[["SNP sites"]] <- report_number(snp_sites[named]) + for (pool in group) { + table[[pool]] <- report_number(depth_at(pool, "depth_harmonic"), decimals = 1) + } + if (length(rest) > 0) { + other <- c(rest_label, report_number(sum(snp_sites[rest])), + report_number(rest_harmonic[group], decimals = 1)) + table <- rbind(table, stats::setNames(as.list(other), names(table))) + } + which <- if (length(pool_groups) > 1) sprintf(" Pools %d to %d of %d.", ends[k] - length(group) + 1, + ends[k], length(pools)) else "" + report_table(table, groups = c("", "", rep("Harmonic depth", length(group))), + caption = paste0("Each pool's harmonic mean depth at the called SNP sites of each ", + "sequence it was read at (depth.tsv).", which)) +} + +if (any(num(depth$unmeasured) > 0)) { + for (k in seq_along(pool_groups)) { + group <- pool_groups[[k]] + table <- data.frame(Sequence = named, check.names = FALSE, stringsAsFactors = FALSE) + table[["SNP sites"]] <- report_number(snp_sites[named]) + for (pool in group) { + table[[pool]] <- report_number(depth_at(pool, "unmeasured")) + } + if (length(rest) > 0) { + other <- c(rest_label, report_number(sum(snp_sites[rest])), + report_number(rest_unmeasured[group])) + table <- rbind(table, stats::setNames(as.list(other), names(table))) + } + report_table(table, groups = c("", "", rep("Unmeasured sites", length(group))), + caption = paste0("The called SNP sites each pool has no reads at, which every ", + "depth above leaves out (depth.tsv).")) + } +} + +# Up to eight pools as colored points; past that a grid, a sequence to a row and a pool to a column. +shown <- depth[depth$chrom %in% named, ] +shown$value <- num(shown$depth_harmonic) +shown <- shown[!is.na(shown$value), ] +if (nrow(shown) > 0) { + shown$chrom <- factor(shown$chrom, levels = rev(named)) + shown$pool <- factor(shown$pool, levels = pools) + if (length(pools) <= 8) { + figure <- ggplot2::ggplot(shown, ggplot2::aes(x = value, y = chrom, color = pool)) + + ggplot2::geom_point(size = 2, alpha = 0.85) + + ggplot2::scale_color_manual(values = report_palette(pools)) + + ggplot2::guides(color = ggplot2::guide_legend( + ncol = max(1, min(length(pools), floor(90 / max(nchar(pools))))))) + + ggplot2::labs(x = "Harmonic mean depth at the called SNP sites", y = NULL, color = NULL) + + ggplot2::theme_bw(base_size = 9) + + ggplot2::theme(legend.position = "bottom") + width <- 7 + } else { + figure <- ggplot2::ggplot(shown, ggplot2::aes(x = pool, y = chrom, fill = value)) + + ggplot2::geom_tile(color = "white") + + ggplot2::scale_fill_gradient(low = "#deebf7", high = "#08306b", + name = "Harmonic mean depth\nat the called SNP sites") + + ggplot2::labs(x = NULL, y = NULL) + + ggplot2::theme_minimal(base_size = 9) + + # The left margin holds the first pool's name, leaning left at 60 degrees, at the + # length of the longest. + ggplot2::theme(axis.text.x = ggplot2::element_text(angle = 60, hjust = 1), + panel.grid = ggplot2::element_blank(), + plot.margin = ggplot2::margin(5.5, 5.5, 5.5, + max(5.5, 2 * max(nchar(pools))))) + width <- 7.5 + } + report_plot(figure, "depth_by_sequence", width = width, + height = min(8.5, 0.2 * length(named) + 1.5 + + if (length(pools) > 8) 0.06 * max(nchar(pools)) else 0), + caption = "Depth by sequence, from depth.tsv as the first table of this section gives it.") +} +``` + +## Called sites by sequence + +```{r basicstats-sites, results = "asis"} +table <- data.frame(Sequence = named, + a = report_number(snp_sites[named]), b = report_number(snp_alleles[named]), + c = report_number(indel_sites[named]), d = report_number(indel_alleles[named]), + stringsAsFactors = FALSE) +if (length(rest) > 0) { + table <- rbind(table, data.frame(Sequence = rest_label, + a = report_number(sum(snp_sites[rest])), + b = report_number(sum(snp_alleles[rest])), + c = report_number(sum(indel_sites[rest])), + d = report_number(sum(indel_alleles[rest])), + stringsAsFactors = FALSE)) +} +table <- rbind(table, data.frame(Sequence = "All sequences", + a = report_number(sum(snp_sites)), b = report_number(sum(snp_alleles)), + c = report_number(sum(indel_sites)), d = report_number(sum(indel_alleles)), + stringsAsFactors = FALSE)) +names(table) <- c("Sequence", "Sites", "Alleles", "Sites ", "Alleles ") +report_table(table, groups = c("", "SNPs", "SNPs", "Indels", "Indels"), + caption = paste0("Sites are records of the depth table and alleles rows of the ", + "frequency table, the reference included (sites.tsv).")) +``` + +```{r basicstats-depth-plots, results = "asis"} +# The depth plots in the order of the sequences they draw, each under the sequence's own name. A +# plot's file name carries the name with every character outside [A-Za-z0-9._-] made _. +plots <- report_files("depth_*.png") +if (length(plots) > 0) { + cat("\n\n## Depth along a sequence\n\n") + drawn <- sub("^depth_(.*)[.]png$", "\\1", basename(plots)) + which_sequence <- match(drawn, gsub("[^A-Za-z0-9._-]", "_", sequences)) + for (i in order(which_sequence, drawn)) { + name <- if (is.na(which_sequence[i])) drawn[i] else sequences[which_sequence[i]] + report_image(plots[i], caption = paste0("Depth at the called SNP sites along ", name, + ", one panel per pool (", basename(plots[i]), ").")) + } +} +``` + +## Design + +```{r basicstats-design, results = "asis"} +table <- data.frame(Pool = pools, Libraries = gsub(";", "\n", design$libraries), + check.names = FALSE, stringsAsFactors = FALSE) +for (v in variables) table[[v]] <- design[[v]] +table[["Pool size"]] <- report_number(design$pool_size) +table[["Ploidy"]] <- report_number(design$ploidy) +table[["Chromosomes"]] <- report_number(design$n_chrom) +table[["Detection limit"]] <- report_number(design$detection_limit, digits = 3) +report_table(table, caption = "The design the results were produced under (design.tsv).") +``` diff --git a/modules/basicstats/test/basicstats.sh b/modules/basicstats/test/basicstats.sh index 9a0b3f3..26072cb 100644 --- a/modules/basicstats/test/basicstats.sh +++ b/modules/basicstats/test/basicstats.sh @@ -5,6 +5,7 @@ # covers: modules/basicstats/ modules/lib/ # covers: test/tools/freq_corpus.py # covers: analysis.nf modules/basicstats/main.nf +# covers: analysis/lib/rmd/ # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # @@ -92,6 +93,109 @@ test_basicstats_publishes_a_row_for_every_pool() { "and each row carries the design and the figures it was produced under" } +# THE REPORT LAYS OUT THE RESULTS, not the list of files. basicstats declares a report of its +# own, and the frame knits it under its header: the sections are the module's, and what fills +# them is read back out of the tables it published. +test_basicstats_reports_its_results() { + basicstats_published || return + if ! have_report_tools; then + skip_case "the analysis environment has no pandoc and typst"; return + fi + if ! command -v pdftotext > /dev/null 2>&1; then + skip_case "no pdftotext to read the report back"; return + fi + + local text; text=$(pdf_text "$BASICSTATS_PUBLISHED/report.pdf") + assert_contains "$text" "Produced by PoolSeqFlow $(analysis_release)," \ + "under the frame's header, naming the release" + local section + for section in "Gene diversity" "Depth and effective sample size" "Depth by sequence" \ + "Called sites by sequence" "Design"; do + assert_contains "$text" "$section" "the report has its '$section' section" + done + assert_contains "$text" "Figure 2" "and draws its figures" + assert_contains "$text" "TestSample6" "with every pool in it" + assert_contains "$text" "Pop1" "and the experimental variables beside them" + # One pool's pi, as the table prints it: four significant digits of diversity.tsv's value. + local pi; pi=$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) if ($i == "pi_per_called_site") c = i; next } + $1 == "TestSample1" { printf "%.4g", $c }' "$BASICSTATS_PUBLISHED/diversity.tsv") + [ -n "$pi" ] || fail_case "diversity.tsv should give TestSample1 a pi to look for" + assert_contains "$text" "$pi" "and the numbers are the ones the tables hold" + assert_eq "" "$(pdf_overlapping_words "$BASICSTATS_PUBLISHED/report.pdf")" \ + "and no word is printed over another" +} + +# THE REPORT'S OWN ARITHMETIC, read out of the typst it is drawn with: no Nextflow and no PDF. +# +# Twenty-three sequences with distinct SNP counts, so the three with the fewest are the ones added +# up in one row; a merged pool, whose genome-wide figures are lower bounds; and unmeasured sites, +# which "Sites read" leaves out. The expected depth of the added-up row is computed here, by awk, +# from depth.tsv's per-sequence harmonic means and the sites each was taken over. +test_basicstats_report_adds_up_and_marks_what_it_should() { + have_analysis_r || { skip_case "no analysis environment"; return; } + local sb; sb=$(guard_path "$TEST_TMPDIR/basicstats-report-md") + rm -rf "$sb"; mkdir -p "$sb/folder" + local f="$sb/folder" i + printf 'pool\tlibraries\tn_libraries\tpool_size\tploidy\tn_chrom\tdetection_limit\n' > "$f/design.tsv" + printf 'P1\tP1\t1\t100\t2\t200\t0.0025\nP2\tP2a;P2b\t2\t100\t2\t200\t0.0025\n' >> "$f/design.tsv" + printf 'chrom\tkind\tsites\talleles\n' > "$f/sites.tsv" + printf 'pool\tchrom\tsites\tunmeasured\tdepth_mean\tdepth_median\tdepth_harmonic\n' > "$f/depth.tsv" + for i in $(seq 1 23); do + printf 'chr%d\tsnp\t%d\t%d\n' "$i" "$((i + 5))" "$((2 * (i + 5)))" >> "$f/sites.tsv" + printf 'P1\tchr%d\t%d\t%d\t30\t30\t%d.5\n' "$i" "$((i + 5))" "$((i % 3))" "$((20 + i))" >> "$f/depth.tsv" + printf 'P2\tchr%d\t%d\t0\t40\t40\t%d.25\n' "$i" "$((i + 5))" "$((30 + 2 * i))" >> "$f/depth.tsv" + done + printf 'pool\tn_chrom\tsites\tunmeasured\tsegregating\tdepth_harmonic\tn_eff_harmonic\th_sum\tpi_per_called_site\n' > "$f/diversity.tsv" + printf 'P1\t200\t391\t24\t300\t28.1\t24.6\t90.5\t0.2459\nP2\t200\t391\t0\t310\t40.2\t33.4\t95.1\t0.2432\n' >> "$f/diversity.tsv" + printf 'level\tid\tpool\tsource\tpositions\tdepth_harmonic\tn_chrom\tn_eff\testimate\n' > "$f/neff.tsv" + printf 'library\tP1\tP1\thistogram\t1000\t7.0\t200\t6.8\texact\n' >> "$f/neff.tsv" + printf 'pool\tP1\tP1\thistogram\t1000\t7.0\t200\t6.8\texact\n' >> "$f/neff.tsv" + printf 'library\tP2a\tP2\thistogram\t1000\t6.0\t200\t5.8\texact\n' >> "$f/neff.tsv" + printf 'library\tP2b\tP2\thistogram\t1000\t7.0\t200\t6.8\texact\n' >> "$f/neff.tsv" + printf 'pool\tP2\tP2\thistogram\t1000\t13.0\t200\t12.3\tlower_bound\n' >> "$f/neff.tsv" + + analysis_render_report "$f" "$sb/out" "$REPO_ROOT/modules/basicstats/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/out/report_knit.log")"; return; } + local md; md=$(cat "$sb/out/report.md") + + # chr1 to chr3 hold 6, 7 and 8 SNP sites, the fewest, and are the ones added up. + local p1 p2 + p1=$(awk -F'\t' '$1 == "P1" && ($2 == "chr1" || $2 == "chr2" || $2 == "chr3") { + n = $3 - $4; s += n; r += n / $7 } END { printf "%.1f", s / r }' "$f/depth.tsv") + p2=$(awk -F'\t' '$1 == "P2" && ($2 == "chr1" || $2 == "chr2" || $2 == "chr3") { + n = $3 - $4; s += n; r += n / $7 } END { printf "%.1f", s / r }' "$f/depth.tsv") + assert_contains "$md" "\"3 other sequences\", \"21\", \"$p1\", \"$p2\"," \ + "the added-up row: the sequences counted, their SNP sites, and each pool's recombined depth" + assert_contains "$md" "\"3 other sequences\", \"21\", \"3\", \"0\"," \ + "and its unmeasured sites summed, 1 + 2 + 0 for P1" + # The sign is written by its code point here: it reaches the file as UTF-8 from R's escape. + local bound=$'\u2265' + assert_contains "$md" "\"P2\", \"0\", \"40.2\", \"33.4\", \"1,000\", \"$bound 13.0\", \"$bound 12.3\"," \ + "the merged pool's genome-wide figures are marked as lower bounds" + assert_contains "$md" "\"P1\", \"24\", \"28.1\", \"24.6\", \"1,000\", \"7.0\", \"6.8\"," \ + "and an exact pool's are not" + assert_contains "$md" "\"P1\", \"200\", \"367\", \"300\", \"90.50\"," \ + "sites read are the sites less the unmeasured ones, 391 - 24" +} + +# A project where no indel survived publishes an indel depth table holding only its header. +# Until 2026-10-08 basicstats stopped on it, so such a project got no results and no report. +test_basicstats_runs_when_no_indel_survived() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/basicstats-no-indel") + rm -rf "$sb"; mkdir -p "$sb" + CORPUS_DIR="$sb" + python3 "$REPO_ROOT/test/tools/freq_corpus.py" "$sb" "$sb" + local table="$sb/Frequencies/Test_indel_depth.tsv" + head -1 "$table" > "$table.header" && mv "$table.header" "$table" + + basicstats_direct "$sb/out" '{"minReads":2,"binSize":100000,"workers":1,"usecpp":false}' "$sb" + assert_file "$sb/out/diversity.tsv" "the module should run: $(cat "$sb/out/out.txt" 2>/dev/null)" + assert_not_contains "$(cat "$sb/out/sites.tsv" 2>/dev/null)" "indel" "counting no indel" + assert_eq "$(corpus_expects sites.chr1.snp.sites)" \ + "$(published_cell2 "$sb/out/sites.tsv" chr1 snp sites)" "and the SNPs as ever" +} + # THE METHODS ARE CITED, NOT ONLY THE SOFTWARE. A diversity estimate a reader cannot trace to a # definition is one they cannot check, and the two n_eff forms in circulation differ. test_basicstats_cites_the_statistics_it_computes() { diff --git a/modules/mds/citations.json b/modules/mds/citations.json index 6f011fc..beb8225 100644 --- a/modules/mds/citations.json +++ b/modules/mds/citations.json @@ -16,7 +16,8 @@ "defined once across the whole manual, and this module's own citation is the ordination it", "makes of that distance.", "", - "ggplot2 and jsonlite are credited in analysis/references.bib for the same reason.", + "ggplot2 and jsonlite are credited in analysis/references.bib for the same reason. ggrepel is", + "this module's alone, and its version is asked of R at run time.", "", "GENERATED from references.bib beside this file. Do not edit it: dev/scripts/bib2citations.py overwrites it, and 00_static fails when the two differ." ], @@ -33,5 +34,15 @@ "year": "1966", "doi": "10.2307/2333639", "note": "The ordination this module draws: a matrix of squared distances double centered into a Gram matrix, whose leading eigenvectors are the coordinates. It is what R's own cmdscale cites and implements, and the source of the negative eigenvalues this module publishes rather than hides - they are what a distance matrix that no flat space holds exactly produces." + }, + "ggrepel": { + "name": "ggrepel", + "type": "misc", + "key": "rggrepel", + "r_package": "ggrepel", + "authors": "Slowikowski, Kamil", + "title": "ggrepel: Automatically Position Non-Overlapping Text Labels with 'ggplot2'", + "url": "https://CRAN.R-project.org/package=ggrepel", + "note": "Places each pool's name in mds.png off its point and off every other name." } } diff --git a/modules/mds/main.nf b/modules/mds/main.nf index ea67789..3646077 100644 --- a/modules/mds/main.nf +++ b/modules/mds/main.nf @@ -22,6 +22,7 @@ def settingDefaults() { return [ dimensions : 2, colorBy : '', shapeBy : '', + labels : true, includeIndels: false, chromosomes : [], binSize : 100000, @@ -82,6 +83,7 @@ process Analyze { options = groovy.json.JsonOutput.toJson([ dimensions : settings.dimensions, colorBy : settings.colorBy, shapeBy : settings.shapeBy, + labels : settings.labels, includeIndels: settings.includeIndels, chromosomes : settings.chromosomes, binSize : settings.binSize, diff --git a/modules/mds/manifest.json b/modules/mds/manifest.json index 7f076dd..a50c3a1 100644 --- a/modules/mds/manifest.json +++ b/modules/mds/manifest.json @@ -1,6 +1,6 @@ { "name": "mds", - "version": "20261006.002", + "version": "20261008.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -33,6 +33,7 @@ "bin/cap_depth.awk keeps reads in coordinate order until the cap rather than subsampling, so at a capped position the retained reads are biased toward those mapping earliest. That is an allele-frequency bias and not only a depth truncation.", "filterFalsePositives.sh admits an allele that clears half of one chromosome's frequency in a fraction of the samples, and MajorAlleleToRef.py re-polarizes on cohort totals. Both depend on which pools were in the run, so a distance from six pools is not the same quantity as one from twelve and the two are not comparable across targets." ], + "report": "report.Rmd", "outputs": [ { "file": "mds.tsv", @@ -51,7 +52,7 @@ }, { "file": "mds.png", - "summary": "the pools on the leading two axes, labeled and optionally colored by an experimental variable, and under it a count of the pairs averaged over fewer than 30 sites when there are any", + "summary": "the pools on the leading two axes, each named unless analysis.modules.mds.labels is false and optionally colored and shaped by experimental variables, and under it a count of the pairs averaged over fewer than 30 sites when there are any", "anchor": "mds-plot" }, { @@ -75,6 +76,7 @@ "r-foreach=1.5.2", "r-future=1.76.0", "r-ggplot2=4.0.3", + "r-ggrepel=0.9.8", "r-jsonlite=2.0.0", "r-rcpp=1.1.2" ] diff --git a/modules/mds/mds.R b/modules/mds/mds.R index c300b14..4340296 100644 --- a/modules/mds/mds.R +++ b/modules/mds/mds.R @@ -68,13 +68,21 @@ if (OPTS$workers > 1) { # it is named here rather than left to surface hundreds of lines later as a comparison against # nothing. `colorBy` and `shapeBy` are absent from this list because an empty string is the # documented value of each and means the points carry no such key. -for (needed in c("dimensions", "includeIndels", "binSize", "workers", "usecpp")) { +for (needed in c("dimensions", "labels", "includeIndels", "binSize", "workers", "usecpp")) { if (is.null(OPTS[[needed]])) { stop("mds.R: options.json carries no '", needed, "'. It is written from ", "analysis.modules.mds by this module's main.nf, which sends every setting the ", "module declares.") } } +if (!isTRUE(OPTS$labels) && !isFALSE(OPTS$labels)) { + stop("mds.R: analysis.modules.mds.labels is ", deparse(OPTS$labels), ", and it is true, to ", + "name each pool on the plot, or false.") +} +if (isTRUE(OPTS$labels) && !requireNamespace("ggrepel", quietly = TRUE)) { + stop("mds.R: analysis.modules.mds.labels is true and ggrepel is not installed. It is one of ", + "the packages this module declares: install the module again, or set labels to false.") +} # ---------------------------------------------------------------------------------------- # What the design says to ordinate. @@ -362,7 +370,8 @@ eigen_table <- data.frame( axis = seq_along(placed$values), eigenvalue = placed$values, share = shares$signed, share_absolute = shares$absolute, cumulative = cumsum(shares$signed), cumulative_absolute = cumsum(shares$absolute), - plotted = as.integer(seq_along(placed$values) <= ncol(placed$coords)), + # The axes mds.png draws: the leading two, and none when fewer than two were written. + plotted = as.integer(seq_along(placed$values) <= 2 & ncol(placed$coords) >= 2), stringsAsFactors = FALSE) write.table(eigen_table, file.path(out, "eigenvalues.tsv"), sep = "\t", quote = FALSE, row.names = FALSE, na = "") @@ -389,8 +398,18 @@ if (ncol(placed$coords) >= 2) { ggplot2::aes() } figure <- ggplot2::ggplot(frame, ggplot2::aes(x = x, y = y)) + - ggplot2::geom_point(mapping, size = 3) + - ggplot2::geom_text(ggplot2::aes(label = pool), vjust = -1, size = 3) + + ggplot2::geom_point(mapping, size = 3) + # Each name is moved off its point and off every other name, with a line back to the point + # when it had to go far. A fixed seed and a fixed number of iterations place the names the + # same way on every machine, unless the time limit stops them first. + if (isTRUE(OPTS$labels)) { + figure <- figure + + ggrepel::geom_text_repel(ggplot2::aes(label = pool), size = 3, seed = 1, + max.overlaps = Inf, max.iter = 3e5, max.time = 60) + } + figure <- figure + + ggplot2::scale_x_continuous(expand = ggplot2::expansion(mult = 0.12)) + + ggplot2::scale_y_continuous(expand = ggplot2::expansion(mult = 0.12)) + ggplot2::coord_fixed() + ggplot2::labs(x = label(1), y = label(2), title = "Nei's minimum distance, classical MDS") + diff --git a/modules/mds/references.bib b/modules/mds/references.bib index 584e3fd..4c6cabb 100644 --- a/modules/mds/references.bib +++ b/modules/mds/references.bib @@ -14,7 +14,8 @@ % defined once across the whole manual, and this module's own citation is the ordination it % makes of that distance. % -% ggplot2 and jsonlite are credited in analysis/references.bib for the same reason. +% ggplot2 and jsonlite are credited in analysis/references.bib for the same reason. ggrepel is +% this module's alone, and its version is asked of R at run time. @article{gower1966mds, group = {The statistics PoolSeqFlow computes}, @@ -30,3 +31,13 @@ @article{gower1966mds doi = {10.2307/2333639}, note = {The ordination this module draws: a matrix of squared distances double centered into a Gram matrix, whose leading eigenvectors are the coordinates. It is what R's own cmdscale cites and implements, and the source of the negative eigenvalues this module publishes rather than hides - they are what a distance matrix that no flat space holds exactly produces.}, } + +@misc{rggrepel, + id = {ggrepel}, + name = {ggrepel}, + r_package = {ggrepel}, + authors = {Slowikowski, Kamil}, + title = {ggrepel: Automatically Position Non-Overlapping Text Labels with 'ggplot2'}, + url = {https://CRAN.R-project.org/package=ggrepel}, + note = {Places each pool's name in mds.png off its point and off every other name.}, +} diff --git a/modules/mds/report.Rmd b/modules/mds/report.Rmd new file mode 100644 index 0000000..7e5a36b --- /dev/null +++ b/modules/mds/report.Rmd @@ -0,0 +1,191 @@ +```{r mds-tables} +# mds' own report, knitted by the frame's report.Rmd with the functions of +# analysis/lib/rmd/report.R. It reads the tables this module published and nothing else. +# +# Pools keep mds.tsv's order. +coords <- report_read("mds.tsv") +pairs <- report_read("distance.tsv") +eigen <- report_read("eigenvalues.tsv") +num <- function(x) suppressWarnings(as.numeric(x)) + +pools <- coords$pool +axes <- grep("^dim[0-9]+$", names(coords), value = TRUE) +variables <- setdiff(names(coords), c("pool", "unit", axes)) +units <- length(unique(coords$unit)) +shared <- num(pairs$sites) +absolute <- num(eigen$share_absolute) +drawn <- num(eigen$plotted) == 1 +carried <- if (any(drawn)) { + paste0("the two axes drawn carry ", report_number(100 * sum(absolute[drawn]), decimals = 1), + "% of the absolute eigenvalue sum") +} else { + paste0("the one axis written carries ", report_number(100 * absolute[1], decimals = 1), + "% of the absolute eigenvalue sum, and one axis draws no plot") +} +# An eigenvalue within rounding of zero counts as zero: n pools have at most n - 1 axes, and the +# last eigenvalue lands either side of zero. +eigenvalues <- num(eigen$eigenvalue) +negative <- !is.na(eigenvalues) & eigenvalues < -1e-10 * max(abs(eigenvalues), na.rm = TRUE) +negative_share <- sum(abs(eigenvalues[negative])) / sum(abs(eigenvalues), na.rm = TRUE) + +# One pool-by-pool matrix out of the pair rows, from `column`, with the diagonal left empty. +pool_matrix <- function(column, ...) { + held <- matrix("", length(pools), length(pools), dimnames = list(pools, pools)) + values <- report_number(pairs[[column]], ...) + held[cbind(match(pairs$pool_a, pools), match(pairs$pool_b, pools))] <- values + held[cbind(match(pairs$pool_b, pools), match(pairs$pool_a, pools))] <- values + held +} + +# The pools a matrix holds as columns, at most eight to a table. +column_groups <- report_columns(pools) +ends <- cumsum(lengths(column_groups)) +matrix_tables <- function(held, label, caption) { + for (k in seq_along(column_groups)) { + group <- column_groups[[k]] + table <- data.frame(Pool = pools, held[, group, drop = FALSE], check.names = FALSE, + stringsAsFactors = FALSE) + which <- if (length(column_groups) > 1) sprintf(" Pools %d to %d of %d as columns.", + ends[k] - length(group) + 1, ends[k], + length(pools)) else "" + report_table(table, groups = c("", rep(label, length(group))), + caption = paste0(caption, which)) + } +} +``` + +`r length(pools)` pools from `r units` `r if (units == 1) "unit" else "units"`. Each distance is averaged over the sites both of its pools were read at, `r if (min(shared) == max(shared)) paste(report_number(min(shared)), "a pair") else paste("from", report_number(min(shared)), "to", report_number(max(shared)), "a pair")`, and `r carried`. + +## The pools on the leading axes + +```{r mds-plot, results = "asis"} +for (path in report_files("mds.png")) { + report_image(path, caption = paste0("The pools on the leading two axes (mds.png, drawn from ", + "mds.tsv and eigenvalues.tsv).")) +} +table <- data.frame(Pool = pools, Unit = coords$unit, check.names = FALSE, + stringsAsFactors = FALSE) +for (v in variables) table[[v]] <- coords[[v]] +for (a in axes) table[[sub("^dim", "Axis ", a)]] <- report_number(coords[[a]], digits = 3) +report_table(table, caption = paste0("Each pool's coordinate on every axis the run wrote (mds.tsv). ", + "Each axis is signed so that its largest coordinate is positive.")) +``` + +## How much each axis carries + +```{r mds-eigenvalues, results = "asis"} +percent <- function(x) paste0(report_number(100 * num(x), decimals = 1), "%") +table <- data.frame(Axis = eigen$axis, + Eigenvalue = report_number(eigen$eigenvalue, digits = 3), + "Share" = percent(eigen$share), + "Share of |sum|" = percent(eigen$share_absolute), + "Cumulative |sum|" = percent(eigen$cumulative_absolute), + Drawn = ifelse(drawn, "yes", ""), + check.names = FALSE, stringsAsFactors = FALSE) +report_table(table, caption = paste0( + "Every eigenvalue with its share of the scatter (eigenvalues.tsv, where the shares are ", + "fractions). Share divides by the signed sum and Share of |sum| by the sum of absolute values, ", + "which the plot's axis labels use. ", + if (any(negative)) paste0("Negative eigenvalues carry ", + report_number(100 * negative_share, decimals = 1), + "% of the absolute sum.") + else if (any(eigenvalues < 0, na.rm = TRUE)) + paste0("No eigenvalue is negative beyond rounding: ", + report_number(min(eigenvalues, na.rm = TRUE), digits = 3), " is zero.") + else "No eigenvalue is negative.")) + +scree <- data.frame(axis = factor(eigen$axis, levels = eigen$axis), share = 100 * absolute, + sign = ifelse(negative, "negative", "positive"), stringsAsFactors = FALSE) +figure <- ggplot2::ggplot(scree, ggplot2::aes(x = axis, y = share, fill = sign)) + + ggplot2::geom_col(width = 0.7) + + ggplot2::scale_fill_manual(values = c(positive = "#4d4d4d", negative = "#d6604d")) + + ggplot2::labs(x = "Axis", y = "% of the absolute sum", fill = NULL) + + ggplot2::theme_bw(base_size = 9) + + ggplot2::theme(legend.position = "bottom") +report_plot(figure, "eigenvalues", width = 6, height = 2.8, + caption = "How much each axis carries, from eigenvalues.tsv.") +``` + +## Distances between pools + +```{r mds-distances, results = "asis"} +matrix_tables(pool_matrix("distance", digits = 3), "Corrected distance", + paste0("Nei's minimum distance between every two pools, corrected for sampling ", + "(distance.tsv). A negative entry reads as indistinguishable at this depth; ", + "distances are not floored at zero.")) + +grid <- data.frame(a = factor(c(pairs$pool_a, pairs$pool_b), levels = pools), + b = factor(c(pairs$pool_b, pairs$pool_a), levels = rev(pools)), + value = num(c(pairs$distance, pairs$distance))) +figure <- ggplot2::ggplot(grid, ggplot2::aes(x = a, y = b, fill = value)) + + ggplot2::geom_tile(color = "white") + + ggplot2::scale_fill_gradient2(low = "#2166ac", mid = "white", high = "#b2182b", midpoint = 0, + name = "Corrected\ndistance") + + ggplot2::labs(x = NULL, y = NULL) + + ggplot2::theme_minimal(base_size = 9) + + # The left margin holds the first pool's name, leaning left at 60 degrees, at the length of + # the longest. + ggplot2::theme(axis.text.x = ggplot2::element_text(angle = 60, hjust = 1), + panel.grid = ggplot2::element_blank(), + plot.margin = ggplot2::margin(5.5, 5.5, 5.5, max(5.5, 2 * max(nchar(pools))))) +# The figure grows by the longest pool name, at about 0.05 in a character in its axes, so the +# tiles keep their size. +name_room <- 0.05 * max(nchar(pools)) +side <- min(7, 0.25 * length(pools) + 2.5) +report_plot(figure, "distances", width = side + 1.2 + name_room, height = side + 0.87 * name_room, + caption = "The distance matrix above, from distance.tsv. Blue below zero, red above.") +``` + +## What each distance rests on + +```{r mds-sites, results = "asis"} +few <- num(pairs$few_sites) == 1 +if (min(shared) == max(shared)) { + if (all(few)) { + cat("**Every pair is averaged over ", report_number(min(shared)), " sites, fewer than ", + "30** (few_sites is 1 throughout distance.tsv), so every distance here is rough ", + "rather than wrong.\n\n", sep = "") + } else { + cat("Every pair is averaged over ", report_number(min(shared)), " sites (distance.tsv).\n\n", + sep = "") + } +} else { + matrix_tables(pool_matrix("sites"), "Shared sites", + paste0("How many sites each pair's distance is averaged over (distance.tsv). A ", + "pair rests on the sites both of its pools were read at, so two pairs in ", + "one run can rest on different counts.")) +} + +if (min(shared) != max(shared) && all(few)) { + cat("**Every pair is averaged over fewer than 30 sites** (few_sites is 1 throughout ", + "distance.tsv), so every distance here is rough rather than wrong.\n\n", sep = "") +} else if (!all(few) && any(few)) { + table <- data.frame("Pool A" = pairs$pool_a[few], "Pool B" = pairs$pool_b[few], + Sites = report_number(pairs$sites[few]), + Distance = report_number(pairs$distance[few], digits = 3), + check.names = FALSE, stringsAsFactors = FALSE) + report_table(table, max_rows = 40, source = "distance.tsv", + caption = paste0("The pairs averaged over fewer than 30 sites (few_sites in ", + "distance.tsv). Their distances are rough rather than wrong.")) +} + +before <- data.frame(raw = num(pairs$raw), corrected = num(pairs$distance), + shared = ifelse(few, "fewer than 30 sites", "30 sites or more")) +# Both axes on one scale from zero, so the line of no correction is on the panel however much the +# correction removed. +limits <- range(c(0, before$raw, before$corrected), na.rm = TRUE) +figure <- ggplot2::ggplot(before, ggplot2::aes(x = raw, y = corrected, shape = shared)) + + ggplot2::geom_abline(slope = 1, intercept = 0, color = "gray60", linetype = "dashed") + + ggplot2::geom_hline(yintercept = 0, color = "gray80") + + ggplot2::geom_point(size = 2) + + ggplot2::scale_shape_manual(values = c("30 sites or more" = 16, "fewer than 30 sites" = 1)) + + ggplot2::coord_fixed(xlim = limits, ylim = limits) + + ggplot2::labs(x = "Uncorrected distance", y = "Corrected distance", shape = NULL) + + ggplot2::theme_bw(base_size = 9) + + # The right margin takes half of a tick label at the panel's edge. + ggplot2::theme(legend.position = "bottom", plot.margin = ggplot2::margin(5.5, 16, 5.5, 5.5)) +report_plot(figure, "correction", width = 4.6, height = 5, + caption = paste0("Every pair before and after the sampling correction, from the raw and ", + "distance columns of distance.tsv. The dashed line is no correction; ", + "how far a point sits below it is what the correction removed.")) +``` diff --git a/modules/mds/test/mds.sh b/modules/mds/test/mds.sh index 8a93f8d..2387be2 100644 --- a/modules/mds/test/mds.sh +++ b/modules/mds/test/mds.sh @@ -5,6 +5,7 @@ # covers: modules/mds/ modules/lib/ # covers: test/tools/freq_corpus.py # covers: analysis.nf modules/mds/main.nf +# covers: analysis/lib/rmd/ # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # @@ -156,7 +157,7 @@ pair_cell() { # The options a case uses unless it is testing one of them. Every key main.nf sends, because the # module refuses one that is missing and a fixture that quietly omits half the contract is not # testing the thing that ships. -MDS_OPTIONS='{"dimensions":2,"colorBy":"","shapeBy":"","includeIndels":false,"chromosomes":[],"binSize":100000,"workers":1,"usecpp":false}' +MDS_OPTIONS='{"dimensions":2,"colorBy":"","shapeBy":"","labels":true,"includeIndels":false,"chromosomes":[],"binSize":100000,"workers":1,"usecpp":false}' # --------------------------------------------------------------------------------------- @@ -309,6 +310,18 @@ test_the_eigenvalues_are_published_whole() { "two axes plotted at the default dimensions" assert_close "1" "$(awk -F'\t' 'NR > 1 { s += $3 } END { print s }' \ "$sb/run/eigenvalues.tsv")" "the signed shares sum to one" + + # plotted is what mds.png draws, not what mds.tsv holds: a third axis written is not drawn, + # and one axis draws no plot at all. It marked every written axis until 2026-10-08. + local plotted='NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["plotted"]) == 1 { n++ } END { print n + 0 }' + mds_direct "$sb/three" "${MDS_OPTIONS/\"dimensions\":2/\"dimensions\":3}" + assert_contains "$(head -1 "$sb/three/mds.tsv")" "dim3" "three axes are written" + assert_eq "2" "$(awk -F'\t' "$plotted" "$sb/three/eigenvalues.tsv")" "and two of them plotted" + mds_direct "$sb/one" "${MDS_OPTIONS/\"dimensions\":2/\"dimensions\":1}" + assert_file "$sb/one/mds.tsv" "one axis is written"$'\n'"$(cat "$sb/one/out.txt")" + assert_no_file "$sb/one/mds.png" "and draws no plot" + assert_eq "0" "$(awk -F'\t' "$plotted" "$sb/one/eigenvalues.tsv")" "so no axis is plotted" } # THE POINTS ARE POOLS AND CARRY THE UNIT THEY BELONG TO, so a reader can see which of them @@ -399,6 +412,45 @@ test_the_points_can_carry_a_color_and_a_shape() { "a color and a shape key changed the coordinates" } +# THE NAMES CAN BE LEFT OFF THE PLOT, and nothing else moves when they are: a name is +# presentation. ggrepel places each name with a fixed seed, so two runs draw the same picture; +# without the seed it takes R's random state, which differs from run to run. +test_the_names_can_be_left_off_the_plot() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/mds-labels") + mds_corpus "$sb" + mds_direct "$sb/named" "$MDS_OPTIONS" + mds_direct "$sb/again" "$MDS_OPTIONS" + mds_direct "$sb/bare" "${MDS_OPTIONS/\"labels\":true/\"labels\":false}" + + local run + for run in named again bare; do + if [ ! -s "$sb/$run/mds.png" ]; then + fail_case "the $run run drew no plot"$'\n'"$(cat "$sb/$run/out.txt")" + return + fi + assert_not_contains "$(cat "$sb/$run/out.txt")" "Warning" "the $run run draws without warning" + done + cmp -s "$sb/named/mds.png" "$sb/again/mds.png" \ + || fail_case "the names must land in the same places on every run" + cmp -s "$sb/named/mds.png" "$sb/bare/mds.png" \ + && fail_case "labels = false must draw the plot without the names" + assert_tables_agree "$sb/named/mds.tsv" "$sb/bare/mds.tsv" \ + "leaving the names off moved the coordinates" +} + +# A labels SETTING THAT IS NOT TRUE OR FALSE IS REFUSED. Written as the string 'false' it would +# otherwise read as false, and the names would be gone with nothing said. +test_a_labels_setting_that_is_not_true_or_false_is_refused() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/mds-labels-word") + mds_corpus "$sb" + mds_direct "$sb/run" "${MDS_OPTIONS/\"labels\":true/\"labels\":\"false\"}" + assert_contains "$(cat "$sb/run/out.txt")" 'analysis.modules.mds.labels is "false"' \ + "the refusal names the setting and what it was given" + assert_no_file "$sb/run/distance.tsv" "and nothing is published" +} + # SIX SHAPES AND NO MORE. ggplot2 assigns none to a seventh level and leaves those pools off the # plot with a warning a task swallows, so a picture would be missing samples and not say so. # @@ -582,4 +634,156 @@ test_mds_runs_through_the_frame() { assert_contains "$(cat "$ANALYSIS_SB/run.out")" \ "15 of 15 pairs of pools rest on fewer than 30 shared sites" \ "the warning on thin pairs must reach the console of the run" + + # mds lays out its own report, and the frame knits it under its header. + if ! have_report_tools; then + skip_case "the analysis environment has no pandoc and typst"; return + fi + if ! command -v pdftotext > /dev/null 2>&1 || ! command -v pdfimages > /dev/null 2>&1; then + skip_case "no pdftotext and pdfimages to read the report back"; return + fi + assert_file "$dir/report.pdf" "the report is built; see $ANALYSIS_SB/run.out" + local text section; text=$(pdf_text "$dir/report.pdf") + for section in "The pools on the leading axes" "How much each axis carries" \ + "Distances between pools" "What each distance rests on"; do + assert_contains "$text" "$section" "the report has its '$section' section" + done + assert_eq "4" "$(pdf_figures "$dir/report.pdf")" "mds.png and the report's own three figures" + assert_eq "" "$(pdf_overlapping_words "$dir/report.pdf")" "and no word is printed over another" + assert_eq "" "$(pdf_missing_words "$dir/report.pdf" Eigenvalue Cumulative Drawn Pool)" \ + "or into its neighbor" +} + +# THE REPORT'S MATRIX, read out of the typst it is drawn with: no Nextflow and no PDF. +# +# A row of the distance matrix is that pool's pairs from distance.tsv, in mds.tsv's pool order, +# with its own cell left empty. The expected rows are assembled here by awk from the two tables, +# each distance to three significant digits as `%#.3g` prints it. Every row is checked, because +# distance.tsv holds each pair once and the rows below the first are mostly the other half. +test_mds_report_lays_out_the_distance_matrix() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/mds-report-md") + mds_corpus "$sb" + mds_direct "$sb/run" "$MDS_OPTIONS" + [ -s "$sb/run/distance.tsv" ] || { fail_case "nothing published: $(cat "$sb/run/out.txt")"; return; } + analysis_render_report "$sb/run" "$sb/report" "$REPO_ROOT/modules/mds/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/report/report_knit.log")"; return; } + local md; md=$(cat "$sb/report/report.md") + + local rows row + rows=$(awk -F'\t' ' + FNR == 1 { split("", c); for (i = 1; i <= NF; i++) c[$i] = i; next } + FILENAME ~ /mds[.]tsv$/ { order[++n] = $(c["pool"]); next } + { a = $(c["pool_a"]); b = $(c["pool_b"]); d[a, b] = d[b, a] = $(c["distance"]) } + END { + for (r = 1; r <= n; r++) { + line = " \"" order[r] "\"" + for (i = 1; i <= n; i++) { + cell = (i == r) ? "" : sprintf("%#.3g", d[order[r], order[i]]) + line = line ", \"" cell "\"" + } + print line "," + } + }' "$sb/run/mds.tsv" "$sb/run/distance.tsv") + assert_eq "6" "$(printf '%s\n' "$rows" | grep -c .)" "the corpus's six pools give six rows" + while IFS= read -r row; do + assert_contains "$md" "$row" "a pool's row is its distances in pool order: $row" + done <<< "$rows" + # The corpus's fourteen sites are every pair's, and under 30: said once rather than drawn as a + # matrix of one number or listed pair by pair. + assert_contains "$md" "**Every pair is averaged over 14 sites, fewer than 30**" \ + "pairs that all rest on one count, under 30, are said to" + assert_not_contains "$md" "Shared sites" "with no matrix of the one count" + assert_not_contains "$md" "The pairs averaged over fewer than 30 sites" "and no list of the pairs" + + # One pair on 40 sites and the last eigenvalue a rounding zero below 0, as an eigensolver + # leaves it about half the time. + local mixed="$sb/mixed" + cp -r "$sb/run" "$mixed" + awk -F'\t' -v OFS='\t' ' + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; print; next } + $(h["pool_a"]) == "TestSample1" && $(h["pool_b"]) == "TestSample2" { + $(h["sites"]) = 40; $(h["few_sites"]) = 0 + } + { print }' "$sb/run/distance.tsv" > "$mixed/distance.tsv" + awk -F'\t' -v OFS='\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; print; next } + { line[NR] = $0 } END { + for (r = 2; r < NR; r++) print line[r] + $0 = line[NR]; $(h["eigenvalue"]) = "-3.14e-18"; print }' \ + "$sb/run/eigenvalues.tsv" > "$mixed/eigenvalues.tsv" + analysis_render_report "$mixed" "$sb/mixed-report" "$REPO_ROOT/modules/mds/report.Rmd" md \ + || { fail_case "the mixed report should knit: $(cat "$sb/mixed-report/report_knit.log")"; return; } + md=$(cat "$sb/mixed-report/report.md") + assert_contains "$md" ' "TestSample1", "", "40", "14", "14", "14", "14",' \ + "pairs on different counts get the matrix, the 40 in its row" + assert_contains "$md" ' "TestSample2", "40", "", "14", "14", "14", "14",' "and in its column" + assert_contains "$md" "The pairs averaged over fewer than 30 sites" "and the pairs under 30 a list" + assert_not_contains "$md" "Every pair is averaged over" "rather than a sentence about all of them" + assert_contains "$md" "No eigenvalue is negative beyond rounding: -3.14e-18 is zero." \ + "and an eigenvalue of -3.14e-18 is zero, not negative" +} + +# THE REST OF THE REPORT, read back against the published tables by awk: the opening sentence, +# every row of the coordinates and of the eigenvalues, and the figures placed. Three axes are +# written and two drawn, so the Drawn column and the opening sentence's share are of the two. +# +# And 25 pools, which make four tables of columns: 7, 6, 6 and 6. Cut eight at a time and the last +# took the remainder, 7, 7, 7 and 4, under a comment that said the tables were of even size. +test_mds_report_reads_back_its_tables() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/mds-report-tables") + mds_corpus "$sb" + mds_direct "$sb/run" "${MDS_OPTIONS/\"dimensions\":2/\"dimensions\":3}" + [ -s "$sb/run/mds.tsv" ] || { fail_case "nothing published: $(cat "$sb/run/out.txt")"; return; } + analysis_render_report "$sb/run" "$sb/report" "$REPO_ROOT/modules/mds/report.Rmd" md \ + || { fail_case "the report should knit: $(cat "$sb/report/report_knit.log")"; return; } + local md; md=$(cat "$sb/report/report.md") + + local expected + expected=$(awk -F'\t' "$REPORT_NUMBER_AWK"' + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["plotted"]) == 1 { drawn += $(h["share_absolute"]) } + END { + printf "6 pools from 6 units. Each distance is averaged over the sites both of its " + printf "pools were read at, 14 a pair, and the two axes drawn carry %s of the ", rpct(drawn) + printf "absolute eigenvalue sum." + }' "$sb/run/eigenvalues.tsv") + assert_contains "$md" "$expected" "the opening sentence: $expected" + + while IFS= read -r expected; do + assert_contains "$md" "$expected" "an axis as eigenvalues.tsv holds it: $expected" + done < <(awk -F'\t' "$REPORT_NUMBER_AWK"' + NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + { + printf " \"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\",\n", $(h["axis"]), + rnum($(h["eigenvalue"]), 3), rpct($(h["share"])), rpct($(h["share_absolute"])), + rpct($(h["cumulative_absolute"])), ($(h["plotted"]) == 1 ? "yes" : "") + }' "$sb/run/eigenvalues.tsv") + + while IFS= read -r expected; do + assert_contains "$md" "$expected" "a pool as mds.tsv holds it: $expected" + done < <(awk -F'\t' "$REPORT_NUMBER_AWK"' + NR == 1 { for (i = 1; i <= NF; i++) { h[$i] = i; name[i] = $i }; next } + { + line = " \"" $(h["pool"]) "\", \"" $(h["unit"]) "\"" + for (i = 1; i <= NF; i++) { + if (name[i] != "pool" && name[i] != "unit" && name[i] !~ /^dim[0-9]+$/) { + line = line ", \"" $i "\"" + } + } + for (i = 1; i <= NF; i++) if (name[i] ~ /^dim[0-9]+$/) line = line ", \"" rnum($i, 3) "\"" + print line "," + }' "$sb/run/mds.tsv") + assert_eq "4" "$(grep -c '^!\[' <<< "$md")" \ + "mds.png and the report's own three figures are placed" + + mds_wide_cohort "$sb/wide" 25 + mds_on_cohort "$sb/wide" "${MDS_OPTIONS/\"labels\":true/\"labels\":false}" "$sb/wide-run" + analysis_render_report "$sb/wide-run" "$sb/wide-report" "$REPO_ROOT/modules/mds/report.Rmd" md \ + || { fail_case "the wide report should knit: $(cat "$sb/wide-report/report_knit.log")"; return; } + md=$(cat "$sb/wide-report/report.md") + local range + for range in "1 to 7" "8 to 13" "14 to 19" "20 to 25"; do + assert_contains "$md" "Pools $range of 25 as columns." "25 pools make tables of 7, 6, 6 and 6" + done } From 86bb11b756898c6525cf0f8f780f070a08a366a6 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Thu, 8 Oct 2026 12:56:28 -0400 Subject: [PATCH 3/6] Filename fixes for modules --- modules/README.md | 12 ++++--- modules/association/report.Rmd | 31 ++++++++++------- modules/association/test/association.sh | 39 +++++++++++---------- modules/basicstats/report.Rmd | 42 +++++++++++++++-------- modules/basicstats/test/basicstats.sh | 45 +++++++++++++++---------- modules/mds/report.Rmd | 43 ++++++++++++++--------- modules/mds/test/mds.sh | 26 +++++++++----- 7 files changed, 147 insertions(+), 91 deletions(-) diff --git a/modules/README.md b/modules/README.md index 8be1367..ecaa50d 100644 --- a/modules/README.md +++ b/modules/README.md @@ -116,14 +116,16 @@ Write a library when more than one module wants the same decision-free arithmeti 15b. **Carry your own `citations.json`. It is required, like `manifest.json` and `main.nf`.** The modules published here do not write it by hand: they carry a `references.bib` and compile it with `dev/scripts/bib2citations.py`, so an entry can be pasted from a publisher and the fields that are not bibliographic -- `r_package`, `note`, `id` -- ride along as extra BibTeX fields. The frame reads only the JSON, so a module built any other way is fine; what it must not be is edited in both places. Every published analysis gets `CITATIONS.md` and `references.bib` written into its folder, in the same rename as the result. The frame contributes PoolSeqFlow, Nextflow and R; **the method your module implements and the R packages it uses are yours to declare**, because you are published separately and the frame cannot hold a list of citations for modules that do not exist yet. At the very least, name the libraries you call -- a user cannot credit a package they never learn they depended on. The shape is `analysis/citations.json`'s, one entry per reference, keyed however you like; an entry naming `"r_package": "vegan"` has its version asked of R at run time rather than taken from what the environment pins, so it stays correct if someone repointed it. A directory without the file is refused at DAG-build time, before the verification clears a results folder, exactly as a missing `main.nf` is, and `modules install` refuses an archive that lacks one. -15c. **Declare an `outputs` entry for every file you publish, and say where it is explained.** Each is `{"file": "design.tsv", "summary": "one row per pool", "anchor": "the-experimental-design"}` -- `file` may be a glob, `summary` is the one-line description the reader gets, and `anchor` names a heading of the manual this release ships. A module published separately has nowhere in that manual to point and gives `"url"` instead, in full. The frame renders one `README.md` from every module's declarations plus what every analysis carries anyway, so a folder found months later says how to read itself and no module invents its own way of saying it. Two things are checked for you: an `anchor` no heading answers to is refused before your module starts, and a `file` you declared and did not publish fails the publish with nothing written. **This is where a number that is not a measurement gets said to be one** -- a bound, an estimate with an assumption behind it, a figure whose meaning changes with a setting. A table cell cannot carry that sentence and a `summary` plus a link can. +15c. **Declare an `outputs` entry for every file you publish, and say where it is explained.** Each is `{"file": "design.tsv", "summary": "one row per pool", "anchor": "the-experimental-design"}` -- `file` may be a glob, `summary` is the one-line description the reader gets, and `anchor` names a heading of the manual this release ships. A module published separately has nowhere in that manual to point and gives `"url"` instead, in full. The frame renders one `README.md` from every module's declarations plus what every analysis carries anyway, so a folder found months later says how to read itself and no module invents its own way of saying it. Two things are checked for you: an `anchor` no heading answers to is refused before your module starts, and a `file` you declared and did not publish fails the publish with nothing written. **Declare the name you write.** The frame publishes every file and directory you hand it under the run's `outputPrefix`, so `mds.tsv` is `Test_mds.tsv` in the folder and in the README, and code keeps its own name: anything ending in one of the script extensions of rule 15, `.cpp`, `.c`, `.h` or `.hpp`. What a directory holds keeps its names, and a declared `file` can only name something at the top of the folder. **This is where a number that is not a measurement gets said to be one** -- a bound, an estimate with an assumption behind it, a figure whose meaning changes with a setting. A table cell cannot carry that sentence and a `summary` plus a link can. -15d. **Lay out your own report by naming an `.Rmd` in your directory as `report`.** Every published folder carries `report.pdf`. Without a `report`, it holds each `.tsv` table and `.png` figure you declared, under its file's name, and leaves any other file out. With one, the frame knits your file under its own header, and what follows the header is yours: tables grouped by what they answer and figures drawn from them, the way `basicstats/report.Rmd`, `association/report.Rmd` and `mds/report.Rmd` do. Your chunks run in the task's directory, in a UTF-8 locale, with the functions of `analysis/lib/rmd/report.R` already loaded, and they read only what you published: +15d. **Lay out your own report by naming an `.Rmd` in your directory as `report`.** Every published folder carries a report, `__report_-.pdf`. Without a `report`, it holds each `.tsv` table and `.png` figure you declared, under its file's name, and leaves any other file out. With one, the frame knits your file under its own header, and what follows the header is yours: tables grouped by what they answer and figures drawn from them, the way `basicstats/report.Rmd`, `association/report.Rmd` and `mds/report.Rmd` do. Your chunks run in the task's directory, in a UTF-8 locale, with the functions of `analysis/lib/rmd/report.R` already loaded, and they read only what you published: | Function | What it does | |---|---| -| `report_read(name)` | one of your published tables, every column as text, or `NULL` when you did not publish it | -| `report_files(pattern)` | the published files a glob matches, in name order | +| `report_read(name)` | one of your published tables, by the name you wrote it under, every column as text, or `NULL` when you did not publish it | +| `report_files(pattern)` | the published files a glob of the names you wrote matches, in name order | +| `report_name(name)` | the name a file you wrote is published under, the prefix in front, for a caption | +| `report_logical(name)` | a published name with the prefix taken off, for reading back something you wrote into a file's name | | `report_number(x, digits, decimals)` | numbers as a table prints them: counts whole with thousands separated, anything else to `digits` significant digits or `decimals` places, a magnitude below 1e-4 in scientific notation, and `NA`, `NaN`, `Inf` and `-Inf` as themselves | | `report_table(table, caption, align, groups, size, max_rows, source, total)` | a table that runs across pages with its header repeated. Typst measures every cell and sets it at the largest of 10 to 7 points at which it fits; past that the headers wrap, and the size goes down to 4 points. A newline in a cell sets it on two lines and nothing else breaks a cell; `groups` prints a label once over a run of columns | | `report_text(x)` | a name as markdown that reads exactly as written; put every name you print outside a table through it, since a `*` or an `@` in one is otherwise emphasis or a citation | @@ -132,7 +134,7 @@ Write a library when more than one module wants the same decision-free arithmeti | `report_columns(items, most)` | items split into as few groups of at most `most` as hold them, sizes differing by at most one, for a table with a column per pool | | `report_palette(pools)` | one color per pool, in the order given | -Call `report_table`, `report_image` and `report_plot` from a chunk with `results = "asis"`. A table of one row per site can run to millions of rows, which `report_read` would hold whole as text; read such a table with `data.table::fread` and only the columns you show, as `association/report.Rmd` does, and declare `r-data.table` in your `packages`. A chunk that fails stops the report rather than printing its error into it, and the analysis is published without a report and with the reason. **Say in each caption which file the numbers come from**, so a reader can always get from the report back to the file. A frame older than `20261008.001` ignores the key and lays out what you declared, file by file, so a module that adds a report still installs and runs on it. +Call `report_table`, `report_image` and `report_plot` from a chunk with `results = "asis"`. A table of one row per site can run to millions of rows, which `report_read` would hold whole as text; read such a table with `data.table::fread` and only the columns you show, as `association/report.Rmd` does, and declare `r-data.table` in your `packages`. A chunk that fails stops the report rather than printing its error into it, and the analysis is published without a report and with the reason. **Say in each caption which file the numbers come from**, by the name `report_name()` gives it, so a reader can always get from the report back to the file the folder holds. A frame older than `20261008.001` ignores the key and lays out what you declared, file by file, so a module that adds a report still installs and runs on it. ### Sharing what you derive diff --git a/modules/association/report.Rmd b/modules/association/report.Rmd index 2e92a00..5f95268 100644 --- a/modules/association/report.Rmd +++ b/modules/association/report.Rmd @@ -14,8 +14,8 @@ phenotypes <- unique(runs$phenotype) stem <- function(p) gsub("[^A-Za-z0-9._-]", "_", p) site_rows <- as.data.frame(data.table::rbindlist(lapply(phenotypes, function(p) { sites <- data.table::fread( - file.path(report_folder(), paste0("association_", stem(p), ".tsv")), sep = "\t", - na.strings = "NA", + file.path(report_folder(), report_name(paste0("association_", stem(p), ".tsv"))), + sep = "\t", na.strings = "NA", select = c("kind", "chrom", "pos", "k", "n_observed", "S", "perm_p", "fdr_p", "max_leverage", "zero_variance"), colClasses = list(character = c("kind", "chrom")), data.table = FALSE) @@ -76,7 +76,7 @@ for (p in phenotypes) { } else "" report_table(table, caption = paste0( p, ", ", rows$kind[1], ", as each pool's cell was written and the number the fit used ", - "(phenotype.tsv)", scale, ".")) + "(", report_name("phenotype.tsv"), ")", scale, ".")) } ``` @@ -109,8 +109,9 @@ measures <- list( table <- data.frame(Measure = names(measures), check.names = FALSE, stringsAsFactors = FALSE) for (i in seq_along(labels)) table[[labels[i]]] <- vapply(measures, function(m) m[i], "") report_table(table, caption = paste0( - "What each fit assumed and absorbed (permutations.tsv). lambda_gc reads above 1 on a clean ", - "table: compare it with the units and alleles a site was read in, not with 1.")) + "What each fit assumed and absorbed (", report_name("permutations.tsv"), "). lambda_gc reads ", + "above 1 on a clean table: compare it with the units and alleles a site was read in, not with ", + "1.")) ``` ## The strongest sites @@ -167,15 +168,17 @@ for (p in phenotypes) { check.names = FALSE, stringsAsFactors = FALSE) report_table(table, align = c("l", "r", "l", "r", "r", "r", "r", "r", "l", "r", "r"), caption = paste0( - lead, " (association_", stem(p), ".tsv), with the allele of the largest |t| at each, or ", - "of the steepest slope at a flagged site, and its slope (alleles_", stem(p), ".tsv). A ", + lead, " (", report_name(paste0("association_", stem(p), ".tsv")), "), with the allele of ", + "the largest |t| at each, or of the steepest slope at a flagged site, and its slope (", + report_name(paste0("alleles_", stem(p), ".tsv")), "). A ", "positive slope means the allele is more frequent ", direction, ". ", reach, " the ", "smallest p this run could report.", if (any(flagged)) { " A flagged site's S is not comparable between implementations and is not printed." }, if (anyNA(best$allele)) { - paste0(" NA is a site alleles_", stem(p), ".tsv holds no row for: reportBelow and ", + paste0(" NA is a site ", report_name(paste0("alleles_", stem(p), ".tsv")), + " holds no row for: reportBelow and ", "reportTop choose the sites it holds.") })) } @@ -186,20 +189,22 @@ for (p in phenotypes) { ```{r association-qq, results = "asis"} for (path in report_files("qq.png")) { report_image(path, caption = paste0("Every tested site's permutation p against the uniform, ", - "a panel per phenotype (qq.png, drawn from the ", - "association\\_ tables). The points sit on a ladder whose ", - "ceiling is the design's floor.")) + "a panel per phenotype (", report_text(basename(path)), + ", drawn from the ", report_text(report_name("association_")), + " tables). The points sit on a ladder whose ceiling is ", + "the design's floor.")) } ``` ```{r association-manhattan, results = "asis"} # The Manhattan plots in the order of the sequences they draw, each under the sequence's own -# name. A plot's file name carries the name with every character outside [A-Za-z0-9._-] made _. +# name. A plot's file name, the prefix taken off, carries the name with every character outside +# [A-Za-z0-9._-] made _. plots <- report_files("manhattan_*.png") if (length(plots) > 0) { cat("\n\n## Association along a sequence\n\n") sequences <- unique(site_rows$chrom) - drawn <- sub("^manhattan_(.*)[.]png$", "\\1", basename(plots)) + drawn <- sub("^manhattan_(.*)[.]png$", "\\1", report_logical(basename(plots))) which_sequence <- match(drawn, gsub("[^A-Za-z0-9._-]", "_", sequences)) for (i in order(which_sequence, drawn)) { name <- if (is.na(which_sequence[i])) drawn[i] else sequences[which_sequence[i]] diff --git a/modules/association/test/association.sh b/modules/association/test/association.sh index 69270cc..cb6470e 100644 --- a/modules/association/test/association.sh +++ b/modules/association/test/association.sh @@ -565,8 +565,8 @@ test_a_depth_table_of_one_site_is_fitted() { analysis_render_report "$sb/alone" "$sb/report" "$REPO_ROOT/modules/association/report.Rmd" md \ || { fail_case "the report should knit: $(cat "$sb/report/report_knit.log")"; return; } local md; md=$(cat "$sb/report/report.md") - assert_contains "$md" "The one tested site of pt_wingspan (association_pt_wingspan.tsv)" \ - "one tested site is the one, not all 1" + assert_contains "$md" "The one tested site of pt_wingspan (Test_association_pt_wingspan.tsv)" \ + "one tested site is the one, not all 1, and its table is named as it is published" assert_contains "$md" "It does not reach the smallest p this run could report." \ "and whether it reaches the floor is said of it alone" } @@ -1061,14 +1061,15 @@ test_the_report_takes_the_paths_a_real_run_takes() { "and could report 1/101 where the design allows 1/720" assert_contains "$md" '"NA", "NA",' "a site the allele table does not hold shows NA" - assert_contains "$md" "NA is a site alleles_pt_wingspan.tsv holds no row for" \ + assert_contains "$md" "NA is a site Test_alleles_pt_wingspan.tsv holds no row for" \ "and the caption says what NA means" + # The sequence is read back out of the published name, the prefix taken off first. assert_contains "$md" "## Association along a sequence" "every named sequence is drawn" local at1 at2 at10 - at1=$(grep -n -F 'Association along chr1 (manhattan\_chr1\.png)' <<< "$md" | cut -d: -f1) - at2=$(grep -n -F 'Association along chr2 (manhattan\_chr2\.png)' <<< "$md" | cut -d: -f1) - at10=$(grep -n -F 'Association along chr10 (manhattan\_chr10\.png)' <<< "$md" | cut -d: -f1) + at1=$(grep -n -F 'Association along chr1 (Test\_manhattan\_chr1\.png)' <<< "$md" | cut -d: -f1) + at2=$(grep -n -F 'Association along chr2 (Test\_manhattan\_chr2\.png)' <<< "$md" | cut -d: -f1) + at10=$(grep -n -F 'Association along chr10 (Test\_manhattan\_chr10\.png)' <<< "$md" | cut -d: -f1) if [ -z "$at1" ] || [ -z "$at2" ] || [ -z "$at10" ]; then fail_case "a Manhattan plot is missing its caption, the file name escaped: $at1 $at2 $at10" elif [ "$at1" -gt "$at2" ] || [ "$at2" -gt "$at10" ]; then @@ -1177,11 +1178,13 @@ CFG assert_status 0 "$status" "association should run; see $ANALYSIS_SB/run.out" local dir="$ANALYSIS_SB/main/Analysis/Results/association" - assert_file "$dir/association_pt_wingspan.tsv" "the site table, named after its phenotype" - assert_file "$dir/alleles_pt_wingspan.tsv" "and its allele table" - assert_file "$dir/permutations.tsv" "the diagnostics that say what it assumed" - assert_file "$dir/phenotype.tsv" "the phenotype as it was fitted" - assert_file "$dir/association.R" "the script that produced them" + assert_file "$dir/Test_association_pt_wingspan.tsv" \ + "the site table, named after its phenotype, under the outputPrefix" + assert_file "$dir/Test_alleles_pt_wingspan.tsv" "and its allele table" + assert_file "$dir/Test_permutations.tsv" "the diagnostics that say what it assumed" + assert_file "$dir/Test_phenotype.tsv" "the phenotype as it was fitted" + assert_file "$dir/Test_qq.png" "and the qq plot" + assert_file "$dir/association.R" "the script that produced them, under its own name" # The one the bug above destroyed: a compiled source is published whether or not the run used # it, so its absence is a broken process rather than a choice about the hot path. assert_file "$dir/allele_frequencies.cpp" "the compiled parse, published either way" @@ -1195,17 +1198,19 @@ CFG if ! command -v pdftotext > /dev/null 2>&1 || ! command -v pdfimages > /dev/null 2>&1; then skip_case "no pdftotext and pdfimages to read the report back"; return fi - assert_file "$dir/report.pdf" "the report is built; see $ANALYSIS_SB/run.out" - local text section; text=$(pdf_text "$dir/report.pdf") + local report; report=$(analysis_published_report "$dir" association) + [ -n "$report" ] || { fail_case "the report should be built; see $ANALYSIS_SB/run.out"; return; } + local text section; text=$(pdf_text "$report") for section in "The phenotype as it was read" "What the design can show" "The strongest sites" \ "The p-values against the uniform"; do assert_contains "$text" "$section" "the report has its '$section' section" done - local floor; floor=$(published_cell "$dir/permutations.tsv" pt_wingspan design_floor) + local floor; floor=$(published_cell "$dir/Test_permutations.tsv" pt_wingspan design_floor) assert_contains "$text" "$(awk -v x="$floor" "$REPORT_NUMBER_AWK"'BEGIN { print rnum(x, 3) }')" \ "the design's floor, $floor, is printed" - assert_eq "1" "$(pdf_figures "$dir/report.pdf")" "qq.png is placed, and no sequence was named" - assert_eq "" "$(pdf_overlapping_words "$dir/report.pdf")" "and no word is printed over another" - assert_eq "" "$(pdf_missing_words "$dir/report.pdf" Measure Leverage Sequence Position)" \ + assert_contains "$text" "Test_permutations.tsv" "and the tables are named as they are published" + assert_eq "1" "$(pdf_figures "$report")" "qq.png is placed, and no sequence was named" + assert_eq "" "$(pdf_overlapping_words "$report")" "and no word is printed over another" + assert_eq "" "$(pdf_missing_words "$report" Measure Leverage Sequence Position)" \ "or into its neighbor" } diff --git a/modules/basicstats/report.Rmd b/modules/basicstats/report.Rmd index 84b6a82..e30fc80 100644 --- a/modules/basicstats/report.Rmd +++ b/modules/basicstats/report.Rmd @@ -10,6 +10,16 @@ depth <- report_read("depth.tsv") sites <- report_read("sites.tsv") num <- function(x) suppressWarnings(as.numeric(x)) +# The names the tables are published under, as plain text and as markdown. +design_tsv <- report_name("design.tsv") +diversity_tsv <- report_name("diversity.tsv") +neff_tsv <- report_name("neff.tsv") +depth_tsv <- report_name("depth.tsv") +sites_tsv <- report_name("sites.tsv") +diversity_md <- report_text(diversity_tsv) +neff_md <- report_text(neff_tsv) +depth_md <- report_text(depth_tsv) + pools <- design$pool fixed <- c("pool", "libraries", "n_libraries", "pool_size", "ploidy", "n_chrom", "detection_limit") @@ -77,7 +87,7 @@ table[["h_sum"]] <- report_number(div$h_sum, decimals = 2) table[["\u03c0 per called site"]] <- report_number(div$pi_per_called_site, digits = 4) report_table(table, caption = paste0( "Gene diversity over the ", report_number(sum(snp_sites)), " called SNP sites, each pool over ", - "the sites it was read at (diversity.tsv). \u03c0 per called site leaves invariant sites out ", + "the sites it was read at (", diversity_tsv, "). \u03c0 per called site leaves invariant sites out ", "of its denominator and is an upper bound on \u03c0.")) ``` @@ -97,8 +107,8 @@ report_table(table, "Genome-wide", "Genome-wide", "Genome-wide"), caption = paste0( "Depth is the harmonic mean, which every n_eff is computed from. At the called SNP sites: ", - "over the sites each pool was read at (diversity.tsv). Genome-wide: over every position ", - "its libraries covered, from the depth histograms (neff.tsv); NA where a pool has none, ", + "over the sites each pool was read at (", diversity_tsv, "). Genome-wide: over every position ", + "its libraries covered, from the depth histograms (", neff_tsv, "); NA where a pool has none, ", "and \u2265 where a merged pool's figure is a lower bound.")) libraries <- neff[neff$level == "library", ] @@ -108,7 +118,7 @@ if (any(num(design$n_libraries) > 1) && nrow(libraries) > 0) { Depth = report_number(libraries$depth_harmonic, decimals = 1), n_eff = report_number(libraries$n_eff, decimals = 1), check.names = FALSE, stringsAsFactors = FALSE) - report_table(table, caption = "Each library on its own, genome-wide (neff.tsv).") + report_table(table, caption = paste0("Each library on its own, genome-wide (", neff_tsv, ").")) } ``` @@ -141,7 +151,8 @@ figure <- ggplot2::ggplot(long, ggplot2::aes(x = value, y = pool, shape = series ggplot2::theme_bw(base_size = 9) + ggplot2::theme(legend.position = "bottom", panel.spacing.x = ggplot2::unit(14, "pt")) report_plot(figure, "pools", width = 7.5, height = min(8.5, 0.22 * length(pools) + 1.6), - caption = "The pools side by side, from diversity.tsv and neff.tsv as the two tables above give them.") + caption = paste0("The pools side by side, from ", diversity_md, " and ", neff_md, + " as the two tables above give them.")) ``` ## Depth by sequence @@ -166,7 +177,7 @@ for (k in seq_along(pool_groups)) { ends[k], length(pools)) else "" report_table(table, groups = c("", "", rep("Harmonic depth", length(group))), caption = paste0("Each pool's harmonic mean depth at the called SNP sites of each ", - "sequence it was read at (depth.tsv).", which)) + "sequence it was read at (", depth_tsv, ").", which)) } if (any(num(depth$unmeasured) > 0)) { @@ -184,7 +195,7 @@ if (any(num(depth$unmeasured) > 0)) { } report_table(table, groups = c("", "", rep("Unmeasured sites", length(group))), caption = paste0("The called SNP sites each pool has no reads at, which every ", - "depth above leaves out (depth.tsv).")) + "depth above leaves out (", depth_tsv, ").")) } } @@ -223,7 +234,8 @@ if (nrow(shown) > 0) { report_plot(figure, "depth_by_sequence", width = width, height = min(8.5, 0.2 * length(named) + 1.5 + if (length(pools) > 8) 0.06 * max(nchar(pools)) else 0), - caption = "Depth by sequence, from depth.tsv as the first table of this section gives it.") + caption = paste0("Depth by sequence, from ", depth_md, + " as the first table of this section gives it.")) } ``` @@ -249,21 +261,23 @@ table <- rbind(table, data.frame(Sequence = "All sequences", names(table) <- c("Sequence", "Sites", "Alleles", "Sites ", "Alleles ") report_table(table, groups = c("", "SNPs", "SNPs", "Indels", "Indels"), caption = paste0("Sites are records of the depth table and alleles rows of the ", - "frequency table, the reference included (sites.tsv).")) + "frequency table, the reference included (", sites_tsv, ").")) ``` ```{r basicstats-depth-plots, results = "asis"} # The depth plots in the order of the sequences they draw, each under the sequence's own name. A -# plot's file name carries the name with every character outside [A-Za-z0-9._-] made _. +# plot's file name, the prefix taken off, carries the name with every character outside +# [A-Za-z0-9._-] made _. plots <- report_files("depth_*.png") if (length(plots) > 0) { cat("\n\n## Depth along a sequence\n\n") - drawn <- sub("^depth_(.*)[.]png$", "\\1", basename(plots)) + drawn <- sub("^depth_(.*)[.]png$", "\\1", report_logical(basename(plots))) which_sequence <- match(drawn, gsub("[^A-Za-z0-9._-]", "_", sequences)) for (i in order(which_sequence, drawn)) { name <- if (is.na(which_sequence[i])) drawn[i] else sequences[which_sequence[i]] - report_image(plots[i], caption = paste0("Depth at the called SNP sites along ", name, - ", one panel per pool (", basename(plots[i]), ").")) + report_image(plots[i], caption = paste0("Depth at the called SNP sites along ", + report_text(name), ", one panel per pool (", + report_text(basename(plots[i])), ").")) } } ``` @@ -278,5 +292,5 @@ table[["Pool size"]] <- report_number(design$pool_size) table[["Ploidy"]] <- report_number(design$ploidy) table[["Chromosomes"]] <- report_number(design$n_chrom) table[["Detection limit"]] <- report_number(design$detection_limit, digits = 3) -report_table(table, caption = "The design the results were produced under (design.tsv).") +report_table(table, caption = paste0("The design the results were produced under (", design_tsv, ").")) ``` diff --git a/modules/basicstats/test/basicstats.sh b/modules/basicstats/test/basicstats.sh index 26072cb..6039cc1 100644 --- a/modules/basicstats/test/basicstats.sh +++ b/modules/basicstats/test/basicstats.sh @@ -80,11 +80,11 @@ test_basicstats_publishes_a_row_for_every_pool() { assert_status 0 "$BASICSTATS_STATUS" "basicstats should run; see $ANALYSIS_SB/run.out" local folder="$BASICSTATS_PUBLISHED" - assert_file "$folder/design.tsv" "the design table is published" - assert_file "$folder/basicstats.R" "and the script that produced it" + assert_file "$folder/Test_design.tsv" "the design table is published, under the outputPrefix" + assert_file "$folder/basicstats.R" "and the script that produced it, under its own name" assert_file "$folder/CITATIONS.md" "and what to cite for it" - local table; table=$(cat "$folder/design.tsv" 2>/dev/null) + local table; table=$(cat "$folder/Test_design.tsv" 2>/dev/null) assert_eq 7 "$(printf '%s\n' "$table" | wc -l)" "a header and one row per pool" assert_contains "$table" "pool libraries n_libraries exp_population exp_time pool_size ploidy n_chrom detection_limit" \ "every experimental variable gets a column of its own, between the pool and its size" @@ -105,7 +105,9 @@ test_basicstats_reports_its_results() { skip_case "no pdftotext to read the report back"; return fi - local text; text=$(pdf_text "$BASICSTATS_PUBLISHED/report.pdf") + local report; report=$(analysis_published_report "$BASICSTATS_PUBLISHED" basicstats) + [ -n "$report" ] || { fail_case "the report should be built; see $ANALYSIS_SB/run.out"; return; } + local text; text=$(pdf_text "$report") assert_contains "$text" "Produced by PoolSeqFlow $(analysis_release)," \ "under the frame's header, naming the release" local section @@ -118,10 +120,11 @@ test_basicstats_reports_its_results() { assert_contains "$text" "Pop1" "and the experimental variables beside them" # One pool's pi, as the table prints it: four significant digits of diversity.tsv's value. local pi; pi=$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) if ($i == "pi_per_called_site") c = i; next } - $1 == "TestSample1" { printf "%.4g", $c }' "$BASICSTATS_PUBLISHED/diversity.tsv") + $1 == "TestSample1" { printf "%.4g", $c }' "$BASICSTATS_PUBLISHED/Test_diversity.tsv") [ -n "$pi" ] || fail_case "diversity.tsv should give TestSample1 a pi to look for" assert_contains "$text" "$pi" "and the numbers are the ones the tables hold" - assert_eq "" "$(pdf_overlapping_words "$BASICSTATS_PUBLISHED/report.pdf")" \ + assert_contains "$text" "Test_diversity.tsv" "under the names the tables are published as" + assert_eq "" "$(pdf_overlapping_words "$report")" \ "and no word is printed over another" } @@ -153,11 +156,18 @@ test_basicstats_report_adds_up_and_marks_what_it_should() { printf 'library\tP2a\tP2\thistogram\t1000\t6.0\t200\t5.8\texact\n' >> "$f/neff.tsv" printf 'library\tP2b\tP2\thistogram\t1000\t7.0\t200\t6.8\texact\n' >> "$f/neff.tsv" printf 'pool\tP2\tP2\thistogram\t1000\t13.0\t200\t12.3\tlower_bound\n' >> "$f/neff.tsv" + # A depth plot, empty: report.md places it without reading it. + : > "$f/depth_chr2.png" analysis_render_report "$f" "$sb/out" "$REPO_ROOT/modules/basicstats/report.Rmd" md \ || { fail_case "the report should knit: $(cat "$sb/out/report_knit.log")"; return; } local md; md=$(cat "$sb/out/report.md") + # The plot is placed under its sequence, which the report reads back out of the published name + # with the prefix taken off; read with it on, chr2 is no sequence and the caption names the file. + assert_contains "$md" 'Depth at the called SNP sites along chr2, one panel per pool (Test\_depth\_chr2\.png).' \ + "a depth plot is placed under its sequence's name" + # chr1 to chr3 hold 6, 7 and 8 SNP sites, the fewest, and are the ones added up. local p1 p2 p1=$(awk -F'\t' '$1 == "P1" && ($2 == "chr1" || $2 == "chr2" || $2 == "chr3") { @@ -231,10 +241,10 @@ test_basicstats_publishes_what_it_measured() { assert_status 0 "$BASICSTATS_STATUS" "basicstats should run; see $ANALYSIS_SB/run.out" local folder="$BASICSTATS_PUBLISHED" - assert_file "$folder/sites.tsv" "the site counts are published" - assert_file "$folder/depth.tsv" "and the depth summaries" - assert_file "$folder/diversity.tsv" "and the diversity" - assert_file "$folder/site_diversity.cpp" "and the compiled path, used or not" + assert_file "$folder/Test_sites.tsv" "the site counts are published" + assert_file "$folder/Test_depth.tsv" "and the depth summaries" + assert_file "$folder/Test_diversity.tsv" "and the diversity" + assert_file "$folder/site_diversity.cpp" "and the compiled path, used or not, under its own name" # THE DEFAULT, END TO END. No flag was given, so the compiled path is what must have run - # and the published script's header is the only record of which one did. @@ -245,21 +255,22 @@ test_basicstats_publishes_what_it_measured() { # chr2, which would misorder every genome whose sequences are numbered. assert_eq "chr1 chr2 chr10" \ "$(awk -F'\t' 'NR > 1 && $2 == "snp" { printf "%s%s", sep, $1; sep = " " }' \ - "$folder/sites.tsv" 2>/dev/null)" \ + "$folder/Test_sites.tsv" 2>/dev/null)" \ "the sequences keep the order the depth table gave them" # One number out of each table, against the corpus's own arithmetic. - assert_close "$(published_cell2 "$folder/sites.tsv" chr2 snp alleles)" \ + assert_close "$(published_cell2 "$folder/Test_sites.tsv" chr2 snp alleles)" \ "$(corpus_expects sites.chr2.snp.alleles)" "chr2 holds a triallelic and a tetrallelic site" - assert_close "$(published_cell2 "$folder/depth.tsv" TestSample1 chr1 depth_harmonic)" \ + assert_close "$(published_cell2 "$folder/Test_depth.tsv" TestSample1 chr1 depth_harmonic)" \ "$(corpus_expects depth.TestSample1.chr1.harmonic)" "the harmonic depth on chr1" - assert_close "$(published_cell "$folder/diversity.tsv" TestSample1 h_sum)" \ + assert_close "$(published_cell "$folder/Test_diversity.tsv" TestSample1 h_sum)" \ "$(corpus_expects pool.TestSample1.h_sum)" "the diversity summed over the called sites" - # The README the frame renders has to name every file, or a reader of the folder has no - # account of one of them. + # The README the frame renders has to name every file as the folder holds it, or a reader of + # the folder has no account of one of them. local readme; readme=$(cat "$folder/README.md" 2>/dev/null) - for name in design.tsv sites.tsv depth.tsv diversity.tsv basicstats.R site_diversity.cpp; do + for name in Test_design.tsv Test_sites.tsv Test_depth.tsv Test_diversity.tsv Test_neff.tsv \ + basicstats.R site_diversity.cpp; do assert_contains "$readme" "\`$name\`" "the README accounts for $name" done } diff --git a/modules/mds/report.Rmd b/modules/mds/report.Rmd index 7e5a36b..21c54fe 100644 --- a/modules/mds/report.Rmd +++ b/modules/mds/report.Rmd @@ -8,6 +8,14 @@ pairs <- report_read("distance.tsv") eigen <- report_read("eigenvalues.tsv") num <- function(x) suppressWarnings(as.numeric(x)) +# The names the three tables are published under, as plain text and as markdown. +mds_tsv <- report_name("mds.tsv") +distance_tsv <- report_name("distance.tsv") +eigenvalues_tsv <- report_name("eigenvalues.tsv") +mds_md <- report_text(mds_tsv) +distance_md <- report_text(distance_tsv) +eigenvalues_md <- report_text(eigenvalues_tsv) + pools <- coords$pool axes <- grep("^dim[0-9]+$", names(coords), value = TRUE) variables <- setdiff(names(coords), c("pool", "unit", axes)) @@ -60,15 +68,17 @@ matrix_tables <- function(held, label, caption) { ```{r mds-plot, results = "asis"} for (path in report_files("mds.png")) { - report_image(path, caption = paste0("The pools on the leading two axes (mds.png, drawn from ", - "mds.tsv and eigenvalues.tsv).")) + report_image(path, caption = paste0("The pools on the leading two axes (", + report_text(basename(path)), ", drawn from ", mds_md, + " and ", eigenvalues_md, ").")) } table <- data.frame(Pool = pools, Unit = coords$unit, check.names = FALSE, stringsAsFactors = FALSE) for (v in variables) table[[v]] <- coords[[v]] for (a in axes) table[[sub("^dim", "Axis ", a)]] <- report_number(coords[[a]], digits = 3) -report_table(table, caption = paste0("Each pool's coordinate on every axis the run wrote (mds.tsv). ", - "Each axis is signed so that its largest coordinate is positive.")) +report_table(table, caption = paste0("Each pool's coordinate on every axis the run wrote (", mds_tsv, + "). Each axis is signed so that its largest coordinate is ", + "positive.")) ``` ## How much each axis carries @@ -83,7 +93,7 @@ table <- data.frame(Axis = eigen$axis, Drawn = ifelse(drawn, "yes", ""), check.names = FALSE, stringsAsFactors = FALSE) report_table(table, caption = paste0( - "Every eigenvalue with its share of the scatter (eigenvalues.tsv, where the shares are ", + "Every eigenvalue with its share of the scatter (", eigenvalues_tsv, ", where the shares are ", "fractions). Share divides by the signed sum and Share of |sum| by the sum of absolute values, ", "which the plot's axis labels use. ", if (any(negative)) paste0("Negative eigenvalues carry ", @@ -103,7 +113,7 @@ figure <- ggplot2::ggplot(scree, ggplot2::aes(x = axis, y = share, fill = sign)) ggplot2::theme_bw(base_size = 9) + ggplot2::theme(legend.position = "bottom") report_plot(figure, "eigenvalues", width = 6, height = 2.8, - caption = "How much each axis carries, from eigenvalues.tsv.") + caption = paste0("How much each axis carries, from ", eigenvalues_md, ".")) ``` ## Distances between pools @@ -111,7 +121,7 @@ report_plot(figure, "eigenvalues", width = 6, height = 2.8, ```{r mds-distances, results = "asis"} matrix_tables(pool_matrix("distance", digits = 3), "Corrected distance", paste0("Nei's minimum distance between every two pools, corrected for sampling ", - "(distance.tsv). A negative entry reads as indistinguishable at this depth; ", + "(", distance_tsv, "). A negative entry reads as indistinguishable at this depth; ", "distances are not floored at zero.")) grid <- data.frame(a = factor(c(pairs$pool_a, pairs$pool_b), levels = pools), @@ -133,7 +143,8 @@ figure <- ggplot2::ggplot(grid, ggplot2::aes(x = a, y = b, fill = value)) + name_room <- 0.05 * max(nchar(pools)) side <- min(7, 0.25 * length(pools) + 2.5) report_plot(figure, "distances", width = side + 1.2 + name_room, height = side + 0.87 * name_room, - caption = "The distance matrix above, from distance.tsv. Blue below zero, red above.") + caption = paste0("The distance matrix above, from ", distance_md, + ". Blue below zero, red above.")) ``` ## What each distance rests on @@ -143,30 +154,30 @@ few <- num(pairs$few_sites) == 1 if (min(shared) == max(shared)) { if (all(few)) { cat("**Every pair is averaged over ", report_number(min(shared)), " sites, fewer than ", - "30** (few_sites is 1 throughout distance.tsv), so every distance here is rough ", + "30** (few_sites is 1 throughout ", distance_md, "), so every distance here is rough ", "rather than wrong.\n\n", sep = "") } else { - cat("Every pair is averaged over ", report_number(min(shared)), " sites (distance.tsv).\n\n", - sep = "") + cat("Every pair is averaged over ", report_number(min(shared)), " sites (", distance_md, + ").\n\n", sep = "") } } else { matrix_tables(pool_matrix("sites"), "Shared sites", - paste0("How many sites each pair's distance is averaged over (distance.tsv). A ", + paste0("How many sites each pair's distance is averaged over (", distance_tsv, "). A ", "pair rests on the sites both of its pools were read at, so two pairs in ", "one run can rest on different counts.")) } if (min(shared) != max(shared) && all(few)) { cat("**Every pair is averaged over fewer than 30 sites** (few_sites is 1 throughout ", - "distance.tsv), so every distance here is rough rather than wrong.\n\n", sep = "") + distance_md, "), so every distance here is rough rather than wrong.\n\n", sep = "") } else if (!all(few) && any(few)) { table <- data.frame("Pool A" = pairs$pool_a[few], "Pool B" = pairs$pool_b[few], Sites = report_number(pairs$sites[few]), Distance = report_number(pairs$distance[few], digits = 3), check.names = FALSE, stringsAsFactors = FALSE) - report_table(table, max_rows = 40, source = "distance.tsv", + report_table(table, max_rows = 40, source = distance_tsv, caption = paste0("The pairs averaged over fewer than 30 sites (few_sites in ", - "distance.tsv). Their distances are rough rather than wrong.")) + distance_tsv, "). Their distances are rough rather than wrong.")) } before <- data.frame(raw = num(pairs$raw), corrected = num(pairs$distance), @@ -186,6 +197,6 @@ figure <- ggplot2::ggplot(before, ggplot2::aes(x = raw, y = corrected, shape = s ggplot2::theme(legend.position = "bottom", plot.margin = ggplot2::margin(5.5, 16, 5.5, 5.5)) report_plot(figure, "correction", width = 4.6, height = 5, caption = paste0("Every pair before and after the sampling correction, from the raw and ", - "distance columns of distance.tsv. The dashed line is no correction; ", + "distance columns of ", distance_md, ". The dashed line is no correction; ", "how far a point sits below it is what the correction removed.")) ``` diff --git a/modules/mds/test/mds.sh b/modules/mds/test/mds.sh index 2387be2..e3d0591 100644 --- a/modules/mds/test/mds.sh +++ b/modules/mds/test/mds.sh @@ -618,10 +618,11 @@ test_mds_runs_through_the_frame() { assert_status 0 "$status" "mds should run; see $ANALYSIS_SB/run.out" local dir="$ANALYSIS_SB/main/Analysis/Results/mds" - assert_file "$dir/distance.tsv" "the distances" - assert_file "$dir/eigenvalues.tsv" "the eigenvalues" - assert_file "$dir/mds.tsv" "the coordinates" - assert_file "$dir/mds.R" "the script that produced them" + assert_file "$dir/Test_distance.tsv" "the distances, under the outputPrefix" + assert_file "$dir/Test_eigenvalues.tsv" "the eigenvalues" + assert_file "$dir/Test_mds.tsv" "the coordinates" + assert_file "$dir/Test_mds.png" "the plot" + assert_file "$dir/mds.R" "the script that produced them, under its own name" assert_file "$dir/allele_frequencies.cpp" "the compiled parse, shipped either way" assert_file "$dir/nei_distance.cpp" "and the compiled distance" assert_contains "$(cat "$dir/mds.R")" "nei_distance <- function" \ @@ -642,15 +643,17 @@ test_mds_runs_through_the_frame() { if ! command -v pdftotext > /dev/null 2>&1 || ! command -v pdfimages > /dev/null 2>&1; then skip_case "no pdftotext and pdfimages to read the report back"; return fi - assert_file "$dir/report.pdf" "the report is built; see $ANALYSIS_SB/run.out" - local text section; text=$(pdf_text "$dir/report.pdf") + local report; report=$(analysis_published_report "$dir" mds) + [ -n "$report" ] || { fail_case "the report should be built; see $ANALYSIS_SB/run.out"; return; } + local text section; text=$(pdf_text "$report") for section in "The pools on the leading axes" "How much each axis carries" \ "Distances between pools" "What each distance rests on"; do assert_contains "$text" "$section" "the report has its '$section' section" done - assert_eq "4" "$(pdf_figures "$dir/report.pdf")" "mds.png and the report's own three figures" - assert_eq "" "$(pdf_overlapping_words "$dir/report.pdf")" "and no word is printed over another" - assert_eq "" "$(pdf_missing_words "$dir/report.pdf" Eigenvalue Cumulative Drawn Pool)" \ + assert_contains "$text" "Test_distance.tsv" "and names the tables as they are published" + assert_eq "4" "$(pdf_figures "$report")" "mds.png and the report's own three figures" + assert_eq "" "$(pdf_overlapping_words "$report")" "and no word is printed over another" + assert_eq "" "$(pdf_missing_words "$report" Eigenvalue Cumulative Drawn Pool)" \ "or into its neighbor" } @@ -776,6 +779,11 @@ test_mds_report_reads_back_its_tables() { }' "$sb/run/mds.tsv") assert_eq "4" "$(grep -c '^!\[' <<< "$md")" \ "mds.png and the report's own three figures are placed" + # A caption names the file as it is published: plain in a table's, which typst sets as text, + # and escaped in a figure's, which is markdown. + assert_contains "$md" "every axis the run wrote (Test_mds.tsv)." "a table's caption names its file" + assert_contains "$md" '(Test\_mds\.png, drawn from Test\_mds\.tsv and Test\_eigenvalues\.tsv)' \ + "and so does a figure's" mds_wide_cohort "$sb/wide" 25 mds_on_cohort "$sb/wide" "${MDS_OPTIONS/\"labels\":true/\"labels\":false}" "$sb/wide-run" From 60d6000f3ea451fe5db9ff9a365f592e775003df Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Thu, 8 Oct 2026 12:58:19 -0400 Subject: [PATCH 4/6] Large commit on releasing, modules install all, and test suite prep for 3.3.0 --- .claude/development-notes/mds-labels.md | 40 + .claude/development-notes/module-reports.md | 142 ++++ .claude/development-notes/output-prefix.md | 61 ++ .../site-follows-releases.md | 35 + .github/workflows/docs.yml | 50 +- CLAUDE.md | 8 +- PoolSeqFlow | 260 ++++-- analysis/frame.config | 7 + analysis/frame.version | 2 +- analysis/lib/nf/citations.nf | 4 +- analysis/lib/nf/modules.nf | 12 + analysis/lib/nf/outputs.nf | 53 +- analysis/lib/nf/paths.nf | 22 + analysis/lib/nf/plan.nf | 21 + analysis/lib/nf/report.nf | 79 +- analysis/lib/nf/results.nf | 30 +- analysis/lib/rmd/report.R | 329 ++++++++ analysis/lib/rmd/report.Rmd | 97 +-- bin/check_parameters.sh | 27 +- bin/config_migrate.sh | 43 +- bin/parse_multirun.py | 21 + dev/RELEASING.md | 35 +- dev/scripts/publish-module.sh | 259 ++++-- dev/scripts/raise-module-floors.sh | 32 +- dev/validation/agree.R | 9 +- install/environment-analysis.yml | 1 + lib/_PoolSeqFlow | 7 +- lib/poolseqflow-completion.bash | 12 +- manual/PoolSeqFlow-manual.md | 121 ++- nextflow.config | 7 + parameters.config.template | 9 +- scripts/0_verify_environment.nf | 2 +- scripts/resolve_parameters.nf | 14 +- scripts/variants.nf | 5 +- test/README.md | 14 +- test/lib/analysis.sh | 235 +++++- test/lib/harness.sh | 30 +- test/lib/sandbox.sh | 1 + test/run_tests.sh | 10 + test/suites/00_static.sh | 784 +++++++++++++++--- test/suites/01_migrate.sh | 55 ++ test/suites/02_launcher.sh | 292 ++++++- test/suites/03_helpers.sh | 80 +- test/suites/05_guards.sh | 57 +- test/suites/09_analysis_modules.sh | 26 + test/suites/15_analysis_results.sh | 445 +++++++++- 46 files changed, 3414 insertions(+), 471 deletions(-) create mode 100644 .claude/development-notes/mds-labels.md create mode 100644 .claude/development-notes/module-reports.md create mode 100644 .claude/development-notes/output-prefix.md create mode 100644 .claude/development-notes/site-follows-releases.md create mode 100644 analysis/lib/rmd/report.R diff --git a/.claude/development-notes/mds-labels.md b/.claude/development-notes/mds-labels.md new file mode 100644 index 0000000..3fb060f --- /dev/null +++ b/.claude/development-notes/mds-labels.md @@ -0,0 +1,40 @@ +# Names on mds.png: ggrepel, and a setting to leave them off + +Written 2026-10-08, against `dev` at `d8b7cee` plus the uncommitted report work of that day (`module-reports.md`). Not updated to follow the code. + +## The ask + +The example reports showed `mds.png` with 24 long pool names printed over each other in every cluster, and the outermost names cut at the edge of the image. I asked whether to add `ggrepel`, label with short numbers and a key, or leave it. Z: *"add ggrepel + add disabling the names"*. + +## What was built + +- **`labels`**, a new mds setting, default `true`. `false` draws the points alone. mds.R refuses anything but true or false, because the string `'false'` would otherwise read as false and the names would vanish with nothing said, and it refuses `true` at start when `ggrepel` is not installed, before any work, naming the setting that avoids it. +- **`ggrepel::geom_text_repel`** in place of `geom_text`, with `seed = 1`, `max.overlaps = Inf`, `max.iter = 3e5` and `max.time = 60`. The labels-run-past-the-panel fix from earlier the same day (`coord_fixed(clip = "off")`) is gone; only the 12% the axes leave on each side remains. +- **`r-ggrepel=0.9.8`** in mds' `packages`, credited in `modules/mds/references.bib` as software with its version asked of R at run time, the way basicstats credits data.table. +- **The report's caption** for `mds.png` no longer says each point is labeled, since it may not be. +- **agree.R** runs mds with `labels = FALSE`: it compares tables, and it runs in association's suite, whose environment need not hold a package only mds declares. + +## Facts, measured + +- **conda-forge** has `r-ggrepel` 0.9.8 built for R 4.5 (`r45h3697838_0`), and 0.9.5 and 0.9.6 for older R. A dry-run install into `PoolSeqFlow-3.2.0-analysis` links that one package and changes nothing else: every dependency (ggplot2 >= 3.5.2, Rcpp, rlang >= 1.1.6, S7, scales >= 1.4.0, withr >= 3.0.2) is already in the baseline. +- **License GPL-3**, under which mds (GPL-3.0-or-later) is already published. +- **It is the first module package the analysis baseline does not carry.** Every package any module declared before this is in `install/environment-analysis.yml`, so the package manager had never installed anything for a user. `check-module-packages.sh` step 1 now does real work at a release. +- **Placement time grows with the square of the pool count**, and these names never settle early, so every run spends its whole iteration budget. Measured on this machine with `max.time` out of the way, 38-character names in four clusters, 3e5 iterations: 12 pools 2.0 s, 24 pools 6.4 s, 48 pools 34.7 s, 72 pools 66.8 s. For 24 pools: 1e4 iterations 0.7 s, 1e5 2.3 to 2.7 s, 3e5 6.6 to 9.5 s, 1e6 23 to 31 s. +- **The placement is byte-identical between runs when the iteration limit stops it** (two runs at each of five budgets gave the same md5) and differs when the time limit does, since a slower machine stops earlier. ggrepel's defaults (0.5 s, 1e4 iterations) are time-bound for a few dozen pools, so the seed alone does not make the plot reproducible. With 3e5 and 60 s, the time limit binds first at about 65 pools. +- **Quality**: at 1e5 iterations the 24-name plot still had a few names over each other, at 3e5 almost none in a full panel, and in mds.png's own panel, narrowed by two legends, a few collisions remain. The manual says so and points to `labels = false`. + +## What the cases check, and that they bite + +`the names can be left off the plot`: two runs with names draw identical PNGs, a run without differs, and the coordinates do not move. `a labels setting that is not true or false is refused`. Reverted in a scratch clone, each failed: the seed removed (two runs differ), the setting ignored, the type check removed. Without `ggrepel` on the library path, mds.R stops with its own message and publishes nothing; the frame case and every direct case need the package. + +## How a module suite gets a package the baseline lacks + +The mds cases first ran here with `R_LIBS` pointing at the package unpacked into the scratchpad from conda-forge's own archive. Z then installed `r-ggrepel` 0.9.8, the pinned build, into `PoolSeqFlow-3.2.0-analysis` the same day, so filtered runs have it. Three places need it, and as written only the first has it: + +- a filtered run uses the installed `PoolSeqFlow--analysis`, which holds it only once mds is installed from a catalogue that declares it, or it is installed by hand; +- a full run builds a scratch environment from `install/environment-analysis.yml` alone; +- `prep-version.sh` hands its freshly built pair to the full run. + +I proposed that the full run's scratch environment take every shipped module's packages after it is built, which is what a user has after `analysis modules install all`; `prep-version.sh`'s environment is the release's and the export refuses module packages in it. Z installed the package by hand, and deferred the full run until the shape of association's output was settled. + +**Settled when the full run was due: `ggrepel` is in the baseline.** Z: *"add ggrepel to the environment file. scratch builds from environment right?"* It does: the full run and `prep-version.sh` both build from `install/environment-analysis.yml`. The line `r-ggrepel=0.9.8=r45h3697838_0` was written into that file by hand, against its header's "do not edit by hand", because the documented route cannot take it. `export-environment.sh` refuses an environment holding a package a module declares that the baseline lacks, which is the guard against folding a module's packages in by accident, and this was the same act done on purpose. With the line in place, `export-environment.sh --check PoolSeqFlow-3.2.0-analysis` accepts, so a later export reproduces it. So every package any shipped module declares is in the baseline again, and the question of how a suite gets one that is not did not have to be answered. diff --git a/.claude/development-notes/module-reports.md b/.claude/development-notes/module-reports.md new file mode 100644 index 0000000..4223e86 --- /dev/null +++ b/.claude/development-notes/module-reports.md @@ -0,0 +1,142 @@ +# Module reports: what the PDF was, and what it became + +Written 2026-10-08 (UTC; the evening of 2026-10-07 in Z's time), against `dev` at `d8b7cee` plus the uncommitted work of that day. Not updated to follow the code. + +## The ask + +Z, after the first `Project/` run with `keepLowDepthAsZero`: *"Basic stats report should display the results under as tables but better organized. Also basic stats pdf has one table and it doesn't render right. the end of the first column renders on top of the beginning of second at times. Report should not be a list of what table is what. It should show the results and plots etc."* + +## What the report had been, since v3.0.0 + +The report and its wiring arrived together in `381541e` (2026-09-04, "F1 basicstats, and the PDF report every analysis carries") and shipped in every release from v3.0.0 to v3.2.0. Every `report.pdf` any of them published held a header and a table naming the declared files, and nothing else. Three causes, one of them the bug and two of them why nobody saw it. + +1. **The folder never reached the template.** `InstallResults` hands `reportShell` the destination as the shell text `$STAGE`. `reportSpec` put it into the JSON spec, and the spec was written with `printf '%s' ''`, single-quoted, so the template read the literal string `$STAGE` as its folder. Every `Sys.glob` matched nothing, and every section was skipped by the branch meant for "an optional output this run did not produce". Rendering the same template against a real folder by hand gave 10 sections where the published one had 0. +2. **The case asserted what the broken report printed anyway.** `test_a_published_folder_carries_one_pdf_of_everything_in_it` looked for `result.tsv` in the PDF text, and the contents table at the top printed every declared file name whatever happened below it. It passed on a report holding nothing. This is the gates-that-stop-checking pattern again: the assertion was aimed at the right word in the wrong place. +3. **A report that failed to build failed in silence.** The reason was printed to stderr, and Nextflow never shows a successful task's stderr; `InstallResults` succeeds by design when the report fails. Found while debugging this work: a typst syntax error in a header-only table left no trace on the console. + +The intended layout would not have satisfied the ask either. Rendered by hand with the folder fixed, it gave every heading a page of its own with the table pushed onto the next (a typst `#pagebreak()` before each section), ran long tables off the page with their last rows overprinted (pandoc wraps a pipe table in a `figure`, and a typst figure does not break across pages), overlapped columns, and printed 15-digit doubles. + +**The column overlap Z saw** is pandoc's: when a pipe table has a line longer than `--columns`, pandoc gives each column a fixed fraction of the width from the separator's dash counts, and typst cannot break `diversity.tsv` set in monospace, so it overflows into the next column. + +## The release read as "unknown" + +A module runs as its own Nextflow pipeline (`nextflow run //main.nf`), and Nextflow reads `nextflow.config` only beside the entry script, so `workflow.manifest.version` is null there. The README said "Produced by PoolSeqFlow unknown", the report said the same, and `CITATIONS.md` named PoolSeqFlow without a version (the `.bib` had no "Version X used" note for it). `releaseVersion()` in `paths.nf` now reads `version = 'X.Y.Z'` out of `${installDir()}/nextflow.config`, with the same regex `bump-version.sh` writes it with, and refuses when it finds none. `checkModuleCompatible()` still skips its environment check in a module's own run; `analysis.nf` makes that check first, so it was left alone. + +## What was decided + +**A module may lay out its own report.** The organization Z asked for is specific to what a module computes: basicstats' per-pool table combines `design.tsv`, `diversity.tsv` and `neff.tsv`, and its by-sequence tables pivot `depth.tsv`. A generic renderer cannot know that. So the manifest gains `report`, the name of an `.Rmd` in the module's directory, and the frame knits it as a child under the frame's own header. A module without one gets the generic layout: each declared table and figure under its file name, with no list of files (Z: *"Report should not be a list of what table is what"*). `readManifest()` refuses a `report` that is not a plain `.Rmd` name or is not there, the way it refuses a missing `main.nf`. + +**The frame keeps the header and the drawing functions.** `analysis/lib/rmd/report.R` holds `report_read`, `report_files`, `report_number`, `report_table`, `report_image`, `report_plot`, `report_palette` and the generic `report_declared`. A child calls them and reads only the published folder (`POOLSEQFLOW_REPORT_FOLDER`). The header does not come from the folder: the release, the frame, the design notes and the child's path are in the spec `reportShell` writes into the task directory, which is not published. Nothing rebuilds a report from a published folder. + +**Figures a report draws exist only in the PDF.** They are views of published tables. Publishing them would have made them declared outputs with manual anchors, and Z asked for the report to show them, not for more files. + +**basicstats' `frame` floor was not raised.** An older frame's `readManifest()` returns a fixed map and ignores keys it does not know, so `report` is invisible to it and the module still installs and runs. Z may want the floor raised so that a 3.2.0 installation never shows the old report; that is a choice, recorded here as not made. + +**Tables are typst code, not pandoc tables.** `report_table` emits a raw typst block: a `table` with `table.header` (repeated on every page), right-aligned numbers, light striping, cells as typst *string literals* so nothing a sample or sequence is called is read as markup, and a figure made breakable by a show rule. + +## What was measured + +All on the analysis environment's pandoc 3.12 and typst 0.15.1, with `pdftotext` to read pages back. + +- **pandoc's typst template** sets US letter, 1.25in margins (a 6in text block) and justified paragraphs. The frame's template sets 1in margins through YAML front matter (6.5in) and turns justification off; a justified line holding a long path spread its words across the page. +- **A table inside `#layout` still breaks across pages**: a 120-row table measured inside one ran over three pages. So typst itself chooses the size. My first attempt estimated widths in R from glyph widths; measured Libertinus Serif at 10pt is about 0.465em a digit, 0.468em a lowercase letter and 0.624em an uppercase one, bold about 10% wider. The R estimate ran roughly 10% high and turned a 24-pool table onto a landscape page it did not need. +- **Turned pages strand headings.** `page(flipped: true)` forces a page break, and the section heading stayed behind on the page before. Dropped. +- **Measuring a whole table unconstrained never wraps.** `measure()` of a table gives its natural width, so headers that could wrap read as too wide and the table was scaled to unreadable. The second rule tried, and the one the review took apart, was: the natural layout at 10 to 7 points; then the columns of numbers sharing what was left equally (`1fr`), measured with their headers stacked and displayed free to wrap; then that scaled down with `scale()`. Each of the three later steps failed on a shape it had not been tested on (below). +- **The rule that holds, measured cell by cell.** Every cell and every header is measured once at 10 points, and widths scale with the type, so each candidate size costs a sum rather than a layout. A column's least width is its widest cell or the widest part of its header (parts end after an underscore or a dot, or at a space), and its most is its widest cell or its whole header. The natural layout (every column at its most) is tried at 10, 9, 8 and 7 points; then explicit widths at the first size from 10 down to 5 at which the least widths fit, each column given its least and a share of what is left over in proportion to what it lacks of its most, capped at its most; then 4 points. Nothing is scaled, so every layout breaks across pages. Across eleven stress folders (the copy of the `Project/` basicstats folder, synthetic ones from 2 to 100 pools, and four the reviewers built and found failing), every overlapping word went to zero, and the 100-pool folder renders in 4 seconds where the whole-table measuring took 8. +- **Zero-width spaces in cells survive into copied text.** `pdftotext` returned `Y_mapped_Scaffold`. Pasted into `analysis.modules.basicstats.chromosomes`, that name would be refused with a list of names that look identical. Break opportunities are inserted in headers only. +- **A breakable figure orphans its caption.** Sweeping filler length so a table's top caption falls at a page foot: 2 of 9 layouts, and 2 of 13 in a second sweep, left the caption alone on one page and the rows on the next. `#show figure.caption.where(kind: table): it => block(sticky: true, it)` brought both to 0. Breakability is set for table figures only: an image figure that was breakable put its caption on the page after the image. +- **knitr's `knit()` defaults to `error = TRUE`**, which writes an R error into the document and carries on; a top-level `if ... else` split across lines in a chunk published a report reading `## Error`. The template sets `error = FALSE`, so a broken chunk stops the knit and the reason is printed. +- **knitr evaluates chunks in the input file's directory** unless `root.dir` is set, which would have written a report's figures into the installation's `analysis/lib/rmd/`. `reportShell` sets `root.dir` to the task directory. +- **`paste0()` recycles a zero-length argument to `""`**, so a header-only table emitted a stray `,` and typst refused the document. The writer fixture's `result.tsv` is one line, which `read.delim` reads as a header with no rows; that is the case that found it. + +## What the review found + +A review workflow (four reviewers on the frame, the table functions, basicstats' report and the documentation, each followed by a skeptic told to refute every finding) confirmed 31 findings, several of them the same defect found twice. All were fixed the same day. The ones that changed the design: + +- **The generic layout read and formatted every row of a table before showing forty.** Measured by the reviewers: 100,000 rows took 60 seconds in the publish step, a million about 13 to 18 minutes and up to 2 GB, to print 40 rows of `association.tsv`. `report_declared()` now reads `nrows = max_rows` and counts the rest by lines. +- **`report_number()` stopped on `Inf`** (`abs(Inf - round(Inf))` is `NaN`, and the subscripted assignment refused it), which association publishes as `t` for a separated allele, and its whole-number test, `abs(x - round(x)) < 1e-9`, printed every value below 1e-9 as 0, `design_floor` at 13 units included. It now prints `Inf`, `-Inf` and `NaN` as themselves and tests `x == round(x)`. +- **The generic layout reformatted identifiers.** A column whose every value parsed as a number was reprinted, so pool `01` read `1`, a year `2,024`, and a long scaffold number gained commas. Only a column holding decimals and no value with a leading zero is reformatted now. +- **The PDF depended on the task's locale.** Under `LANG` unset or `C`, knitr's capture wrote the report's own escapes as ``, `` and `` and an accented pool name as bytes; the build succeeded. The template now switches `LC_CTYPE` to the first UTF-8 locale it finds and refuses to build without one. +- **Equal shares for the columns of numbers still overlapped**: a design table with three experimental variables printed "Chromosomes" 3.6pt over "Ploidy", Z's own symptom. **The scaled layout was one unbreakable block**: 60 to 100 long-named pools ran past the page foot and smeared. **And typst's own auto columns squeeze a column below its unbreakable content** when a table is wider than the page, so long pool names ran into the next column. That last one I found myself, after switching to auto columns on a reviewer's suggestion, by running a word-overlap check (`pdftotext -bbox`) over the reviewers' own renders. Together these three are why the rule above measures cells. +- **Sequence order is the tables', not the reference's.** basicstats.R builds `chrom_levels` from the indel table first and the SNP table after, so a sequence with SNPs and no indels lands after the last indel-bearing one. The manual sentence said "the order your reference gives them" and now says "the order the tables give them"; basicstats.R's order was left as it is. The depth plots had been in file-name order (`chr10` before `chr2`) and are now in the tables' order. +- **basicstats.R stopped on a header-only indel table**, which is what a project with no surviving indel publishes, so such a project got no results at all. Pre-existing, outside the report work; one guard in `site_rows`. + +Smaller ones: figure 1 drew a merged pool's lower bound as an ordinary point, adjacent panels' tick labels collided (3 of 42 random projects), the grid's first rotated pool name was clipped, the report was quadratic in the number of sequences (2 minutes at 20,000), three figure captions named no file, and the cases had gaps: nothing failed on overlapping columns, on `error = TRUE`, on the design notes, or on the report's own arithmetic. One reviewer left two directories, `long__/` and `short__/`, in the repository root from a render with a relative path; they were removed. + +## The cases, and that they bite + +`15_analysis_results`: the generic report now asserts the file's own text (`analysis of Output`) and the release, a module's own report is laid out instead of the file list, the README and the `.bib` name the release, a failed report's reason reaches the console, and two R-only cases check `report_number` and `typst_string`. `09_analysis_modules`: a declared report that is missing, and one named by a path, are refused. `basicstats`: the report has its five sections, every pool, the experimental variables and a pi value as `diversity.tsv` holds it. `00_static`: every declared module report is in its directory, failing when no module declares one. + +Each fix was reverted in a scratch clone and the case aimed at it failed: the folder pointed at the task directory (the original bug's shape: files missed, sections skipped), "unknown" restored in the README, the citations and the report header separately, the reason back on stderr, and each of `readManifest()`'s two report checks removed. + +After the review, more: `report_number()` with Inf, NaN and tiny values; `report_format()` with zero-padded and four-digit identifiers; a cell drawn by `report_table()` itself rather than the helper it calls; the design notes; a wide-table case holding all three layout shapes that broke, checked with a word-overlap reader of the PDF (`pdf_overlapping_words` in `test/lib/analysis.sh`); a chunk that fails; basicstats' added-up row, lower-bound marks and sites read, read out of `report.md` against values computed by awk; and basicstats run on a header-only indel table. Each was proven the same way: the old whole-number test, formatting every numeric column, a cell written as markup, the notes chunk disabled, equal shares, a scaled block, typst's auto columns, an arithmetic mean for the added-up row, the lower-bound mark dropped, sites read without the unmeasured subtracted, and the empty-table guard removed. The wide-table case first passed the scaled and auto mutants, because its tables never reached those layouts; each table in it now is a shape that did. + +## Open + +- mds and association have no report of their own and get the generic layout. What each should show first is Z's to decide. +- A table running onto a new page can carry a single row there under a repeated header; typst has no orphan control for table rows. + +## mds and association, the same day + +Written later on 2026-10-08, against the same tree plus everything above. Z, on the basicstats report: *"This looks good. Let's do this for both association and mds too"*. That settles the first open item: both modules now declare `report.Rmd`, at version `20261008.001`, and neither `frame` floor was raised, for the reason given for basicstats. + +**What each shows first** follows the order the manual already asks a reader to take. association opens with the phenotype as it was read, because `phenotype.tsv` is where a reversed coding is caught; then what the design can show, both floors among them, so the bold sentence about a floor above 0.05 comes before any site; then the strongest sites per phenotype, `qq.png` and the Manhattan plots. mds opens with the plot and every pool's coordinates; then the eigenvalues as a table and as bars colored by sign; then the corrected distances as a pool-by-pool matrix with a heatmap (diverging at zero, since a negative distance is a result); then what each distance rests on: the shared sites as a matrix, the pairs on fewer than 30 sites, and every pair before and after the correction. A matrix holds at most eight pools as columns, split into tables of even size, and when every pair is on fewer than 30 sites that is one sentence rather than a table of all of them. + +**association's site table is read by `data.table::fread`**, for the eleven columns the report shows. Measured on a synthetic `association.tsv` of 1,020,000 rows: the report renders in 3 seconds. data.table is in the analysis baseline, and association's manifest now declares `r-data.table=1.18.6.1` as basicstats' does. + +**The allele named at each site.** The table shows, beside each site, the allele with the largest |t| and its slope. A biallelic site's reference row carries the negated slope of the other, so the two |t| are one number up to rounding, and which is larger is noise. In the corpus, pt_wingspan's chr1:700 has |t| 5.346774501780389599 for both A and T read back from the table, and the first rule, a stable order on |t|, named the first row, A at -0.0587; at chr10:1800 the two are 0.802679592830802 and 0.802679592830801, and that rule decided in the fifteenth digit. The rule now compares at six significant digits and, between alleles that tie, names the one whose slope is positive, the same reasoning as mds pinning each axis's sign. At a flagged site *t* is not comparable between implementations, so the steepest slope decides instead. Ties in `perm_p` go to the larger `S`, with every flagged site first in file order, since its `S` is Inf on one implementation and 1e16 on another. No case covers that last rule: the corpus has no two flagged sites at one p. + +**One defect found writing the manual section**: "Selected at FDR 0.05" labeled `selected`, which counts `fdr_p` at most 0.05 under whatever `fdr` names, Bonferroni included; the row is now labeled by the column. I also made the bold floor sentence per phenotype, believing a run could mix phenotypes at different floors and print a smaller floor beside "above 0.05". It cannot: `design_floor` is 1/units! for every fit of a run (`association.R`), and ties keep a site above the floor without moving it. The review below found that, and the sentence went back to one for the run. + +**The count of sites at the floor** compares `perm_p` with 1.5 times the run's `floor`. A permutation p is a whole number of rearrangements over the count, the observed one always among them, so it is a whole multiple of the floor and the next value above the floor is twice it; the midpoint survives any printing. A relative tolerance (1e-9, then 1e-6) depended on how many digits the two were printed with. association.R prints both to 15 and they agree, but the synthetic three-unit folder printed p to six digits and the report counted 0 of its sites at the floor while every site it showed sat there. + +**Found on the way, outside the reports:** + +- **`report_table`'s `total = nrow(table)` was evaluated lazily**, at its first use, after the table had been cut to `max_rows`, so a cut table never said it was cut. mds' table of pairs on few sites relied on the default. `force(total)` at the top of the function. +- **`mds.png` clipped the labels of the outermost pools** at the panel edge; each label is drawn above its point. The first fix let the labels run past the panel; it was replaced the same day by `ggrepel` (below), and only the 12% the axes leave on each side remains of it. +- **association stopped on a depth table of one site.** The weights are built by `vapply`, which returns a bare vector for one row, so `colnames<-` refused it, and a run whose indel table held one indel published nothing, SNPs included. A review found it on 2026-10-05 and it was reported and left (`depth-masking-proposal.md`, `missing-cell-validation.md`); a synthetic three-unit cohort built for the report reached it. The weights are now built as a matrix. `dev/validation/lib.R` had already guarded its own copy of the same `vapply`. + +**Renders, all with 0 overlapping words** by `pdf_overlapping_words`: the corpus through each module (mds 4 pages, association 3), the copy of the `Project/` mds folder, 24 pools with long names (mds, 10 pages), three units (association), and the 1,020,000-row table. + +**The cases, and that they bite.** `mds`: every row of the distance matrix, read out of `report.md` against `distance.tsv` in `mds.tsv`'s pool order, assembled by awk; the frame case checks the four sections and overlaps. `association`: the design's floor in every column, the count of sites at the floor per phenotype, five site rows built by awk from the two tables (two flagged, the tie at chr1:700, a triallelic site), the level coded 1, and the floor sentence after raising one phenotype's floor to 1/6, a state no run produces, dropped after the review; a depth table of one site checked against the corpus oracle; the frame case checks the four sections, the design's floor and overlaps. `15_analysis_results`: a table cut short says by how much. Each was reverted in a scratch clone and its case failed: the first-row allele rule, the floor counted strictly below, a flagged `S` printed as a number, the coded level dropped, the `vapply` restored, the matrix's lower half left unfilled, the matrix at four digits, `force(total)` removed, and the floor sentence taken from the run's minimum. The first mds case checked only the first row, which the lower-half mutation cannot reach; it now checks every row. + +**Settled by Z the same day:** association's `chromosomes` restricts every fit to the named sequences (`association.R`, the depth table filtered before parsing), while the manual's settings table said it decides "which sequences get a plot". Z: *"User should be able to pick chromosomes correct. Otherwise with scaffolds etc it is messy."* The code was right and the manual wrong: the settings row, the Manhattan section and the site table's "always complete" now say the fit is restricted and what that does to `fdr_p` and the dispersion, and `test_naming_sequences_restricts_the_fit_to_them` pins it (removing the filter fails it). Before ruling, Z asked how the two collide; measured on the corpus SNP table, pt_wingspan, dispersion estimated: with no sequence named, 14 sites read, 13 tested, dispersion 0.206, lambda_gc 2.28, 2 selected; with `chr10` named to draw its Manhattan plot, 6 read, 5 tested, dispersion 0.267, lambda_gc 2.02, 1 selected, and the eight sites off chr10 gone from `association.tsv`. On chr10 itself every number moved with the dispersion and the smaller FDR family: chr10:500 S 0.504 to 0.513, perm_p 0.593 to 0.589, fdr_p 0.652 to 0.603; chr10:1700 fdr_p 0.00903 to 0.00694. + +## The review of these two reports + +The same evening, against the tree above. Four reviewers (association, mds, the cases, the documentation), each followed by a skeptic told to refute every finding; all eight on Sonnet, read-only on the repository, rendering in the scratchpad. 44 findings, 34 confirmed, several of them one defect found twice. Four were already fixed by then, from looking at the example reports: the rounding-zero eigenvalue, "SNP sites" with indels in, and two copies of each. + +**What changed in the reports**, each confirmed by a reproduction: + +- **An ordinal phenotype's slope read "more frequent in the units coded mid"** for low < mid < high. The frame hands an ordinal phenotype its level's index, so the slope is per step along the order and `coded_one`, the second level, is no direction at all. The caption now says "further along low < mid < high", the phenotype table says each level is read as its place in the order, and the manual's two sentences were corrected; the one under `association_alleles.tsv` predates this work. +- **The floor sentence went back to one for the run**, as recorded above. +- **NA in the allele columns is explained in the caption.** At three units the allele table's selection and the report's ranking disagree, so most of the twenty rows can be NA (below). +- **One tested site** reads "The one tested site of", not "All 1 tested sites"; **an allele past 20 bases** is shown by its first 12 and its length, where one of about 140 pushed the table off the page. +- **Names reach markdown escaped** (`report_text()`): a sequence called `@x` stopped the PDF build, and `*` or a leading `_` rewrote a caption. +- **A bracket in the project's own path dropped every figure** with nothing said: the report builds in a folder beside the results, so the path is part of every glob. Files are listed by a pattern made from the glob, and a figure's path is in angle brackets, where a parenthesis ended it. +- **A quote in a cell** opened a quoted string that swallowed the rows after it: tables are read with `quote = ""`, as they are written. +- **A figure taller than the page was clipped**: `qq.png` at four phenotypes is 6 by 11 inches. A PNG at full width is now kept to 7.5 inches tall, read from its header. +- **The column split** of both reports said "even" and gave 7, 7, 7 and 4 for 25 pools; `report_columns()` gives 7, 6, 6 and 6, and basicstats uses it too. +- **`plotted` in eigenvalues.tsv marked every axis written**, so at `dimensions = 3` the report said the axes drawn carry 95.0% above a plot whose own labels add to 84.2%. Fixed at the source in `mds.R`, which the manual already described: `plotted` is the two axes `mds.png` draws, and none at one axis. +- **The heatmap's tiles collapsed** under pool names of 50 characters or more; the figure grows by the longest name. +- **Comments**: "takes the first pool's name" where the code takes the longest, in both reports. + +**What changed in the cases.** The association case now checks every row of the design table and the phenotype table, and the order of each site table against an oracle that sorts association.tsv by the caption's rule. It also checks seven site rows through the leverage, two of them chosen because their |t| differ in the fifteenth digit, plus a planted flagged triallelic site whose largest |t| and steepest slope name different alleles. A new case takes the paths a real run takes: 26 tested sites cut to 20, a sampled run whose floor parts from the design's, an allele table of three sites, and three Manhattan plots in the tables' order. Another reads the floor sentence off a real three-unit run and an ordinal phenotype's wording. The mds cases read back the opening sentence, every coordinate and eigenvalue row, the figures placed, a 25-pool split, and `plotted` at one and three axes. Suite 15 adds a declared table counted and not read whole, and a path holding brackets and parentheses, a quoted cell, an escaped name and a tall figure. The three module suites now declare `analysis/lib/rmd/` in `# covers:`, which `--changed` had skipped. The awk oracles print numbers through `REPORT_NUMBER_AWK` in `test/lib/analysis.sh`, which follows `report_number()` where `%#.3g` did not (1234.5 is `1,230`, not `1.23e+03`). + +**And that they bite**: 24 mutants, one per new assertion, each made in a scratch clone, and every case named failed: the order reversed, the cut at 15, sites read and tested swapped, the two floors swapped, leverage at five digits, |t| at full precision, |t| at a flagged site, NA unexplained, Manhattan plots in file-name order, names unescaped, the ordinal read as coded 1, the floor sentence never printed, one site as "all 1", `plotted` marking every axis, columns cut eight at a time, the opening sentence on the wrong axes, a declared table read quoted, the header counted as a row, an uncut table counting its rows, files globbed through the path, a table read quoted, a tall figure at full width, a path outside angle brackets, and columns 8pt short of their text. + +**`pdftotext` fuses words overprinted by up to 5pt**, so `pdf_overlapping_words`, which compares word boxes, cannot see them; the review measured it and I reproduced it with `Chromosomes#h(-Xpt)Ploidy` at 10pt. At 1, 2, 3.5 and 5pt the PDF reads `ChromosomesPloidy` and the box check is empty; at 6pt the words split and the box check names them. `pdf_missing_words` names a word that does not stand alone, which covers the first regime, and the frame cases and the wide-table case now use both. Squeezing `report_table`'s columns could not produce the fused state: 4 and 6pt short of their text is absorbed by the 5pt insets, and 7pt short is a box overlap. The helper also prints a line for a PDF with no words, which a missing one is. + +**Refuted**, ten: among them that the pairs table's "whole table is in distance.tsv" was untrue, that an all-negative ordination printed a negative share, that the README omits `report_folder()`, and that the note's 19-digit |t| could not have been read back from a 15-digit table (R reads the 15 digits and prints the double it parsed). + +**Open, and Z's: the allele table's selection.** `association.R` keeps the sites at `perm_p` at most `reportBelow` and the first `reportTop` of all phenotypes and kinds together by `perm_p`, ties in file order. With two phenotypes the first can take every slot and the second get none, and at three units, where every site ties at the floor, the report's top twenty by S are mostly not among them. Ranking per phenotype with ties broken by S would make the two agree; that changes what `association_alleles.tsv` holds, so it was reported and not changed. + +## One file per phenotype + +Z, on that question: *"Isn't the association data like GWAS, a per site value for a given phenotype? It can be a matrix no? sites as rows, and phenotypes as columns?"* I laid out three shapes: one wide table with a column group per phenotype, the long table with each site's strongest allele and slope added, and one file per phenotype. Z: *"OK. Let's do one file per phenotype. That sounds better. Joining files for comparing sounds good."* + +**What association publishes now**: `association_.tsv` and `alleles_.tsv` for each phenotype fitted, the stem its name with every character outside `[A-Za-z0-9._-]` made `_`, as the Manhattan plots already name their sequences. Neither carries a phenotype column. `alleles_` and not `association_alleles_`, because a declared output is a glob and `association_*.tsv` would match both. Two phenotypes whose stems are equal are refused before the fit, naming both; a `pt_` name has no character rules beyond its prefix (`bin/parse_metadata.py`), so `pt_wing span` and `pt_wing_span` can both exist. A phenotype with no site read still gets both files, as a header. `permutations.tsv`, `phenotype.tsv`, `qq.png` and the Manhattan plots are unchanged; the figures keep a panel per phenotype. + +**The selection is now each phenotype's own**: the sites at `perm_p` of `reportBelow` or under, and the first `reportTop` by `perm_p`, ties to the larger S with every flagged site first, the order the report ranks them in. So with `reportTop` at 20 or more, every site the report shows has its alleles. Measured on the corpus as three units of two, where most sites tie at 1/6: for pt_resistant the tied site first in file order is chr2:400, and the one with the largest S is chr10:500. The published `association.R` names the selection in its header, which the comment beside it had claimed and main.nf had never written. + +**What read the old names**: the report (captions name each phenotype's files), `dev/validation/agree.R` (`association_pt_y.tsv`), the manifest's two outputs, the manual's two sections, its settings row and its report section, and 48 places in the association suite, renamed by one announced sed with the multi-phenotype helpers and the planted-table edits done by hand; one `awk` there read `kind` as the second column. New cases: each phenotype in files of its own, without the column, its allele table holding its own first site in the report's order; and two names under one file name refused. Reverted in a scratch clone, each failed: ties in file order, the column kept, the clash let through. The Nextflow case asserts both files by name, and the static suite is unchanged at 66. diff --git a/.claude/development-notes/output-prefix.md b/.claude/development-notes/output-prefix.md new file mode 100644 index 0000000..73d19a8 --- /dev/null +++ b/.claude/development-notes/output-prefix.md @@ -0,0 +1,61 @@ +# outputPrefix: one name at the start of every output, and dated report names + +Written 2026-10-08, against `dev` at `1b04cb1` plus that day's uncommitted work (`module-reports.md`, `mds-labels.md`). Not updated to follow the code. + +## The ask + +Z: *"At the top we will add params.outputPrefix = "Test". We will make vcf.filename = params.outputPrefix. Then all the other output files freq depth etc will be correctly named with the prefix. We will add output prefix in front of all analysis outputs. Also the reports coming from analysis should be named [prefix]_[analysis]_report_[datetime as in 20261007-103602].pdf eg. Test_mds_report_20261007-103602.pdf"* + +I proposed renaming `vcf.fileName` to `outputPrefix` outright, one setting so the VCF and the analysis files could not disagree. Z ruled against it: *"If someone wants to change VCF name they still should be able to. vcf.fileName should stay in the template."* Z then wrote the template line as `fileName = "${params.outputPrefix}"`, the interpolated form the template's other derived lines use, *"Easier for the end user to change."* My other three points Z accepted: the analysis layer, not each module, applies the prefix (*"is fine as long as we get the end result right"*), and upgrading projects rerunning is *"not a problem"*. + +## What was built + +**The pipeline side.** `outputPrefix` is the first line of `params`. `vcf.fileName` follows it in the template, and `nextflow.config` fills either for a config that lacks it: `outputPrefix` from `vcf.fileName` and then `'Test'`, `vcf.fileName` from `outputPrefix`. A run table's `outputPrefix` reaches that run's VCF name through `deriveRunPaths()`, which re-derives it unless the global `vcf.fileName` differs from the global `outputPrefix`, which is the same pin test the other derived values use. `vcf.fileName` joined `derivedParameterNames()`. `outputPrefix` joined step 7's artifact identity, so one results directory never holds runs of two prefixes and every analysis target has exactly one; `targetPrefix()` in `plan.nf` throws if that ever fails. + +**The check.** `check_parameters.sh` fails an empty `outputPrefix` or `vcf.fileName` and one holding anything outside `[A-Za-z0-9._-]`. It judges only a key the input carries. Step 0 runs that check over the global parameters only, so a run table's `outputPrefix` is not seen there; `targetPrefix()` applies the same rule per results directory, which is where a run-table value would first become a file name in the analysis layer. + +**The migration.** `renamed()` maps `outputPrefix` to the old `vcf.fileName`, so an upgrading project's prefix is its existing VCF name and nothing of its gets renamed; `vcf.fileName` then takes the template's derived line, which yields the same name. A note explains it. `pinned()` keeps a literal `vcf.fileName` when the old config already has `outputPrefix`. Without that, the migration's rule that a template-computed line beats the user's literal would undo a deliberately named VCF at every later migration, which is exactly what Z's ruling says must stay possible. + +**The analysis side.** `InstallResults` renames every top-level file the module produced to `_` right after copying it into the stage, before the README, the citations and the report are written. Code keeps its name: the script extensions the script check already uses, plus `.cpp`, `.c`, `.h`, `.hpp`. So do the frame's own files, `README.md`, `CITATIONS.md`, `references.bib` and the verification record, because they are written after the rename. Manifests keep the names a module writes; `publishedName()` gives the README and the declared-output check the published ones. + +**The report name** is `__report_.pdf`, the time taken once in the `InstallResults` script block from the local clock, so the README and the PDF carry the same stamp. The middle part is the module name, not `folderName`. Both were my calls, stated to Z with the plan. + +**Inside the report**, `report_read()` and `report_files()` take the name a module wrote and read the prefixed file; `report_name()` gives a caption the published name, and `report_logical()` takes the prefix off a name a report parses a sequence out of (basicstats' depth plots, association's Manhattan plots). `POOLSEQFLOW_REPORT_PREFIX` carries the prefix and **an unset one stops the report**. Every report from v3.0.0 to v3.2.0 was blank because the folder reached the template as a literal that matched nothing (`module-reports.md`); a prefix that silently defaulted to none would do the same thing again, every section looking for a name the folder does not hold. + +## Found on the way + +**The manual said changing `vcf.fileName` after a run re-runs the VCF branch beside the old files. Step 0 refuses instead.** `analysisParams()` excludes paths, resources and tool locations and nothing else, so `vcf.fileName` has been in the recorded manifest all along and any change to it fails the parameter guard. The new naming section says what happens. The same now holds for `outputPrefix`. + +**Step 0's line after a FAIL finding said the setting "would make this run produce nothing"**, which a name that cannot start a file name does not do; it stops the run partway. Widened, and the manual's quote of it and its level table with it. `fastqc.memory` carrying a unit was already that kind of FAIL. + +## Tests + +`analysis_render_report` stages the folder as `InstallResults` publishes it, as links named under `Test`, so every report case outside Nextflow reads prefixed files. Its copy of the code rule is a second definition of `InstallResults`'; the cases that publish through Nextflow are what hold the two together. + +A third analysis baseline, `prefixed`, sets `outputPrefix = 'Pfx'` and `vcf.fileName = 'Calls'`. Everywhere else both are `Test`, so a frame that took the VCF name would pass every other case. The case checks the config took both substitutions before trusting what it publishes. + +The report-name case brackets the run with the clock on either side and requires the stamp between them, which catches a fixed string, a date alone, or UTC on a machine that is not on it. + +Module versions were not moved. Z: *"You don't need to bump versions."* Step 4 of the release names what is behind. + +Fifteen mutants of the guards above, each killed: eleven without a JVM (the check's empty and absent cases, the migration's rename and pin, `report_read`, `report_files`, `report_logical` and the unset prefix, an mds caption, both reports' plot-name parsing) and four through Nextflow (the prefix taken from the VCF name, a date-only stamp, a script given the prefix, a run's prefix kept from its VCF). + +## The review, the same day + +Three read-only reviewers went over the change before Z had seen it. What they found, and what changed. **Where this contradicts a section above, this is the current state**; the sections above say what was built first. + +**A module never got the default.** A module is its own pipeline and reads no `nextflow.config`, which `shell-and-nextflow-gotchas.md` records and `analysis/frame.config` says in its header; the frame config carried `params.cores` and nothing for `outputPrefix`. A config without the key, which a project that skipped `migrate_config` has, published every file as `null_*`: `"${run.outputPrefix}"` makes a missing key the string `null`, and the name rule accepted it. The verification, run beside `nextflow.config`, saw the right prefix, so nothing looked wrong. The three default lines are now in `frame.config` too, and a fourth analysis baseline, `legacy`, runs a module on such a config. + +**The name rule was too loose in three ways.** A leading `.` passed, and `list.files()` does not list hidden files, so a report lost every figure while the folder was complete. A leading `-` reads as an option. And `null` passed, which is also what the template's own `fileName = "${params.outputPrefix}"` reads as when someone deletes or comments out the `outputPrefix` line, so such a config ran as `null.vcf`. A name now starts with a letter or a digit and `null` is refused with that cause named, in `check_parameters.sh`, `targetPrefix()` and the run-table parser alike. + +**A run table's cells were never checked.** Step 0 runs `check_parameters.sh` over the global parameters only. Above I wrote that `targetPrefix()` is where a run-table value would first become a file name; that was wrong, because `deriveRunPaths()` makes a run's prefix its VCF name, and step 6 passes that name to `bcftools` unquoted, so a cell holding a space failed at variant calling after trimming and alignment. `parse_multirun.py` now holds `outputPrefix` and `vcf.fileName` cells to the rule, beside its RunID check, before anything runs. + +**The migration still renamed a VCF silently in two shapes.** A current config whose `vcf.fileName` is an expression of its own, `"${params.outputPrefix}_v2"`, was given the template's line with no report entry; `pinned()` only kept literals. And a config from before the prefix whose name was an expression lost it the same way, with no note, because the rename into `outputPrefix` only carries a literal and nothing else reported it. Now a config that has `outputPrefix` keeps whatever `vcf.fileName` it wrote; a config without it keeps an expression where it is and gets the template's prefix, with a note saying so; and only a literal name moves into `outputPrefix`. + +**The empty-name finding had an empty middle field**, and both callers read findings with `IFS= read`, where two tabs are one, so the explanation landed in the detail column. It carries `no name` now, and a case reads a finding the way the callers do. + +**Smaller.** The pin test compares as text, since an unquoted prefix is a number and a GString is not equal to one. Three `complete` cases still asserted `result.tsv` in a published folder; I had run selected cases of the suite and not all of it. The manual now says that varying `outputPrefix` in a run table varies the VCF name and so repeats variant calling, and that a fixed `fileName` keeps it shared; it no longer names step 0 in the new prose; it widens "produce nothing" wherever it describes the check; and it lists `vcf.fileName` among the derived values. + +**Left as it is.** The rename covers what sits at the top of the folder, which is all any module produces; the contract text now says a directory's contents keep their names rather than the frame refusing a directory. `mds.png` still says `few_sites in distance.tsv` in its own caption, which is module output that does not know the prefix. + +Nine more mutants, of these fixes, each killed: six without a JVM (`null` and a leading dot accepted, the empty detail field, the run-table cells unchecked, the pin back to literals only, an expression moved into the prefix) and three through Nextflow (no default in `frame.config`, the prefix out of step 7's identity, the pin compared as values). A number written as the prefix cannot be varied by a run-table cell holding text: a cell takes the type of the config's value, and that conversion fails for every parameter alike. diff --git a/.claude/development-notes/site-follows-releases.md b/.claude/development-notes/site-follows-releases.md new file mode 100644 index 0000000..5469ec6 --- /dev/null +++ b/.claude/development-notes/site-follows-releases.md @@ -0,0 +1,35 @@ +# The site follows releases + +**Written 2026-10-07 against `dev` at `d8b7cee` plus uncommitted work**: `.github/workflows/docs.yml`, its 00_static case `test_the_site_deploys_only_for_a_released_version`, and the lines in `dev/RELEASING.md`, `modules/README.md` and `dev/scripts/publish-module.sh` that described the old wiring. + +## What happened + +`docs.yml` deployed the site on any push to `main` that touched the manual, the CHANGELOG, the catalogue or the build. A release pushes `main` before it tags: the step-6 merge and the step-9 commit. On 2026-10-07 the merge `84def44` and the version-bump commit `babb475` each deployed, so the site carried the 3.3.0 manual and a 3.3.0 CHANGELOG entry. The v3.3.0 tag then failed `release.yml` at its analysis version gate, and no release was made. The site went on describing a release that did not exist, with v3.2.0 still the latest. + +Z, the same day: "If a release fails website should not be published." + +## What was weighed + +The site holds one manual, and the module pages (basicstats, association, mds, the modules index) are sections of it, deployed together with the catalogue. Modules are meant to publish between releases, page and catalogue row together. Three shapes were put to Z: + +- pipeline pages and the CHANGELOG from the latest successful release, module pages and the catalogue from `main`, which needs `build_docs.py` to assemble one site from two commits; +- the whole site from the latest successful release, with only the catalogue moving between releases, so a module published between releases has no page until the next release; +- keep deploying from `main`, and hold every deploy while a release is in flight, which needs "a release is in flight" read off `main`'s history. + +Z: "Isn't it a single manual anyway?" It is, and that settles it: the site is built from one commit, and the only question is when it deploys. + +## What was built + +`docs.yml` deploys in two cases. After `release.yml` completes, through `workflow_run`, when that run was a pushed tag and succeeded, building the commit the release was made from. And when run by hand, which is how a module published between releases reaches the site. In both cases a step first asks GitHub whether the version the commit declares (`VERSION=` in `./PoolSeqFlow`) has a published, non-draft release, and refuses otherwise. A push to `main` and a pull request still build with `--strict`, so a broken page fails early, and deploy nothing. A `release.yml` run by hand checks a commit without publishing it, so it does not deploy the site either. + +`workflow_run` runs from the default branch's copy of the workflow and checks out the released commit itself. A deploy job inside `release.yml` would run on the tag instead, and a `github-pages` environment limited to `main`, which is how Pages environments are commonly set up, would refuse it. That limit was not checked against this repository's settings; `workflow_run` avoids depending on it. + +## What it costs + +**A module published between releases needs the Documentation workflow run by hand** once its commit is on `main`. Before, the push deployed it. + +**The hand run checks only the declared version.** Between the step-6 merge and the step-7 bump, `main` holds the next release's manual while still declaring the released version, and a hand run then would deploy it. Nothing in the release procedure runs the workflow by hand, and no automatic path deploys in that window. + +## How it is checked + +`test_the_site_deploys_only_for_a_released_version` reads the workflow and requires every path to the deploy to go through the release check: the deploy job's condition, the build's output, the release check's events, the Pages steps' conditions, the `workflow_run` gate on a successful pushed tag, and `workflow_run` naming `release.yml` by its `name:`. It then runs the release check's own script against a stand-in `gh` answering published, draft and missing. Four mutants of the workflow were each caught when it was built: deploy on anything but a pull request, the pushed-tag condition dropped, the Pages steps ungated, and `workflow_run` following another name. diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index d8e02c6..68538a2 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -1,7 +1,7 @@ name: Documentation -# Builds the MkDocs site and publishes it straight from main. No gh-pages branch: the -# site is uploaded as an artifact and deployed by actions/deploy-pages. +# Builds the MkDocs site, and deploys it only for a version that has been released. No +# gh-pages branch: the site is uploaded as an artifact and deployed by actions/deploy-pages. # # One-time repository setting this depends on: # Settings -> Pages -> Build and deployment -> Source: "GitHub Actions" @@ -9,15 +9,17 @@ name: Documentation # It cannot be "Deploy from a branch -> main -> /docs", which serves the folder as-is; # docs/ holds Markdown, not finished HTML. # -# Pull requests build with --strict but do not deploy. +# It deploys in two cases: after release.yml publishes a release, built from the commit that +# release was made from, and when run by hand on main (Actions -> Documentation -> Run +# workflow), which is how a module published between releases reaches the site. Both refuse a +# commit whose declared version has no published release. A push to main and a pull request +# build with --strict and deploy nothing. on: push: branches: [main] paths: - "manual/**" - # The module repository is deployed with the site, so publishing a module is a change - # here. Without this line the catalogue would advertise a tarball no deploy had carried. - "modules/repo/**" - "mkdocs.yml" - "CHANGELOG.md" @@ -31,6 +33,11 @@ on: - "CHANGELOG.md" - "dev/scripts/build_docs.py" - ".github/workflows/docs.yml" + # Runs from main's copy of this file whenever release.yml finishes; the build job decides + # whether that run published anything. + workflow_run: + workflows: ["Publish a release"] + types: [completed] workflow_dispatch: permissions: @@ -45,9 +52,36 @@ concurrency: jobs: build: + # After release.yml, only a run that published: a pushed tag with every step green. Run by + # hand, release.yml checks a commit and publishes nothing. + if: >- + github.event_name != 'workflow_run' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.event == 'push') runs-on: ubuntu-latest + outputs: + deploy: ${{ steps.released.outputs.deploy }} steps: - uses: actions/checkout@v7 + with: + ref: ${{ github.event.workflow_run.head_sha || github.sha }} + + # The version this commit declares has to have a published release before its pages do. + - name: Check this version is released + id: released + if: github.event_name == 'workflow_run' || github.event_name == 'workflow_dispatch' + env: + GH_TOKEN: ${{ github.token }} + run: | + declared=$(sed -n 's/^VERSION="\(.*\)"$/\1/p' PoolSeqFlow) + draft=$(gh release view "v$declared" --repo "$GITHUB_REPOSITORY" \ + --json isDraft --jq '.isDraft' 2>/dev/null || true) + if [ "$draft" != "false" ]; then + echo "This commit declares $declared, and v$declared has no published release," >&2 + echo "so its pages are not deployed. The site deploys once release.yml publishes it." >&2 + exit 1 + fi + echo "deploy=true" >> "$GITHUB_OUTPUT" - uses: actions/setup-python@v5 with: @@ -69,15 +103,15 @@ jobs: # Fails early and clearly if Pages is not enabled. - uses: actions/configure-pages@v6 - if: github.event_name != 'pull_request' + if: steps.released.outputs.deploy == 'true' - uses: actions/upload-pages-artifact@v5 - if: github.event_name != 'pull_request' + if: steps.released.outputs.deploy == 'true' with: path: site deploy: - if: github.event_name != 'pull_request' + if: needs.build.outputs.deploy == 'true' needs: build runs-on: ubuntu-latest environment: diff --git a/CLAUDE.md b/CLAUDE.md index 68238c1..859c1bb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,7 +70,7 @@ Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different ## Other standing rules - **`dev/RELEASING.md` is a living procedure, not a record.** Correct it when a release teaches you something; it carries no date and describes no particular version. It holds only what a person does: a command, what it must show, and what to do when it does not. Everything else belongs in `.claude/development-notes/` or in the script's own header, and a step must never point into a dated note -- that is how it came to promise an audit that did not exist. -- **Several release gates live in the suite rather than in that document**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`), the analysis versions (`check-analysis-versions.sh`) and the version-consistency case all run inside `00_static`. A green suite is how they report. Do not add a step for one. +- **Several release gates live in the suite rather than in that document**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`) and the version-consistency case all run inside `00_static`. A green suite is how they report. Do not add a step for one. **`check-analysis-versions.sh` is the exception**: a module is legitimately behind between releases, so the suite tests the script only, and the real tree is checked at steps 4 and 11 of `dev/RELEASING.md` and by `release.yml`. This line used to list it with the others, and nothing ran it between the floors being raised at step 7 and the tag, which is where v3.3.0 failed. - **The manual keeps real typographic characters. Everything else in the repository is ASCII.** `manual/PoolSeqFlow-manual.md` is rendered to a website and read as prose, so an arrow is `→`, a reverse arrow `←`, a product `×`, an ellipsis `…`, a negative `−`, a comparison `≥`. **Never ASCII-ify the manual**, and never let a sweep do it: a pass asked to remove em dashes has twice taken 53 arrows, 29 products and 16 ellipses with them. Two carve-outs inside the manual, and only two. **Em dashes are out on style** -- rewrite the sentence, do not substitute `--`. And **a command someone copies and runs stays ASCII**: that is the test, not whether something sits in a fenced block, because the manual fences its diagrams and formulas too. `distance = (J_A + J_B) / 2 − J_AB` and the labels inside a ```mermaid block keep their glyphs. Every other file, including the templates, the notes and this one, is ASCII as the global rule says. - **A ```mermaid diagram exists on the site only.** `mkdocs.yml` carries the custom fence for it and material loads the renderer from a CDN, so the diagram needs a network to draw and the shipped `manual/PoolSeqFlow-manual.md` shows its `graph TD` source. Use it for what ASCII cannot draw. Anything a reader must have in the tarball stays as text, or as an `.svg` under `manual/assets/` like the five depth-histogram figures. @@ -89,7 +89,7 @@ Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different Not one line of code may change. Per file, diff the non-comment lines against `HEAD`. For Python, a docstring is not a `#` comment and the line count misleads -- prove it with an AST comparison that strips docstrings. -Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts -- files linted, cases passed -- against the run before it rather than only the exit status: a filter that matches nothing also reports success. Neither number is written down here, because both move with every file added. +Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts -- files linted, cases passed -- against the run before it rather than only the exit status: a filter that matches nothing now stops the run, but one narrower than you meant still reports success. Neither number is written down here, because both move with every file added. **Lint through the suite, at zero errors and zero warnings.** Seven seconds: @@ -109,7 +109,7 @@ The two cases behind that filter are `nextflow lint is clean`, which lints the t bash test/run_tests.sh --suite 07_analysis --case citation ``` -`--case` matches any part of a case name and filters at dispatch, so the suite's fixtures are still built by the cases that define them. Three cases in under a minute against ten for the suite they live in -- which is the difference between checking a change and avoiding checking it. +`--case` matches any part of a case's function name, so it takes underscores (`all_pending`, never `"all pending"`), and filters at dispatch, so the suite's fixtures are still built by the cases that define them. **A case that cannot run fails the run, except under `--fast`**: outside it, a missing environment is a red run, not a skip. Three cases in under a minute against ten for the suite they live in -- which is the difference between checking a change and avoiding checking it. **The full suite belongs to a release**, not to a step. `--fast` is the cheap sweep when a change reaches beyond its own step -- the wrapper, packaging, a shared library -- and a whole `--suite ` is for when the step *is* that suite. @@ -130,7 +130,7 @@ bash test/run_tests.sh --suite 07_analysis --case citation **`bash test/run_tests.sh --changed` picks the suites for you**, from what each suite declares it runs expanded through the include graph. `dev/scripts/select-tests.py ` shows the reasoning without running anything. It errs wide -- a change to `test/lib/` or to the selector selects everything -- so a narrow answer is trustworthy and a wide one is only expensive. -**Every suite declares what it may cost** -- `static`, `jvm` or `pipeline` -- in a `# cost:` line in its own header. `--cost static` is the set that completes with nothing installed: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose. +**Every suite declares what it may cost** -- `static`, `jvm` or `pipeline` -- in a `# cost:` line in its own header. `--cost static` is the set that builds nothing: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose: with nothing installed, `--cost static --fast` completes, while `--cost static` alone fails the cases that want a tool. **`--suite` and `--case` accumulate and match by name, not by number.** `--suite analysis_time --suite analysis_series` runs both, and `--suite analysis` runs all nine. Every run prints the scope it selected, so a narrowed run cannot be mistaken for a full one; renumbering a suite therefore costs nothing, because nothing addresses one by its number. diff --git a/PoolSeqFlow b/PoolSeqFlow index 32286b4..3f12f9d 100755 --- a/PoolSeqFlow +++ b/PoolSeqFlow @@ -59,7 +59,7 @@ usage_analysis() { # The module store's own words. usage_modules() { - echo "Usage: $0 analysis modules {list|available|install [version]|uninstall }" + echo "Usage: $0 analysis modules {list|available|install [version]|install all|uninstall }" exit 1 } @@ -339,6 +339,12 @@ deploy_payload() { # analysis environment comes out first, while the manifests declaring it still exist. # After this the store and the environment are both back to what the release ships. gone=$(store_packages "$dest/analysis/modules" | cut -d= -f1 | sort -u) + # Less the release's own baseline, as a module uninstall leaves it: what a module + # declares overlaps it by design, and removing that strips the environment the release + # built. Read from the copy being replaced, which is what built it. + gone=$(printf '%s\n' "$gone" \ + | grep -vxF "$(ANALYSIS_ENV_FILE="$dest/install/environment-analysis.yml" \ + baseline_packages)" || true) if [ -n "$gone" ] && env_exists "$ANALYSIS_ENV"; then echo "Removing what the installed modules added to '$ANALYSIS_ENV':" printf ' %s\n' $gone @@ -354,6 +360,18 @@ deploy_payload() { for item in $PAYLOAD_ITEMS; do cp -r "$INSTALL/$item" "$dest"/ done + # From a checkout, the copy also carries what git ignores inside the payload: the + # checkout's own module store and compiled Python, neither of which a release holds. + # Taken back out, so the store starts empty. Only for a checkout of its own: a release + # unpacked inside some other repository is not judged by that repository's rules. + if [ "$(git -C "$INSTALL" rev-parse --show-toplevel 2>/dev/null)" = "$here" ]; then + # shellcheck disable=SC2086 + git -C "$INSTALL" ls-files -z --others --ignored --exclude-standard --directory \ + -- $PAYLOAD_ITEMS | while IFS= read -r -d '' ignored; do + rm -rf "${dest:?}/$ignored" + echo " left out, ignored by git here: $ignored" + done + fi # Stamps each copy with its own location. for item in $WRAPPERS; do sed -i "s|^POOLSEQFLOW_INSTALLED_HOME=\"\"$|POOLSEQFLOW_INSTALLED_HOME=\"$dest\"|" \ @@ -683,8 +701,51 @@ module_index_names() { $2 == "" || $2 == "module" { print $1 }' | sort -u } +# One module as this installation sees it: the version `install ` would take, that row's +# contract and summary, and a state that is empty when it can be installed, otherwise "installed" +# or why this release cannot run it. One line, fields separated by MODULE_INDEX_SEP. `available` +# and `install all` both read it, so what one lists and the other installs cannot drift. +# +# Nothing runnable falls back to the NEWEST row, so the state can say why rather than leaving the +# module out. A module published for a later release is the case this exists for: being told to +# upgrade is an answer, and a blank is not. +module_state() { # index name contract frame_here + local index="$1" name="$2" contract="$3" frame_here="$4" + local row best="" newest_row="" _n _k _v version row_contract row_frame row_env url sha summary + local state + while IFS= read -r row; do + [ -n "$row" ] || continue + [ -n "$newest_row" ] || newest_row="$row" + IFS="$MODULE_INDEX_SEP" read -r _n _k _v row_contract row_frame row_env url sha summary <` would take, asked with the same helper install_module asks with, so the # two cannot come to different answers. - local row best newest_row _n _k _v while IFS= read -r name; do [ -n "$name" ] || continue rows=$((rows + 1)) - best=""; newest_row="" - while IFS= read -r row; do - [ -n "$row" ] || continue - [ -n "$newest_row" ] || newest_row="$row" - IFS="$MODULE_INDEX_SEP" read -r _n _k _v row_contract row_frame row_env url sha summary <` in a process of its own: install_module ends the +# script on a refusal, and one module that will not install must not stop the others. What this +# release cannot run is named and passed over with the reason `available` gives. A module already +# in the store is left as it is whatever the catalogue says, as install_module leaves it. +install_every_module() { + local index contract frame_here name version row_contract summary state + local installed="" already="" refused="" passed="" + index=$(mktemp) + trap 'rm -f "$index"' RETURN + get_module_index "$index" + require_module_index_format "$index" + contract=$(module_contract) + if [ -z "$contract" ]; then + echo "ERROR: cannot tell which table contract this release speaks." >&2 + echo " Expected to read it from $INSTALL/analysis/lib/nf/modules.nf." >&2 + echo " This installation is incomplete; install it again." >&2 + exit 1 + fi + frame_here=$(installed_frame_version) + + while IFS= read -r name; do + [ -n "$name" ] || continue + if [ -d "$MODULE_STORE/$name" ]; then + already="$already $name" + continue + fi + IFS="$MODULE_INDEX_SEP" read -r version row_contract summary state <&2 + exit 1 + fi +} + +# Takes back out the libraries one module install put in the store, when that install fails. +# Only names it installed itself are given, so no other module can be using them. +remove_new_libraries() { + local lib + for lib in "$@"; do + rm -rf "$(library_store)/${lib:?}" + echo " library $lib taken back out" >&2 + done +} + install_module() { local name="$1" want="${2:-}" index contract line version row_contract url sha summary local tmp tarball unpacked dest specs spec @@ -980,7 +1082,7 @@ ROW # # A library already in the store is left alone: it is shared, another module may be using it, # and every version of it published for this frame computes the same thing. - local libs lib + local libs lib added="" libs=$(module_libraries "$unpacked/$name/manifest.json") if [ -n "$libs" ]; then mkdir -p "$(library_store)" @@ -990,11 +1092,14 @@ ROW continue fi if ! install_library "$lib" "$index"; then + # shellcheck disable=SC2086 + remove_new_libraries $added echo "" >&2 echo "ERROR: $name v$version needs the library '$lib', which could not be" >&2 - echo " installed. The module was not installed." >&2 + echo " installed. The module was not installed, nor any library it brought." >&2 exit 1 fi + added="$added $lib" done fi @@ -1028,10 +1133,12 @@ ROW # shellcheck disable=SC2086 if ! conda_install_packages "$ANALYSIS_ENV" $specs; then rm -rf "$dest" + # shellcheck disable=SC2086 + remove_new_libraries $added echo "" >&2 echo "ERROR: $name v$version needs packages this release's analysis environment" >&2 echo " cannot take without moving something already in it. The module was not" >&2 - echo " installed and the environment was not changed." >&2 + echo " installed, nor any library it brought, and the environment was not changed." >&2 echo "" >&2 echo " That is a compatibility question between this release and this module," >&2 echo " not something to solve here: install a build of $name published for" >&2 @@ -1406,25 +1513,48 @@ populate_project() { # Every PoolSeqFlow environment and installation on the machine. A function rather than an arm # of its own: `uninstall all` is one command with a word after it, and the arm list has to stay # the same set as the usage line. +# The wrapper links in $PREFIX/bin that point into an installation under $PREFIX/opt, whether or +# not that installation is still there. One path a line. +poolseqflow_links() { + local prefix="$1" link + for link in "$prefix/bin/PoolSeqFlow" "$prefix/bin/PoolSeqFlow-"*; do + [ -L "$link" ] || continue + case "$(readlink "$link")" in + "$prefix/opt/PoolSeqFlow-"*) printf '%s\n' "$link" ;; + esac + done +} + uninstall_every_version() { INSTALLED=$(poolseqflow_envs) - if [ -z "$INSTALLED" ]; then - echo "No PoolSeqFlow conda environments are installed. Nothing to do." - exit 0 - fi PREFIX=$(install_prefix) PAYLOADS=$(installed_payload_versions "$PREFIX") + LINKS=$(poolseqflow_links "$PREFIX") + # Each of the three can outlive the others, and any one of them is something to remove. + if [ -z "$INSTALLED" ] && [ -z "$PAYLOADS" ] && [ -z "$LINKS" ]; then + echo "No PoolSeqFlow environment, installation or wrapper is here. Nothing to do." + exit 0 + fi echo "" - echo "This removes every PoolSeqFlow conda environment on this machine:" - echo "$INSTALLED" | sed 's/^/ /' + echo "This removes everything PoolSeqFlow installed on this machine:" + if [ -n "$INSTALLED" ]; then + echo "" + echo " every PoolSeqFlow conda environment:" + echo "$INSTALLED" | sed 's/^/ /' + fi if [ -n "$PAYLOADS" ]; then echo "" - echo "and every installed pipeline under $PREFIX/opt:" + echo " every installed pipeline under $PREFIX/opt:" for v in $PAYLOADS; do echo " v$v"; done + fi + if [ -n "$LINKS" ]; then echo "" - echo "including the symlinks in $PREFIX/bin." + echo " every wrapper in $PREFIX/bin:" + echo "$LINKS" | sed 's/^/ /' fi echo "" + echo " and the tab completions." + echo "" echo "Only the software goes. Your project directories - parameters.config, Data/," echo "Reference/ - and everything in storageDir are untouched: none of them live" echo "inside an installation." @@ -1446,20 +1576,36 @@ uninstall_every_version() { esac # An active environment has to be stepped out of first; there may be none. conda deactivate 2>/dev/null || true - # A plain `for`, not `... | while read`: a piped loop is a subshell, where `set -e` - # would not stop the script. Environment names cannot contain whitespace. + # Every removal is attempted whatever the one before it did, so one refusal cannot leave + # the rest installed. Environment names cannot contain whitespace. for env in $INSTALLED; do echo "Removing $env..." - conda env remove -n "$env" -y + if ! said=$(conda env remove -n "$env" -y 2>&1); then + echo " could not remove $env. conda said:" >&2 + printf '%s\n' "$said" | grep -v '^[[:space:]]*$' | tail -n 3 | sed 's/^/ /' >&2 + fi done - # remove_payload re-points the plain wrapper each time and removes it with the last - # version, so nothing is left on PATH pointing at a directory that has gone. for v in $PAYLOADS; do remove_payload "$v" || true done + # The links remove_payload could not reach: those whose installation was already gone. + poolseqflow_links "$PREFIX" | while IFS= read -r link; do + rm -f "$link" && echo "Removed $link" + done remove_completion echo "" - echo "All PoolSeqFlow environments and installations removed." + # What is left, read back from conda and the prefix. + LEFT_ENVS=$(poolseqflow_envs) + LEFT=$(installed_payload_versions "$PREFIX") + LEFT_LINKS=$(poolseqflow_links "$PREFIX") + if [ -n "$LEFT_ENVS$LEFT$LEFT_LINKS" ]; then + echo "Not everything is gone, for the reasons given above:" >&2 + for env in $LEFT_ENVS; do echo " conda environment $env" >&2; done + for v in $LEFT; do echo " $PREFIX/opt/PoolSeqFlow-$v" >&2; done + [ -z "$LEFT_LINKS" ] || printf '%s\n' "$LEFT_LINKS" | sed 's/^/ /' >&2 + exit 1 + fi + echo "Everything PoolSeqFlow installed is removed." } case $COMMAND in @@ -1878,8 +2024,14 @@ EOF *) usage_modules ;; esac require_analysis_env - install_module "${ANALYSIS_ARGS[1]}" "${ANALYSIS_ARGS[2]:-}" - conda deactivate + if [ "${ANALYSIS_ARGS[1]}" = all ]; then + [ "${#ANALYSIS_ARGS[@]}" -eq 2 ] || usage_modules + conda deactivate + install_every_module + else + install_module "${ANALYSIS_ARGS[1]}" "${ANALYSIS_ARGS[2]:-}" + conda deactivate + fi ;; uninstall) [ "${#ANALYSIS_ARGS[@]}" -eq 2 ] || usage_modules diff --git a/analysis/frame.config b/analysis/frame.config index a6fc22d..e421cd5 100644 --- a/analysis/frame.config +++ b/analysis/frame.config @@ -27,6 +27,13 @@ includeConfig "${launchDir}/parameters.config" // A ternary and not an `if`: the strict parser rejects `if` statements in config. params.cores = params.containsKey('cores') && params.cores != null ? params.cores : [:] +// The outputPrefix of a project whose config predates it: the VCF name it already has, and the +// template's 'Test' when it names neither. vcf.fileName, when a config leaves it out, follows the +// prefix. The same three lines as nextflow.config's, guarded the same way. +params.vcf = params.containsKey('vcf') && params.vcf != null ? params.vcf : [:] +params.outputPrefix = params.containsKey('outputPrefix') && params.outputPrefix != null ? params.outputPrefix : (params.vcf.containsKey('fileName') && params.vcf.fileName != null ? params.vcf.fileName : 'Test') +params.vcf.fileName = params.vcf.containsKey('fileName') && params.vcf.fileName != null ? params.vcf.fileName : params.outputPrefix + // The helpers in bin/, which parameters.config places beside the entry script. For a module the // entry script is the module, so this is the installation instead. // diff --git a/analysis/frame.version b/analysis/frame.version index 707b3c3..29e43b4 100644 --- a/analysis/frame.version +++ b/analysis/frame.version @@ -24,4 +24,4 @@ # not touch this file. Each module carries its own version in the same form, in its manifest.json. # # Blank lines and lines starting with # are ignored; the first line left is the version. -20260910.001 +20261008.001 diff --git a/analysis/lib/nf/citations.nf b/analysis/lib/nf/citations.nf index 45177d0..8c755ca 100644 --- a/analysis/lib/nf/citations.nf +++ b/analysis/lib/nf/citations.nf @@ -8,7 +8,7 @@ nextflow.enable.dsl=2 -include { installDir } from './paths.nf' +include { installDir; releaseVersion } from './paths.nf' include { moduleStore } from './modules.nf' // The entries citations/citations.json holds that an ANALYSIS run also invokes. It lists every @@ -56,7 +56,7 @@ def citationShell(String module, String dest) { def json = groovy.json.JsonOutput.toJson(merged).replace("'", "'\\''") def packages = merged.findAll { _key, entry -> entry instanceof Map && entry.r_package } .collect { key, entry -> "${key}=${entry.r_package}" } - def release = "${workflow.manifest.version ?: ''}" + def release = releaseVersion() def nextflowVersion = "${workflow.nextflow.version ?: ''}" def lines = [] diff --git a/analysis/lib/nf/modules.nf b/analysis/lib/nf/modules.nf index 5e30c8e..a3551b6 100644 --- a/analysis/lib/nf/modules.nf +++ b/analysis/lib/nf/modules.nf @@ -176,6 +176,17 @@ def readManifest(Object dir) { "know them for a module published separately.\n" + "Install the module again, or remove the directory.") } + def report = "${parsed.report ?: ''}".trim() + if (!report.isEmpty() && !(report ==~ /[A-Za-z0-9._-]+\.Rmd/)) { + throw new IllegalStateException( + "${manifest} gives 'report' as '${report}'. It is the name of an .Rmd file in the " + + "module's own directory, which the frame knits into the published report.") + } + if (!report.isEmpty() && !file("${dir}/${report}").exists()) { + throw new IllegalStateException( + "${manifest} declares its report as '${report}', and ${dir} has no such file.\n" + + "Install the module again, or remove the directory.") + } return [ summary : "${parsed.summary}".toString(), version : "${parsed.version}".toString(), contract : "${parsed.contract}".toString(), @@ -187,6 +198,7 @@ def readManifest(Object dir) { gates : parsed.gates ?: [], outputs : parsed.outputs ?: [], packages : (parsed.packages ?: []).collect { spec -> "${spec}".toString() }, + report : report.toString(), builtin : false, dir : "${dir}".toString(), entry : "${entry}".toString() ] diff --git a/analysis/lib/nf/outputs.nf b/analysis/lib/nf/outputs.nf index e452bca..b309eb7 100644 --- a/analysis/lib/nf/outputs.nf +++ b/analysis/lib/nf/outputs.nf @@ -9,7 +9,7 @@ nextflow.enable.dsl=2 -include { installDir; verificationRecordName } from './paths.nf' +include { installDir; releaseVersion; verificationRecordName } from './paths.nf' include { moduleEntry } from './modules.nf' // The manual this release ships, installed beside the code. Authoritative for this release; the @@ -23,8 +23,27 @@ def manualSite() { return 'https://ozankiratli.github.io/PoolSeqFlow/' } -// What every published analysis carries whichever module produced it. -def frameOutputs() { +// What counts as the script that produced a result, by extension: R, and the shell, Python and +// Julia a module may drive it from. +def scriptSuffixes() { + return ['R', 'r', 'Rmd', 'rmd', 'sh', 'py', 'jl'] +} + +// The code a result was produced by: a script, and a source a script compiles. +def codeSuffixes() { + return scriptSuffixes() + ['cpp', 'c', 'h', 'hpp'] +} + +// The name a file a module produced is published under: the target's outputPrefix and an +// underscore in front of it, and code under its own name. A glob gives a glob. +def publishedName(Map target, String file) { + def code = codeSuffixes().any { suffix -> file.endsWith(".${suffix}") } + return (code ? file : "${target.prefix}_${file}").toString() +} + +// What every published analysis carries whichever module produced it. `report` is the name its PDF +// is published under. +def frameOutputs(String report) { return [ [ file : verificationRecordName(), summary: 'the checks that cleared this folder, and what they were run against', @@ -35,8 +54,8 @@ def frameOutputs() { [ file : 'references.bib', summary: 'the same list as BibTeX', anchor : 'citing-the-tools-it-runs' ], - [ file : 'report.pdf', - summary : 'every result in this folder in one document, each under its own file name', + [ file : report, + summary : 'the results laid out to be read: the module\'s own layout where it has one, and each table and figure under its file\'s name where it does not', anchor : 'analysis-report', optional: true ], ] @@ -164,13 +183,14 @@ def readmeDesignLines(Map target) { return lines } -// The README a published folder carries: every file in it, what it is, and where to read about -// it. Rendered from the module's declarations and the frame's own, so no module writes its own. -def readmeText(String module, Map target, List outputs) { +// The README a published folder carries: every file in it under its published name, what it is, +// and where to read about it. Rendered from the module's declarations and the frame's own, so no +// module writes its own. +def readmeText(String module, Map target, List outputs, String report) { def entry = moduleEntry(module) def lines = ["# ${target.label}: ${module}".toString(), '', - "Produced by PoolSeqFlow ${workflow.manifest.version ?: 'unknown'}, module " + + "Produced by PoolSeqFlow ${releaseVersion()}, module " + "${module} v${entry.version}, reading".toString(), '', " ${target.dir}".toString(), @@ -181,7 +201,8 @@ def readmeText(String module, Map target, List outputs) { '|---|---|---|'] // An optional output is named whether or not this run produced one, and its row says so. - (outputs + frameOutputs()).each { output -> + def published = outputs.collect { output -> output + [ file: publishedName(target, "${output.file}") ] } + (published + frameOutputs(report)).each { output -> def what = "${output.summary ?: ''}${output.optional ? ' *(only when the setting that produces it is set)*' : ''}" lines << "| `${output.file}` | ${what} | ${outputLink(output)} |".toString() } @@ -195,10 +216,11 @@ def readmeText(String module, Map target, List outputs) { } // The shell that writes README.md into `dest`, and refuses a module that declared a file it did -// not publish. Rendered into the task's own script, the way citationShell() is. -def readmeShell(String module, Map target, String dest) { +// not publish. Rendered into the task's own script, the way citationShell() is. `report` is the +// name the PDF is published under. +def readmeShell(String module, Map target, String dest, String report) { def outputs = moduleOutputs(module) - def text = readmeText(module, target, outputs).replace("'", "'\\''") + def text = readmeText(module, target, outputs, report).replace("'", "'\\''") def lines = [] outputs.each { output -> @@ -206,9 +228,10 @@ def readmeShell(String module, Map target, String dest) { // named, and none when you named none. Its absence is not a failure. if (output.optional) return // The declared name may be a glob, so the test counts what it matched. - lines << "MATCHED=\$(find \"${dest}\" -maxdepth 1 -name '${output.file}' | wc -l)" + def name = publishedName(target, "${output.file}") + lines << "MATCHED=\$(find \"${dest}\" -maxdepth 1 -name '${name}' | wc -l)" lines << "if [ \"\$MATCHED\" -eq 0 ]; then" - lines << " echo \"PUBLISHING ${target.label}: ERROR: ${module} declares it publishes '${output.file}',\" >&2" + lines << " echo \"PUBLISHING ${target.label}: ERROR: ${module} declares it publishes '${output.file}', here '${name}',\" >&2" lines << " echo \"PUBLISHING ${target.label}: and this analysis produced nothing matching it.\" >&2" lines << " echo \"PUBLISHING ${target.label}: What it produced:\" >&2" lines << " find \"${dest}\" -mindepth 1 -maxdepth 1 | sed 's|.*/| |' >&2" diff --git a/analysis/lib/nf/paths.nf b/analysis/lib/nf/paths.nf index b2e6d8d..1b46b98 100644 --- a/analysis/lib/nf/paths.nf +++ b/analysis/lib/nf/paths.nf @@ -236,6 +236,28 @@ def frameVersion() { return version } +// The PoolSeqFlow release this installation is, from the manifest in its nextflow.config. A +// module's own run has no workflow.manifest.version: Nextflow reads that file only for an entry +// script beside it. +def releaseVersion() { + def config = file("${installDir()}/nextflow.config") + def version = config.exists() + ? config.readLines().findResult { line -> + def hit = (line =~ /^\s*version\s*=\s*'([^']*)'/) + hit ? "${hit[0][1]}".toString() : null } ?: '' + : '' + if (!(version ==~ /\d+\.\d+\.\d+/)) { + throw new IllegalStateException( + "the installation does not say which release it is.\n" + + " ${config}\n" + + "should carry the release as `version = 'X.Y.Z'` in its manifest" + + (version.isEmpty() ? ', and carries none' : ", and carries '${version}'") + ".\n" + + "Every published analysis names the release that produced it. Install this release " + + "again.") + } + return version +} + // The installation this invocation was launched from, and a refusal when it cannot be found. A // module is its own pipeline, so ${projectDir} is the module's own directory and nothing // Nextflow computes points at the installation. The wrapper exports POOLSEQFLOW_HOME. diff --git a/analysis/lib/nf/plan.nf b/analysis/lib/nf/plan.nf index 4731b3d..c849b9f 100644 --- a/analysis/lib/nf/plan.nf +++ b/analysis/lib/nf/plan.nf @@ -94,6 +94,26 @@ def directoryLabel(Map variant) { return "${variant.dir.outputs}".tokenize('/').last() } +// The outputPrefix the files published from one results directory start with. Every run sharing +// the directory has the same one, as it is part of what step 7 shares on. Held to the rule +// bin/check_parameters.sh applies, 'null' included. +def targetPrefix(String label, List covered) { + def prefixes = covered.collect { run -> "${run.outputPrefix}".toString() }.unique() + if (prefixes.size() != 1) { + throw new IllegalStateException( + "the runs that share ${label} have ${prefixes.size()} outputPrefix values: " + + "${prefixes.join(', ')}. The plan and the run table disagree about which runs share work.") + } + def prefix = prefixes.first() + if (!(prefix ==~ /[A-Za-z0-9][A-Za-z0-9._-]*/) || prefix == 'null') { + throw new IllegalArgumentException( + "the outputPrefix of ${label} is '${prefix}', and it starts the name of every file " + + "an analysis publishes there. It may hold letters, digits, '.', '_' and '-', and " + + "starts with a letter or a digit.") + } + return prefix +} + // One entry per results directory the selected runs produced, and where each artifact class sits // inside it. Two runs whose tables are the same file share a directory and are covered once. // `needs` names the classes the module cannot run without. @@ -126,6 +146,7 @@ def resultsTargets(Map plan, List runDefs, List selected, String module, List ne checkTargetDesign(label, rows) return [ label : label, module : module, + prefix : targetPrefix(label, covered), dir : "${variant.dir.outputs}".toString(), members : variant.members.collect { member -> runToken(member) }, selected: variant.members.findAll { member -> wanted.contains(member) } diff --git a/analysis/lib/nf/report.nf b/analysis/lib/nf/report.nf index 028e97e..31812da 100644 --- a/analysis/lib/nf/report.nf +++ b/analysis/lib/nf/report.nf @@ -1,62 +1,77 @@ -// The PDF a published analysis carries: every result in the folder, each under its own file -// name. -// -// A FRAME CAPABILITY, not a module's: it is built from the same output declarations as the README, -// so a module writes nothing to get one. +// The PDF a published analysis carries. A module that declares a `report` in its manifest lays +// out its own results in it; any other module gets each table and figure it declared, under the +// file's name. // // Knitted by knitr and turned into a PDF by pandoc with typst as the engine. Not // rmarkdown::pdf_document, which is bound to LaTeX and ignores the engine it is given. nextflow.enable.dsl=2 -include { installDir; frameVersion } from './paths.nf' +include { installDir; frameVersion; releaseVersion } from './paths.nf' include { moduleEntry } from './modules.nf' include { moduleOutputs; frameOutputs } from './outputs.nf' -def reportName() { - return 'report.pdf' +// The name the PDF is published under: the target's outputPrefix, the module, and `stamp`, the +// time it was built as yyyyMMdd-HHmmss. +def reportName(Map target, String stamp) { + return "${target.prefix}_${target.module}_report_${stamp}.pdf".toString() } def reportTemplate() { return "${installDir()}/analysis/lib/rmd/report.Rmd".toString() } -// What the template renders from: where the folder is, what made it, and every file the module -// declared with the summary it declared. `report.pdf` itself is left out. -def reportSpec(String module, Map target, String folder) { +// The functions every report is drawn with, the frame's and a module's alike. +def reportLibrary() { + return "${installDir()}/analysis/lib/rmd/report.R".toString() +} + +// What the template renders from: what made the folder, the warnings its design carries, the +// module's own report if it declares one, and every file the module declared with the summary it +// declared, under the name the module wrote it. The report itself is left out. The folder is not in +// it: it exists only once the task runs, and reaches the template as POOLSEQFLOW_REPORT_FOLDER. +def reportSpec(String module, Map target, String report) { def entry = moduleEntry(module) - def outputs = (moduleOutputs(module) + frameOutputs()) - .findAll { output -> "${output.file}" != reportName() } + def outputs = (moduleOutputs(module) + frameOutputs(report)) + .findAll { output -> "${output.file}" != report } .collect { output -> [ file: "${output.file}".toString(), summary: "${output.summary ?: ''}".toString() ] } + def child = entry.report ? "${entry.dir}/${entry.report}".toString() : '' + def warnings = (target.design?.warnings ?: []) + .collect { note -> "${note.detail}".split('\n').join(' ').toString() } return groovy.json.JsonOutput.toJson([ - label : "${target.label}".toString(), - module : module, - version: "${entry.version}".toString(), - release: "${workflow.manifest.version ?: 'unknown'}".toString(), - frame : frameVersion(), - source : "${target.dir}".toString(), - folder : folder, - outputs: outputs ]) + label : "${target.label}".toString(), + module : module, + version : "${entry.version}".toString(), + release : releaseVersion(), + frame : frameVersion(), + source : "${target.dir}".toString(), + warnings: warnings, + library : reportLibrary(), + child : child, + outputs : outputs ]) } -// The shell that writes the report into `dest`. +// The shell that writes the report into `dest` as `report`. The files it reads are published +// under the target's outputPrefix, which reaches the template as POOLSEQFLOW_REPORT_PREFIX. // -// It does not fail the publish, and prints the reason when it cannot build one. -def reportShell(String module, Map target, String dest) { - def spec = reportSpec(module, target, dest).replace("'", "'\\''") +// It does not fail the publish, and prints the reason when it cannot build one. On stdout: the +// task succeeds, and Nextflow shows a successful task's stdout under `debug` and never its stderr. +def reportShell(String module, Map target, String dest, String report) { + def spec = reportSpec(module, target, report).replace("'", "'\\''") def lines = [] lines << "printf '%s' '${spec}' > report_spec.json" - lines << "POOLSEQFLOW_REPORT_SPEC=\"\$PWD/report_spec.json\" \\" - lines << " Rscript --vanilla -e 'knitr::knit(commandArgs(TRUE)[1], \"report.md\", quiet = TRUE)' \\" + lines << "POOLSEQFLOW_REPORT_SPEC=\"\$PWD/report_spec.json\" POOLSEQFLOW_REPORT_FOLDER=\"${dest}\" \\" + lines << " POOLSEQFLOW_REPORT_PREFIX='${target.prefix}' \\" + lines << " Rscript --vanilla -e 'knitr::opts_knit\$set(root.dir = getwd()); knitr::knit(commandArgs(TRUE)[1], \"report.md\", quiet = TRUE)' \\" lines << " '${reportTemplate()}' > report_knit.log 2>&1 \\" - lines << " && pandoc report.md -o \"${dest}/${reportName()}\" --pdf-engine=typst \\" + lines << " && pandoc report.md -o \"${dest}/${report}\" --pdf-engine=typst \\" lines << " >> report_knit.log 2>&1 \\" lines << " || {" - lines << " echo \"PUBLISHING ${target.label}: the PDF report could not be built. The analysis\" >&2" - lines << " echo \"PUBLISHING ${target.label}: is published without it; every number in it is a file\" >&2" - lines << " echo \"PUBLISHING ${target.label}: in the folder already.\" >&2" - lines << " sed 's/^/PUBLISHING ${target.label}: /' report_knit.log >&2 || true" + lines << " echo \"PUBLISHING ${target.label}: the PDF report could not be built. The analysis\"" + lines << " echo \"PUBLISHING ${target.label}: is published without it; every number in it is a file\"" + lines << " echo \"PUBLISHING ${target.label}: in the folder already.\"" + lines << " sed 's/^/PUBLISHING ${target.label}: /' report_knit.log || true" lines << " }" return lines.join('\n ') } diff --git a/analysis/lib/nf/results.nf b/analysis/lib/nf/results.nf index 6aca25a..4a09ebc 100644 --- a/analysis/lib/nf/results.nf +++ b/analysis/lib/nf/results.nf @@ -8,14 +8,8 @@ nextflow.enable.dsl=2 include { verificationRecordName } from './paths.nf' include { citationShell } from './citations.nf' -include { readmeShell } from './outputs.nf' -include { reportShell } from './report.nf' - -// What counts as the script that produced a result, by extension: R, and the shell, Python and -// Julia a module may drive it from. -def scriptSuffixes() { - return ['R', 'r', 'Rmd', 'rmd', 'sh', 'py', 'jl'] -} +include { readmeShell; scriptSuffixes; codeSuffixes } from './outputs.nf' +include { reportShell; reportName } from './report.nf' // One results directory's analysis, moved into place. process InstallResults { @@ -34,10 +28,15 @@ process InstallResults { // `find -name` tests, one per accepted extension, joined for a single walk of the stage. script_test = scriptSuffixes().collect { s -> "-name '*.${s}'" }.join(' -o ') script_list = scriptSuffixes().collect { s -> "*.${s}" }.join(', ') + // The code a module produced, as the patterns of a `case`. + code_case = codeSuffixes().collect { s -> "*.${s}" }.join('|') + // When the analysis is published, in local time, which the report's name carries. + stamp = java.time.LocalDateTime.now().format(java.time.format.DateTimeFormatter.ofPattern('yyyyMMdd-HHmmss')) + report_name = reportName(target, stamp) // Written into the STAGE, so the citations arrive in the same rename as the analysis. citations = citationShell("${target.module}", '$STAGE') - readme = readmeShell("${target.module}", target, '$STAGE') - report = reportShell("${target.module}", target, '$STAGE') + readme = readmeShell("${target.module}", target, '$STAGE', report_name) + report = reportShell("${target.module}", target, '$STAGE', report_name) """ set -eo pipefail @@ -67,6 +66,17 @@ process InstallResults { exit 1 fi + # Every file and directory the module produced starts with the outputPrefix, and the code it + # was produced by keeps its own name. What a directory holds keeps its names. + for FILE in "\$STAGE"/*; do + [ -e "\$FILE" ] || continue + NAME=\$(basename "\$FILE") + case "\$NAME" in + ${code_case}) ;; + *) mv -- "\$FILE" "\$STAGE/${target.prefix}_\$NAME" ;; + esac + done + # Every file the module said it would publish is here, or nothing is published. ${readme} diff --git a/analysis/lib/rmd/report.R b/analysis/lib/rmd/report.R new file mode 100644 index 0000000..315f805 --- /dev/null +++ b/analysis/lib/rmd/report.R @@ -0,0 +1,329 @@ +# The functions a report is drawn with. report.Rmd sources this before anything is knitted, so the +# frame's own layout and a module's report call the same ones. +# +# Every function that draws prints markdown or a raw typst block, so it is called from a chunk with +# results = "asis". + +# The published folder the report describes. +report_folder <- function() { + folder <- Sys.getenv("POOLSEQFLOW_REPORT_FOLDER") + if (!nzchar(folder) || !dir.exists(folder)) { + stop("report: POOLSEQFLOW_REPORT_FOLDER names no folder: '", folder, "'") + } + folder +} + +# The outputPrefix every file the module computed is published under. +report_prefix <- function() { + prefix <- Sys.getenv("POOLSEQFLOW_REPORT_PREFIX") + if (!nzchar(prefix)) stop("report: POOLSEQFLOW_REPORT_PREFIX is not set") + prefix +} + +# The name a file the module wrote is published under, the prefix and an underscore in front. A +# glob gives a glob. +report_name <- function(name) { + paste0(report_prefix(), "_", name) +} + +# A published name with the prefix taken off: the name the module wrote. +report_logical <- function(name) { + lead <- paste0(report_prefix(), "_") + ifelse(startsWith(name, lead), substring(name, nchar(lead) + 1), name) +} + +# A published table, by the name the module wrote it under, with every column as text, so a +# sequence called 4 stays a name. NULL when the folder does not hold it. The tables are written +# unquoted, so a quote in a cell is a character. +report_read <- function(name) { + path <- file.path(report_folder(), report_name(name)) + if (!file.exists(path)) return(NULL) + utils::read.delim(path, check.names = FALSE, colClasses = "character", quote = "", + comment.char = "") +} + +# The published files a glob of the names the module wrote matches, in name order. The glob is +# matched against the names alone, so a bracket in the folder's own path is a character. +report_files <- function(pattern) { + sort(list.files(report_folder(), pattern = utils::glob2rx(report_name(pattern)), + full.names = TRUE)) +} + +# Text as markdown that reads exactly as written: every ASCII punctuation mark escaped, so a name +# holding * or @ is not emphasis or a citation. +report_text <- function(x) { + gsub("([[:punct:]])", "\\\\\\1", as.character(x), perl = TRUE) +} + +# Numbers as a table shows them: NA, NaN, Inf and -Inf as themselves. With `decimals`, that many +# places after the point. Without, a whole number is printed whole, a magnitude below 1e-4 or from +# 1e15 in scientific notation, and anything else to `digits` significant digits. Thousands are +# separated. +report_number <- function(x, digits = 3, decimals = NULL) { + x <- suppressWarnings(as.numeric(x)) + out <- rep("NA", length(x)) + out[is.nan(x)] <- "NaN" + out[!is.na(x) & x == Inf] <- "Inf" + out[!is.na(x) & x == -Inf] <- "-Inf" + ok <- is.finite(x) + # A value that rounds to zero prints unsigned; formatC writes it as -0. + if (!is.null(decimals)) x[ok] <- round(x[ok], decimals) + x[ok & x == 0] <- 0 + if (!is.null(decimals)) { + out[ok] <- formatC(x[ok], format = "f", digits = decimals, big.mark = ",") + return(out) + } + whole <- ok & x == round(x) & abs(x) < 1e15 + out[whole] <- formatC(x[whole], format = "f", digits = 0, big.mark = ",") + rest <- ok & !whole + out[rest] <- vapply(x[rest], function(v) { + if (abs(v) < 1e-4 || abs(v) >= 1e15) return(formatC(v, format = "e", digits = digits - 1)) + sub("[.]$", "", formatC(signif(v, digits), digits = digits, format = "fg", flag = "#", + big.mark = ",")) + }, "") + out +} + +# Every column of a table that holds decimals, printed by report_number(), for a table the report +# knows nothing about. A column of whole numbers is left as written, and so is any column holding a +# value like 01. +report_format <- function(table, digits = 4) { + for (name in names(table)) { + values <- table[[name]] + held <- values[!is.na(values) & nzchar(values) & !(values %in% c("NA", "NaN", "Inf", "-Inf"))] + if (length(held) == 0 || anyNA(suppressWarnings(as.numeric(held)))) next + if (!any(grepl(".", held, fixed = TRUE)) || any(grepl("^[-+]?0[0-9]", held))) next + table[[name]] <- ifelse(is.na(values) | !nzchar(values), values, + report_number(values, digits = digits)) + } + table +} + +# The rows of a published table below its header, counted without reading them as a table. +count_rows <- function(path) { + con <- file(path, "r") + on.exit(close(con)) + lines <- 0L + repeat { + chunk <- readLines(con, n = 100000L, warn = FALSE) + if (length(chunk) == 0) break + lines <- lines + length(chunk) + } + max(lines - 1L, 0L) +} + +# Text as a typst string literal, which typst sets as plain text: no character in it is markup. +typst_string <- function(x) { + x <- ifelse(is.na(x), "NA", as.character(x)) + x <- gsub("\\", "\\\\", x, fixed = TRUE) + x <- gsub("\"", "\\\"", x, fixed = TRUE) + x <- gsub("\n", "\\n", x, fixed = TRUE) + paste0("\"", x, "\"") +} + +# A column is aligned right when every value in it is a number, NA or empty, a column holding +# nothing else included. +numeric_column <- function(values) { + held <- values[!is.na(values) & nzchar(values) & values != "NA"] + all(grepl("^[-+\u2265\u2264<>~ ]*[0-9][0-9,.]*(e[-+]?[0-9]+)?%?$", held)) +} + +# A table, as typst draws it: the header repeated on every page the table runs onto, a cell +# holding a newline set on two lines, and no cell broken anywhere else. +# +# Typst measures every cell and every header once and keeps the first layout that fits the text +# width: each column as wide as what is in it, at 10, 9, 8 or 7 points; then each column at least +# as wide as its widest cell and the widest part of its header, with the space left over shared in +# proportion to what each column lacks of its full width and the headers wrapping after an +# underscore, a dot or a space, at 10 down to 5 points; then that at 4 points. Every one of them +# breaks across pages. +# +# `align` is one letter per column, "l" or "r"; without it a column of numbers is aligned right and +# anything else left. `groups` is one label per column, and a run of equal labels is printed once +# above the columns it spans; "" spans nothing. `size` fixes the text size in points, with the +# headers on one line. A table longer than `max_rows` shows its first rows. `total` is how many +# rows the table has when it was read cut short, and the table says how many `source` holds. +report_table <- function(table, caption = NULL, align = NULL, groups = NULL, size = NULL, + max_rows = Inf, source = NULL, total = nrow(table)) { + stopifnot(is.data.frame(table)) + # Evaluated here: its default counts the rows before the cut below. + force(total) + if (ncol(table) == 0) { + cat("\n*Nothing to show", if (!is.null(caption)) paste0(": ", caption), ".*\n\n", sep = "") + return(invisible()) + } + if (nrow(table) > max_rows) table <- utils::head(table, max_rows) + if (is.null(align)) align <- vapply(table, function(col) if (numeric_column(col)) "r" else "l", "") + alignment <- ifelse(align == "r", "right", "left") + + n <- ncol(table) + as_array <- function(items) paste0("(", paste(c(items, ""), collapse = ", "), ")") + values <- lapply(table, function(col) typst_string(ifelse(is.na(col), "NA", as.character(col)))) + labels <- names(table) + # A header's parts: where it may wrap, after an underscore or a dot, or at a space. + parts <- lapply(labels, function(label) { + pieces <- strsplit(gsub("([_.])", "\\1\001", label), "\001| ", perl = TRUE)[[1]] + pieces[nzchar(pieces)] + }) + + # The header rows, and the rule under each group label. + names_row <- paste(sprintf("strong(if wrapped { soft.at(%d) } else { whole.at(%d) })", + seq_len(n) - 1, seq_len(n) - 1), collapse = ", ") + header_rows <- 1 + if (is.null(groups)) { + header <- paste0("table.header(", names_row, ")") + } else { + stopifnot(length(groups) == ncol(table)) + runs <- rle(groups) + starts <- cumsum(c(0, utils::head(runs$lengths, -1))) + spans <- character(0) + rules <- character(0) + # A label is broken between words to about the width of the columns under it, counted in + # characters. + chars <- vapply(seq_along(table), function(j) { + max(nchar(names(table)[j]), nchar(as.character(table[[j]])), na.rm = TRUE) + 2 + }, 0) + for (i in seq_along(runs$lengths)) { + if (!nzchar(runs$values[i])) { + spans <- c(spans, rep("[]", runs$lengths[i])) + next + } + room <- sum(chars[starts[i] + seq_len(runs$lengths[i])]) + label <- paste(strwrap(runs$values[i], width = max(room, 1)), collapse = "\n") + spans <- c(spans, sprintf("table.cell(colspan: %d, align: center, strong(%s))", + runs$lengths[i], typst_string(label))) + rules <- c(rules, sprintf("table.hline(y: 1, start: %d, end: %d, stroke: 0.4pt)", + starts[i], starts[i] + runs$lengths[i])) + } + header <- paste0("table.header(", paste(c(spans, rules), collapse = ", "), ", ", + names_row, ")") + header_rows <- 2 + } + + # The cells a row to a line, and the same cells a column to an array for measuring. + rows <- if (nrow(table) > 0) paste0(" ", do.call(paste, c(values, sep = ", ")), ",") + else character(0) + data <- c( + " let flat = (", + rows, + " )", + sprintf(" let cells = range(%d).map(j => range(%d).map(i => flat.at(i * %d + j)))", + n, nrow(table), n), + sprintf(" let whole = %s", as_array(typst_string(labels))), + sprintf(" let soft = %s", as_array(typst_string(gsub("([_.])", "\\1\u200b", labels)))), + sprintf(" let parts = %s", as_array(vapply(parts, function(p) as_array(typst_string(p)), "")))) + make <- c( + " let make(s, columns, wrapped) = text(size: s, table(", + " columns: columns,", + sprintf(" align: %s,", as_array(alignment)), + " stroke: none,", + " inset: (x: 5pt, y: 3.5pt),", + sprintf(" fill: (_, y) => if y >= %d and calc.odd(y - %d) { luma(245) },", + header_rows, header_rows), + " table.hline(),", + paste0(" ", header, ","), + " table.hline(stroke: 0.5pt),", + " ..flat,", + " table.hline(),", + " ))") + # Widths are measured at 10 points and scale with the text; each column also carries its + # 5pt inset on either side. + shown <- if (!is.null(size)) sprintf(" let shown = make(%spt, %d, false)", size, n) else c( + " let width(x, bold) = measure(text(size: 10pt, if bold { strong(x) } else { x })).width", + " let widest(xs, bold) = xs.fold(0pt, (m, x) => calc.max(m, width(x, bold)))", + sprintf(" let least = range(%d).map(j => calc.max(widest(cells.at(j), false), widest(parts.at(j), true)))", n), + sprintf(" let most = range(%d).map(j => calc.max(widest(cells.at(j), false), width(whole.at(j), true)))", n), + " let at(ws, s) = ws.map(w => w * (s / 10pt) + 10pt)", + " let fits(ws, s) = at(ws, s).sum() <= region.width", + " let natural = (10pt, 9pt, 8pt, 7pt).find(s => fits(most, s))", + sprintf(" let shown = if natural != none { make(natural, %d, false) } else {", n), + " let found = (10pt, 9pt, 8pt, 7pt, 6pt, 5pt).find(s => fits(least, s))", + " let s = if found != none { found } else { 4pt }", + " let low = at(least, s)", + " let lacking = at(most, s).zip(low).map(((h, l)) => h - l)", + " let spare = calc.max(0pt, region.width - low.sum())", + " let total = lacking.sum()", + " let widths = low.zip(lacking).map(((l, d)) =>", + " if total > 0pt { l + calc.min(d, spare * (d / total)) } else { l })", + " make(s, widths, true)", + " }") + placed <- if (is.null(caption)) " shown" + else paste0(" figure(kind: table, caption: ", typst_string(caption), ", shown)") + cat("\n\n```{=typst}\n#layout(region => {\n", paste(c(data, make, shown, placed), collapse = "\n"), + "\n})\n```\n\n", sep = "") + if (total > nrow(table)) { + cat(nrow(table), " of ", total, " rows", + if (!is.null(source)) paste0(". The whole table is in `", source, "`"), ".\n\n", sep = "") + } + invisible() +} + +# A PNG's width and height in pixels, from its header; NULL for any other file. +png_size <- function(path) { + head <- readBin(path, "raw", n = 24L) + if (length(head) < 24L || !identical(head[2:4], charToRaw("PNG"))) return(NULL) + number <- function(bytes) sum(as.numeric(bytes) * 256^(3:0)) + c(number(head[17:20]), number(head[21:24])) +} + +# A figure already on disk, scaled to `width` of the text width, and a PNG at the full width kept +# to 7.5 in tall so a caption fits under it. The template's 1 in margins on letter paper leave a +# text block 6.5 in wide and 9 in tall. The path is in angle brackets, so a parenthesis in it is a +# character. +report_image <- function(path, caption = NULL, width = "100%") { + size <- png_size(path) + if (identical(width, "100%") && !is.null(size) && 7.5 * size[1] / size[2] < 6.5) { + width <- sprintf("%.2fin", 7.5 * size[1] / size[2]) + } + cat("\n\n![", if (is.null(caption)) "" else caption, "](<", path, ">){width=", width, "}\n\n", + sep = "") + invisible() +} + +# A ggplot drawn into report_figures/ beside the report and placed like report_image(). `width` +# and `height` are the size it is drawn at, in inches. +report_plot <- function(plot, name, width = 7, height = 4, caption = NULL) { + dir.create("report_figures", showWarnings = FALSE) + path <- file.path("report_figures", paste0(name, ".png")) + ggplot2::ggsave(path, plot, width = width, height = height, dpi = 200) + report_image(path, caption) +} + +# Items split into as few groups of at most `most` as hold them all, in the order given, the sizes +# differing by at most one: 25 pools make tables of 7, 6, 6 and 6 columns. +report_columns <- function(items, most = 8) { + groups <- max(1, ceiling(length(items) / most)) + sizes <- length(items) %/% groups + (seq_len(groups) <= length(items) %% groups) + split(items, rep(seq_len(groups), sizes)) +} + +# One color per pool, in the order given: the Okabe-Ito palette for up to eight pools, and evenly +# spaced hues past that. +report_palette <- function(pools) { + okabe_ito <- c("#E69F00", "#56B4E9", "#009E73", "#0072B2", "#D55E00", "#CC79A7", + "#F0E442", "#000000") + colors <- if (length(pools) <= length(okabe_ito)) okabe_ito[seq_along(pools)] + else grDevices::hcl.colors(length(pools), "Dark 3") + stats::setNames(colors, pools) +} + +# Each declared .tsv table and .png figure under its own file name, which is the whole report of a +# module that declares no report of its own. A table is read only as far as the rows it shows. +report_declared <- function(outputs, max_rows = 40) { + for (output in outputs) { + for (path in report_files(output$file)) { + name <- basename(path) + if (grepl("[.]tsv$", name)) { + cat("\n\n## ", report_text(name), "\n\n", sep = "") + table <- utils::read.delim(path, check.names = FALSE, colClasses = "character", + nrows = max_rows, quote = "", comment.char = "") + report_table(report_format(table), caption = output$summary, source = name, + total = count_rows(path)) + } else if (grepl("[.]png$", name)) { + cat("\n\n## ", report_text(name), "\n\n", sep = "") + report_image(path, caption = output$summary) + } + } + } + invisible() +} diff --git a/analysis/lib/rmd/report.Rmd b/analysis/lib/rmd/report.Rmd index 047cb43..6f4adbf 100644 --- a/analysis/lib/rmd/report.Rmd +++ b/analysis/lib/rmd/report.Rmd @@ -1,63 +1,66 @@ +--- +margin: + x: 1in + y: 1in +--- + ```{r setup, include = FALSE} -# The PDF the frame writes beside every published analysis: every result it holds, each under -# the name of the file it came from. +# The PDF the frame writes beside every published analysis. # # Knitted to markdown and handed to pandoc, never rendered by rmarkdown::pdf_document, which is # bound to LaTeX and ignores --pdf-engine. # -# The spec is written by analysis/lib/nf/report.nf and names the published folder, what -# produced it, and every file the module declared with the summary it declared. Nothing here -# knows what any particular module computes. -knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE) +# The spec is written by analysis/lib/nf/report.nf: what produced the folder, the warnings its +# design carries, the module's own report if it declares one, and every file the module declared +# with the summary it declared. The folder itself is POOLSEQFLOW_REPORT_FOLDER. +knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, error = FALSE) + +# Characters are read and written as UTF-8 whatever locale the task inherited. Outside one, R +# writes the report's own pi and its breaks as and , and a name with an accent in +# it as bytes. +if (!isTRUE(l10n_info()[["UTF-8"]])) { + for (locale in c("C.UTF-8", "C.utf8", "en_US.UTF-8", "en_US.utf8")) { + if (nzchar(suppressWarnings(Sys.setlocale("LC_CTYPE", locale)))) break + } + if (!isTRUE(l10n_info()[["UTF-8"]])) { + stop("report: this machine has no UTF-8 locale (tried C.UTF-8, C.utf8, en_US.UTF-8 ", + "and en_US.utf8), and the report cannot print its own symbols or a pool name ", + "with an accent without one") + } +} + spec <- jsonlite::fromJSON(Sys.getenv("POOLSEQFLOW_REPORT_SPEC"), simplifyVector = FALSE) -folder <- spec$folder +source(spec$library) +``` -# A table is shown in full up to this many rows. Past it the head is shown and the full row -# count is printed with it. -ROW_CAP <- 40 +```{=typst} +#set par(justify: false) +#show figure.where(kind: table): set figure.caption(position: top) +#show figure.where(kind: table): set block(breakable: true) +#show figure.caption.where(kind: table): it => block(sticky: true, it) +#show heading.where(level: 2): set block(above: 1.8em, below: 0.9em) ``` -# `r spec$label`: `r spec$module` +# `r spec$module`: `r spec$label` -Produced by PoolSeqFlow `r spec$release`, module `r spec$module` v`r spec$version`, -analysis frame `r spec$frame`, on `r format(Sys.time(), "%Y-%m-%d %H:%M")`. +Produced by PoolSeqFlow `r spec$release`, module `r spec$module` v`r spec$version` and analysis frame `r spec$frame`, on `r format(Sys.time(), "%Y-%m-%d %H:%M")`. -Read from `r spec$source`. +Read from `r paste0("\x60", spec$source, "\x60")`. -```{r contents} -rows <- do.call(rbind, lapply(spec$outputs, function(o) data.frame( - File = sprintf("`%s`", o$file), What = o$summary %||% "", stringsAsFactors = FALSE))) -knitr::kable(rows, caption = "What this folder holds") +```{r warnings, results = "asis"} +if (length(spec$warnings) > 0) { + cat("\n## Read these before the numbers\n\n") + for (note in spec$warnings) cat("- ", note, "\n", sep = "") + cat("\n") +} ``` -```{r results = "asis"} -`%||%` <- function(a, b) if (is.null(a)) b else a - -for (output in spec$outputs) { - matched <- Sys.glob(file.path(folder, output$file)) - if (length(matched) == 0) next # an optional output this run did not produce - for (path in sort(matched)) { - name <- basename(path) - # A typst raw block, not \newpage: pandoc passes LaTeX commands only to LaTeX and - # drops this one in silence, so the report would run together with nothing to show it. - cat("\n\n```{=typst}\n#pagebreak()\n```\n\n## ", name, "\n\n", sep = "") - if (!is.null(output$summary)) cat("*", output$summary, "*\n\n", sep = "") - - if (grepl("[.]tsv$", name)) { - table <- utils::read.delim(path, check.names = FALSE, colClasses = "character") - shown <- utils::head(table, ROW_CAP) - print(knitr::kable(shown)) - if (nrow(table) > ROW_CAP) { - cat("\n\n", nrow(shown), " of ", nrow(table), - " rows. The whole table is in `", name, "`.\n", sep = "") - } - } else if (grepl("[.]png$", name)) { - # Scaled to the text width, not placed at its own size: a figure sized for looking - # at on its own pushes the heading naming it onto the previous page. - cat("\n![](", path, "){width=90%}\n", sep = "") - } else { - cat("\nNot rendered here - read `", name, "` itself.\n", sep = "") - } - } +```{r body, results = "asis"} +if (nzchar(spec$child)) { + cat(knitr::knit_child(spec$child, quiet = TRUE), sep = "\n") +} else { + report_declared(spec$outputs) } ``` + +Every number here is in a file in this folder, and `README.md` says which file holds what. diff --git a/bin/check_parameters.sh b/bin/check_parameters.sh index 1d7f08e..ea24a66 100755 --- a/bin/check_parameters.sh +++ b/bin/check_parameters.sh @@ -15,9 +15,9 @@ # # Silence is the answer for a parameter set with nothing wrong with it. # -# WHAT BELONGS HERE: a setting that makes the run produce NOTHING, or that silently changes what -# a published number means. What does not: a setting that merely produces fewer sites. minDP 20 -# against minDP 5 is a scientific choice and this has no opinion on it. +# WHAT BELONGS HERE: a setting that makes the run produce NOTHING or stops it partway, or that +# silently changes what a published number means. What does not: a setting that merely produces +# fewer sites. minDP 20 against minDP 5 is a scientific choice and this has no opinion on it. # # VALUES ARE READ FROM THE COMPOSED OPTION STRINGS WHERE THERE IS ONE. A project may pin # variantCall.mpileupOptions by hand, which makes scaleMapQ and varQualMin inert, so judging the @@ -146,4 +146,25 @@ if [ -n "$fqmem" ] && ! is_int "$fqmem"; then "FastQC takes megabytes as a bare integer and refuses a value carrying a unit, so write 2048 rather than 2G." fi +# ------------------------------------------------------------------- the names -- +# +# outputPrefix and vcf.fileName each start file names: letters, digits, '.', '_' and '-', the +# first a letter or a digit. 'null' is what the template's vcf.fileName line holds when the +# config has no outputPrefix line. Judged only when the input carries the key: an absent key is +# not an empty name. +for key in outputPrefix vcf.fileName; do + [ -n "${P[$key]+set}" ] || continue + name=$(get "$key") + if [ -z "$name" ]; then + say FAIL "$key" "EMPTY" "no name" \ + "It starts the name of the files it covers, so they would be named from nothing. Write a name: letters, digits, '.', '_' and '-', starting with a letter or a digit." + elif [ "$name" = "null" ]; then + say FAIL "$key" "NOT SET" "null" \ + "null is what the template's vcf.fileName line reads when parameters.config has no outputPrefix line. Put the line back, or write a name." + elif ! printf '%s' "$name" | grep -qx '[A-Za-z0-9][A-Za-z0-9._-]*'; then + say FAIL "$key" "NOT A FILE NAME" "$name" \ + "It starts the name of the files it covers, so it may hold letters, digits, '.', '_' and '-', and starts with a letter or a digit." + fi +done + exit "$FAILED" diff --git a/bin/config_migrate.sh b/bin/config_migrate.sh index 98444de..8b1ad2d 100755 --- a/bin/config_migrate.sh +++ b/bin/config_migrate.sh @@ -78,6 +78,13 @@ awk -v OLDF="$OLD" -v REPORT="$REPORT" ' return (k == "fastqc.memory" || k == "variantCall.maxDepth") } + # A value of yours where the template computes one, kept as written: vcf.fileName in a config + # that already has outputPrefix, whatever it holds, and an expression in one that does not. + # A literal in a config without outputPrefix is the old name, and outputPrefix takes it. + function pinned(k) { + return (k == "vcf.fileName" && (("outputPrefix" in oldval) || !literal(oldval[k]))) + } + # Parameters renamed in this release, keyed by their CURRENT name and returning the name # they had before. Needs a line per rename: without one, a rename reads as one DROPPED plus # one NEW. A rename that also changed the meaning goes in reformatted() too, which runs @@ -87,6 +94,10 @@ awk -v OLDF="$OLD" -v REPORT="$REPORT" ' if (k == "vcffilter.minQUAL") return "vcftools.minQUAL" if (k == "storageDir") return "projectDir" if (k == "ploidy") return "diploidy" + # vcf.fileName stays, and the template now derives it from outputPrefix. Only a name + # moves: an expression stays where it is. + if (k == "outputPrefix") + return (("vcf.fileName" in oldval) && literal(oldval["vcf.fileName"])) ? "vcf.fileName" : "" # Whole-scope renames: every field follows its prefix, so none is listed by name. old = k if (sub(/^cleanBAM\./, "samtools.", old)) return old @@ -146,14 +157,14 @@ awk -v OLDF="$OLD" -v REPORT="$REPORT" ' if (src in oldval) { used[src] = 1 - if (!literal(val)) { + if (!literal(val) && !(pinned(full) && oldval[src] != val)) { # this release computes it - the template wins if (literal(oldval[src]) && oldval[src] != val) printf "COMPUTED\t%s\t%s\t%s\n", full, val, oldval[src] >> REPORT } else if (reformatted(full)) { if (oldval[src] != val) printf "REFORMAT\t%s\t%s\t%s\n", full, val, oldval[src] >> REPORT - } else if (literal(oldval[src])) { + } else if (literal(oldval[src]) || pinned(full)) { if (was_renamed) printf "RENAMED\t%s\t%s\t%s\n", full, oldval[src], src >> REPORT else if (oldval[src] != val) @@ -369,4 +380,32 @@ if [ -n "$NEW_KEEPLOW" ]; then echo fi +# Read back from the report, so this fires exactly when outputPrefix took the old vcf.fileName. +NEW_PREFIX=$(awk -F'\t' '$1 == "RENAMED" && $2 == "outputPrefix" { print $3 }' "$REPORT") +NEW_PREFIX=${NEW_PREFIX//[\'\"]/} +if [ -n "$NEW_PREFIX" ]; then + note_heading + echo " outputPrefix IS NEW, and it took your vcf.fileName, $NEW_PREFIX." + echo + echo " It starts the name of the files a run writes. vcf.fileName now follows it, so your" + echo " VCF and its frequency and depth tables keep the names they have. What changes is the" + echo " analysis layer: every file an analysis publishes now starts with it, so mds.tsv is" + echo " ${NEW_PREFIX}_mds.tsv, and each report is named ${NEW_PREFIX}__report_-