diff --git a/.claude/development-notes/changelog-3.3.0-prose.md b/.claude/development-notes/changelog-3.3.0-prose.md new file mode 100644 index 0000000..1901f78 --- /dev/null +++ b/.claude/development-notes/changelog-3.3.0-prose.md @@ -0,0 +1,51 @@ +# The 3.3.0 release notes, kept while the bump is reverted + +**Written 2026-10-07 against `main` at `6550cb5`.** The prose of the `## [3.3.0]` section of `CHANGELOG.md` as Z left it, from its heading down to `### Commits`, copied verbatim below. + +## Why it is here + +v3.3.0 was tagged at `babb475` on 2026-10-07 and never released. `release.yml` stopped at its analysis version gate: `raise-module-floors.sh` had moved the `environment` floor of six manifests (association, basicstats, mds, allele_frequencies, n_eff, nei_distance) from 3.0.0 to 3.3.0, and those edits landed in the version-bump commit, which moved none of their versions. No GitHub release was created, and the tag was deleted locally and on origin. + +A review of the new `publish-module.sh --all-pending` then found defects, and Z decided they are to be fixed before releasing: "This is more serious." So the bump is reverted on `dev`, and `bump-version.sh --revert` deletes this section along with it. + +The revert does not fix what stopped the release. Raising the floors at the next bump puts the same edit in the same kind of commit, and the gate fails the same way until `raise-module-floors.sh` or the gate changes. + +## What to check when it goes back + +The next bump writes its own heading, date and commit list, and this prose goes back above `### Commits`. + +- The module versions under Analysis modules are the ones prepared for this cycle. A module bumped again before the release carries a new version, and its line has to name that one. +- Anything that lands on `dev` before the next bump and changes what a user sees needs a line of its own. `publish-module.sh --all-pending` is a maintainer tool and does not. + +## The section + +```markdown +## [3.3.0] - 2026-10-06 + +**The shallowest library no longer has to decide which sites the whole cohort keeps.** Until now one pool below `vcffilter.minDP` removed a site for every pool. `vcffilter.keepLowDepthAsZero` keeps the site instead and writes that pool's reads there as unread. It is off by default, so nothing changes until you turn it on. + +**Two things to read before upgrading**, both under Changed: the default of `variantCall.scaleMapQ` moved from 50 to 100, and a migrated `parameters.config` keeps its 50; and `init_multi` and `uninstall_all` are now two words. + +### Added + +- **`vcffilter.keepLowDepthAsZero` and `vcffilter.minSamples`.** With the switch on, every cell below `minDP` (one pool's reads at one site) is written as unread, which the depth table carries as zeros and the frequency table as `NA`, and a site is kept when at least `minSamples` of its cells reach `minDP`. `minSamples` defaults to 2 and runs from 1 to the number of pools with reads; anything else is refused before the run starts. Every read still counts toward deciding that an allele exists, because the false-positive filter runs first; only the cells at the floor count toward measuring its frequency. The manual's depth and quality filter section has the details, including what it does to `TOTAL_AD`. +- **Settings that would make a run produce nothing are refused at step 0**, and `PoolSeqFlow check project` reports the same findings before you run. A `scaleMapQ` below `varQualMin` discards every read, a `sampleThreshold` above 1 removes every site, a `ploidy` or `poolSize` below 1 breaks every detection limit, and a `fastqc.memory` with a unit is refused by FastQC. Settings that work but cost more than they look, or change what a number means, are warned about and the run continues. The manual lists them all. +- **Files saved on Windows.** A `metadata.csv` or `runs.csv` that Excel saved as "CSV UTF-8" is read as it is, and a UTF-16 one is refused with what to save it as instead. A `parameters.config` that begins with a byte-order mark is named by `check project`, with the command that removes it; Nextflow on its own refuses the file over a character nothing displays. +- **Native zsh completion**, with a description beside each command. The installer prints the one line `~/.zshrc` needs. + +### Changed + +- **`variantCall.scaleMapQ` defaults to 100, where it was 50.** It is the `-C` that lowers the mapping quality of reads with many mismatches. Measured on real pools, 100 removes the sites that are artifacts (Ti/Tv 0.848, nearly four times the soft-clip bias of the sites kept), while 50 removed five times as many and took real variants with them. 50 also lowered the frequencies it kept, by 0.035 on average against 0.008 at 100, and by 0.081 in pools between 0.50 and 0.75. **A migrated `parameters.config` keeps 50**: `migrate_config` carries your value across, as it does every setting that still exists, and reports it under `Kept your value`. Set it to 100 before the first run under 3.3.0. The manual has the full measurements. +- **`init_multi` is `init multi`, and `uninstall_all` is `uninstall all`.** The old spellings stop with a message giving the new one. +- **A pool with no reads at a site is `NA` in the frequency table**, where it was 0, because a frequency over zero reads does not exist. By default no such cell reaches a table: the depth filter now also drops a site with an unread pool at `minDP` 0, which 3.2.0 kept and published as 0. With `keepLowDepthAsZero` on, `NA` is what an unread cell reads. +- **Every step writes its log as it runs.** A step that fails, or is killed, leaves what it had reached in `Logs/`. Before this, a step copied its log there only at the end, so a failure lost exactly the log that would have explained it. Each attempt starts with a line naming the run, the session and the time. +- **Capping is about seven times faster on real data**, with identical output. +- **A call set or filter that leaves nothing stops the run** and says which settings to look at, instead of finishing as though it had succeeded. Nothing is published from it. +- **Environments.** FastQC 0.12.1 to 0.13.0, Perl 5.32.1 to 5.44.0, pandoc 3.11 to 3.12, R's future and doFuture to their next minor versions, and patch updates beneath them. bwa, samtools, bcftools, cutadapt, Trim Galore and snpEff did not move. + +### Analysis modules + +- **association `20261006.001`.** Two ways it published `perm_p` 0, which no permutation p can be. A site where any unit had no reads came out 0, and `fdr_p` with it; in 3.2.0 that took `minDP` 0. And the strongest sites in a table, far beyond any rearrangement, failed to count themselves and came out 0 too. Each site is now tested over the units it was read in, and not tested when fewer than three were read. `design_floor` is 1/n!, where it said 2/n!, so four units can reach 0.042. +- **mds `20261006.002`.** A pair of pools averaged over fewer than 30 shared sites is flagged, in `distance.tsv`, on the console and under the plot. In simulation, a distance near 0.02 measured over 30 shared sites is off by about half itself, and real sites are noisier. +- **basicstats `20261006.001`.** A pool's depth summaries cover only the sites it was read at, and `depth.tsv` and `diversity.tsv` count the rest in a new `unmeasured` column. A site with no reads for a pool was averaged in as a depth of 0, and one such site was enough to turn that pool's harmonic depth into 0 and its effective sample size into NA; in 3.2.0 that took `minDP` 0. +``` diff --git a/.claude/development-notes/mds-labels.md b/.claude/development-notes/mds-labels.md new file mode 100644 index 0000000..3fb060f --- /dev/null +++ b/.claude/development-notes/mds-labels.md @@ -0,0 +1,40 @@ +# Names on mds.png: ggrepel, and a setting to leave them off + +Written 2026-10-08, against `dev` at `d8b7cee` plus the uncommitted report work of that day (`module-reports.md`). Not updated to follow the code. + +## The ask + +The example reports showed `mds.png` with 24 long pool names printed over each other in every cluster, and the outermost names cut at the edge of the image. I asked whether to add `ggrepel`, label with short numbers and a key, or leave it. Z: *"add ggrepel + add disabling the names"*. + +## What was built + +- **`labels`**, a new mds setting, default `true`. `false` draws the points alone. mds.R refuses anything but true or false, because the string `'false'` would otherwise read as false and the names would vanish with nothing said, and it refuses `true` at start when `ggrepel` is not installed, before any work, naming the setting that avoids it. +- **`ggrepel::geom_text_repel`** in place of `geom_text`, with `seed = 1`, `max.overlaps = Inf`, `max.iter = 3e5` and `max.time = 60`. The labels-run-past-the-panel fix from earlier the same day (`coord_fixed(clip = "off")`) is gone; only the 12% the axes leave on each side remains. +- **`r-ggrepel=0.9.8`** in mds' `packages`, credited in `modules/mds/references.bib` as software with its version asked of R at run time, the way basicstats credits data.table. +- **The report's caption** for `mds.png` no longer says each point is labeled, since it may not be. +- **agree.R** runs mds with `labels = FALSE`: it compares tables, and it runs in association's suite, whose environment need not hold a package only mds declares. + +## Facts, measured + +- **conda-forge** has `r-ggrepel` 0.9.8 built for R 4.5 (`r45h3697838_0`), and 0.9.5 and 0.9.6 for older R. A dry-run install into `PoolSeqFlow-3.2.0-analysis` links that one package and changes nothing else: every dependency (ggplot2 >= 3.5.2, Rcpp, rlang >= 1.1.6, S7, scales >= 1.4.0, withr >= 3.0.2) is already in the baseline. +- **License GPL-3**, under which mds (GPL-3.0-or-later) is already published. +- **It is the first module package the analysis baseline does not carry.** Every package any module declared before this is in `install/environment-analysis.yml`, so the package manager had never installed anything for a user. `check-module-packages.sh` step 1 now does real work at a release. +- **Placement time grows with the square of the pool count**, and these names never settle early, so every run spends its whole iteration budget. Measured on this machine with `max.time` out of the way, 38-character names in four clusters, 3e5 iterations: 12 pools 2.0 s, 24 pools 6.4 s, 48 pools 34.7 s, 72 pools 66.8 s. For 24 pools: 1e4 iterations 0.7 s, 1e5 2.3 to 2.7 s, 3e5 6.6 to 9.5 s, 1e6 23 to 31 s. +- **The placement is byte-identical between runs when the iteration limit stops it** (two runs at each of five budgets gave the same md5) and differs when the time limit does, since a slower machine stops earlier. ggrepel's defaults (0.5 s, 1e4 iterations) are time-bound for a few dozen pools, so the seed alone does not make the plot reproducible. With 3e5 and 60 s, the time limit binds first at about 65 pools. +- **Quality**: at 1e5 iterations the 24-name plot still had a few names over each other, at 3e5 almost none in a full panel, and in mds.png's own panel, narrowed by two legends, a few collisions remain. The manual says so and points to `labels = false`. + +## What the cases check, and that they bite + +`the names can be left off the plot`: two runs with names draw identical PNGs, a run without differs, and the coordinates do not move. `a labels setting that is not true or false is refused`. Reverted in a scratch clone, each failed: the seed removed (two runs differ), the setting ignored, the type check removed. Without `ggrepel` on the library path, mds.R stops with its own message and publishes nothing; the frame case and every direct case need the package. + +## How a module suite gets a package the baseline lacks + +The mds cases first ran here with `R_LIBS` pointing at the package unpacked into the scratchpad from conda-forge's own archive. Z then installed `r-ggrepel` 0.9.8, the pinned build, into `PoolSeqFlow-3.2.0-analysis` the same day, so filtered runs have it. Three places need it, and as written only the first has it: + +- a filtered run uses the installed `PoolSeqFlow--analysis`, which holds it only once mds is installed from a catalogue that declares it, or it is installed by hand; +- a full run builds a scratch environment from `install/environment-analysis.yml` alone; +- `prep-version.sh` hands its freshly built pair to the full run. + +I proposed that the full run's scratch environment take every shipped module's packages after it is built, which is what a user has after `analysis modules install all`; `prep-version.sh`'s environment is the release's and the export refuses module packages in it. Z installed the package by hand, and deferred the full run until the shape of association's output was settled. + +**Settled when the full run was due: `ggrepel` is in the baseline.** Z: *"add ggrepel to the environment file. scratch builds from environment right?"* It does: the full run and `prep-version.sh` both build from `install/environment-analysis.yml`. The line `r-ggrepel=0.9.8=r45h3697838_0` was written into that file by hand, against its header's "do not edit by hand", because the documented route cannot take it. `export-environment.sh` refuses an environment holding a package a module declares that the baseline lacks, which is the guard against folding a module's packages in by accident, and this was the same act done on purpose. With the line in place, `export-environment.sh --check PoolSeqFlow-3.2.0-analysis` accepts, so a later export reproduces it. So every package any shipped module declares is in the baseline again, and the question of how a suite gets one that is not did not have to be answered. diff --git a/.claude/development-notes/module-reports.md b/.claude/development-notes/module-reports.md new file mode 100644 index 0000000..4223e86 --- /dev/null +++ b/.claude/development-notes/module-reports.md @@ -0,0 +1,142 @@ +# Module reports: what the PDF was, and what it became + +Written 2026-10-08 (UTC; the evening of 2026-10-07 in Z's time), against `dev` at `d8b7cee` plus the uncommitted work of that day. Not updated to follow the code. + +## The ask + +Z, after the first `Project/` run with `keepLowDepthAsZero`: *"Basic stats report should display the results under as tables but better organized. Also basic stats pdf has one table and it doesn't render right. the end of the first column renders on top of the beginning of second at times. Report should not be a list of what table is what. It should show the results and plots etc."* + +## What the report had been, since v3.0.0 + +The report and its wiring arrived together in `381541e` (2026-09-04, "F1 basicstats, and the PDF report every analysis carries") and shipped in every release from v3.0.0 to v3.2.0. Every `report.pdf` any of them published held a header and a table naming the declared files, and nothing else. Three causes, one of them the bug and two of them why nobody saw it. + +1. **The folder never reached the template.** `InstallResults` hands `reportShell` the destination as the shell text `$STAGE`. `reportSpec` put it into the JSON spec, and the spec was written with `printf '%s' ''`, single-quoted, so the template read the literal string `$STAGE` as its folder. Every `Sys.glob` matched nothing, and every section was skipped by the branch meant for "an optional output this run did not produce". Rendering the same template against a real folder by hand gave 10 sections where the published one had 0. +2. **The case asserted what the broken report printed anyway.** `test_a_published_folder_carries_one_pdf_of_everything_in_it` looked for `result.tsv` in the PDF text, and the contents table at the top printed every declared file name whatever happened below it. It passed on a report holding nothing. This is the gates-that-stop-checking pattern again: the assertion was aimed at the right word in the wrong place. +3. **A report that failed to build failed in silence.** The reason was printed to stderr, and Nextflow never shows a successful task's stderr; `InstallResults` succeeds by design when the report fails. Found while debugging this work: a typst syntax error in a header-only table left no trace on the console. + +The intended layout would not have satisfied the ask either. Rendered by hand with the folder fixed, it gave every heading a page of its own with the table pushed onto the next (a typst `#pagebreak()` before each section), ran long tables off the page with their last rows overprinted (pandoc wraps a pipe table in a `figure`, and a typst figure does not break across pages), overlapped columns, and printed 15-digit doubles. + +**The column overlap Z saw** is pandoc's: when a pipe table has a line longer than `--columns`, pandoc gives each column a fixed fraction of the width from the separator's dash counts, and typst cannot break `diversity.tsv` set in monospace, so it overflows into the next column. + +## The release read as "unknown" + +A module runs as its own Nextflow pipeline (`nextflow run //main.nf`), and Nextflow reads `nextflow.config` only beside the entry script, so `workflow.manifest.version` is null there. The README said "Produced by PoolSeqFlow unknown", the report said the same, and `CITATIONS.md` named PoolSeqFlow without a version (the `.bib` had no "Version X used" note for it). `releaseVersion()` in `paths.nf` now reads `version = 'X.Y.Z'` out of `${installDir()}/nextflow.config`, with the same regex `bump-version.sh` writes it with, and refuses when it finds none. `checkModuleCompatible()` still skips its environment check in a module's own run; `analysis.nf` makes that check first, so it was left alone. + +## What was decided + +**A module may lay out its own report.** The organization Z asked for is specific to what a module computes: basicstats' per-pool table combines `design.tsv`, `diversity.tsv` and `neff.tsv`, and its by-sequence tables pivot `depth.tsv`. A generic renderer cannot know that. So the manifest gains `report`, the name of an `.Rmd` in the module's directory, and the frame knits it as a child under the frame's own header. A module without one gets the generic layout: each declared table and figure under its file name, with no list of files (Z: *"Report should not be a list of what table is what"*). `readManifest()` refuses a `report` that is not a plain `.Rmd` name or is not there, the way it refuses a missing `main.nf`. + +**The frame keeps the header and the drawing functions.** `analysis/lib/rmd/report.R` holds `report_read`, `report_files`, `report_number`, `report_table`, `report_image`, `report_plot`, `report_palette` and the generic `report_declared`. A child calls them and reads only the published folder (`POOLSEQFLOW_REPORT_FOLDER`). The header does not come from the folder: the release, the frame, the design notes and the child's path are in the spec `reportShell` writes into the task directory, which is not published. Nothing rebuilds a report from a published folder. + +**Figures a report draws exist only in the PDF.** They are views of published tables. Publishing them would have made them declared outputs with manual anchors, and Z asked for the report to show them, not for more files. + +**basicstats' `frame` floor was not raised.** An older frame's `readManifest()` returns a fixed map and ignores keys it does not know, so `report` is invisible to it and the module still installs and runs. Z may want the floor raised so that a 3.2.0 installation never shows the old report; that is a choice, recorded here as not made. + +**Tables are typst code, not pandoc tables.** `report_table` emits a raw typst block: a `table` with `table.header` (repeated on every page), right-aligned numbers, light striping, cells as typst *string literals* so nothing a sample or sequence is called is read as markup, and a figure made breakable by a show rule. + +## What was measured + +All on the analysis environment's pandoc 3.12 and typst 0.15.1, with `pdftotext` to read pages back. + +- **pandoc's typst template** sets US letter, 1.25in margins (a 6in text block) and justified paragraphs. The frame's template sets 1in margins through YAML front matter (6.5in) and turns justification off; a justified line holding a long path spread its words across the page. +- **A table inside `#layout` still breaks across pages**: a 120-row table measured inside one ran over three pages. So typst itself chooses the size. My first attempt estimated widths in R from glyph widths; measured Libertinus Serif at 10pt is about 0.465em a digit, 0.468em a lowercase letter and 0.624em an uppercase one, bold about 10% wider. The R estimate ran roughly 10% high and turned a 24-pool table onto a landscape page it did not need. +- **Turned pages strand headings.** `page(flipped: true)` forces a page break, and the section heading stayed behind on the page before. Dropped. +- **Measuring a whole table unconstrained never wraps.** `measure()` of a table gives its natural width, so headers that could wrap read as too wide and the table was scaled to unreadable. The second rule tried, and the one the review took apart, was: the natural layout at 10 to 7 points; then the columns of numbers sharing what was left equally (`1fr`), measured with their headers stacked and displayed free to wrap; then that scaled down with `scale()`. Each of the three later steps failed on a shape it had not been tested on (below). +- **The rule that holds, measured cell by cell.** Every cell and every header is measured once at 10 points, and widths scale with the type, so each candidate size costs a sum rather than a layout. A column's least width is its widest cell or the widest part of its header (parts end after an underscore or a dot, or at a space), and its most is its widest cell or its whole header. The natural layout (every column at its most) is tried at 10, 9, 8 and 7 points; then explicit widths at the first size from 10 down to 5 at which the least widths fit, each column given its least and a share of what is left over in proportion to what it lacks of its most, capped at its most; then 4 points. Nothing is scaled, so every layout breaks across pages. Across eleven stress folders (the copy of the `Project/` basicstats folder, synthetic ones from 2 to 100 pools, and four the reviewers built and found failing), every overlapping word went to zero, and the 100-pool folder renders in 4 seconds where the whole-table measuring took 8. +- **Zero-width spaces in cells survive into copied text.** `pdftotext` returned `Y_mapped_Scaffold`. Pasted into `analysis.modules.basicstats.chromosomes`, that name would be refused with a list of names that look identical. Break opportunities are inserted in headers only. +- **A breakable figure orphans its caption.** Sweeping filler length so a table's top caption falls at a page foot: 2 of 9 layouts, and 2 of 13 in a second sweep, left the caption alone on one page and the rows on the next. `#show figure.caption.where(kind: table): it => block(sticky: true, it)` brought both to 0. Breakability is set for table figures only: an image figure that was breakable put its caption on the page after the image. +- **knitr's `knit()` defaults to `error = TRUE`**, which writes an R error into the document and carries on; a top-level `if ... else` split across lines in a chunk published a report reading `## Error`. The template sets `error = FALSE`, so a broken chunk stops the knit and the reason is printed. +- **knitr evaluates chunks in the input file's directory** unless `root.dir` is set, which would have written a report's figures into the installation's `analysis/lib/rmd/`. `reportShell` sets `root.dir` to the task directory. +- **`paste0()` recycles a zero-length argument to `""`**, so a header-only table emitted a stray `,` and typst refused the document. The writer fixture's `result.tsv` is one line, which `read.delim` reads as a header with no rows; that is the case that found it. + +## What the review found + +A review workflow (four reviewers on the frame, the table functions, basicstats' report and the documentation, each followed by a skeptic told to refute every finding) confirmed 31 findings, several of them the same defect found twice. All were fixed the same day. The ones that changed the design: + +- **The generic layout read and formatted every row of a table before showing forty.** Measured by the reviewers: 100,000 rows took 60 seconds in the publish step, a million about 13 to 18 minutes and up to 2 GB, to print 40 rows of `association.tsv`. `report_declared()` now reads `nrows = max_rows` and counts the rest by lines. +- **`report_number()` stopped on `Inf`** (`abs(Inf - round(Inf))` is `NaN`, and the subscripted assignment refused it), which association publishes as `t` for a separated allele, and its whole-number test, `abs(x - round(x)) < 1e-9`, printed every value below 1e-9 as 0, `design_floor` at 13 units included. It now prints `Inf`, `-Inf` and `NaN` as themselves and tests `x == round(x)`. +- **The generic layout reformatted identifiers.** A column whose every value parsed as a number was reprinted, so pool `01` read `1`, a year `2,024`, and a long scaffold number gained commas. Only a column holding decimals and no value with a leading zero is reformatted now. +- **The PDF depended on the task's locale.** Under `LANG` unset or `C`, knitr's capture wrote the report's own escapes as ``, `` and `` and an accented pool name as bytes; the build succeeded. The template now switches `LC_CTYPE` to the first UTF-8 locale it finds and refuses to build without one. +- **Equal shares for the columns of numbers still overlapped**: a design table with three experimental variables printed "Chromosomes" 3.6pt over "Ploidy", Z's own symptom. **The scaled layout was one unbreakable block**: 60 to 100 long-named pools ran past the page foot and smeared. **And typst's own auto columns squeeze a column below its unbreakable content** when a table is wider than the page, so long pool names ran into the next column. That last one I found myself, after switching to auto columns on a reviewer's suggestion, by running a word-overlap check (`pdftotext -bbox`) over the reviewers' own renders. Together these three are why the rule above measures cells. +- **Sequence order is the tables', not the reference's.** basicstats.R builds `chrom_levels` from the indel table first and the SNP table after, so a sequence with SNPs and no indels lands after the last indel-bearing one. The manual sentence said "the order your reference gives them" and now says "the order the tables give them"; basicstats.R's order was left as it is. The depth plots had been in file-name order (`chr10` before `chr2`) and are now in the tables' order. +- **basicstats.R stopped on a header-only indel table**, which is what a project with no surviving indel publishes, so such a project got no results at all. Pre-existing, outside the report work; one guard in `site_rows`. + +Smaller ones: figure 1 drew a merged pool's lower bound as an ordinary point, adjacent panels' tick labels collided (3 of 42 random projects), the grid's first rotated pool name was clipped, the report was quadratic in the number of sequences (2 minutes at 20,000), three figure captions named no file, and the cases had gaps: nothing failed on overlapping columns, on `error = TRUE`, on the design notes, or on the report's own arithmetic. One reviewer left two directories, `long__/` and `short__/`, in the repository root from a render with a relative path; they were removed. + +## The cases, and that they bite + +`15_analysis_results`: the generic report now asserts the file's own text (`analysis of Output`) and the release, a module's own report is laid out instead of the file list, the README and the `.bib` name the release, a failed report's reason reaches the console, and two R-only cases check `report_number` and `typst_string`. `09_analysis_modules`: a declared report that is missing, and one named by a path, are refused. `basicstats`: the report has its five sections, every pool, the experimental variables and a pi value as `diversity.tsv` holds it. `00_static`: every declared module report is in its directory, failing when no module declares one. + +Each fix was reverted in a scratch clone and the case aimed at it failed: the folder pointed at the task directory (the original bug's shape: files missed, sections skipped), "unknown" restored in the README, the citations and the report header separately, the reason back on stderr, and each of `readManifest()`'s two report checks removed. + +After the review, more: `report_number()` with Inf, NaN and tiny values; `report_format()` with zero-padded and four-digit identifiers; a cell drawn by `report_table()` itself rather than the helper it calls; the design notes; a wide-table case holding all three layout shapes that broke, checked with a word-overlap reader of the PDF (`pdf_overlapping_words` in `test/lib/analysis.sh`); a chunk that fails; basicstats' added-up row, lower-bound marks and sites read, read out of `report.md` against values computed by awk; and basicstats run on a header-only indel table. Each was proven the same way: the old whole-number test, formatting every numeric column, a cell written as markup, the notes chunk disabled, equal shares, a scaled block, typst's auto columns, an arithmetic mean for the added-up row, the lower-bound mark dropped, sites read without the unmeasured subtracted, and the empty-table guard removed. The wide-table case first passed the scaled and auto mutants, because its tables never reached those layouts; each table in it now is a shape that did. + +## Open + +- mds and association have no report of their own and get the generic layout. What each should show first is Z's to decide. +- A table running onto a new page can carry a single row there under a repeated header; typst has no orphan control for table rows. + +## mds and association, the same day + +Written later on 2026-10-08, against the same tree plus everything above. Z, on the basicstats report: *"This looks good. Let's do this for both association and mds too"*. That settles the first open item: both modules now declare `report.Rmd`, at version `20261008.001`, and neither `frame` floor was raised, for the reason given for basicstats. + +**What each shows first** follows the order the manual already asks a reader to take. association opens with the phenotype as it was read, because `phenotype.tsv` is where a reversed coding is caught; then what the design can show, both floors among them, so the bold sentence about a floor above 0.05 comes before any site; then the strongest sites per phenotype, `qq.png` and the Manhattan plots. mds opens with the plot and every pool's coordinates; then the eigenvalues as a table and as bars colored by sign; then the corrected distances as a pool-by-pool matrix with a heatmap (diverging at zero, since a negative distance is a result); then what each distance rests on: the shared sites as a matrix, the pairs on fewer than 30 sites, and every pair before and after the correction. A matrix holds at most eight pools as columns, split into tables of even size, and when every pair is on fewer than 30 sites that is one sentence rather than a table of all of them. + +**association's site table is read by `data.table::fread`**, for the eleven columns the report shows. Measured on a synthetic `association.tsv` of 1,020,000 rows: the report renders in 3 seconds. data.table is in the analysis baseline, and association's manifest now declares `r-data.table=1.18.6.1` as basicstats' does. + +**The allele named at each site.** The table shows, beside each site, the allele with the largest |t| and its slope. A biallelic site's reference row carries the negated slope of the other, so the two |t| are one number up to rounding, and which is larger is noise. In the corpus, pt_wingspan's chr1:700 has |t| 5.346774501780389599 for both A and T read back from the table, and the first rule, a stable order on |t|, named the first row, A at -0.0587; at chr10:1800 the two are 0.802679592830802 and 0.802679592830801, and that rule decided in the fifteenth digit. The rule now compares at six significant digits and, between alleles that tie, names the one whose slope is positive, the same reasoning as mds pinning each axis's sign. At a flagged site *t* is not comparable between implementations, so the steepest slope decides instead. Ties in `perm_p` go to the larger `S`, with every flagged site first in file order, since its `S` is Inf on one implementation and 1e16 on another. No case covers that last rule: the corpus has no two flagged sites at one p. + +**One defect found writing the manual section**: "Selected at FDR 0.05" labeled `selected`, which counts `fdr_p` at most 0.05 under whatever `fdr` names, Bonferroni included; the row is now labeled by the column. I also made the bold floor sentence per phenotype, believing a run could mix phenotypes at different floors and print a smaller floor beside "above 0.05". It cannot: `design_floor` is 1/units! for every fit of a run (`association.R`), and ties keep a site above the floor without moving it. The review below found that, and the sentence went back to one for the run. + +**The count of sites at the floor** compares `perm_p` with 1.5 times the run's `floor`. A permutation p is a whole number of rearrangements over the count, the observed one always among them, so it is a whole multiple of the floor and the next value above the floor is twice it; the midpoint survives any printing. A relative tolerance (1e-9, then 1e-6) depended on how many digits the two were printed with. association.R prints both to 15 and they agree, but the synthetic three-unit folder printed p to six digits and the report counted 0 of its sites at the floor while every site it showed sat there. + +**Found on the way, outside the reports:** + +- **`report_table`'s `total = nrow(table)` was evaluated lazily**, at its first use, after the table had been cut to `max_rows`, so a cut table never said it was cut. mds' table of pairs on few sites relied on the default. `force(total)` at the top of the function. +- **`mds.png` clipped the labels of the outermost pools** at the panel edge; each label is drawn above its point. The first fix let the labels run past the panel; it was replaced the same day by `ggrepel` (below), and only the 12% the axes leave on each side remains of it. +- **association stopped on a depth table of one site.** The weights are built by `vapply`, which returns a bare vector for one row, so `colnames<-` refused it, and a run whose indel table held one indel published nothing, SNPs included. A review found it on 2026-10-05 and it was reported and left (`depth-masking-proposal.md`, `missing-cell-validation.md`); a synthetic three-unit cohort built for the report reached it. The weights are now built as a matrix. `dev/validation/lib.R` had already guarded its own copy of the same `vapply`. + +**Renders, all with 0 overlapping words** by `pdf_overlapping_words`: the corpus through each module (mds 4 pages, association 3), the copy of the `Project/` mds folder, 24 pools with long names (mds, 10 pages), three units (association), and the 1,020,000-row table. + +**The cases, and that they bite.** `mds`: every row of the distance matrix, read out of `report.md` against `distance.tsv` in `mds.tsv`'s pool order, assembled by awk; the frame case checks the four sections and overlaps. `association`: the design's floor in every column, the count of sites at the floor per phenotype, five site rows built by awk from the two tables (two flagged, the tie at chr1:700, a triallelic site), the level coded 1, and the floor sentence after raising one phenotype's floor to 1/6, a state no run produces, dropped after the review; a depth table of one site checked against the corpus oracle; the frame case checks the four sections, the design's floor and overlaps. `15_analysis_results`: a table cut short says by how much. Each was reverted in a scratch clone and its case failed: the first-row allele rule, the floor counted strictly below, a flagged `S` printed as a number, the coded level dropped, the `vapply` restored, the matrix's lower half left unfilled, the matrix at four digits, `force(total)` removed, and the floor sentence taken from the run's minimum. The first mds case checked only the first row, which the lower-half mutation cannot reach; it now checks every row. + +**Settled by Z the same day:** association's `chromosomes` restricts every fit to the named sequences (`association.R`, the depth table filtered before parsing), while the manual's settings table said it decides "which sequences get a plot". Z: *"User should be able to pick chromosomes correct. Otherwise with scaffolds etc it is messy."* The code was right and the manual wrong: the settings row, the Manhattan section and the site table's "always complete" now say the fit is restricted and what that does to `fdr_p` and the dispersion, and `test_naming_sequences_restricts_the_fit_to_them` pins it (removing the filter fails it). Before ruling, Z asked how the two collide; measured on the corpus SNP table, pt_wingspan, dispersion estimated: with no sequence named, 14 sites read, 13 tested, dispersion 0.206, lambda_gc 2.28, 2 selected; with `chr10` named to draw its Manhattan plot, 6 read, 5 tested, dispersion 0.267, lambda_gc 2.02, 1 selected, and the eight sites off chr10 gone from `association.tsv`. On chr10 itself every number moved with the dispersion and the smaller FDR family: chr10:500 S 0.504 to 0.513, perm_p 0.593 to 0.589, fdr_p 0.652 to 0.603; chr10:1700 fdr_p 0.00903 to 0.00694. + +## The review of these two reports + +The same evening, against the tree above. Four reviewers (association, mds, the cases, the documentation), each followed by a skeptic told to refute every finding; all eight on Sonnet, read-only on the repository, rendering in the scratchpad. 44 findings, 34 confirmed, several of them one defect found twice. Four were already fixed by then, from looking at the example reports: the rounding-zero eigenvalue, "SNP sites" with indels in, and two copies of each. + +**What changed in the reports**, each confirmed by a reproduction: + +- **An ordinal phenotype's slope read "more frequent in the units coded mid"** for low < mid < high. The frame hands an ordinal phenotype its level's index, so the slope is per step along the order and `coded_one`, the second level, is no direction at all. The caption now says "further along low < mid < high", the phenotype table says each level is read as its place in the order, and the manual's two sentences were corrected; the one under `association_alleles.tsv` predates this work. +- **The floor sentence went back to one for the run**, as recorded above. +- **NA in the allele columns is explained in the caption.** At three units the allele table's selection and the report's ranking disagree, so most of the twenty rows can be NA (below). +- **One tested site** reads "The one tested site of", not "All 1 tested sites"; **an allele past 20 bases** is shown by its first 12 and its length, where one of about 140 pushed the table off the page. +- **Names reach markdown escaped** (`report_text()`): a sequence called `@x` stopped the PDF build, and `*` or a leading `_` rewrote a caption. +- **A bracket in the project's own path dropped every figure** with nothing said: the report builds in a folder beside the results, so the path is part of every glob. Files are listed by a pattern made from the glob, and a figure's path is in angle brackets, where a parenthesis ended it. +- **A quote in a cell** opened a quoted string that swallowed the rows after it: tables are read with `quote = ""`, as they are written. +- **A figure taller than the page was clipped**: `qq.png` at four phenotypes is 6 by 11 inches. A PNG at full width is now kept to 7.5 inches tall, read from its header. +- **The column split** of both reports said "even" and gave 7, 7, 7 and 4 for 25 pools; `report_columns()` gives 7, 6, 6 and 6, and basicstats uses it too. +- **`plotted` in eigenvalues.tsv marked every axis written**, so at `dimensions = 3` the report said the axes drawn carry 95.0% above a plot whose own labels add to 84.2%. Fixed at the source in `mds.R`, which the manual already described: `plotted` is the two axes `mds.png` draws, and none at one axis. +- **The heatmap's tiles collapsed** under pool names of 50 characters or more; the figure grows by the longest name. +- **Comments**: "takes the first pool's name" where the code takes the longest, in both reports. + +**What changed in the cases.** The association case now checks every row of the design table and the phenotype table, and the order of each site table against an oracle that sorts association.tsv by the caption's rule. It also checks seven site rows through the leverage, two of them chosen because their |t| differ in the fifteenth digit, plus a planted flagged triallelic site whose largest |t| and steepest slope name different alleles. A new case takes the paths a real run takes: 26 tested sites cut to 20, a sampled run whose floor parts from the design's, an allele table of three sites, and three Manhattan plots in the tables' order. Another reads the floor sentence off a real three-unit run and an ordinal phenotype's wording. The mds cases read back the opening sentence, every coordinate and eigenvalue row, the figures placed, a 25-pool split, and `plotted` at one and three axes. Suite 15 adds a declared table counted and not read whole, and a path holding brackets and parentheses, a quoted cell, an escaped name and a tall figure. The three module suites now declare `analysis/lib/rmd/` in `# covers:`, which `--changed` had skipped. The awk oracles print numbers through `REPORT_NUMBER_AWK` in `test/lib/analysis.sh`, which follows `report_number()` where `%#.3g` did not (1234.5 is `1,230`, not `1.23e+03`). + +**And that they bite**: 24 mutants, one per new assertion, each made in a scratch clone, and every case named failed: the order reversed, the cut at 15, sites read and tested swapped, the two floors swapped, leverage at five digits, |t| at full precision, |t| at a flagged site, NA unexplained, Manhattan plots in file-name order, names unescaped, the ordinal read as coded 1, the floor sentence never printed, one site as "all 1", `plotted` marking every axis, columns cut eight at a time, the opening sentence on the wrong axes, a declared table read quoted, the header counted as a row, an uncut table counting its rows, files globbed through the path, a table read quoted, a tall figure at full width, a path outside angle brackets, and columns 8pt short of their text. + +**`pdftotext` fuses words overprinted by up to 5pt**, so `pdf_overlapping_words`, which compares word boxes, cannot see them; the review measured it and I reproduced it with `Chromosomes#h(-Xpt)Ploidy` at 10pt. At 1, 2, 3.5 and 5pt the PDF reads `ChromosomesPloidy` and the box check is empty; at 6pt the words split and the box check names them. `pdf_missing_words` names a word that does not stand alone, which covers the first regime, and the frame cases and the wide-table case now use both. Squeezing `report_table`'s columns could not produce the fused state: 4 and 6pt short of their text is absorbed by the 5pt insets, and 7pt short is a box overlap. The helper also prints a line for a PDF with no words, which a missing one is. + +**Refuted**, ten: among them that the pairs table's "whole table is in distance.tsv" was untrue, that an all-negative ordination printed a negative share, that the README omits `report_folder()`, and that the note's 19-digit |t| could not have been read back from a 15-digit table (R reads the 15 digits and prints the double it parsed). + +**Open, and Z's: the allele table's selection.** `association.R` keeps the sites at `perm_p` at most `reportBelow` and the first `reportTop` of all phenotypes and kinds together by `perm_p`, ties in file order. With two phenotypes the first can take every slot and the second get none, and at three units, where every site ties at the floor, the report's top twenty by S are mostly not among them. Ranking per phenotype with ties broken by S would make the two agree; that changes what `association_alleles.tsv` holds, so it was reported and not changed. + +## One file per phenotype + +Z, on that question: *"Isn't the association data like GWAS, a per site value for a given phenotype? It can be a matrix no? sites as rows, and phenotypes as columns?"* I laid out three shapes: one wide table with a column group per phenotype, the long table with each site's strongest allele and slope added, and one file per phenotype. Z: *"OK. Let's do one file per phenotype. That sounds better. Joining files for comparing sounds good."* + +**What association publishes now**: `association_.tsv` and `alleles_.tsv` for each phenotype fitted, the stem its name with every character outside `[A-Za-z0-9._-]` made `_`, as the Manhattan plots already name their sequences. Neither carries a phenotype column. `alleles_` and not `association_alleles_`, because a declared output is a glob and `association_*.tsv` would match both. Two phenotypes whose stems are equal are refused before the fit, naming both; a `pt_` name has no character rules beyond its prefix (`bin/parse_metadata.py`), so `pt_wing span` and `pt_wing_span` can both exist. A phenotype with no site read still gets both files, as a header. `permutations.tsv`, `phenotype.tsv`, `qq.png` and the Manhattan plots are unchanged; the figures keep a panel per phenotype. + +**The selection is now each phenotype's own**: the sites at `perm_p` of `reportBelow` or under, and the first `reportTop` by `perm_p`, ties to the larger S with every flagged site first, the order the report ranks them in. So with `reportTop` at 20 or more, every site the report shows has its alleles. Measured on the corpus as three units of two, where most sites tie at 1/6: for pt_resistant the tied site first in file order is chr2:400, and the one with the largest S is chr10:500. The published `association.R` names the selection in its header, which the comment beside it had claimed and main.nf had never written. + +**What read the old names**: the report (captions name each phenotype's files), `dev/validation/agree.R` (`association_pt_y.tsv`), the manifest's two outputs, the manual's two sections, its settings row and its report section, and 48 places in the association suite, renamed by one announced sed with the multi-phenotype helpers and the planted-table edits done by hand; one `awk` there read `kind` as the second column. New cases: each phenotype in files of its own, without the column, its allele table holding its own first site in the report's order; and two names under one file name refused. Reverted in a scratch clone, each failed: ties in file order, the column kept, the clash let through. The Nextflow case asserts both files by name, and the static suite is unchanged at 66. diff --git a/.claude/development-notes/output-prefix.md b/.claude/development-notes/output-prefix.md new file mode 100644 index 0000000..73d19a8 --- /dev/null +++ b/.claude/development-notes/output-prefix.md @@ -0,0 +1,61 @@ +# outputPrefix: one name at the start of every output, and dated report names + +Written 2026-10-08, against `dev` at `1b04cb1` plus that day's uncommitted work (`module-reports.md`, `mds-labels.md`). Not updated to follow the code. + +## The ask + +Z: *"At the top we will add params.outputPrefix = "Test". We will make vcf.filename = params.outputPrefix. Then all the other output files freq depth etc will be correctly named with the prefix. We will add output prefix in front of all analysis outputs. Also the reports coming from analysis should be named [prefix]_[analysis]_report_[datetime as in 20261007-103602].pdf eg. Test_mds_report_20261007-103602.pdf"* + +I proposed renaming `vcf.fileName` to `outputPrefix` outright, one setting so the VCF and the analysis files could not disagree. Z ruled against it: *"If someone wants to change VCF name they still should be able to. vcf.fileName should stay in the template."* Z then wrote the template line as `fileName = "${params.outputPrefix}"`, the interpolated form the template's other derived lines use, *"Easier for the end user to change."* My other three points Z accepted: the analysis layer, not each module, applies the prefix (*"is fine as long as we get the end result right"*), and upgrading projects rerunning is *"not a problem"*. + +## What was built + +**The pipeline side.** `outputPrefix` is the first line of `params`. `vcf.fileName` follows it in the template, and `nextflow.config` fills either for a config that lacks it: `outputPrefix` from `vcf.fileName` and then `'Test'`, `vcf.fileName` from `outputPrefix`. A run table's `outputPrefix` reaches that run's VCF name through `deriveRunPaths()`, which re-derives it unless the global `vcf.fileName` differs from the global `outputPrefix`, which is the same pin test the other derived values use. `vcf.fileName` joined `derivedParameterNames()`. `outputPrefix` joined step 7's artifact identity, so one results directory never holds runs of two prefixes and every analysis target has exactly one; `targetPrefix()` in `plan.nf` throws if that ever fails. + +**The check.** `check_parameters.sh` fails an empty `outputPrefix` or `vcf.fileName` and one holding anything outside `[A-Za-z0-9._-]`. It judges only a key the input carries. Step 0 runs that check over the global parameters only, so a run table's `outputPrefix` is not seen there; `targetPrefix()` applies the same rule per results directory, which is where a run-table value would first become a file name in the analysis layer. + +**The migration.** `renamed()` maps `outputPrefix` to the old `vcf.fileName`, so an upgrading project's prefix is its existing VCF name and nothing of its gets renamed; `vcf.fileName` then takes the template's derived line, which yields the same name. A note explains it. `pinned()` keeps a literal `vcf.fileName` when the old config already has `outputPrefix`. Without that, the migration's rule that a template-computed line beats the user's literal would undo a deliberately named VCF at every later migration, which is exactly what Z's ruling says must stay possible. + +**The analysis side.** `InstallResults` renames every top-level file the module produced to `_` right after copying it into the stage, before the README, the citations and the report are written. Code keeps its name: the script extensions the script check already uses, plus `.cpp`, `.c`, `.h`, `.hpp`. So do the frame's own files, `README.md`, `CITATIONS.md`, `references.bib` and the verification record, because they are written after the rename. Manifests keep the names a module writes; `publishedName()` gives the README and the declared-output check the published ones. + +**The report name** is `__report_.pdf`, the time taken once in the `InstallResults` script block from the local clock, so the README and the PDF carry the same stamp. The middle part is the module name, not `folderName`. Both were my calls, stated to Z with the plan. + +**Inside the report**, `report_read()` and `report_files()` take the name a module wrote and read the prefixed file; `report_name()` gives a caption the published name, and `report_logical()` takes the prefix off a name a report parses a sequence out of (basicstats' depth plots, association's Manhattan plots). `POOLSEQFLOW_REPORT_PREFIX` carries the prefix and **an unset one stops the report**. Every report from v3.0.0 to v3.2.0 was blank because the folder reached the template as a literal that matched nothing (`module-reports.md`); a prefix that silently defaulted to none would do the same thing again, every section looking for a name the folder does not hold. + +## Found on the way + +**The manual said changing `vcf.fileName` after a run re-runs the VCF branch beside the old files. Step 0 refuses instead.** `analysisParams()` excludes paths, resources and tool locations and nothing else, so `vcf.fileName` has been in the recorded manifest all along and any change to it fails the parameter guard. The new naming section says what happens. The same now holds for `outputPrefix`. + +**Step 0's line after a FAIL finding said the setting "would make this run produce nothing"**, which a name that cannot start a file name does not do; it stops the run partway. Widened, and the manual's quote of it and its level table with it. `fastqc.memory` carrying a unit was already that kind of FAIL. + +## Tests + +`analysis_render_report` stages the folder as `InstallResults` publishes it, as links named under `Test`, so every report case outside Nextflow reads prefixed files. Its copy of the code rule is a second definition of `InstallResults`'; the cases that publish through Nextflow are what hold the two together. + +A third analysis baseline, `prefixed`, sets `outputPrefix = 'Pfx'` and `vcf.fileName = 'Calls'`. Everywhere else both are `Test`, so a frame that took the VCF name would pass every other case. The case checks the config took both substitutions before trusting what it publishes. + +The report-name case brackets the run with the clock on either side and requires the stamp between them, which catches a fixed string, a date alone, or UTC on a machine that is not on it. + +Module versions were not moved. Z: *"You don't need to bump versions."* Step 4 of the release names what is behind. + +Fifteen mutants of the guards above, each killed: eleven without a JVM (the check's empty and absent cases, the migration's rename and pin, `report_read`, `report_files`, `report_logical` and the unset prefix, an mds caption, both reports' plot-name parsing) and four through Nextflow (the prefix taken from the VCF name, a date-only stamp, a script given the prefix, a run's prefix kept from its VCF). + +## The review, the same day + +Three read-only reviewers went over the change before Z had seen it. What they found, and what changed. **Where this contradicts a section above, this is the current state**; the sections above say what was built first. + +**A module never got the default.** A module is its own pipeline and reads no `nextflow.config`, which `shell-and-nextflow-gotchas.md` records and `analysis/frame.config` says in its header; the frame config carried `params.cores` and nothing for `outputPrefix`. A config without the key, which a project that skipped `migrate_config` has, published every file as `null_*`: `"${run.outputPrefix}"` makes a missing key the string `null`, and the name rule accepted it. The verification, run beside `nextflow.config`, saw the right prefix, so nothing looked wrong. The three default lines are now in `frame.config` too, and a fourth analysis baseline, `legacy`, runs a module on such a config. + +**The name rule was too loose in three ways.** A leading `.` passed, and `list.files()` does not list hidden files, so a report lost every figure while the folder was complete. A leading `-` reads as an option. And `null` passed, which is also what the template's own `fileName = "${params.outputPrefix}"` reads as when someone deletes or comments out the `outputPrefix` line, so such a config ran as `null.vcf`. A name now starts with a letter or a digit and `null` is refused with that cause named, in `check_parameters.sh`, `targetPrefix()` and the run-table parser alike. + +**A run table's cells were never checked.** Step 0 runs `check_parameters.sh` over the global parameters only. Above I wrote that `targetPrefix()` is where a run-table value would first become a file name; that was wrong, because `deriveRunPaths()` makes a run's prefix its VCF name, and step 6 passes that name to `bcftools` unquoted, so a cell holding a space failed at variant calling after trimming and alignment. `parse_multirun.py` now holds `outputPrefix` and `vcf.fileName` cells to the rule, beside its RunID check, before anything runs. + +**The migration still renamed a VCF silently in two shapes.** A current config whose `vcf.fileName` is an expression of its own, `"${params.outputPrefix}_v2"`, was given the template's line with no report entry; `pinned()` only kept literals. And a config from before the prefix whose name was an expression lost it the same way, with no note, because the rename into `outputPrefix` only carries a literal and nothing else reported it. Now a config that has `outputPrefix` keeps whatever `vcf.fileName` it wrote; a config without it keeps an expression where it is and gets the template's prefix, with a note saying so; and only a literal name moves into `outputPrefix`. + +**The empty-name finding had an empty middle field**, and both callers read findings with `IFS= read`, where two tabs are one, so the explanation landed in the detail column. It carries `no name` now, and a case reads a finding the way the callers do. + +**Smaller.** The pin test compares as text, since an unquoted prefix is a number and a GString is not equal to one. Three `complete` cases still asserted `result.tsv` in a published folder; I had run selected cases of the suite and not all of it. The manual now says that varying `outputPrefix` in a run table varies the VCF name and so repeats variant calling, and that a fixed `fileName` keeps it shared; it no longer names step 0 in the new prose; it widens "produce nothing" wherever it describes the check; and it lists `vcf.fileName` among the derived values. + +**Left as it is.** The rename covers what sits at the top of the folder, which is all any module produces; the contract text now says a directory's contents keep their names rather than the frame refusing a directory. `mds.png` still says `few_sites in distance.tsv` in its own caption, which is module output that does not know the prefix. + +Nine more mutants, of these fixes, each killed: six without a JVM (`null` and a leading dot accepted, the empty detail field, the run-table cells unchecked, the pin back to literals only, an expression moved into the prefix) and three through Nextflow (no default in `frame.config`, the prefix out of step 7's identity, the pin compared as values). A number written as the prefix cannot be varied by a run-table cell holding text: a cell takes the type of the config's value, and that conversion fails for every parameter alike. diff --git a/.claude/development-notes/site-follows-releases.md b/.claude/development-notes/site-follows-releases.md new file mode 100644 index 0000000..ce88b4a --- /dev/null +++ b/.claude/development-notes/site-follows-releases.md @@ -0,0 +1,51 @@ +# The site follows releases + +**Written 2026-10-07 against `dev` at `d8b7cee` plus uncommitted work**: `.github/workflows/docs.yml`, its 00_static case `test_the_site_deploys_only_for_a_released_version`, and the lines in `dev/RELEASING.md`, `modules/README.md` and `dev/scripts/publish-module.sh` that described the old wiring. + +## What happened + +`docs.yml` deployed the site on any push to `main` that touched the manual, the CHANGELOG, the catalogue or the build. A release pushes `main` before it tags: the step-6 merge and the step-9 commit. On 2026-10-07 the merge `84def44` and the version-bump commit `babb475` each deployed, so the site carried the 3.3.0 manual and a 3.3.0 CHANGELOG entry. The v3.3.0 tag then failed `release.yml` at its analysis version gate, and no release was made. The site went on describing a release that did not exist, with v3.2.0 still the latest. + +Z, the same day: "If a release fails website should not be published." + +## What was weighed + +The site holds one manual, and the module pages (basicstats, association, mds, the modules index) are sections of it, deployed together with the catalogue. Modules are meant to publish between releases, page and catalogue row together. Three shapes were put to Z: + +- pipeline pages and the CHANGELOG from the latest successful release, module pages and the catalogue from `main`, which needs `build_docs.py` to assemble one site from two commits; +- the whole site from the latest successful release, with only the catalogue moving between releases, so a module published between releases has no page until the next release; +- keep deploying from `main`, and hold every deploy while a release is in flight, which needs "a release is in flight" read off `main`'s history. + +Z: "Isn't it a single manual anyway?" It is, and that settles it: the site is built from one commit, and the only question is when it deploys. + +## What was built + +`docs.yml` deploys in two cases. After `release.yml` completes, through `workflow_run`, when that run was a pushed tag and succeeded, building the commit the release was made from. And when run by hand, which is how a module published between releases reaches the site. In both cases a step first asks GitHub whether the version the commit declares (`VERSION=` in `./PoolSeqFlow`) has a published, non-draft release, and refuses otherwise. A push to `main` and a pull request still build with `--strict`, so a broken page fails early, and deploy nothing. A `release.yml` run by hand checks a commit without publishing it, so it does not deploy the site either. + +`workflow_run` runs from the default branch's copy of the workflow and checks out the released commit itself. A deploy job inside `release.yml` would run on the tag instead, and a `github-pages` environment limited to `main`, which is how Pages environments are commonly set up, would refuse it. That limit was not checked against this repository's settings; `workflow_run` avoids depending on it. + +## What it costs + +**A module published between releases needs the Documentation workflow run by hand** once its commit is on `main`. Before, the push deployed it. + +**The hand run checks only the declared version.** Between the step-6 merge and the step-7 bump, `main` holds the next release's manual while still declaring the released version, and a hand run then would deploy it. Nothing in the release procedure runs the workflow by hand, and no automatic path deploys in that window. + +## How it is checked + +`test_the_site_deploys_only_for_a_released_version` reads the workflow and requires every path to the deploy to go through the release check: the deploy job's condition, the build's output, the release check's events, the Pages steps' conditions, the `workflow_run` gate on a successful pushed tag, and `workflow_run` naming `release.yml` by its `name:`. It then runs the release check's own script against a stand-in `gh` answering published, draft and missing. Four mutants of the workflow were each caught when it was built: deploy on anything but a pull request, the pushed-tag condition dropped, the Pages steps ungated, and `workflow_run` following another name. + +## The case needed PyYAML, and the 3.3.0 prep run found it (2026-10-08) + +**`prep-version.sh` stopped at step 2 on this case alone: 710 passed, 1 failed, `ModuleNotFoundError: No module named 'yaml'`.** The case read both workflows with `yaml.safe_load`. `00_static` declares no environment, so it runs on whichever `python3` the launching shell finds, and Z's shell had `PoolSeqFlow-3.2.0-analysis` active (the run's `system-before.txt` records `CONDA_DEFAULT_ENV`). Neither PoolSeqFlow environment carries PyYAML. Every run before it, mine included, had used `/usr/bin/python3`, which has PyYAML 6.0.3 for reasons of its own, so the dependency was never declared and never seen. Nothing else in the suite imports outside the standard library: checked by parsing every Python file in `bin/`, `test/tools/` and `dev/scripts/` and every Python heredoc and `-c` body in the suites, a scan that does report `yaml` in the committed case. + +**The fix reads the workflows with `test/tools/workflow_yaml.py`, stdlib only, and refuses what it does not read.** It takes the part of YAML the three workflow files use and exits naming the line for anything else, rather than adding PyYAML to an environment that ships for one test case. Checked against PyYAML on all three workflow files and on the reader case's own sample, with the same value everywhere after reading every scalar as text, which is the one difference by design: PyYAML types `on` as true and `false` as a boolean. The reader has a case of its own, and five mutants were each caught: the deploy job's condition and the release check's event condition removed from `docs.yml`, and in the reader the folding of a more-indented line, strip chomping, and the refusal of an anchor. + +**`00_static` also gained the case that would have caught it first**: `test_every_python_the_suite_runs_imports_the_standard_library_alone`, which reads every Python file the suite runs and every Python snippet a suite hands `python3`, and refuses an import outside the standard library and the repository's own scripts. `dev/RELEASING.md` asks for exactly that when a gate turns out wrong. + +## The reader's own review, the same day + +Two read-only reviewers went over the reader before Z had seen it, one comparing it with PyYAML over generated and damaged documents, one reading the code and the two cases. **The first version misread valid YAML in five ways**: plain items of a flow sequence skipped every check a block value gets, so `paths: [*.md]` and `x: [a: b]` read as text; folding lost the line break beside an empty line; spaces past a block's indentation on a white-space-only line were dropped; a block ending a file with no final line break gained one; and characters outside ASCII were read, among them the line separators PyYAML breaks a line at. A file that was not UTF-8 crashed with a traceback. **The cases claimed more than they checked**: the reader's case passed with 42 of the reader's rules mutated away, since its refusals asserted only that something was refused somewhere; the site case passed a gate whose `&&` was made `||`, a second job deploying Pages ungated, the release looked up without its `v`, and draft read from another field; and the import scan missed `python3 -c "..."`. + +The reader now refuses every character outside printable ASCII but the tab, which removes the whole class of invisible characters rather than reading it, as the repository's own files are ASCII. Folding follows YAML's rule. The other three misreads are fixed, and a bad encoding or nesting deeper than Python recurses is refused by name. Each refusal in the case now names its line and its reason, the sample carries every construct the reviewers found uncovered, and the site case reads every job, compares the gate whole, and records what `gh` was asked. + +Measured against PyYAML afterwards: 80,000 documents, generated, damaged, and damaged in ASCII alone, with no misread and no crash. Every valid document it refused, 1,987 of 5,000 generated, was refused for a character outside ASCII. What it reads that PyYAML rejects is damaged input: mostly a tab where PyYAML's scanner refuses one, and flow items holding `?` or a stray bracket. That is the reader being more lenient than the library, never reading a valid document differently. diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index d8e02c6..68538a2 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -1,7 +1,7 @@ name: Documentation -# Builds the MkDocs site and publishes it straight from main. No gh-pages branch: the -# site is uploaded as an artifact and deployed by actions/deploy-pages. +# Builds the MkDocs site, and deploys it only for a version that has been released. No +# gh-pages branch: the site is uploaded as an artifact and deployed by actions/deploy-pages. # # One-time repository setting this depends on: # Settings -> Pages -> Build and deployment -> Source: "GitHub Actions" @@ -9,15 +9,17 @@ name: Documentation # It cannot be "Deploy from a branch -> main -> /docs", which serves the folder as-is; # docs/ holds Markdown, not finished HTML. # -# Pull requests build with --strict but do not deploy. +# It deploys in two cases: after release.yml publishes a release, built from the commit that +# release was made from, and when run by hand on main (Actions -> Documentation -> Run +# workflow), which is how a module published between releases reaches the site. Both refuse a +# commit whose declared version has no published release. A push to main and a pull request +# build with --strict and deploy nothing. on: push: branches: [main] paths: - "manual/**" - # The module repository is deployed with the site, so publishing a module is a change - # here. Without this line the catalogue would advertise a tarball no deploy had carried. - "modules/repo/**" - "mkdocs.yml" - "CHANGELOG.md" @@ -31,6 +33,11 @@ on: - "CHANGELOG.md" - "dev/scripts/build_docs.py" - ".github/workflows/docs.yml" + # Runs from main's copy of this file whenever release.yml finishes; the build job decides + # whether that run published anything. + workflow_run: + workflows: ["Publish a release"] + types: [completed] workflow_dispatch: permissions: @@ -45,9 +52,36 @@ concurrency: jobs: build: + # After release.yml, only a run that published: a pushed tag with every step green. Run by + # hand, release.yml checks a commit and publishes nothing. + if: >- + github.event_name != 'workflow_run' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.event == 'push') runs-on: ubuntu-latest + outputs: + deploy: ${{ steps.released.outputs.deploy }} steps: - uses: actions/checkout@v7 + with: + ref: ${{ github.event.workflow_run.head_sha || github.sha }} + + # The version this commit declares has to have a published release before its pages do. + - name: Check this version is released + id: released + if: github.event_name == 'workflow_run' || github.event_name == 'workflow_dispatch' + env: + GH_TOKEN: ${{ github.token }} + run: | + declared=$(sed -n 's/^VERSION="\(.*\)"$/\1/p' PoolSeqFlow) + draft=$(gh release view "v$declared" --repo "$GITHUB_REPOSITORY" \ + --json isDraft --jq '.isDraft' 2>/dev/null || true) + if [ "$draft" != "false" ]; then + echo "This commit declares $declared, and v$declared has no published release," >&2 + echo "so its pages are not deployed. The site deploys once release.yml publishes it." >&2 + exit 1 + fi + echo "deploy=true" >> "$GITHUB_OUTPUT" - uses: actions/setup-python@v5 with: @@ -69,15 +103,15 @@ jobs: # Fails early and clearly if Pages is not enabled. - uses: actions/configure-pages@v6 - if: github.event_name != 'pull_request' + if: steps.released.outputs.deploy == 'true' - uses: actions/upload-pages-artifact@v5 - if: github.event_name != 'pull_request' + if: steps.released.outputs.deploy == 'true' with: path: site deploy: - if: github.event_name != 'pull_request' + if: needs.build.outputs.deploy == 'true' needs: build runs-on: ubuntu-latest environment: diff --git a/CHANGELOG.md b/CHANGELOG.md index 0407370..b2c67c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,70 +10,6 @@ In practice a version number says what upgrading will cost you: a third number i --- -## [3.3.0] - 2026-10-06 - -**The shallowest library no longer has to decide which sites the whole cohort keeps.** Until now one pool below `vcffilter.minDP` removed a site for every pool. `vcffilter.keepLowDepthAsZero` keeps the site instead and writes that pool's reads there as unread. It is off by default, so nothing changes until you turn it on. - -**Two things to read before upgrading**, both under Changed: the default of `variantCall.scaleMapQ` moved from 50 to 100, and a migrated `parameters.config` keeps its 50; and `init_multi` and `uninstall_all` are now two words. - -### Added - -- **`vcffilter.keepLowDepthAsZero` and `vcffilter.minSamples`.** With the switch on, every cell below `minDP` (one pool's reads at one site) is written as unread, which the depth table carries as zeros and the frequency table as `NA`, and a site is kept when at least `minSamples` of its cells reach `minDP`. `minSamples` defaults to 2 and runs from 1 to the number of pools with reads; anything else is refused before the run starts. Every read still counts toward deciding that an allele exists, because the false-positive filter runs first; only the cells at the floor count toward measuring its frequency. The manual's depth and quality filter section has the details, including what it does to `TOTAL_AD`. -- **Settings that would make a run produce nothing are refused at step 0**, and `PoolSeqFlow check project` reports the same findings before you run. A `scaleMapQ` below `varQualMin` discards every read, a `sampleThreshold` above 1 removes every site, a `ploidy` or `poolSize` below 1 breaks every detection limit, and a `fastqc.memory` with a unit is refused by FastQC. Settings that work but cost more than they look, or change what a number means, are warned about and the run continues. The manual lists them all. -- **Files saved on Windows.** A `metadata.csv` or `runs.csv` that Excel saved as "CSV UTF-8" is read as it is, and a UTF-16 one is refused with what to save it as instead. A `parameters.config` that begins with a byte-order mark is named by `check project`, with the command that removes it; Nextflow on its own refuses the file over a character nothing displays. -- **Native zsh completion**, with a description beside each command. The installer prints the one line `~/.zshrc` needs. - -### Changed - -- **`variantCall.scaleMapQ` defaults to 100, where it was 50.** It is the `-C` that lowers the mapping quality of reads with many mismatches. Measured on real pools, 100 removes the sites that are artifacts (Ti/Tv 0.848, nearly four times the soft-clip bias of the sites kept), while 50 removed five times as many and took real variants with them. 50 also lowered the frequencies it kept, by 0.035 on average against 0.008 at 100, and by 0.081 in pools between 0.50 and 0.75. **A migrated `parameters.config` keeps 50**: `migrate_config` carries your value across, as it does every setting that still exists, and reports it under `Kept your value`. Set it to 100 before the first run under 3.3.0. The manual has the full measurements. -- **`init_multi` is `init multi`, and `uninstall_all` is `uninstall all`.** The old spellings stop with a message giving the new one. -- **A pool with no reads at a site is `NA` in the frequency table**, where it was 0, because a frequency over zero reads does not exist. By default no such cell reaches a table: the depth filter now also drops a site with an unread pool at `minDP` 0, which 3.2.0 kept and published as 0. With `keepLowDepthAsZero` on, `NA` is what an unread cell reads. -- **Every step writes its log as it runs.** A step that fails, or is killed, leaves what it had reached in `Logs/`. Before this, a step copied its log there only at the end, so a failure lost exactly the log that would have explained it. Each attempt starts with a line naming the run, the session and the time. -- **Capping is about seven times faster on real data**, with identical output. -- **A call set or filter that leaves nothing stops the run** and says which settings to look at, instead of finishing as though it had succeeded. Nothing is published from it. -- **Environments.** FastQC 0.12.1 to 0.13.0, Perl 5.32.1 to 5.44.0, pandoc 3.11 to 3.12, R's future and doFuture to their next minor versions, and patch updates beneath them. bwa, samtools, bcftools, cutadapt, Trim Galore and snpEff did not move. - -### Analysis modules - -- **association `20261006.001`.** Two ways it published `perm_p` 0, which no permutation p can be. A site where any unit had no reads came out 0, and `fdr_p` with it; in 3.2.0 that took `minDP` 0. And the strongest sites in a table, far beyond any rearrangement, failed to count themselves and came out 0 too. Each site is now tested over the units it was read in, and not tested when fewer than three were read. `design_floor` is 1/n!, where it said 2/n!, so four units can reach 0.042. -- **mds `20261006.002`.** A pair of pools averaged over fewer than 30 shared sites is flagged, in `distance.tsv`, on the console and under the plot. In simulation, a distance near 0.02 measured over 30 shared sites is off by about half itself, and real sites are noisier. -- **basicstats `20261006.001`.** A pool's depth summaries cover only the sites it was read at, and `depth.tsv` and `diversity.tsv` count the rest in a new `unmeasured` column. A site with no reads for a pool was averaged in as a depth of 0, and one such site was enough to turn that pool's harmonic depth into 0 and its effective sample size into NA; in 3.2.0 that took `minDP` 0. - -### Commits - -- (2a15aea) small change in releasing, test for linting -- (b96bfca) zsh completion added -- (c5bacb4) dos2unix for parameters, metadata, and runs files -- (51b649c) manual and documents passes -- (8d0447c) dropZeroDept toggle added -- (b6e0ce6) Comments shortened -- (ebd69af) Comments shortened -- (eedf598) scaleMapQ decisions are recorded -- (772db1f) Analysis modules are updated to be compatible with the zero depth sites change -- (85eb380) cap_depth is fixed for performance -- (bffc6a0) Empty vcf, depth, and freq files now produce error -- (ad85fb1) Logging reworked -- (aaeab55) Test suite for dropZeroDepth added -- (6746280) major allele to ref tests -- (f54a5ae) SampleID RG_Sample clarifications -- (8e8a078) parameter checks added to step0 -- (9b80793) config migrate is updated after dropZeroDepth -- (6127b53) Full release cycle rework -- (b96b818) wrapper tidiness -- (2b50c1c) Releasing improvements -- (87edc97) releasing cycle streamlined further -- (f2c8187) Manual rewrite -- (998c307) prep version is rescoped -- (1224341) Minor fix on how prep version works -- (d76ba2f) Prep for v3.3.0 -- (7419281) module publishing in release cycle is handled with a script now -- (1e36baf) mds site diversity limit established -- (e87e742) factorial calculation fixed -- (e04215b) dropZeroDepth replaced with keepLowDepthAsZero, and minSamples is added to vcfFilter -- (30aa234) Prep for v3.3.0 - ---- - ## [3.2.0] - 2026-09-24 **Your raw reads can stay where they were handed to you.** `Data/` no longer has to be one flat folder (one folder per sample, one per sequencing run, nested as deep as you like, or all together as before). Nothing you already have needs moving, not one pinned tool changed version, and no result is affected. As with every release, the new installation's module store starts empty and your modules are installed into it again. @@ -603,7 +539,6 @@ Major upgrade to **Nextflow 26** and **Trim Galore 2.x**. This release is not ba --- -[3.3.0]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.3.0 [3.2.0]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.2.0 [3.1.2]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.1.2 [3.1.1]: https://github.com/ozankiratli/PoolSeqFlow/releases/tag/v3.1.1 diff --git a/CLAUDE.md b/CLAUDE.md index 68238c1..859c1bb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,7 +70,7 @@ Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different ## Other standing rules - **`dev/RELEASING.md` is a living procedure, not a record.** Correct it when a release teaches you something; it carries no date and describes no particular version. It holds only what a person does: a command, what it must show, and what to do when it does not. Everything else belongs in `.claude/development-notes/` or in the script's own header, and a step must never point into a dated note -- that is how it came to promise an audit that did not exist. -- **Several release gates live in the suite rather than in that document**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`), the analysis versions (`check-analysis-versions.sh`) and the version-consistency case all run inside `00_static`. A green suite is how they report. Do not add a step for one. +- **Several release gates live in the suite rather than in that document**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`) and the version-consistency case all run inside `00_static`. A green suite is how they report. Do not add a step for one. **`check-analysis-versions.sh` is the exception**: a module is legitimately behind between releases, so the suite tests the script only, and the real tree is checked at steps 4 and 11 of `dev/RELEASING.md` and by `release.yml`. This line used to list it with the others, and nothing ran it between the floors being raised at step 7 and the tag, which is where v3.3.0 failed. - **The manual keeps real typographic characters. Everything else in the repository is ASCII.** `manual/PoolSeqFlow-manual.md` is rendered to a website and read as prose, so an arrow is `→`, a reverse arrow `←`, a product `×`, an ellipsis `…`, a negative `−`, a comparison `≥`. **Never ASCII-ify the manual**, and never let a sweep do it: a pass asked to remove em dashes has twice taken 53 arrows, 29 products and 16 ellipses with them. Two carve-outs inside the manual, and only two. **Em dashes are out on style** -- rewrite the sentence, do not substitute `--`. And **a command someone copies and runs stays ASCII**: that is the test, not whether something sits in a fenced block, because the manual fences its diagrams and formulas too. `distance = (J_A + J_B) / 2 − J_AB` and the labels inside a ```mermaid block keep their glyphs. Every other file, including the templates, the notes and this one, is ASCII as the global rule says. - **A ```mermaid diagram exists on the site only.** `mkdocs.yml` carries the custom fence for it and material loads the renderer from a CDN, so the diagram needs a network to draw and the shipped `manual/PoolSeqFlow-manual.md` shows its `graph TD` source. Use it for what ASCII cannot draw. Anything a reader must have in the tarball stays as text, or as an `.svg` under `manual/assets/` like the five depth-histogram figures. @@ -89,7 +89,7 @@ Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different Not one line of code may change. Per file, diff the non-comment lines against `HEAD`. For Python, a docstring is not a `#` comment and the line count misleads -- prove it with an AST comparison that strips docstrings. -Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts -- files linted, cases passed -- against the run before it rather than only the exit status: a filter that matches nothing also reports success. Neither number is written down here, because both move with every file added. +Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts -- files linted, cases passed -- against the run before it rather than only the exit status: a filter that matches nothing now stops the run, but one narrower than you meant still reports success. Neither number is written down here, because both move with every file added. **Lint through the suite, at zero errors and zero warnings.** Seven seconds: @@ -109,7 +109,7 @@ The two cases behind that filter are `nextflow lint is clean`, which lints the t bash test/run_tests.sh --suite 07_analysis --case citation ``` -`--case` matches any part of a case name and filters at dispatch, so the suite's fixtures are still built by the cases that define them. Three cases in under a minute against ten for the suite they live in -- which is the difference between checking a change and avoiding checking it. +`--case` matches any part of a case's function name, so it takes underscores (`all_pending`, never `"all pending"`), and filters at dispatch, so the suite's fixtures are still built by the cases that define them. **A case that cannot run fails the run, except under `--fast`**: outside it, a missing environment is a red run, not a skip. Three cases in under a minute against ten for the suite they live in -- which is the difference between checking a change and avoiding checking it. **The full suite belongs to a release**, not to a step. `--fast` is the cheap sweep when a change reaches beyond its own step -- the wrapper, packaging, a shared library -- and a whole `--suite ` is for when the step *is* that suite. @@ -130,7 +130,7 @@ bash test/run_tests.sh --suite 07_analysis --case citation **`bash test/run_tests.sh --changed` picks the suites for you**, from what each suite declares it runs expanded through the include graph. `dev/scripts/select-tests.py ` shows the reasoning without running anything. It errs wide -- a change to `test/lib/` or to the selector selects everything -- so a narrow answer is trustworthy and a wide one is only expensive. -**Every suite declares what it may cost** -- `static`, `jvm` or `pipeline` -- in a `# cost:` line in its own header. `--cost static` is the set that completes with nothing installed: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose. +**Every suite declares what it may cost** -- `static`, `jvm` or `pipeline` -- in a `# cost:` line in its own header. `--cost static` is the set that builds nothing: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose: with nothing installed, `--cost static --fast` completes, while `--cost static` alone fails the cases that want a tool. **`--suite` and `--case` accumulate and match by name, not by number.** `--suite analysis_time --suite analysis_series` runs both, and `--suite analysis` runs all nine. Every run prints the scope it selected, so a narrowed run cannot be mistaken for a full one; renumbering a suite therefore costs nothing, because nothing addresses one by its number. diff --git a/PoolSeqFlow b/PoolSeqFlow index 3987dbe..3f12f9d 100755 --- a/PoolSeqFlow +++ b/PoolSeqFlow @@ -1,6 +1,6 @@ #!/bin/bash # -# Version: 3.3.0 +# Version: 3.2.0 # Author : Ozan L. Z. Kiratli # email : ozankiratli@protonmail.com # @@ -43,7 +43,7 @@ set -e eval "$(conda shell.bash hook)" -VERSION="3.3.0" +VERSION="3.2.0" usage() { echo "Usage: $0 {install|init [multi]|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall [all]}" @@ -59,7 +59,7 @@ usage_analysis() { # The module store's own words. usage_modules() { - echo "Usage: $0 analysis modules {list|available|install [version]|uninstall }" + echo "Usage: $0 analysis modules {list|available|install [version]|install all|uninstall }" exit 1 } @@ -339,6 +339,12 @@ deploy_payload() { # analysis environment comes out first, while the manifests declaring it still exist. # After this the store and the environment are both back to what the release ships. gone=$(store_packages "$dest/analysis/modules" | cut -d= -f1 | sort -u) + # Less the release's own baseline, as a module uninstall leaves it: what a module + # declares overlaps it by design, and removing that strips the environment the release + # built. Read from the copy being replaced, which is what built it. + gone=$(printf '%s\n' "$gone" \ + | grep -vxF "$(ANALYSIS_ENV_FILE="$dest/install/environment-analysis.yml" \ + baseline_packages)" || true) if [ -n "$gone" ] && env_exists "$ANALYSIS_ENV"; then echo "Removing what the installed modules added to '$ANALYSIS_ENV':" printf ' %s\n' $gone @@ -354,6 +360,18 @@ deploy_payload() { for item in $PAYLOAD_ITEMS; do cp -r "$INSTALL/$item" "$dest"/ done + # From a checkout, the copy also carries what git ignores inside the payload: the + # checkout's own module store and compiled Python, neither of which a release holds. + # Taken back out, so the store starts empty. Only for a checkout of its own: a release + # unpacked inside some other repository is not judged by that repository's rules. + if [ "$(git -C "$INSTALL" rev-parse --show-toplevel 2>/dev/null)" = "$here" ]; then + # shellcheck disable=SC2086 + git -C "$INSTALL" ls-files -z --others --ignored --exclude-standard --directory \ + -- $PAYLOAD_ITEMS | while IFS= read -r -d '' ignored; do + rm -rf "${dest:?}/$ignored" + echo " left out, ignored by git here: $ignored" + done + fi # Stamps each copy with its own location. for item in $WRAPPERS; do sed -i "s|^POOLSEQFLOW_INSTALLED_HOME=\"\"$|POOLSEQFLOW_INSTALLED_HOME=\"$dest\"|" \ @@ -683,8 +701,51 @@ module_index_names() { $2 == "" || $2 == "module" { print $1 }' | sort -u } +# One module as this installation sees it: the version `install ` would take, that row's +# contract and summary, and a state that is empty when it can be installed, otherwise "installed" +# or why this release cannot run it. One line, fields separated by MODULE_INDEX_SEP. `available` +# and `install all` both read it, so what one lists and the other installs cannot drift. +# +# Nothing runnable falls back to the NEWEST row, so the state can say why rather than leaving the +# module out. A module published for a later release is the case this exists for: being told to +# upgrade is an answer, and a blank is not. +module_state() { # index name contract frame_here + local index="$1" name="$2" contract="$3" frame_here="$4" + local row best="" newest_row="" _n _k _v version row_contract row_frame row_env url sha summary + local state + while IFS= read -r row; do + [ -n "$row" ] || continue + [ -n "$newest_row" ] || newest_row="$row" + IFS="$MODULE_INDEX_SEP" read -r _n _k _v row_contract row_frame row_env url sha summary <` would take, asked with the same helper install_module asks with, so the # two cannot come to different answers. - local row best newest_row _n _k _v while IFS= read -r name; do [ -n "$name" ] || continue rows=$((rows + 1)) - best=""; newest_row="" - while IFS= read -r row; do - [ -n "$row" ] || continue - [ -n "$newest_row" ] || newest_row="$row" - IFS="$MODULE_INDEX_SEP" read -r _n _k _v row_contract row_frame row_env url sha summary <` in a process of its own: install_module ends the +# script on a refusal, and one module that will not install must not stop the others. What this +# release cannot run is named and passed over with the reason `available` gives. A module already +# in the store is left as it is whatever the catalogue says, as install_module leaves it. +install_every_module() { + local index contract frame_here name version row_contract summary state + local installed="" already="" refused="" passed="" + index=$(mktemp) + trap 'rm -f "$index"' RETURN + get_module_index "$index" + require_module_index_format "$index" + contract=$(module_contract) + if [ -z "$contract" ]; then + echo "ERROR: cannot tell which table contract this release speaks." >&2 + echo " Expected to read it from $INSTALL/analysis/lib/nf/modules.nf." >&2 + echo " This installation is incomplete; install it again." >&2 + exit 1 + fi + frame_here=$(installed_frame_version) + + while IFS= read -r name; do + [ -n "$name" ] || continue + if [ -d "$MODULE_STORE/$name" ]; then + already="$already $name" + continue + fi + IFS="$MODULE_INDEX_SEP" read -r version row_contract summary state <&2 + exit 1 + fi +} + +# Takes back out the libraries one module install put in the store, when that install fails. +# Only names it installed itself are given, so no other module can be using them. +remove_new_libraries() { + local lib + for lib in "$@"; do + rm -rf "$(library_store)/${lib:?}" + echo " library $lib taken back out" >&2 + done +} + install_module() { local name="$1" want="${2:-}" index contract line version row_contract url sha summary local tmp tarball unpacked dest specs spec @@ -980,7 +1082,7 @@ ROW # # A library already in the store is left alone: it is shared, another module may be using it, # and every version of it published for this frame computes the same thing. - local libs lib + local libs lib added="" libs=$(module_libraries "$unpacked/$name/manifest.json") if [ -n "$libs" ]; then mkdir -p "$(library_store)" @@ -990,11 +1092,14 @@ ROW continue fi if ! install_library "$lib" "$index"; then + # shellcheck disable=SC2086 + remove_new_libraries $added echo "" >&2 echo "ERROR: $name v$version needs the library '$lib', which could not be" >&2 - echo " installed. The module was not installed." >&2 + echo " installed. The module was not installed, nor any library it brought." >&2 exit 1 fi + added="$added $lib" done fi @@ -1028,10 +1133,12 @@ ROW # shellcheck disable=SC2086 if ! conda_install_packages "$ANALYSIS_ENV" $specs; then rm -rf "$dest" + # shellcheck disable=SC2086 + remove_new_libraries $added echo "" >&2 echo "ERROR: $name v$version needs packages this release's analysis environment" >&2 echo " cannot take without moving something already in it. The module was not" >&2 - echo " installed and the environment was not changed." >&2 + echo " installed, nor any library it brought, and the environment was not changed." >&2 echo "" >&2 echo " That is a compatibility question between this release and this module," >&2 echo " not something to solve here: install a build of $name published for" >&2 @@ -1406,25 +1513,48 @@ populate_project() { # Every PoolSeqFlow environment and installation on the machine. A function rather than an arm # of its own: `uninstall all` is one command with a word after it, and the arm list has to stay # the same set as the usage line. +# The wrapper links in $PREFIX/bin that point into an installation under $PREFIX/opt, whether or +# not that installation is still there. One path a line. +poolseqflow_links() { + local prefix="$1" link + for link in "$prefix/bin/PoolSeqFlow" "$prefix/bin/PoolSeqFlow-"*; do + [ -L "$link" ] || continue + case "$(readlink "$link")" in + "$prefix/opt/PoolSeqFlow-"*) printf '%s\n' "$link" ;; + esac + done +} + uninstall_every_version() { INSTALLED=$(poolseqflow_envs) - if [ -z "$INSTALLED" ]; then - echo "No PoolSeqFlow conda environments are installed. Nothing to do." - exit 0 - fi PREFIX=$(install_prefix) PAYLOADS=$(installed_payload_versions "$PREFIX") + LINKS=$(poolseqflow_links "$PREFIX") + # Each of the three can outlive the others, and any one of them is something to remove. + if [ -z "$INSTALLED" ] && [ -z "$PAYLOADS" ] && [ -z "$LINKS" ]; then + echo "No PoolSeqFlow environment, installation or wrapper is here. Nothing to do." + exit 0 + fi echo "" - echo "This removes every PoolSeqFlow conda environment on this machine:" - echo "$INSTALLED" | sed 's/^/ /' + echo "This removes everything PoolSeqFlow installed on this machine:" + if [ -n "$INSTALLED" ]; then + echo "" + echo " every PoolSeqFlow conda environment:" + echo "$INSTALLED" | sed 's/^/ /' + fi if [ -n "$PAYLOADS" ]; then echo "" - echo "and every installed pipeline under $PREFIX/opt:" + echo " every installed pipeline under $PREFIX/opt:" for v in $PAYLOADS; do echo " v$v"; done + fi + if [ -n "$LINKS" ]; then echo "" - echo "including the symlinks in $PREFIX/bin." + echo " every wrapper in $PREFIX/bin:" + echo "$LINKS" | sed 's/^/ /' fi echo "" + echo " and the tab completions." + echo "" echo "Only the software goes. Your project directories - parameters.config, Data/," echo "Reference/ - and everything in storageDir are untouched: none of them live" echo "inside an installation." @@ -1446,20 +1576,36 @@ uninstall_every_version() { esac # An active environment has to be stepped out of first; there may be none. conda deactivate 2>/dev/null || true - # A plain `for`, not `... | while read`: a piped loop is a subshell, where `set -e` - # would not stop the script. Environment names cannot contain whitespace. + # Every removal is attempted whatever the one before it did, so one refusal cannot leave + # the rest installed. Environment names cannot contain whitespace. for env in $INSTALLED; do echo "Removing $env..." - conda env remove -n "$env" -y + if ! said=$(conda env remove -n "$env" -y 2>&1); then + echo " could not remove $env. conda said:" >&2 + printf '%s\n' "$said" | grep -v '^[[:space:]]*$' | tail -n 3 | sed 's/^/ /' >&2 + fi done - # remove_payload re-points the plain wrapper each time and removes it with the last - # version, so nothing is left on PATH pointing at a directory that has gone. for v in $PAYLOADS; do remove_payload "$v" || true done + # The links remove_payload could not reach: those whose installation was already gone. + poolseqflow_links "$PREFIX" | while IFS= read -r link; do + rm -f "$link" && echo "Removed $link" + done remove_completion echo "" - echo "All PoolSeqFlow environments and installations removed." + # What is left, read back from conda and the prefix. + LEFT_ENVS=$(poolseqflow_envs) + LEFT=$(installed_payload_versions "$PREFIX") + LEFT_LINKS=$(poolseqflow_links "$PREFIX") + if [ -n "$LEFT_ENVS$LEFT$LEFT_LINKS" ]; then + echo "Not everything is gone, for the reasons given above:" >&2 + for env in $LEFT_ENVS; do echo " conda environment $env" >&2; done + for v in $LEFT; do echo " $PREFIX/opt/PoolSeqFlow-$v" >&2; done + [ -z "$LEFT_LINKS" ] || printf '%s\n' "$LEFT_LINKS" | sed 's/^/ /' >&2 + exit 1 + fi + echo "Everything PoolSeqFlow installed is removed." } case $COMMAND in @@ -1878,8 +2024,14 @@ EOF *) usage_modules ;; esac require_analysis_env - install_module "${ANALYSIS_ARGS[1]}" "${ANALYSIS_ARGS[2]:-}" - conda deactivate + if [ "${ANALYSIS_ARGS[1]}" = all ]; then + [ "${#ANALYSIS_ARGS[@]}" -eq 2 ] || usage_modules + conda deactivate + install_every_module + else + install_module "${ANALYSIS_ARGS[1]}" "${ANALYSIS_ARGS[2]:-}" + conda deactivate + fi ;; uninstall) [ "${#ANALYSIS_ARGS[@]}" -eq 2 ] || usage_modules diff --git a/analysis/frame.config b/analysis/frame.config index a6fc22d..e421cd5 100644 --- a/analysis/frame.config +++ b/analysis/frame.config @@ -27,6 +27,13 @@ includeConfig "${launchDir}/parameters.config" // A ternary and not an `if`: the strict parser rejects `if` statements in config. params.cores = params.containsKey('cores') && params.cores != null ? params.cores : [:] +// The outputPrefix of a project whose config predates it: the VCF name it already has, and the +// template's 'Test' when it names neither. vcf.fileName, when a config leaves it out, follows the +// prefix. The same three lines as nextflow.config's, guarded the same way. +params.vcf = params.containsKey('vcf') && params.vcf != null ? params.vcf : [:] +params.outputPrefix = params.containsKey('outputPrefix') && params.outputPrefix != null ? params.outputPrefix : (params.vcf.containsKey('fileName') && params.vcf.fileName != null ? params.vcf.fileName : 'Test') +params.vcf.fileName = params.vcf.containsKey('fileName') && params.vcf.fileName != null ? params.vcf.fileName : params.outputPrefix + // The helpers in bin/, which parameters.config places beside the entry script. For a module the // entry script is the module, so this is the installation instead. // diff --git a/analysis/frame.version b/analysis/frame.version index 707b3c3..29e43b4 100644 --- a/analysis/frame.version +++ b/analysis/frame.version @@ -24,4 +24,4 @@ # not touch this file. Each module carries its own version in the same form, in its manifest.json. # # Blank lines and lines starting with # are ignored; the first line left is the version. -20260910.001 +20261008.001 diff --git a/analysis/lib/nf/citations.nf b/analysis/lib/nf/citations.nf index 45177d0..8c755ca 100644 --- a/analysis/lib/nf/citations.nf +++ b/analysis/lib/nf/citations.nf @@ -8,7 +8,7 @@ nextflow.enable.dsl=2 -include { installDir } from './paths.nf' +include { installDir; releaseVersion } from './paths.nf' include { moduleStore } from './modules.nf' // The entries citations/citations.json holds that an ANALYSIS run also invokes. It lists every @@ -56,7 +56,7 @@ def citationShell(String module, String dest) { def json = groovy.json.JsonOutput.toJson(merged).replace("'", "'\\''") def packages = merged.findAll { _key, entry -> entry instanceof Map && entry.r_package } .collect { key, entry -> "${key}=${entry.r_package}" } - def release = "${workflow.manifest.version ?: ''}" + def release = releaseVersion() def nextflowVersion = "${workflow.nextflow.version ?: ''}" def lines = [] diff --git a/analysis/lib/nf/modules.nf b/analysis/lib/nf/modules.nf index 5e30c8e..a3551b6 100644 --- a/analysis/lib/nf/modules.nf +++ b/analysis/lib/nf/modules.nf @@ -176,6 +176,17 @@ def readManifest(Object dir) { "know them for a module published separately.\n" + "Install the module again, or remove the directory.") } + def report = "${parsed.report ?: ''}".trim() + if (!report.isEmpty() && !(report ==~ /[A-Za-z0-9._-]+\.Rmd/)) { + throw new IllegalStateException( + "${manifest} gives 'report' as '${report}'. It is the name of an .Rmd file in the " + + "module's own directory, which the frame knits into the published report.") + } + if (!report.isEmpty() && !file("${dir}/${report}").exists()) { + throw new IllegalStateException( + "${manifest} declares its report as '${report}', and ${dir} has no such file.\n" + + "Install the module again, or remove the directory.") + } return [ summary : "${parsed.summary}".toString(), version : "${parsed.version}".toString(), contract : "${parsed.contract}".toString(), @@ -187,6 +198,7 @@ def readManifest(Object dir) { gates : parsed.gates ?: [], outputs : parsed.outputs ?: [], packages : (parsed.packages ?: []).collect { spec -> "${spec}".toString() }, + report : report.toString(), builtin : false, dir : "${dir}".toString(), entry : "${entry}".toString() ] diff --git a/analysis/lib/nf/outputs.nf b/analysis/lib/nf/outputs.nf index e452bca..b309eb7 100644 --- a/analysis/lib/nf/outputs.nf +++ b/analysis/lib/nf/outputs.nf @@ -9,7 +9,7 @@ nextflow.enable.dsl=2 -include { installDir; verificationRecordName } from './paths.nf' +include { installDir; releaseVersion; verificationRecordName } from './paths.nf' include { moduleEntry } from './modules.nf' // The manual this release ships, installed beside the code. Authoritative for this release; the @@ -23,8 +23,27 @@ def manualSite() { return 'https://ozankiratli.github.io/PoolSeqFlow/' } -// What every published analysis carries whichever module produced it. -def frameOutputs() { +// What counts as the script that produced a result, by extension: R, and the shell, Python and +// Julia a module may drive it from. +def scriptSuffixes() { + return ['R', 'r', 'Rmd', 'rmd', 'sh', 'py', 'jl'] +} + +// The code a result was produced by: a script, and a source a script compiles. +def codeSuffixes() { + return scriptSuffixes() + ['cpp', 'c', 'h', 'hpp'] +} + +// The name a file a module produced is published under: the target's outputPrefix and an +// underscore in front of it, and code under its own name. A glob gives a glob. +def publishedName(Map target, String file) { + def code = codeSuffixes().any { suffix -> file.endsWith(".${suffix}") } + return (code ? file : "${target.prefix}_${file}").toString() +} + +// What every published analysis carries whichever module produced it. `report` is the name its PDF +// is published under. +def frameOutputs(String report) { return [ [ file : verificationRecordName(), summary: 'the checks that cleared this folder, and what they were run against', @@ -35,8 +54,8 @@ def frameOutputs() { [ file : 'references.bib', summary: 'the same list as BibTeX', anchor : 'citing-the-tools-it-runs' ], - [ file : 'report.pdf', - summary : 'every result in this folder in one document, each under its own file name', + [ file : report, + summary : 'the results laid out to be read: the module\'s own layout where it has one, and each table and figure under its file\'s name where it does not', anchor : 'analysis-report', optional: true ], ] @@ -164,13 +183,14 @@ def readmeDesignLines(Map target) { return lines } -// The README a published folder carries: every file in it, what it is, and where to read about -// it. Rendered from the module's declarations and the frame's own, so no module writes its own. -def readmeText(String module, Map target, List outputs) { +// The README a published folder carries: every file in it under its published name, what it is, +// and where to read about it. Rendered from the module's declarations and the frame's own, so no +// module writes its own. +def readmeText(String module, Map target, List outputs, String report) { def entry = moduleEntry(module) def lines = ["# ${target.label}: ${module}".toString(), '', - "Produced by PoolSeqFlow ${workflow.manifest.version ?: 'unknown'}, module " + + "Produced by PoolSeqFlow ${releaseVersion()}, module " + "${module} v${entry.version}, reading".toString(), '', " ${target.dir}".toString(), @@ -181,7 +201,8 @@ def readmeText(String module, Map target, List outputs) { '|---|---|---|'] // An optional output is named whether or not this run produced one, and its row says so. - (outputs + frameOutputs()).each { output -> + def published = outputs.collect { output -> output + [ file: publishedName(target, "${output.file}") ] } + (published + frameOutputs(report)).each { output -> def what = "${output.summary ?: ''}${output.optional ? ' *(only when the setting that produces it is set)*' : ''}" lines << "| `${output.file}` | ${what} | ${outputLink(output)} |".toString() } @@ -195,10 +216,11 @@ def readmeText(String module, Map target, List outputs) { } // The shell that writes README.md into `dest`, and refuses a module that declared a file it did -// not publish. Rendered into the task's own script, the way citationShell() is. -def readmeShell(String module, Map target, String dest) { +// not publish. Rendered into the task's own script, the way citationShell() is. `report` is the +// name the PDF is published under. +def readmeShell(String module, Map target, String dest, String report) { def outputs = moduleOutputs(module) - def text = readmeText(module, target, outputs).replace("'", "'\\''") + def text = readmeText(module, target, outputs, report).replace("'", "'\\''") def lines = [] outputs.each { output -> @@ -206,9 +228,10 @@ def readmeShell(String module, Map target, String dest) { // named, and none when you named none. Its absence is not a failure. if (output.optional) return // The declared name may be a glob, so the test counts what it matched. - lines << "MATCHED=\$(find \"${dest}\" -maxdepth 1 -name '${output.file}' | wc -l)" + def name = publishedName(target, "${output.file}") + lines << "MATCHED=\$(find \"${dest}\" -maxdepth 1 -name '${name}' | wc -l)" lines << "if [ \"\$MATCHED\" -eq 0 ]; then" - lines << " echo \"PUBLISHING ${target.label}: ERROR: ${module} declares it publishes '${output.file}',\" >&2" + lines << " echo \"PUBLISHING ${target.label}: ERROR: ${module} declares it publishes '${output.file}', here '${name}',\" >&2" lines << " echo \"PUBLISHING ${target.label}: and this analysis produced nothing matching it.\" >&2" lines << " echo \"PUBLISHING ${target.label}: What it produced:\" >&2" lines << " find \"${dest}\" -mindepth 1 -maxdepth 1 | sed 's|.*/| |' >&2" diff --git a/analysis/lib/nf/paths.nf b/analysis/lib/nf/paths.nf index b2e6d8d..1b46b98 100644 --- a/analysis/lib/nf/paths.nf +++ b/analysis/lib/nf/paths.nf @@ -236,6 +236,28 @@ def frameVersion() { return version } +// The PoolSeqFlow release this installation is, from the manifest in its nextflow.config. A +// module's own run has no workflow.manifest.version: Nextflow reads that file only for an entry +// script beside it. +def releaseVersion() { + def config = file("${installDir()}/nextflow.config") + def version = config.exists() + ? config.readLines().findResult { line -> + def hit = (line =~ /^\s*version\s*=\s*'([^']*)'/) + hit ? "${hit[0][1]}".toString() : null } ?: '' + : '' + if (!(version ==~ /\d+\.\d+\.\d+/)) { + throw new IllegalStateException( + "the installation does not say which release it is.\n" + + " ${config}\n" + + "should carry the release as `version = 'X.Y.Z'` in its manifest" + + (version.isEmpty() ? ', and carries none' : ", and carries '${version}'") + ".\n" + + "Every published analysis names the release that produced it. Install this release " + + "again.") + } + return version +} + // The installation this invocation was launched from, and a refusal when it cannot be found. A // module is its own pipeline, so ${projectDir} is the module's own directory and nothing // Nextflow computes points at the installation. The wrapper exports POOLSEQFLOW_HOME. diff --git a/analysis/lib/nf/plan.nf b/analysis/lib/nf/plan.nf index 4731b3d..c849b9f 100644 --- a/analysis/lib/nf/plan.nf +++ b/analysis/lib/nf/plan.nf @@ -94,6 +94,26 @@ def directoryLabel(Map variant) { return "${variant.dir.outputs}".tokenize('/').last() } +// The outputPrefix the files published from one results directory start with. Every run sharing +// the directory has the same one, as it is part of what step 7 shares on. Held to the rule +// bin/check_parameters.sh applies, 'null' included. +def targetPrefix(String label, List covered) { + def prefixes = covered.collect { run -> "${run.outputPrefix}".toString() }.unique() + if (prefixes.size() != 1) { + throw new IllegalStateException( + "the runs that share ${label} have ${prefixes.size()} outputPrefix values: " + + "${prefixes.join(', ')}. The plan and the run table disagree about which runs share work.") + } + def prefix = prefixes.first() + if (!(prefix ==~ /[A-Za-z0-9][A-Za-z0-9._-]*/) || prefix == 'null') { + throw new IllegalArgumentException( + "the outputPrefix of ${label} is '${prefix}', and it starts the name of every file " + + "an analysis publishes there. It may hold letters, digits, '.', '_' and '-', and " + + "starts with a letter or a digit.") + } + return prefix +} + // One entry per results directory the selected runs produced, and where each artifact class sits // inside it. Two runs whose tables are the same file share a directory and are covered once. // `needs` names the classes the module cannot run without. @@ -126,6 +146,7 @@ def resultsTargets(Map plan, List runDefs, List selected, String module, List ne checkTargetDesign(label, rows) return [ label : label, module : module, + prefix : targetPrefix(label, covered), dir : "${variant.dir.outputs}".toString(), members : variant.members.collect { member -> runToken(member) }, selected: variant.members.findAll { member -> wanted.contains(member) } diff --git a/analysis/lib/nf/report.nf b/analysis/lib/nf/report.nf index 028e97e..31812da 100644 --- a/analysis/lib/nf/report.nf +++ b/analysis/lib/nf/report.nf @@ -1,62 +1,77 @@ -// The PDF a published analysis carries: every result in the folder, each under its own file -// name. -// -// A FRAME CAPABILITY, not a module's: it is built from the same output declarations as the README, -// so a module writes nothing to get one. +// The PDF a published analysis carries. A module that declares a `report` in its manifest lays +// out its own results in it; any other module gets each table and figure it declared, under the +// file's name. // // Knitted by knitr and turned into a PDF by pandoc with typst as the engine. Not // rmarkdown::pdf_document, which is bound to LaTeX and ignores the engine it is given. nextflow.enable.dsl=2 -include { installDir; frameVersion } from './paths.nf' +include { installDir; frameVersion; releaseVersion } from './paths.nf' include { moduleEntry } from './modules.nf' include { moduleOutputs; frameOutputs } from './outputs.nf' -def reportName() { - return 'report.pdf' +// The name the PDF is published under: the target's outputPrefix, the module, and `stamp`, the +// time it was built as yyyyMMdd-HHmmss. +def reportName(Map target, String stamp) { + return "${target.prefix}_${target.module}_report_${stamp}.pdf".toString() } def reportTemplate() { return "${installDir()}/analysis/lib/rmd/report.Rmd".toString() } -// What the template renders from: where the folder is, what made it, and every file the module -// declared with the summary it declared. `report.pdf` itself is left out. -def reportSpec(String module, Map target, String folder) { +// The functions every report is drawn with, the frame's and a module's alike. +def reportLibrary() { + return "${installDir()}/analysis/lib/rmd/report.R".toString() +} + +// What the template renders from: what made the folder, the warnings its design carries, the +// module's own report if it declares one, and every file the module declared with the summary it +// declared, under the name the module wrote it. The report itself is left out. The folder is not in +// it: it exists only once the task runs, and reaches the template as POOLSEQFLOW_REPORT_FOLDER. +def reportSpec(String module, Map target, String report) { def entry = moduleEntry(module) - def outputs = (moduleOutputs(module) + frameOutputs()) - .findAll { output -> "${output.file}" != reportName() } + def outputs = (moduleOutputs(module) + frameOutputs(report)) + .findAll { output -> "${output.file}" != report } .collect { output -> [ file: "${output.file}".toString(), summary: "${output.summary ?: ''}".toString() ] } + def child = entry.report ? "${entry.dir}/${entry.report}".toString() : '' + def warnings = (target.design?.warnings ?: []) + .collect { note -> "${note.detail}".split('\n').join(' ').toString() } return groovy.json.JsonOutput.toJson([ - label : "${target.label}".toString(), - module : module, - version: "${entry.version}".toString(), - release: "${workflow.manifest.version ?: 'unknown'}".toString(), - frame : frameVersion(), - source : "${target.dir}".toString(), - folder : folder, - outputs: outputs ]) + label : "${target.label}".toString(), + module : module, + version : "${entry.version}".toString(), + release : releaseVersion(), + frame : frameVersion(), + source : "${target.dir}".toString(), + warnings: warnings, + library : reportLibrary(), + child : child, + outputs : outputs ]) } -// The shell that writes the report into `dest`. +// The shell that writes the report into `dest` as `report`. The files it reads are published +// under the target's outputPrefix, which reaches the template as POOLSEQFLOW_REPORT_PREFIX. // -// It does not fail the publish, and prints the reason when it cannot build one. -def reportShell(String module, Map target, String dest) { - def spec = reportSpec(module, target, dest).replace("'", "'\\''") +// It does not fail the publish, and prints the reason when it cannot build one. On stdout: the +// task succeeds, and Nextflow shows a successful task's stdout under `debug` and never its stderr. +def reportShell(String module, Map target, String dest, String report) { + def spec = reportSpec(module, target, report).replace("'", "'\\''") def lines = [] lines << "printf '%s' '${spec}' > report_spec.json" - lines << "POOLSEQFLOW_REPORT_SPEC=\"\$PWD/report_spec.json\" \\" - lines << " Rscript --vanilla -e 'knitr::knit(commandArgs(TRUE)[1], \"report.md\", quiet = TRUE)' \\" + lines << "POOLSEQFLOW_REPORT_SPEC=\"\$PWD/report_spec.json\" POOLSEQFLOW_REPORT_FOLDER=\"${dest}\" \\" + lines << " POOLSEQFLOW_REPORT_PREFIX='${target.prefix}' \\" + lines << " Rscript --vanilla -e 'knitr::opts_knit\$set(root.dir = getwd()); knitr::knit(commandArgs(TRUE)[1], \"report.md\", quiet = TRUE)' \\" lines << " '${reportTemplate()}' > report_knit.log 2>&1 \\" - lines << " && pandoc report.md -o \"${dest}/${reportName()}\" --pdf-engine=typst \\" + lines << " && pandoc report.md -o \"${dest}/${report}\" --pdf-engine=typst \\" lines << " >> report_knit.log 2>&1 \\" lines << " || {" - lines << " echo \"PUBLISHING ${target.label}: the PDF report could not be built. The analysis\" >&2" - lines << " echo \"PUBLISHING ${target.label}: is published without it; every number in it is a file\" >&2" - lines << " echo \"PUBLISHING ${target.label}: in the folder already.\" >&2" - lines << " sed 's/^/PUBLISHING ${target.label}: /' report_knit.log >&2 || true" + lines << " echo \"PUBLISHING ${target.label}: the PDF report could not be built. The analysis\"" + lines << " echo \"PUBLISHING ${target.label}: is published without it; every number in it is a file\"" + lines << " echo \"PUBLISHING ${target.label}: in the folder already.\"" + lines << " sed 's/^/PUBLISHING ${target.label}: /' report_knit.log || true" lines << " }" return lines.join('\n ') } diff --git a/analysis/lib/nf/results.nf b/analysis/lib/nf/results.nf index 6aca25a..4a09ebc 100644 --- a/analysis/lib/nf/results.nf +++ b/analysis/lib/nf/results.nf @@ -8,14 +8,8 @@ nextflow.enable.dsl=2 include { verificationRecordName } from './paths.nf' include { citationShell } from './citations.nf' -include { readmeShell } from './outputs.nf' -include { reportShell } from './report.nf' - -// What counts as the script that produced a result, by extension: R, and the shell, Python and -// Julia a module may drive it from. -def scriptSuffixes() { - return ['R', 'r', 'Rmd', 'rmd', 'sh', 'py', 'jl'] -} +include { readmeShell; scriptSuffixes; codeSuffixes } from './outputs.nf' +include { reportShell; reportName } from './report.nf' // One results directory's analysis, moved into place. process InstallResults { @@ -34,10 +28,15 @@ process InstallResults { // `find -name` tests, one per accepted extension, joined for a single walk of the stage. script_test = scriptSuffixes().collect { s -> "-name '*.${s}'" }.join(' -o ') script_list = scriptSuffixes().collect { s -> "*.${s}" }.join(', ') + // The code a module produced, as the patterns of a `case`. + code_case = codeSuffixes().collect { s -> "*.${s}" }.join('|') + // When the analysis is published, in local time, which the report's name carries. + stamp = java.time.LocalDateTime.now().format(java.time.format.DateTimeFormatter.ofPattern('yyyyMMdd-HHmmss')) + report_name = reportName(target, stamp) // Written into the STAGE, so the citations arrive in the same rename as the analysis. citations = citationShell("${target.module}", '$STAGE') - readme = readmeShell("${target.module}", target, '$STAGE') - report = reportShell("${target.module}", target, '$STAGE') + readme = readmeShell("${target.module}", target, '$STAGE', report_name) + report = reportShell("${target.module}", target, '$STAGE', report_name) """ set -eo pipefail @@ -67,6 +66,17 @@ process InstallResults { exit 1 fi + # Every file and directory the module produced starts with the outputPrefix, and the code it + # was produced by keeps its own name. What a directory holds keeps its names. + for FILE in "\$STAGE"/*; do + [ -e "\$FILE" ] || continue + NAME=\$(basename "\$FILE") + case "\$NAME" in + ${code_case}) ;; + *) mv -- "\$FILE" "\$STAGE/${target.prefix}_\$NAME" ;; + esac + done + # Every file the module said it would publish is here, or nothing is published. ${readme} diff --git a/analysis/lib/rmd/report.R b/analysis/lib/rmd/report.R new file mode 100644 index 0000000..315f805 --- /dev/null +++ b/analysis/lib/rmd/report.R @@ -0,0 +1,329 @@ +# The functions a report is drawn with. report.Rmd sources this before anything is knitted, so the +# frame's own layout and a module's report call the same ones. +# +# Every function that draws prints markdown or a raw typst block, so it is called from a chunk with +# results = "asis". + +# The published folder the report describes. +report_folder <- function() { + folder <- Sys.getenv("POOLSEQFLOW_REPORT_FOLDER") + if (!nzchar(folder) || !dir.exists(folder)) { + stop("report: POOLSEQFLOW_REPORT_FOLDER names no folder: '", folder, "'") + } + folder +} + +# The outputPrefix every file the module computed is published under. +report_prefix <- function() { + prefix <- Sys.getenv("POOLSEQFLOW_REPORT_PREFIX") + if (!nzchar(prefix)) stop("report: POOLSEQFLOW_REPORT_PREFIX is not set") + prefix +} + +# The name a file the module wrote is published under, the prefix and an underscore in front. A +# glob gives a glob. +report_name <- function(name) { + paste0(report_prefix(), "_", name) +} + +# A published name with the prefix taken off: the name the module wrote. +report_logical <- function(name) { + lead <- paste0(report_prefix(), "_") + ifelse(startsWith(name, lead), substring(name, nchar(lead) + 1), name) +} + +# A published table, by the name the module wrote it under, with every column as text, so a +# sequence called 4 stays a name. NULL when the folder does not hold it. The tables are written +# unquoted, so a quote in a cell is a character. +report_read <- function(name) { + path <- file.path(report_folder(), report_name(name)) + if (!file.exists(path)) return(NULL) + utils::read.delim(path, check.names = FALSE, colClasses = "character", quote = "", + comment.char = "") +} + +# The published files a glob of the names the module wrote matches, in name order. The glob is +# matched against the names alone, so a bracket in the folder's own path is a character. +report_files <- function(pattern) { + sort(list.files(report_folder(), pattern = utils::glob2rx(report_name(pattern)), + full.names = TRUE)) +} + +# Text as markdown that reads exactly as written: every ASCII punctuation mark escaped, so a name +# holding * or @ is not emphasis or a citation. +report_text <- function(x) { + gsub("([[:punct:]])", "\\\\\\1", as.character(x), perl = TRUE) +} + +# Numbers as a table shows them: NA, NaN, Inf and -Inf as themselves. With `decimals`, that many +# places after the point. Without, a whole number is printed whole, a magnitude below 1e-4 or from +# 1e15 in scientific notation, and anything else to `digits` significant digits. Thousands are +# separated. +report_number <- function(x, digits = 3, decimals = NULL) { + x <- suppressWarnings(as.numeric(x)) + out <- rep("NA", length(x)) + out[is.nan(x)] <- "NaN" + out[!is.na(x) & x == Inf] <- "Inf" + out[!is.na(x) & x == -Inf] <- "-Inf" + ok <- is.finite(x) + # A value that rounds to zero prints unsigned; formatC writes it as -0. + if (!is.null(decimals)) x[ok] <- round(x[ok], decimals) + x[ok & x == 0] <- 0 + if (!is.null(decimals)) { + out[ok] <- formatC(x[ok], format = "f", digits = decimals, big.mark = ",") + return(out) + } + whole <- ok & x == round(x) & abs(x) < 1e15 + out[whole] <- formatC(x[whole], format = "f", digits = 0, big.mark = ",") + rest <- ok & !whole + out[rest] <- vapply(x[rest], function(v) { + if (abs(v) < 1e-4 || abs(v) >= 1e15) return(formatC(v, format = "e", digits = digits - 1)) + sub("[.]$", "", formatC(signif(v, digits), digits = digits, format = "fg", flag = "#", + big.mark = ",")) + }, "") + out +} + +# Every column of a table that holds decimals, printed by report_number(), for a table the report +# knows nothing about. A column of whole numbers is left as written, and so is any column holding a +# value like 01. +report_format <- function(table, digits = 4) { + for (name in names(table)) { + values <- table[[name]] + held <- values[!is.na(values) & nzchar(values) & !(values %in% c("NA", "NaN", "Inf", "-Inf"))] + if (length(held) == 0 || anyNA(suppressWarnings(as.numeric(held)))) next + if (!any(grepl(".", held, fixed = TRUE)) || any(grepl("^[-+]?0[0-9]", held))) next + table[[name]] <- ifelse(is.na(values) | !nzchar(values), values, + report_number(values, digits = digits)) + } + table +} + +# The rows of a published table below its header, counted without reading them as a table. +count_rows <- function(path) { + con <- file(path, "r") + on.exit(close(con)) + lines <- 0L + repeat { + chunk <- readLines(con, n = 100000L, warn = FALSE) + if (length(chunk) == 0) break + lines <- lines + length(chunk) + } + max(lines - 1L, 0L) +} + +# Text as a typst string literal, which typst sets as plain text: no character in it is markup. +typst_string <- function(x) { + x <- ifelse(is.na(x), "NA", as.character(x)) + x <- gsub("\\", "\\\\", x, fixed = TRUE) + x <- gsub("\"", "\\\"", x, fixed = TRUE) + x <- gsub("\n", "\\n", x, fixed = TRUE) + paste0("\"", x, "\"") +} + +# A column is aligned right when every value in it is a number, NA or empty, a column holding +# nothing else included. +numeric_column <- function(values) { + held <- values[!is.na(values) & nzchar(values) & values != "NA"] + all(grepl("^[-+\u2265\u2264<>~ ]*[0-9][0-9,.]*(e[-+]?[0-9]+)?%?$", held)) +} + +# A table, as typst draws it: the header repeated on every page the table runs onto, a cell +# holding a newline set on two lines, and no cell broken anywhere else. +# +# Typst measures every cell and every header once and keeps the first layout that fits the text +# width: each column as wide as what is in it, at 10, 9, 8 or 7 points; then each column at least +# as wide as its widest cell and the widest part of its header, with the space left over shared in +# proportion to what each column lacks of its full width and the headers wrapping after an +# underscore, a dot or a space, at 10 down to 5 points; then that at 4 points. Every one of them +# breaks across pages. +# +# `align` is one letter per column, "l" or "r"; without it a column of numbers is aligned right and +# anything else left. `groups` is one label per column, and a run of equal labels is printed once +# above the columns it spans; "" spans nothing. `size` fixes the text size in points, with the +# headers on one line. A table longer than `max_rows` shows its first rows. `total` is how many +# rows the table has when it was read cut short, and the table says how many `source` holds. +report_table <- function(table, caption = NULL, align = NULL, groups = NULL, size = NULL, + max_rows = Inf, source = NULL, total = nrow(table)) { + stopifnot(is.data.frame(table)) + # Evaluated here: its default counts the rows before the cut below. + force(total) + if (ncol(table) == 0) { + cat("\n*Nothing to show", if (!is.null(caption)) paste0(": ", caption), ".*\n\n", sep = "") + return(invisible()) + } + if (nrow(table) > max_rows) table <- utils::head(table, max_rows) + if (is.null(align)) align <- vapply(table, function(col) if (numeric_column(col)) "r" else "l", "") + alignment <- ifelse(align == "r", "right", "left") + + n <- ncol(table) + as_array <- function(items) paste0("(", paste(c(items, ""), collapse = ", "), ")") + values <- lapply(table, function(col) typst_string(ifelse(is.na(col), "NA", as.character(col)))) + labels <- names(table) + # A header's parts: where it may wrap, after an underscore or a dot, or at a space. + parts <- lapply(labels, function(label) { + pieces <- strsplit(gsub("([_.])", "\\1\001", label), "\001| ", perl = TRUE)[[1]] + pieces[nzchar(pieces)] + }) + + # The header rows, and the rule under each group label. + names_row <- paste(sprintf("strong(if wrapped { soft.at(%d) } else { whole.at(%d) })", + seq_len(n) - 1, seq_len(n) - 1), collapse = ", ") + header_rows <- 1 + if (is.null(groups)) { + header <- paste0("table.header(", names_row, ")") + } else { + stopifnot(length(groups) == ncol(table)) + runs <- rle(groups) + starts <- cumsum(c(0, utils::head(runs$lengths, -1))) + spans <- character(0) + rules <- character(0) + # A label is broken between words to about the width of the columns under it, counted in + # characters. + chars <- vapply(seq_along(table), function(j) { + max(nchar(names(table)[j]), nchar(as.character(table[[j]])), na.rm = TRUE) + 2 + }, 0) + for (i in seq_along(runs$lengths)) { + if (!nzchar(runs$values[i])) { + spans <- c(spans, rep("[]", runs$lengths[i])) + next + } + room <- sum(chars[starts[i] + seq_len(runs$lengths[i])]) + label <- paste(strwrap(runs$values[i], width = max(room, 1)), collapse = "\n") + spans <- c(spans, sprintf("table.cell(colspan: %d, align: center, strong(%s))", + runs$lengths[i], typst_string(label))) + rules <- c(rules, sprintf("table.hline(y: 1, start: %d, end: %d, stroke: 0.4pt)", + starts[i], starts[i] + runs$lengths[i])) + } + header <- paste0("table.header(", paste(c(spans, rules), collapse = ", "), ", ", + names_row, ")") + header_rows <- 2 + } + + # The cells a row to a line, and the same cells a column to an array for measuring. + rows <- if (nrow(table) > 0) paste0(" ", do.call(paste, c(values, sep = ", ")), ",") + else character(0) + data <- c( + " let flat = (", + rows, + " )", + sprintf(" let cells = range(%d).map(j => range(%d).map(i => flat.at(i * %d + j)))", + n, nrow(table), n), + sprintf(" let whole = %s", as_array(typst_string(labels))), + sprintf(" let soft = %s", as_array(typst_string(gsub("([_.])", "\\1\u200b", labels)))), + sprintf(" let parts = %s", as_array(vapply(parts, function(p) as_array(typst_string(p)), "")))) + make <- c( + " let make(s, columns, wrapped) = text(size: s, table(", + " columns: columns,", + sprintf(" align: %s,", as_array(alignment)), + " stroke: none,", + " inset: (x: 5pt, y: 3.5pt),", + sprintf(" fill: (_, y) => if y >= %d and calc.odd(y - %d) { luma(245) },", + header_rows, header_rows), + " table.hline(),", + paste0(" ", header, ","), + " table.hline(stroke: 0.5pt),", + " ..flat,", + " table.hline(),", + " ))") + # Widths are measured at 10 points and scale with the text; each column also carries its + # 5pt inset on either side. + shown <- if (!is.null(size)) sprintf(" let shown = make(%spt, %d, false)", size, n) else c( + " let width(x, bold) = measure(text(size: 10pt, if bold { strong(x) } else { x })).width", + " let widest(xs, bold) = xs.fold(0pt, (m, x) => calc.max(m, width(x, bold)))", + sprintf(" let least = range(%d).map(j => calc.max(widest(cells.at(j), false), widest(parts.at(j), true)))", n), + sprintf(" let most = range(%d).map(j => calc.max(widest(cells.at(j), false), width(whole.at(j), true)))", n), + " let at(ws, s) = ws.map(w => w * (s / 10pt) + 10pt)", + " let fits(ws, s) = at(ws, s).sum() <= region.width", + " let natural = (10pt, 9pt, 8pt, 7pt).find(s => fits(most, s))", + sprintf(" let shown = if natural != none { make(natural, %d, false) } else {", n), + " let found = (10pt, 9pt, 8pt, 7pt, 6pt, 5pt).find(s => fits(least, s))", + " let s = if found != none { found } else { 4pt }", + " let low = at(least, s)", + " let lacking = at(most, s).zip(low).map(((h, l)) => h - l)", + " let spare = calc.max(0pt, region.width - low.sum())", + " let total = lacking.sum()", + " let widths = low.zip(lacking).map(((l, d)) =>", + " if total > 0pt { l + calc.min(d, spare * (d / total)) } else { l })", + " make(s, widths, true)", + " }") + placed <- if (is.null(caption)) " shown" + else paste0(" figure(kind: table, caption: ", typst_string(caption), ", shown)") + cat("\n\n```{=typst}\n#layout(region => {\n", paste(c(data, make, shown, placed), collapse = "\n"), + "\n})\n```\n\n", sep = "") + if (total > nrow(table)) { + cat(nrow(table), " of ", total, " rows", + if (!is.null(source)) paste0(". The whole table is in `", source, "`"), ".\n\n", sep = "") + } + invisible() +} + +# A PNG's width and height in pixels, from its header; NULL for any other file. +png_size <- function(path) { + head <- readBin(path, "raw", n = 24L) + if (length(head) < 24L || !identical(head[2:4], charToRaw("PNG"))) return(NULL) + number <- function(bytes) sum(as.numeric(bytes) * 256^(3:0)) + c(number(head[17:20]), number(head[21:24])) +} + +# A figure already on disk, scaled to `width` of the text width, and a PNG at the full width kept +# to 7.5 in tall so a caption fits under it. The template's 1 in margins on letter paper leave a +# text block 6.5 in wide and 9 in tall. The path is in angle brackets, so a parenthesis in it is a +# character. +report_image <- function(path, caption = NULL, width = "100%") { + size <- png_size(path) + if (identical(width, "100%") && !is.null(size) && 7.5 * size[1] / size[2] < 6.5) { + width <- sprintf("%.2fin", 7.5 * size[1] / size[2]) + } + cat("\n\n![", if (is.null(caption)) "" else caption, "](<", path, ">){width=", width, "}\n\n", + sep = "") + invisible() +} + +# A ggplot drawn into report_figures/ beside the report and placed like report_image(). `width` +# and `height` are the size it is drawn at, in inches. +report_plot <- function(plot, name, width = 7, height = 4, caption = NULL) { + dir.create("report_figures", showWarnings = FALSE) + path <- file.path("report_figures", paste0(name, ".png")) + ggplot2::ggsave(path, plot, width = width, height = height, dpi = 200) + report_image(path, caption) +} + +# Items split into as few groups of at most `most` as hold them all, in the order given, the sizes +# differing by at most one: 25 pools make tables of 7, 6, 6 and 6 columns. +report_columns <- function(items, most = 8) { + groups <- max(1, ceiling(length(items) / most)) + sizes <- length(items) %/% groups + (seq_len(groups) <= length(items) %% groups) + split(items, rep(seq_len(groups), sizes)) +} + +# One color per pool, in the order given: the Okabe-Ito palette for up to eight pools, and evenly +# spaced hues past that. +report_palette <- function(pools) { + okabe_ito <- c("#E69F00", "#56B4E9", "#009E73", "#0072B2", "#D55E00", "#CC79A7", + "#F0E442", "#000000") + colors <- if (length(pools) <= length(okabe_ito)) okabe_ito[seq_along(pools)] + else grDevices::hcl.colors(length(pools), "Dark 3") + stats::setNames(colors, pools) +} + +# Each declared .tsv table and .png figure under its own file name, which is the whole report of a +# module that declares no report of its own. A table is read only as far as the rows it shows. +report_declared <- function(outputs, max_rows = 40) { + for (output in outputs) { + for (path in report_files(output$file)) { + name <- basename(path) + if (grepl("[.]tsv$", name)) { + cat("\n\n## ", report_text(name), "\n\n", sep = "") + table <- utils::read.delim(path, check.names = FALSE, colClasses = "character", + nrows = max_rows, quote = "", comment.char = "") + report_table(report_format(table), caption = output$summary, source = name, + total = count_rows(path)) + } else if (grepl("[.]png$", name)) { + cat("\n\n## ", report_text(name), "\n\n", sep = "") + report_image(path, caption = output$summary) + } + } + } + invisible() +} diff --git a/analysis/lib/rmd/report.Rmd b/analysis/lib/rmd/report.Rmd index 047cb43..6f4adbf 100644 --- a/analysis/lib/rmd/report.Rmd +++ b/analysis/lib/rmd/report.Rmd @@ -1,63 +1,66 @@ +--- +margin: + x: 1in + y: 1in +--- + ```{r setup, include = FALSE} -# The PDF the frame writes beside every published analysis: every result it holds, each under -# the name of the file it came from. +# The PDF the frame writes beside every published analysis. # # Knitted to markdown and handed to pandoc, never rendered by rmarkdown::pdf_document, which is # bound to LaTeX and ignores --pdf-engine. # -# The spec is written by analysis/lib/nf/report.nf and names the published folder, what -# produced it, and every file the module declared with the summary it declared. Nothing here -# knows what any particular module computes. -knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE) +# The spec is written by analysis/lib/nf/report.nf: what produced the folder, the warnings its +# design carries, the module's own report if it declares one, and every file the module declared +# with the summary it declared. The folder itself is POOLSEQFLOW_REPORT_FOLDER. +knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, error = FALSE) + +# Characters are read and written as UTF-8 whatever locale the task inherited. Outside one, R +# writes the report's own pi and its breaks as and , and a name with an accent in +# it as bytes. +if (!isTRUE(l10n_info()[["UTF-8"]])) { + for (locale in c("C.UTF-8", "C.utf8", "en_US.UTF-8", "en_US.utf8")) { + if (nzchar(suppressWarnings(Sys.setlocale("LC_CTYPE", locale)))) break + } + if (!isTRUE(l10n_info()[["UTF-8"]])) { + stop("report: this machine has no UTF-8 locale (tried C.UTF-8, C.utf8, en_US.UTF-8 ", + "and en_US.utf8), and the report cannot print its own symbols or a pool name ", + "with an accent without one") + } +} + spec <- jsonlite::fromJSON(Sys.getenv("POOLSEQFLOW_REPORT_SPEC"), simplifyVector = FALSE) -folder <- spec$folder +source(spec$library) +``` -# A table is shown in full up to this many rows. Past it the head is shown and the full row -# count is printed with it. -ROW_CAP <- 40 +```{=typst} +#set par(justify: false) +#show figure.where(kind: table): set figure.caption(position: top) +#show figure.where(kind: table): set block(breakable: true) +#show figure.caption.where(kind: table): it => block(sticky: true, it) +#show heading.where(level: 2): set block(above: 1.8em, below: 0.9em) ``` -# `r spec$label`: `r spec$module` +# `r spec$module`: `r spec$label` -Produced by PoolSeqFlow `r spec$release`, module `r spec$module` v`r spec$version`, -analysis frame `r spec$frame`, on `r format(Sys.time(), "%Y-%m-%d %H:%M")`. +Produced by PoolSeqFlow `r spec$release`, module `r spec$module` v`r spec$version` and analysis frame `r spec$frame`, on `r format(Sys.time(), "%Y-%m-%d %H:%M")`. -Read from `r spec$source`. +Read from `r paste0("\x60", spec$source, "\x60")`. -```{r contents} -rows <- do.call(rbind, lapply(spec$outputs, function(o) data.frame( - File = sprintf("`%s`", o$file), What = o$summary %||% "", stringsAsFactors = FALSE))) -knitr::kable(rows, caption = "What this folder holds") +```{r warnings, results = "asis"} +if (length(spec$warnings) > 0) { + cat("\n## Read these before the numbers\n\n") + for (note in spec$warnings) cat("- ", note, "\n", sep = "") + cat("\n") +} ``` -```{r results = "asis"} -`%||%` <- function(a, b) if (is.null(a)) b else a - -for (output in spec$outputs) { - matched <- Sys.glob(file.path(folder, output$file)) - if (length(matched) == 0) next # an optional output this run did not produce - for (path in sort(matched)) { - name <- basename(path) - # A typst raw block, not \newpage: pandoc passes LaTeX commands only to LaTeX and - # drops this one in silence, so the report would run together with nothing to show it. - cat("\n\n```{=typst}\n#pagebreak()\n```\n\n## ", name, "\n\n", sep = "") - if (!is.null(output$summary)) cat("*", output$summary, "*\n\n", sep = "") - - if (grepl("[.]tsv$", name)) { - table <- utils::read.delim(path, check.names = FALSE, colClasses = "character") - shown <- utils::head(table, ROW_CAP) - print(knitr::kable(shown)) - if (nrow(table) > ROW_CAP) { - cat("\n\n", nrow(shown), " of ", nrow(table), - " rows. The whole table is in `", name, "`.\n", sep = "") - } - } else if (grepl("[.]png$", name)) { - # Scaled to the text width, not placed at its own size: a figure sized for looking - # at on its own pushes the heading naming it onto the previous page. - cat("\n![](", path, "){width=90%}\n", sep = "") - } else { - cat("\nNot rendered here - read `", name, "` itself.\n", sep = "") - } - } +```{r body, results = "asis"} +if (nzchar(spec$child)) { + cat(knitr::knit_child(spec$child, quiet = TRUE), sep = "\n") +} else { + report_declared(spec$outputs) } ``` + +Every number here is in a file in this folder, and `README.md` says which file holds what. diff --git a/bin/check_parameters.sh b/bin/check_parameters.sh index 1d7f08e..ea24a66 100755 --- a/bin/check_parameters.sh +++ b/bin/check_parameters.sh @@ -15,9 +15,9 @@ # # Silence is the answer for a parameter set with nothing wrong with it. # -# WHAT BELONGS HERE: a setting that makes the run produce NOTHING, or that silently changes what -# a published number means. What does not: a setting that merely produces fewer sites. minDP 20 -# against minDP 5 is a scientific choice and this has no opinion on it. +# WHAT BELONGS HERE: a setting that makes the run produce NOTHING or stops it partway, or that +# silently changes what a published number means. What does not: a setting that merely produces +# fewer sites. minDP 20 against minDP 5 is a scientific choice and this has no opinion on it. # # VALUES ARE READ FROM THE COMPOSED OPTION STRINGS WHERE THERE IS ONE. A project may pin # variantCall.mpileupOptions by hand, which makes scaleMapQ and varQualMin inert, so judging the @@ -146,4 +146,25 @@ if [ -n "$fqmem" ] && ! is_int "$fqmem"; then "FastQC takes megabytes as a bare integer and refuses a value carrying a unit, so write 2048 rather than 2G." fi +# ------------------------------------------------------------------- the names -- +# +# outputPrefix and vcf.fileName each start file names: letters, digits, '.', '_' and '-', the +# first a letter or a digit. 'null' is what the template's vcf.fileName line holds when the +# config has no outputPrefix line. Judged only when the input carries the key: an absent key is +# not an empty name. +for key in outputPrefix vcf.fileName; do + [ -n "${P[$key]+set}" ] || continue + name=$(get "$key") + if [ -z "$name" ]; then + say FAIL "$key" "EMPTY" "no name" \ + "It starts the name of the files it covers, so they would be named from nothing. Write a name: letters, digits, '.', '_' and '-', starting with a letter or a digit." + elif [ "$name" = "null" ]; then + say FAIL "$key" "NOT SET" "null" \ + "null is what the template's vcf.fileName line reads when parameters.config has no outputPrefix line. Put the line back, or write a name." + elif ! printf '%s' "$name" | grep -qx '[A-Za-z0-9][A-Za-z0-9._-]*'; then + say FAIL "$key" "NOT A FILE NAME" "$name" \ + "It starts the name of the files it covers, so it may hold letters, digits, '.', '_' and '-', and starts with a letter or a digit." + fi +done + exit "$FAILED" diff --git a/bin/config_migrate.sh b/bin/config_migrate.sh index 98444de..8b1ad2d 100755 --- a/bin/config_migrate.sh +++ b/bin/config_migrate.sh @@ -78,6 +78,13 @@ awk -v OLDF="$OLD" -v REPORT="$REPORT" ' return (k == "fastqc.memory" || k == "variantCall.maxDepth") } + # A value of yours where the template computes one, kept as written: vcf.fileName in a config + # that already has outputPrefix, whatever it holds, and an expression in one that does not. + # A literal in a config without outputPrefix is the old name, and outputPrefix takes it. + function pinned(k) { + return (k == "vcf.fileName" && (("outputPrefix" in oldval) || !literal(oldval[k]))) + } + # Parameters renamed in this release, keyed by their CURRENT name and returning the name # they had before. Needs a line per rename: without one, a rename reads as one DROPPED plus # one NEW. A rename that also changed the meaning goes in reformatted() too, which runs @@ -87,6 +94,10 @@ awk -v OLDF="$OLD" -v REPORT="$REPORT" ' if (k == "vcffilter.minQUAL") return "vcftools.minQUAL" if (k == "storageDir") return "projectDir" if (k == "ploidy") return "diploidy" + # vcf.fileName stays, and the template now derives it from outputPrefix. Only a name + # moves: an expression stays where it is. + if (k == "outputPrefix") + return (("vcf.fileName" in oldval) && literal(oldval["vcf.fileName"])) ? "vcf.fileName" : "" # Whole-scope renames: every field follows its prefix, so none is listed by name. old = k if (sub(/^cleanBAM\./, "samtools.", old)) return old @@ -146,14 +157,14 @@ awk -v OLDF="$OLD" -v REPORT="$REPORT" ' if (src in oldval) { used[src] = 1 - if (!literal(val)) { + if (!literal(val) && !(pinned(full) && oldval[src] != val)) { # this release computes it - the template wins if (literal(oldval[src]) && oldval[src] != val) printf "COMPUTED\t%s\t%s\t%s\n", full, val, oldval[src] >> REPORT } else if (reformatted(full)) { if (oldval[src] != val) printf "REFORMAT\t%s\t%s\t%s\n", full, val, oldval[src] >> REPORT - } else if (literal(oldval[src])) { + } else if (literal(oldval[src]) || pinned(full)) { if (was_renamed) printf "RENAMED\t%s\t%s\t%s\n", full, oldval[src], src >> REPORT else if (oldval[src] != val) @@ -369,4 +380,32 @@ if [ -n "$NEW_KEEPLOW" ]; then echo fi +# Read back from the report, so this fires exactly when outputPrefix took the old vcf.fileName. +NEW_PREFIX=$(awk -F'\t' '$1 == "RENAMED" && $2 == "outputPrefix" { print $3 }' "$REPORT") +NEW_PREFIX=${NEW_PREFIX//[\'\"]/} +if [ -n "$NEW_PREFIX" ]; then + note_heading + echo " outputPrefix IS NEW, and it took your vcf.fileName, $NEW_PREFIX." + echo + echo " It starts the name of the files a run writes. vcf.fileName now follows it, so your" + echo " VCF and its frequency and depth tables keep the names they have. What changes is the" + echo " analysis layer: every file an analysis publishes now starts with it, so mds.tsv is" + echo " ${NEW_PREFIX}_mds.tsv, and each report is named ${NEW_PREFIX}__report_-