From 741928142dc5bece1a0e34263b53f1c4107c7e96 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Mon, 5 Oct 2026 07:55:43 -0400 Subject: [PATCH 1/5] module publishing in release cycle is handled with a script now --- dev/RELEASING.md | 69 +++++++++-- dev/scripts/prep-version.sh | 29 +++-- dev/scripts/raise-module-floors.sh | 177 +++++++++++++++++++++++++++++ 3 files changed, 255 insertions(+), 20 deletions(-) create mode 100755 dev/scripts/raise-module-floors.sh diff --git a/dev/RELEASING.md b/dev/RELEASING.md index d4f8429..050f198 100644 --- a/dev/RELEASING.md +++ b/dev/RELEASING.md @@ -45,7 +45,9 @@ dev/scripts/prep-version.sh - reads what each environment requires of its host and exports both files - then proves the files it wrote: `check-exported-floor.sh` solves each from nothing and reads its floor, and `check-module-packages.sh` puts the module pins through a baseline built from the new file -The scratch environments are removed on every exit. `--no-cleanup` keeps them, to reproduce a failure against. The conda package cache is left alone. +The scratch environments are removed on every exit. The conda package cache is left alone. + +**Run it with `--no-cleanup`**, and the pair survives for step 5 to reuse. Three full suite runs in a cycle otherwise means three solves of the same two files, and the second of them proves nothing the first did not. It prints the two `export` lines to use; keep them in the shell you carry through to step 5. Then read `dev/logs/prep--/`, including the per-environment table of what moved. @@ -137,10 +139,36 @@ A full run solves **both** `install/environment.yml` and `install/environment-an Only a full run does this. `--fast` and `--suite` resolve this tree's exact version and skip what is not installed, so they stay cheap, and neither ever borrows another release's environment. +**Reuse step 2's pair rather than solving a third one.** If step 2 ran with `--no-cleanup`, it printed two `export` lines; with those set, this run uses them and starts testing at once: + +``` +export TEST_CONDA_ENV= +export TEST_ANALYSIS_ENV= +bash test/run_tests.sh +``` + +Nothing is lost by it: the files have not changed since step 2 exported them, so a fresh solve would produce the same pair. What this step is actually re-checking is whatever steps 3 and 4 changed. + +**Then remove them**, because `prep-version.sh` refuses to start while they exist: + +``` +conda env remove -n PoolSeqFlow-update -y +conda env remove -n PoolSeqFlow-update-analysis -y +``` + ## 6. Merge to `main` + +``` +git add -A && git commit -m "Prep for vX.X.X" +``` + Open the merge request, review the diff as a whole, merge. +``` +git switch main +git pull +``` ## 7. Bump the version @@ -150,37 +178,60 @@ dev/scripts/bump-version.sh Abandoning the cycle after this: `dev/scripts/bump-version.sh --revert` undoes the bump and the CHANGELOG section it added. It refuses once the version has been tagged. -**Then raise the environment floor of every module step 2 named.** A moved pin names a package version only this release's environment holds, so those modules require this release and nothing earlier -- and `environment` in their manifests is what tells a user on the older release to upgrade, instead of letting the install reach conda and fail there. +**Then raise the environment floor of everything this release republishes** -- libraries as well as modules: ``` -"environment": "" in each modules//manifest.json +dev/scripts/raise-module-floors.sh --dry-run # what it would change +dev/scripts/raise-module-floors.sh ``` +Republished means its manifest version moved since the previous tag, which is the same set `publish-module.sh --list` acts on at step 11. Anything in it was proven only against this release's environment, and `environment` is what tells an older installation to upgrade instead of letting the install reach conda and be refused there with nothing explained. + +**Libraries are in it for a reason beyond their own pins.** The catalogue resolves a module and its libraries **independently**, each taking the newest row whose floor the installation clears. A republished library left at an old floor could therefore be paired with the *previous* release's module on an older installation. Giving everything republished the same floor makes the release the unit: an installation gets the whole of one or the whole of the other. + +`moved-modules.txt` from step 2 is read as a check, not as the list -- anything step 2 recorded moving a pin in must be in the republished set, and a disagreement stops the script. + **It has to be here and not at step 2.** `analysis/lib/nf/modules.nf` refuses at run time any module whose floor is above the running release, so setting it before the bump makes every one of those modules unrunnable in its own repository: ```text 'mds' v20261004.001 needs the analysis environment of PoolSeqFlow 3.3.0 or newer, and this is 3.2.0. ``` +which is six failures across the three module suites. The script refuses a version above the one the tree declares, for the same reason. + No second version bump is needed -- the module's version already moved at step 2 and nothing is committed yet, and the gate compares committed state. ## 8. Run the full suite -**Install the new version first**, and here that is the point rather than a prerequisite: this step is the only one that measures the environments a user actually receives, instead of a scratch solve of the files that describe them. +On `main`, at the new version. **Two different questions, and the suite no longer answers the second one.** + +### The suite, which brings its own environments + +``` +bash test/run_tests.sh +``` + +A full run builds a scratch pair from `install/environment.yml` and `install/environment-analysis.yml` and removes them, so it neither needs an installation nor touches one. Expect minutes before the first case. Nothing has to be installed for this. + +**`0 skipped` and `0 failed` are the numbers that matter.** + +### The installation, which is the other question ``` ./PoolSeqFlow install ./PoolSeqFlow analysis install dev/scripts/check-host-floor.sh - -bash test/run_tests.sh ``` -On `main`, at the new version. The suite resolves both environments and runs tests on them. +Whether an install of the new version works, and what the environment it creates requires of a host. `install` verifies itself and fails if anything is missing, so there is nothing to read but its exit. + +Order between the two does not matter, because neither reaches the other. If you run the suite first you can skip installing until you need the commands. + +Note for step 10: having `PoolSeqFlow-` installed is what makes `check-release-archive.sh` reuse the environment rather than create one, which its own header records as a limit. Installing here is not free of consequence further down. -**`0 skipped` and `O failed` are the numbers that matter.** +### If it fails -If it fails, nothing is published yet: undo the bump, remove what you just installed, and triage on `dev`. +Nothing is published yet: undo the bump, remove what you installed, and triage on `dev`. ``` dev/scripts/bump-version.sh --revert diff --git a/dev/scripts/prep-version.sh b/dev/scripts/prep-version.sh index 72b1f21..a3ae941 100755 --- a/dev/scripts/prep-version.sh +++ b/dev/scripts/prep-version.sh @@ -117,12 +117,17 @@ cleanup() { if [ "$NO_CLEANUP" -eq 1 ]; then printf 'Scratch environments kept, --no-cleanup:\n' >&2 if [ -n "${ENV_PREFIX:-}" ] && [ -n "${ANALYSIS_PREFIX:-}" ]; then - printf ' TEST_CONDA_ENV=%s \\\n' "$ENV_PREFIX" >&2 - printf ' TEST_ANALYSIS_ENV=%s \\\n' "$ANALYSIS_PREFIX" >&2 - printf ' ./test/run_tests.sh --suite \n' >&2 + printf ' export TEST_CONDA_ENV=%s\n' "$ENV_PREFIX" >&2 + printf ' export TEST_ANALYSIS_ENV=%s\n' "$ANALYSIS_PREFIX" >&2 + printf '\n' >&2 + printf 'Named that way, a run uses these instead of building its own - which is how\n' >&2 + printf 'the full suite later in the release reuses this pair rather than solving a\n' >&2 + printf 'second one, and how one suite is re-run against them to chase a failure:\n' >&2 + printf ' ./test/run_tests.sh\n' >&2 + printf ' ./test/run_tests.sh --suite \n' >&2 printf '\n' >&2 fi - printf 'Remove them when you are done:\n' >&2 + printf 'Remove them before preparing a version again - this refuses with leftovers:\n' >&2 for e in $CREATED; do printf ' conda env remove -n %s\n' "$e" >&2; done exit "$status" fi @@ -379,14 +384,16 @@ else # 'mds' v... needs the analysis environment of PoolSeqFlow 3.3.0 or newer, and this is 3.2.0. # # So the pin moves here, where the solve decides it, and the floor moves after the bump. - say " AFTER dev/scripts/bump-version.sh, raise the environment floor in each:" - for NAME in $MOVED; do - say " modules/$NAME/manifest.json \"environment\": \"$NEW\"" - done + # Recorded as well as printed, because the step that acts on it runs after the bump and + # after a merge. One name per line, which is what raise-module-floors.sh reads. + printf '%s\n' $MOVED > "$LOGDIR/moved-modules.txt" + say " Their environment floor has to follow, and cannot be written yet: the frame" + say " refuses a module whose floor is above the running release, and that is still" + say " the old version until the bump. AFTER dev/scripts/bump-version.sh:" + say "" + say " dev/scripts/raise-module-floors.sh" say "" - say " Not now: the frame refuses a module whose floor is above the running release," - say " and that is still the old version until the bump. Each module is republished" - say " at the end of the release, which is when the floor has to be true." + say " which reads $LOGDIR/moved-modules.txt." fi fi say "" diff --git a/dev/scripts/raise-module-floors.sh b/dev/scripts/raise-module-floors.sh new file mode 100755 index 0000000..42a4dc1 --- /dev/null +++ b/dev/scripts/raise-module-floors.sh @@ -0,0 +1,177 @@ +#!/usr/bin/env bash +# +# Raise the `environment` floor of every module whose package pins this release moved. +# +# Usage: dev/scripts/raise-module-floors.sh [version] [--dry-run] +# default: the version this working copy declares, which after the bump is the new one +# +# RUN THIS AFTER dev/scripts/bump-version.sh AND BEFORE PUBLISHING. Both halves of that matter. +# +# `environment` is the oldest release whose analysis environment holds what a module needs, and +# the wrapper compares it against the installed version: a module above it is reported as +# "needs PoolSeqFlow " instead of being installed. When step 2 moves a pin to a version +# only this release's environment holds, that module requires this release, and the field has to +# say so - otherwise a user on the previous release installs it, reaches conda, and is refused +# with "already holds" having been told nothing. +# +# AND IT CANNOT BE DONE AT STEP 2. analysis/lib/nf/modules.nf refuses at run time any module whose +# floor is above the running release, so writing the new floor before the bump makes every one of +# those modules unrunnable in its own repository: +# +# 'mds' v20261004.001 needs the analysis environment of PoolSeqFlow 3.3.0 or newer, +# and this is 3.2.0. +# +# which is six failures across the three module suites. Hence a step of its own, after the bump. +# +# WHICH ARTIFACTS: EVERY MODULE AND LIBRARY THIS RELEASE REPUBLISHES, not only the ones whose +# pins moved. +# +# A moved pin is one reason a floor has to rise. The general one is that an artifact published +# from this release was only ever proven against this release's environment, and the catalogue +# resolves modules and libraries INDEPENDENTLY - PoolSeqFlow's library lookup takes the newest row +# whose floor the installation clears, exactly as the module lookup does. So if a republished +# library kept an old floor while the module using it rose, an older installation could pair that +# release's library code with the previous release's module. Giving everything republished the +# same floor makes the release the unit, and an installation gets the whole of one or the whole of +# the other. +# +# Republished means its manifest version moved since the previous tag, which is what +# dev/scripts/publish-module.sh --list acts on. Derived from git rather than from a log, so a +# pruned log cannot make this answer wrong. +# +# moved-modules.txt from step 2 is read as a CHECK, not as the list: anything step 2 recorded +# itself moving must appear in the derived set, and a disagreement stops this rather than being +# resolved silently. +# +# It does not bump a version. Step 2 bumped what it changed, and until the release is committed +# the gate compares committed state, so one bump covers the pin and the floor together. + +set -euo pipefail + +ROOT="$(git rev-parse --show-toplevel)" +cd "$ROOT" + +DRY=0 +VERSION="" +while [ $# -gt 0 ]; do + case "$1" in + --dry-run) DRY=1 ;; + -*) echo "unknown option: $1" >&2; exit 1 ;; + *) [ -z "$VERSION" ] || { echo "ERROR: two versions given: $VERSION and $1" >&2; exit 1; } + VERSION="$1" ;; + esac + shift +done + +TREE="$(sed -n 's/^VERSION="\(.*\)"$/\1/p' PoolSeqFlow | head -1)" +[ -n "$TREE" ] || { echo "ERROR: no VERSION= line in ./PoolSeqFlow" >&2; exit 1; } +VERSION="${VERSION:-$TREE}" +if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "ERROR: '$VERSION' is not a release version" >&2 + exit 1 +fi + +# THE GUARD FOR THE MISTAKE IN THE HEADER. A floor above the version the tree declares is a +# module its own repository cannot run, so this refuses rather than writing it. +if [ "$VERSION" != "$TREE" ]; then + python3 - "$VERSION" "$TREE" <<'PY' || exit 1 +import sys +want = [int(n) for n in sys.argv[1].split(".")] +tree = [int(n) for n in sys.argv[2].split(".")] +if want > tree: + sys.exit("ERROR: %s is above the version this tree declares (%s). A module whose floor is\n" + " above the running release refuses to run, so bump the version first." % + (sys.argv[1], sys.argv[2])) +PY +fi + +PREV=$(git tag --list 'v*' --sort=-v:refname | grep -vxF "v$VERSION" | head -1) +[ -n "$PREV" ] || { echo "ERROR: no previous v* tag to compare against" >&2; exit 1; } + +# Everything whose manifest version moved since the previous tag: what this release republishes. +# git run from inside python rather than piped in, because `python3 -` takes its PROGRAM on stdin +# and a pipe into it arrives at a stream the interpreter has already read to the end. +MODULES=$(python3 - "$PREV" <<'PY' +import glob, json, pathlib, subprocess, sys + +previous_tag = sys.argv[1] + + +def at_tag(path): + out = subprocess.run(["git", "show", "%s:%s" % (previous_tag, path)], + capture_output=True, text=True) + return out.stdout if out.returncode == 0 else "" + + +for path in sorted(glob.glob("modules/*/manifest.json")) + \ + sorted(glob.glob("modules/lib/*/manifest.json")): + now = json.loads(pathlib.Path(path).read_text(encoding="utf-8")) + was = at_tag(path) + # Absent at the previous tag means new in this release, which is republished by definition. + if not was or json.loads(was).get("version") != now.get("version"): + print(now["name"]) +PY +) + +# Step 2's own record, as a check on the above. Newest first, because a release is prepared more +# than once. +for d in $(ls -dt dev/logs/prep-"$VERSION"-*/ 2>/dev/null); do + [ -f "$d/moved-modules.txt" ] || continue + while read -r NAME; do + [ -n "$NAME" ] || continue + printf '%s\n' "$MODULES" | grep -qxF "$NAME" && continue + echo "ERROR: step 2 recorded moving a pin in '$NAME', and its version has not moved" >&2 + echo " since $PREV, so it would not be republished and its floor would stay behind." >&2 + echo " Recorded in: $d/moved-modules.txt" >&2 + exit 1 + done < "$d/moved-modules.txt" + echo "Checked against $d/moved-modules.txt" + break +done + +echo "Raising the environment floor to $VERSION" +echo " republished since $PREV, so proven only against this release's environment" +echo "" + +if [ -z "$MODULES" ]; then + echo " No module's pins moved in this release, so no floor has to move." + exit 0 +fi + +CHANGED=0 +for NAME in $MODULES; do + MANIFEST="" + for candidate in "modules/$NAME/manifest.json" "modules/lib/$NAME/manifest.json"; do + [ -f "$candidate" ] && MANIFEST="$candidate" && break + done + if [ -z "$MANIFEST" ]; then + echo "ERROR: no module or library '$NAME'" >&2 + exit 1 + fi + WAS=$(sed -n 's/.*"environment"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$MANIFEST" | head -1) + if [ -z "$WAS" ]; then + echo "ERROR: $MANIFEST has no 'environment'" >&2 + exit 1 + fi + if [ "$WAS" = "$VERSION" ]; then + printf ' %-20s already %s\n' "$NAME" "$VERSION" + continue + fi + if [ "$DRY" -eq 1 ]; then + printf ' %-20s %s -> %s (--dry-run, not written)\n' "$NAME" "$WAS" "$VERSION" + continue + fi + sed -i -E "s|(\"environment\"[[:space:]]*:[[:space:]]*\")[^\"]*(\")|\1${VERSION}\2|" "$MANIFEST" + IS=$(sed -n 's/.*"environment"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$MANIFEST" | head -1) + [ "$IS" = "$VERSION" ] \ + || { echo "ERROR: could not write $MANIFEST (still '$IS')" >&2; exit 1; } + printf ' %-20s %s -> %s\n' "$NAME" "$WAS" "$VERSION" + CHANGED=$((CHANGED + 1)) +done + +echo "" +if [ "$DRY" -eq 1 ]; then + echo "Nothing written. Drop --dry-run to apply." + exit 0 +fi +echo "$CHANGED manifest(s) changed, uncommitted. Each is republished with the release." From 1e36baf72bbc1458394aa45d5d5af85471a5c16f Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Tue, 6 Oct 2026 18:50:48 -0400 Subject: [PATCH 2/5] mds site diversity limit established --- modules/mds/main.nf | 10 +++-- modules/mds/manifest.json | 9 +++-- modules/mds/mds.R | 22 +++++++++++ modules/mds/test/mds.sh | 78 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 112 insertions(+), 7 deletions(-) diff --git a/modules/mds/main.nf b/modules/mds/main.nf index 709bddf..ea67789 100644 --- a/modules/mds/main.nf +++ b/modules/mds/main.nf @@ -39,8 +39,7 @@ def useCompiled(Map settings) { // The depth tables of one results directory, by the artifact class the manifest asked for. // // Every published frequency is a six-significant-digit rendering of a ratio these hold exactly, -// and a cell with no reads is published there as 0 rather than as missing, so this module reads -// the depth tables and nothing else. +// so this module reads the depth tables and nothing else. def depthTables(Map target) { def spec = target.classes.depths def found = file("${spec.dir}/${spec.pattern}") @@ -59,6 +58,8 @@ def depthTables(Map target) { // the settings the project declared, so every module in a project reads one answer. process Analyze { tag "${target.label}" + // What mds.R writes to stdout, on the console of the run. + debug true input: tuple val(target), path(depths) @@ -96,6 +97,7 @@ process Analyze { "${settings.includeIndels ? 'SNPs and indels' : 'SNPs'}; ${settings.dimensions} axes", "# a distance is a per-site mean over the sites BOTH pools were read at, and is " + "not floored at zero"].join('\n').replace("'", "'\\''") + target_label = "${target.label}".replace("'", "'\\''") """ mkdir -p published @@ -116,7 +118,9 @@ process Analyze { Rscript --vanilla published/mds.R --design design.json --pools pools.json \\ --options options.json --cpp-frequencies published/allele_frequencies.cpp \\ --cpp-distance published/nei_distance.cpp \\ - --depths '${depths.collect { path -> path.name }.join(',')}' --out published + --depths '${depths.collect { path -> path.name }.join(',')}' --out published \\ + > notes.txt + awk -v label='${target_label}' '{ print "MDS " label ": " \$0 }' notes.txt """ } diff --git a/modules/mds/manifest.json b/modules/mds/manifest.json index a2c83fe..bb38092 100644 --- a/modules/mds/manifest.json +++ b/modules/mds/manifest.json @@ -1,6 +1,6 @@ { "name": "mds", - "version": "20261004.001", + "version": "20261006.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -20,12 +20,13 @@ "IT IS NOT FLOORED AT ZERO AND SMALL NEGATIVE DISTANCES ARE CORRECT. The correction subtracts each pool's own sampling noise, so two pools drawn from one population estimate a true distance of zero and land either side of it. A negative entry in distance.tsv means the pools are indistinguishable at this depth, not that something failed.", "The correction is what keeps DEPTH out of the picture. Uncorrected, two shallow pools drawn from one population come out further apart than two deep ones and the first axis is depth rather than biology; the raw column beside each distance is the uncorrected sum, so how much was removed is readable rather than assumed.", "Every pair is averaged over ITS OWN sites, and distance.tsv reports that count per pair. A pool with no reads at a site drops it for that pool's pairs and leaves the others intact, so two distances in one run can rest on different numbers of sites.", + "A PAIR AVERAGED OVER FEWER THAN 30 SITES IS FLAGGED: few_sites in distance.tsv, a line on the console of the run, and a count under the plot. Its distance is rough rather than wrong. Simulated at a distance near 0.02, the error of one distance was about half the distance at 30 shared sites and a quarter at 100, and real sites, linked along a genome, are noisier than those were. A pool read at few sites can land at the edge of the plot for that reason alone.", "A site where a pool has one read is dropped for that pool's pairs. Effective sample size is exactly 1 at depth 1 whatever the pool holds, and one gene copy carries no within-pool diversity for the correction to work from.", "THE ORDINATION PLACES POOLS, NOT UNITS. Two pools of one biological unit landing apart is what an ordination is read to see, so nothing is collapsed first; mds.tsv carries each pool's unit so a reader can tell which points should have coincided.", "Classical scaling assumes the distances fit a flat space and these need not. Negative eigenvalues are published in full rather than dropped, and every share is reported against both denominators in use - the signed sum, which can put two axes above 100%, and the sum of absolute values, which cannot. The plot's axis labels use the second.", "An axis has a direction and not a sign, so each is turned to make its largest coordinate positive. Without that the same data plots mirrored on two machines and reads as though it changed.", "AT SIX POOLS AN ORDINATION IS SIX POINTS AND WILL ALWAYS SHOW STRUCTURE. Nothing here says whether the arrangement is more than sampling noise; a block bootstrap over linkage blocks with Procrustes alignment is what would give a confidence region and it is not in this release.", - "Computed from the DEPTH table alone. Every published frequency is a six-significant-digit rendering of a ratio the depth table holds exactly, and a cell with no reads is published there as 0 rather than as missing.", + "Computed from the DEPTH table alone. Every published frequency is a six-significant-digit rendering of a ratio the depth table holds exactly.", "Read over the SNP tables alone unless analysis.modules.mds.includeIndels says otherwise, matching how every other module in this frame counts indels apart.", "A spanning-deletion allele reaching the SNP table as a * or an X is one allele of that site like any other: it carries reads, so it carries frequency, and dropping it would renormalize the site to something no pool was sequenced at.", "Pool sizes and ploidy come from the project's metadata as the frame resolved them, not from anything this module infers. Ploidy enters once, through n_chrom = ploidy * poolSize inside n_eff, so nothing here is limited to diploids. A pool of one chromosome is refused: the correction has nothing to divide by at any site.", @@ -40,7 +41,7 @@ }, { "file": "distance.tsv", - "summary": "one row per pair of pools: the corrected distance, the uncorrected sum beside it, what the correction removed, and how many sites the pair was averaged over", + "summary": "one row per pair of pools: the corrected distance, the uncorrected sum beside it, what the correction removed, how many sites the pair was averaged over, and whether that was fewer than 30", "anchor": "mds-distance" }, { @@ -50,7 +51,7 @@ }, { "file": "mds.png", - "summary": "the pools on the leading two axes, labeled and optionally colored by an experimental variable", + "summary": "the pools on the leading two axes, labeled and optionally colored by an experimental variable, and under it a count of the pairs averaged over fewer than 30 sites when there are any", "anchor": "mds-plot" }, { diff --git a/modules/mds/mds.R b/modules/mds/mds.R index b224d0c..c300b14 100644 --- a/modules/mds/mds.R +++ b/modules/mds/mds.R @@ -330,17 +330,34 @@ mds_table <- cbind( write.table(mds_table, file.path(out, "mds.tsv"), sep = "\t", quote = FALSE, row.names = FALSE, na = "") +# A pair averaged over fewer sites than this is flagged in distance.tsv, named on stdout, and +# counted under the plot. +FEW_SITES <- 30L + pairs <- which(upper.tri(corrected), arr.ind = TRUE) distance_table <- data.frame( pool_a = pool_order[pairs[, 1]], pool_b = pool_order[pairs[, 2]], sites = accumulated$sites[pairs], distance = corrected[pairs], raw = raw[pairs], correction = raw[pairs] - corrected[pairs], + few_sites = as.integer(accumulated$sites[pairs] < FEW_SITES), stringsAsFactors = FALSE) write.table(distance_table[order(distance_table$pool_a, distance_table$pool_b), ], file.path(out, "distance.tsv"), sep = "\t", quote = FALSE, row.names = FALSE, na = "") +# Written to stdout, which main.nf prints under the target's name. Nextflow shows nothing a +# task that succeeds wrote to stderr. +few <- distance_table[distance_table$few_sites == 1, , drop = FALSE] +if (nrow(few) > 0) { + cat(sprintf(paste0("%d of %d pairs of pools rest on fewer than %d shared sites, and their ", + "distances are rough. few_sites in distance.tsv marks them:\n"), + nrow(few), nrow(distance_table), FEW_SITES)) + shown <- head(few[order(few$sites, few$pool_a, few$pool_b), , drop = FALSE], 10) + cat(sprintf(" %s and %s: %d sites\n", shown$pool_a, shown$pool_b, shown$sites), sep = "") + if (nrow(few) > nrow(shown)) cat(sprintf(" and %d more\n", nrow(few) - nrow(shown))) +} + eigen_table <- data.frame( axis = seq_along(placed$values), eigenvalue = placed$values, share = shares$signed, share_absolute = shares$absolute, @@ -385,5 +402,10 @@ if (ncol(placed$coords) >= 2) { figure <- figure + ggplot2::labs(shape = shape_by) + ggplot2::scale_shape_manual(values = SHAPE_VALUES[seq_len(shape_levels)]) } + if (nrow(few) > 0) { + figure <- figure + ggplot2::labs(caption = sprintf( + "%d of %d pairs rest on fewer than %d shared sites: few_sites in distance.tsv", + nrow(few), nrow(distance_table), FEW_SITES)) + } ggplot2::ggsave(file.path(out, "mds.png"), figure, width = 7, height = 6, dpi = 150) } diff --git a/modules/mds/test/mds.sh b/modules/mds/test/mds.sh index 19cced9..8a93f8d 100644 --- a/modules/mds/test/mds.sh +++ b/modules/mds/test/mds.sh @@ -484,6 +484,78 @@ test_two_identical_pools_come_out_negative() { "and it is exactly minus the correction" } +# A cohort of the corpus's six pools over forty sites into $1, every cell read except the sixth +# pool's at the first $2 sites, so every pair with the sixth pool rests on 40 - $2 sites and every +# other pair on 40. The counts vary by site and pool so that no distance is degenerate. +mds_thin_cohort() { + mkdir -p "$1/Frequencies" + cp "$CORPUS_DIR/design.json" "$CORPUS_DIR/pools.json" "$1/" + python3 - "$1/Frequencies/Test_snp_depth.tsv" "$2" <<'PY' +import sys +path, emptied = sys.argv[1], int(sys.argv[2]) +pools = ["TestSample%d" % n for n in range(1, 7)] +rows = ["\t".join(["CHROM", "POS", "REF", "ALT", "TOTAL_AD"] + pools)] +for site in range(40): + cells = [] + for pool in range(6): + ref, alt = 50 + (site * 7 + pool * 13) % 40, 30 + (site * 11 + pool * 5) % 50 + if pool == 5 and site < emptied: + ref, alt = 0, 0 + cells.append((ref, alt)) + total = "%d,%d" % (sum(c[0] for c in cells), sum(c[1] for c in cells)) + rows.append("\t".join(["chr1", str(100 + site), "A", "G", total] + + ["%d,%d" % cell for cell in cells])) +open(path, "w").write("\n".join(rows) + "\n") +PY +} + +# A PAIR AVERAGED OVER FEWER THAN 30 SITES IS FLAGGED, and one over 30 is not. Z set the warning +# at 30 on 2026-10-06, from the validation harness's measurement that a distance near 0.02 is off +# by about half itself at 30 shared sites. The boundary is tested on both sides, 29 against 30, +# because the comparison is the whole rule: `<=` for `<` would flag the second run, and a flag +# column that never says 1 would pass every other case in this suite, where the corpus's fourteen +# sites put every pair under the line. +test_a_pair_on_few_sites_is_flagged() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/mds-few-sites") + mds_corpus "$sb" + + mds_thin_cohort "$sb/thin" 11 + mds_on_cohort "$sb/thin" "$MDS_OPTIONS" "$sb/run29" + local table="$sb/run29/distance.tsv" pool + if [ ! -s "$table" ]; then + fail_case "nothing published"$'\n'"$(cat "$sb/run29/out.txt")" + return + fi + for pool in TestSample1 TestSample2 TestSample3 TestSample4 TestSample5; do + assert_eq "29" "$(pair_cell "$table" "$pool" TestSample6 sites)" \ + "$pool and TestSample6 share the 29 sites both were read at" + assert_eq "1" "$(pair_cell "$table" "$pool" TestSample6 few_sites)" \ + "so $pool and TestSample6 are flagged" + done + assert_eq "40" "$(pair_cell "$table" TestSample1 TestSample2 sites)" \ + "TestSample1 and TestSample2 share all forty" + assert_eq "0" "$(pair_cell "$table" TestSample1 TestSample2 few_sites)" \ + "so they are not" + assert_contains "$(cat "$sb/run29/out.txt")" \ + "5 of 15 pairs of pools rest on fewer than 30 shared sites" \ + "the run must say how many pairs are flagged" + assert_contains "$(cat "$sb/run29/out.txt")" "TestSample1 and TestSample6: 29 sites" \ + "and name them" + + mds_thin_cohort "$sb/edge" 10 + mds_on_cohort "$sb/edge" "$MDS_OPTIONS" "$sb/run30" + table="$sb/run30/distance.tsv" + assert_eq "30" "$(pair_cell "$table" TestSample1 TestSample6 sites)" \ + "at ten sites emptied the sixth pool's pairs rest on thirty" + local flagged + flagged=$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) if ($i == "few_sites") c = i; next } + { n += $c } END { print n + 0 }' "$table") + assert_eq "0" "$flagged" "and thirty is not fewer than thirty: no pair is flagged" + assert_not_contains "$(cat "$sb/run30/out.txt")" "rest on fewer than" \ + "and the run says nothing" +} + # AND ONCE THROUGH NEXTFLOW, which is what proves main.nf assembles what the direct cases check. test_mds_runs_through_the_frame() { analysis_ready single || return @@ -504,4 +576,10 @@ test_mds_runs_through_the_frame() { "the shared library must be folded into the published script" assert_contains "$(cat "$dir/mds.R")" "not floored at zero" \ "and the header must say what a negative distance means" + # The corpus's fourteen sites put every pair under 30. A task that succeeds shows nothing on + # the console unless its process sets debug, and then only its stdout, so this is the line + # that proves the warning reaches the person running it. + assert_contains "$(cat "$ANALYSIS_SB/run.out")" \ + "15 of 15 pairs of pools rest on fewer than 30 shared sites" \ + "the warning on thin pairs must reach the console of the run" } From e87e742f8f45582aa0906d3bb350c40c56725fa3 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Tue, 6 Oct 2026 18:52:51 -0400 Subject: [PATCH 3/5] factorial calculation fixed --- modules/association/association.R | 179 ++++++++-- modules/association/main.nf | 3 +- modules/association/manifest.json | 12 +- modules/association/test/association.sh | 439 ++++++++++++++++++++++-- 4 files changed, 568 insertions(+), 65 deletions(-) diff --git a/modules/association/association.R b/modules/association/association.R index 9eb2ead..f3fb250 100644 --- a/modules/association/association.R +++ b/modules/association/association.R @@ -87,6 +87,10 @@ for (needed in c("permutations", "fdr", "reportBelow", "reportTop", "binSize", " # Collapse each unit's pools onto one weighted value: the weights summed, the frequency their # weighted mean. Where a unit holds one pool this returns that pool unchanged, so a design with # no biological replication takes the same path and gets the same answer. +# +# Over the pools that were READ at the site. An unread pool has no frequency and an n_eff of NA, +# so it drops out of both sums and the unit is the weighted mean of the rest. A unit none of +# whose pools were read has no weight and no frequency there: both are NA. roll_up <- function(freq, weight, site, groups) { held <- matrix(0, nrow = nrow(weight), ncol = length(groups), dimnames = list(NULL, names(groups))) @@ -94,9 +98,11 @@ roll_up <- function(freq, weight, site, groups) { dimnames = list(NULL, names(groups))) for (unit in seq_along(groups)) { mine <- weight[, groups[[unit]], drop = FALSE] - held[, unit] <- rowSums(mine) + held[, unit] <- rowSums(mine, na.rm = TRUE) + held[rowSums(!is.na(mine)) == 0, unit] <- NA_real_ value[, unit] <- rowSums(mine[site, , drop = FALSE] * - freq[, groups[[unit]], drop = FALSE]) / held[site, unit] + freq[, groups[[unit]], drop = FALSE], na.rm = TRUE) / + held[site, unit] } value[!is.finite(value)] <- NA_real_ list(weight = held, freq = value) @@ -162,13 +168,24 @@ site_statistic <- function(t, site, sites) { # does not shrink with depth, so the truth is Var(f) = p(1-p) * (theta + 1/n_eff). This is theta # by method of moments: the scatter the units actually show, less the sampling variance the # weights predict. Clamped at zero. +# +# Each site over the units it was read in, and a site read in fewer than two has no scatter and +# takes no part. NA when no site has two. dispersion_of <- function(freq, weight, site) { - held <- ncol(freq) - center <- rowMeans(freq) - spread <- (rowSums(freq * freq) - held * center * center) / (held - 1) + read <- !is.na(weight) & weight > 0 + here <- read[site, , drop = FALSE] + count <- rowSums(here) + kept <- freq + kept[!here] <- 0 + center <- rowSums(kept) / count + spread <- (rowSums(kept * kept) - count * center * center) / (count - 1) scale <- center * (1 - center) - excess <- (spread - scale * rowMeans(1 / weight)[site]) / scale - max(0, mean(excess[is.finite(excess)])) + inverse <- 1 / weight + inverse[!read] <- 0 + excess <- (spread - scale * (rowSums(inverse) / rowSums(read))[site]) / scale + usable <- excess[is.finite(excess) & count >= 2] + if (length(usable) == 0) return(NA_real_) + max(0, mean(usable)) } # Every rearrangement of the units, as index rows, enumerated whole while it fits. @@ -206,6 +223,11 @@ rearrangements <- function(units, budget) { # and no permutation p can be zero. permutation_p <- function(freq, weight, site, sites, y, observed, budget) { moves <- rearrangements(ncol(weight), budget) + # A unit not read at a site is given weight 0 there and no frequency, exactly as + # fit_alleles() gives it, so it has no residual and the center is the weighted mean of the + # units read. + weight[is.na(weight) | weight <= 0] <- 0 + freq[is.na(freq) | weight[site, , drop = FALSE] <= 0] <- 0 root <- sqrt(weight) center <- rowSums(weight[site, , drop = FALSE] * freq) / rowSums(weight)[site] z <- (freq - center) * root[site, , drop = FALSE] @@ -214,18 +236,48 @@ permutation_p <- function(freq, weight, site, sites, y, observed, budget) { # to the null: a bin holds a contiguous run of sites and every bin sees the same # rearrangements in the same order. ranges <- chunk_ranges(sites, OPTS$binSize) + units <- seq_len(ncol(weight)) tally <- function(range) { rows <- which(site >= range[1] & site <= range[2]) held <- weight[range[1]:range[2], , drop = FALSE] index <- site[rows] - range[1] + 1L here <- range[2] - range[1] + 1L seen <- observed[range[1]:range[2]] + # The observed statistic less rounding, absolute below 1 and relative above it: rebuilt, + # the identity itself lands up to 4e-14 of S away from S, which at S of 400 is past 1e-12. + reach <- pmin(seen - 1e-12, seen * (1 - 1e-12)) found <- integer(here) + + # EACH SITE IS REARRANGED AMONG THE UNITS IT WAS READ IN, in the order they take inside + # the rearrangement of all of them: uniform when that one is, and each order met equally + # often across the enumerated set. A unit not read keeps its own zero residual in place. + pattern <- apply(held > 0, 1, function(m) paste(which(m), collapse = ",")) + groups <- lapply(split(seq_along(rows), pattern[index]), function(at) { + mine <- which(held[index[at[1]], ] > 0) + list(at = at, mine = mine, all = length(mine) == length(units)) + }) + whole <- length(groups) == 1 && groups[[1]]$all + for (row in seq_len(nrow(moves$rows))) { - rebuilt <- center[rows] + z[rows, moves$rows[row, ], drop = FALSE] / - root[site[rows], , drop = FALSE] + move <- moves$rows[row, ] + if (whole) { + rebuilt <- center[rows] + z[rows, move, drop = FALSE] / + root[site[rows], , drop = FALSE] + } else { + rebuilt <- matrix(0, nrow = length(rows), ncol = length(units)) + for (group in groups) { + taken <- move + if (!group$all) { + taken <- units + taken[group$mine] <- move[move %in% group$mine] + } + at <- group$at + rebuilt[at, ] <- center[rows[at]] + z[rows[at], taken, drop = FALSE] / + root[site[rows[at]], , drop = FALSE] + } + } under <- site_statistic(fit_alleles(rebuilt, held, index, y)$t, index, here) - found <- found + (!is.na(under) & under >= seen - 1e-12) + found <- found + (!is.na(under) & under >= reach) } found } @@ -329,16 +381,35 @@ read_depth_table <- function(path) { table } -# The largest weighted leverage at a site, over units. It costs one pass and it is the -# diagnostic that says whether one unit is carrying the slope by itself, which at six of them -# is the question a reader has. +# Whether the phenotype varies over the units read at each site. Compared on the stored values: +# a weighted mean of one repeated value need not return it exactly, and a sum of squares about +# that mean then finds a spread of 1e-28 where there is none. +spread_of <- function(weight, y) { + read <- !is.na(weight) & weight > 0 + phen <- matrix(y, nrow = nrow(weight), ncol = ncol(weight), byrow = TRUE) + high <- phen + high[!read] <- -Inf + low <- phen + low[!read] <- Inf + columns <- seq_len(ncol(phen)) + do.call(pmax, lapply(columns, function(j) high[, j])) > + do.call(pmin, lapply(columns, function(j) low[, j])) +} + +# The largest weighted leverage at a site, over the units read there. It costs one pass and it +# is the diagnostic that says whether one unit is carrying the slope by itself, which at six of +# them is the question a reader has. NA where the phenotype does not vary over those units. top_leverage <- function(weight, y) { + weight[is.na(weight)] <- 0 total <- rowSums(weight) phen <- matrix(y, nrow = nrow(weight), ncol = ncol(weight), byrow = TRUE) centered <- phen - rowSums(weight * phen) / total sxx <- rowSums(weight * centered * centered) held <- weight * (1 / total + centered * centered / sxx) - do.call(pmax, c(lapply(seq_len(ncol(held)), function(j) held[, j]), list(na.rm = TRUE))) + out <- do.call(pmax, c(lapply(seq_len(ncol(held)), function(j) held[, j]), + list(na.rm = TRUE))) + out[!spread_of(weight, y)] <- NA_real_ + out } n_chrom <- vapply(pool_order, function(name) { @@ -383,14 +454,26 @@ for (entry in design$phenotypes[declared %in% chosen]) { # analysis.modules.association.dispersion to 0 recovers the plain n_eff weights. theta <- if (is.null(OPTS$dispersion)) dispersion_of(held$freq, held$weight, parsed$site) else as.numeric(OPTS$dispersion) - held$weight <- 1 / (theta + 1 / held$weight) + # NA when no site was read in two units, and the weights are then left as they are. + if (!is.na(theta)) held$weight <- 1 / (theta + 1 / held$weight) fit <- fit_alleles(held$freq, held$weight, parsed$site, y) observed <- site_statistic(fit$t, parsed$site, sites) + # A site read in fewer than three units leaves no residual degree of freedom, and one + # whose read units all carry one phenotype value has no slope: neither is tested. Each + # keeps its row and its n_observed. + varied <- spread_of(held$weight, y) + untestable <- fit$observed < 3 | !varied + observed[untestable] <- NA_real_ shuffled <- permutation_p(held$freq, held$weight, parsed$site, sites, y, observed, budget) tested <- sum(!is.na(observed)) + if (tested == 0) { + message("association.R: no site of ", entry$column, " (", kind, ") was tested: none ", + "was read in three units that differ in the phenotype. Every row is still ", + "published, and n_observed says how many units each was read in.") + } adjusted <- rep(NA_real_, sites) adjusted[!is.na(observed)] <- p.adjust(shuffled$p[!is.na(observed)], method = method, n = tested) @@ -401,38 +484,55 @@ for (entry in design$phenotypes[declared %in% chosen]) { # zero or on 1e-18 decides between t = Inf and t = 8e15. spent <- !is.finite(fit$t) | fit$variance <= 0 | fit$exhausted flagged <- as.integer(tapply(spent, parsed$site, any)) + flagged[!varied] <- NA_integer_ site_rows[[length(site_rows) + 1]] <- data.frame( phenotype = entry$column, kind = kind, chrom = table$CHROM, pos = as.integer(table$POS), k = parsed$alleles, n_observed = fit$observed, n_units = length(unit_pools), S = observed, perm_p = shuffled$p, fdr_p = adjusted, - mean_weight = rowMeans(held$weight), max_leverage = top_leverage(held$weight, y), + mean_weight = rowMeans(held$weight, na.rm = TRUE), + max_leverage = top_leverage(held$weight, y), zero_variance = flagged, stringsAsFactors = FALSE, check.names = FALSE) alleles <- unlist(lapply(strsplit(paste(table$REF, table$ALT, sep = ","), ",", fixed = TRUE), identity), use.names = FALSE) + # The alleles of a site with no test carry no statistic either. + untested <- untestable[parsed$site] + slope <- fit$b1 + error <- fit$se + ratio <- fit$t + slope[untested] <- NA_real_ + error[untested] <- NA_real_ + ratio[untested] <- NA_real_ + closed <- 2 * pt(-abs(ratio), fit$df[parsed$site]) allele_rows[[length(allele_rows) + 1]] <- data.frame( phenotype = entry$column, kind = kind, chrom = table$CHROM[parsed$site], pos = as.integer(table$POS)[parsed$site], allele = alleles, - b1 = fit$b1, se = fit$se, t = fit$t, - p = 2 * pt(-abs(fit$t), fit$df[parsed$site]), + b1 = slope, se = error, t = ratio, p = closed, stringsAsFactors = FALSE, check.names = FALSE) # `design_floor` is what the design supports by rearrangement alone and `floor` what this # run could reach. They differ only when the set was too large to enumerate. - # THE SMALLEST P THIS DESIGN CAN REACH BY REARRANGEMENT AT ALL. Reversing the phenotype - # negates every slope and leaves |t| alone, so the reversal always ties with the observed - # arrangement and the floor is TWO over the count, never one. - limit <- 2 / factorial(length(unit_pools)) + # THE SMALLEST P THIS DESIGN CAN REACH BY REARRANGEMENT AT ALL: one over the count. + limit <- 1 / factorial(length(unit_pools)) if (limit > 0.05) { message("association.R: ", length(unit_pools), " units allow ", factorial(length(unit_pools)), " rearrangements, so the smallest p any site ", "can reach is ", signif(limit, 3), ". Nothing here can be significant at ", "0.05 and the table is a RANKING of effect sizes, not a test. More units is ", "the only thing that changes it.") + } else { + short <- sum(!is.na(observed) & 1 / factorial(fit$observed) > 0.05) + if (short > 0) { + message("association.R: ", short, " tested site(s) of ", entry$column, " (", + kind, ") were read in too few units to reach 0.05. A site read in m ", + "units is rearranged among those m alone and has the floor of an m-unit ", + "design, 1/m!; n_observed in association.tsv says which sites these are. ", + "Their p ranks them and cannot make them significant.") + } } alive <- !is.na(observed) @@ -442,7 +542,8 @@ for (entry in design$phenotypes[declared %in% chosen]) { units = length(unit_pools), permutations = shuffled$count, exhaustive = shuffled$exhaustive, floor = shuffled$floor, design_floor = limit, dispersion = theta, - depth_phenotype_cor = suppressWarnings(cor(colMeans(held$weight), y)), + depth_phenotype_cor = suppressWarnings(cor(colMeans(held$weight, na.rm = TRUE), y, + use = "pairwise.complete.obs")), lambda_gc = if (any(alive)) { median(observed[alive]^2, na.rm = TRUE) / qchisq(0.5, 1) } else NA_real_, @@ -512,7 +613,9 @@ if (!drawable) { "they would have shown is in association.tsv and permutations.tsv.") } -if (drawable && any(!is.na(sites_table$perm_p))) { +# Drawn when no site was tested as well: qq.png is a declared output, and the frame refuses a +# folder that lacks one. +if (drawable) { frame <- do.call(rbind, lapply(split(sites_table, sites_table$phenotype), function(part) { seen <- sort(part$perm_p[!is.na(part$perm_p)]) if (length(seen) == 0) return(NULL) @@ -520,17 +623,25 @@ if (drawable && any(!is.na(sites_table$perm_p))) { expected = -log10(ppoints(length(seen))), observed = -log10(seen), stringsAsFactors = FALSE) })) - figure <- ggplot2::ggplot(frame, ggplot2::aes(x = expected, y = observed)) + - ggplot2::geom_abline(slope = 1, intercept = 0, color = "gray60") + - ggplot2::geom_point(size = 0.5, alpha = 0.6) + - ggplot2::facet_wrap(~ phenotype, ncol = 1) + - ggplot2::labs(title = "Permutation p-values against the uniform they should follow", - subtitle = paste0("A permutation p is discrete, so the points sit on a ", - "ladder; the ceiling is the design's own floor."), - x = "Expected -log10(p)", y = "Observed -log10(p)") + - ggplot2::theme_bw(base_size = 9) + figure <- if (is.null(frame)) { + ggplot2::ggplot() + + ggplot2::labs(title = "Permutation p-values against the uniform they should follow", + subtitle = paste0("No site was tested: none was read in three units ", + "that differ in the phenotype.")) + + ggplot2::theme_bw(base_size = 9) + } else { + ggplot2::ggplot(frame, ggplot2::aes(x = expected, y = observed)) + + ggplot2::geom_abline(slope = 1, intercept = 0, color = "gray60") + + ggplot2::geom_point(size = 0.5, alpha = 0.6) + + ggplot2::facet_wrap(~ phenotype, ncol = 1) + + ggplot2::labs(title = "Permutation p-values against the uniform they should follow", + subtitle = paste0("A permutation p is discrete, so the points sit on ", + "a ladder; the ceiling is the design's own floor."), + x = "Expected -log10(p)", y = "Observed -log10(p)") + + ggplot2::theme_bw(base_size = 9) + } ggplot2::ggsave(file.path(out, "qq.png"), figure, width = 6, - height = 2.6 * length(unique(frame$phenotype)) + 0.6, dpi = 150) + height = 2.6 * max(1, length(unique(frame$phenotype))) + 0.6, dpi = 150) } # Nothing is plotted along a sequence until one is named in the module's `chromosomes` setting. diff --git a/modules/association/main.nf b/modules/association/main.nf index fc10310..95907ba 100644 --- a/modules/association/main.nf +++ b/modules/association/main.nf @@ -40,8 +40,7 @@ def useCompiled(Map settings) { // The depth tables of one results directory, by the artifact class the manifest asked for. // // Every published frequency is a six-significant-digit rendering of a ratio these hold exactly, -// and a cell with no reads is published there as 0 rather than as missing, so this module reads -// the depth tables and nothing else. +// so this module reads the depth tables and nothing else. def depthTables(Map target) { def spec = target.classes.depths def found = file("${spec.dir}/${spec.pattern}") diff --git a/modules/association/manifest.json b/modules/association/manifest.json index 0f97d3d..9657bdc 100644 --- a/modules/association/manifest.json +++ b/modules/association/manifest.json @@ -1,6 +1,6 @@ { "name": "association", - "version": "20261004.001", + "version": "20261005.001", "contract": "freq-1", "license": "GPL-3.0-or-later", "frame": "20260910.001", @@ -16,13 +16,13 @@ ], "gates": [ "THIS IS A MODEL-BASED TEST AND NOT AN ASSUMPTION-LIGHT ONE. It asserts that the variance of a pool's frequency is p(1-p) times (dispersion + 1/n_eff), that units are independent, and that a site's alleles are several views of one comparison. Where those do not describe your experiment the numbers will still be produced and will still be wrong; the diagnostics in permutations.tsv are what say so.", - "The fit is on UNITS, not pools. Each unit's pools are collapsed onto one value first, weighted by their effective sizes, and the degrees of freedom are the units carrying data at that site minus two. Where a unit holds one pool the collapse changes nothing.", - "The published p is a permutation p, and the permutation moves the STANDARDISED RESIDUALS rather than the phenotype labels. Units read at different depths are not interchangeable however correctly they are weighted, so permuting labels across them is invalid; measured at twice its nominal rate for a design whose depths line up with the phenotype.", + "The fit is on UNITS, not pools. Each unit's pools are collapsed onto one value first, over the pools read at that site and weighted by their effective sizes, and the degrees of freedom are the units carrying data at that site minus two. A site carrying data in fewer than three units, or in units that all carry one value of the phenotype, is not tested: its row is published with n_observed and no statistic. Where a unit holds one pool the collapse changes nothing.", + "The published p is a permutation p, and the permutation moves the STANDARDIZED RESIDUALS rather than the phenotype labels. Units read at different depths are not interchangeable however correctly they are weighted, so permuting labels across them is invalid; measured at twice its nominal rate for a design whose depths line up with the phenotype. A site is rearranged among the units it was read in, so a site read in m of them is tested as an m-unit design would be.", "The permutation set is enumerated whole when it fits inside analysis.modules.association.permutations, and sampled above it as (1 + reached) / (1 + draws). Sampling a set small enough to enumerate can return a p below the smallest value the design supports, so this is a correctness requirement rather than a speed one.", - "THE SMALLEST P ANY DESIGN CAN REACH IS 2 / (units factorial), because reversing the phenotype negates every slope and leaves the statistic alone. Four units put that at 0.083, so nothing in such a run can be significant at 0.05 and the table is a ranking of effect sizes rather than a test. permutations.tsv reports it as design_floor.", - "Weights are n_eff at that pool's own depth plus the excess dispersion uneven pooling leaves behind, estimated across sites unless analysis.modules.association.dispersion sets it. Only the RATIO between units is ever a claim: multiplying every weight by a constant leaves the statistic unchanged, so a pool size wrong by the same factor everywhere costs nothing and one wrong differently per pool costs calibration.", + "THE SMALLEST P ANY DESIGN CAN REACH IS 1 / (units factorial), because the observed arrangement is always among those counted. Three units put that at 0.17, so nothing in such a run can be significant at 0.05 and the table is a ranking of effect sizes rather than a test; four put it at 0.042. A site read in fewer units than the design holds has the floor of the units it was read in. Ties keep a site above the floor: where every unit carries the same weight, a phenotype symmetric about its mean ties with its own reversal, and a categorical one with every reshuffle inside a level and with the swap of its levels when they are the same size, so three against three at equal depths cannot go below 0.1. permutations.tsv reports the design's floor as design_floor.", + "Weights are n_eff at that pool's own depth plus the excess dispersion uneven pooling leaves behind, estimated across sites, each over the units it was read in, unless analysis.modules.association.dispersion sets it. Only the RATIO between units is ever a claim: multiplying every weight by a constant leaves the statistic unchanged, so a pool size wrong by the same factor everywhere costs nothing and one wrong differently per pool costs calibration.", "The site statistic is the largest |t| over EVERY allele including the reference. A site's frequencies sum to one, so the reference row carries the negated sum of the others and is where a signal spread across several alternates appears; dropping it costs 40% of the power at such a site and corrects nothing.", - "Computed from the DEPTH table alone. Every published frequency is a six-significant-digit rendering of a ratio the depth table holds exactly, and a cell with no reads is published there as 0 rather than as missing.", + "Computed from the DEPTH table alone. Every published frequency is a six-significant-digit rendering of a ratio the depth table holds exactly. A pool with no reads at a site has no frequency there and no weight either: it takes no part in that site's fit, its rearrangements or the averages published for it.", "A site where an allele has no residual left is flagged rather than read. Two alleles of one perfectly separated site are algebraically the same test, and whether each lands on exactly zero or on 1e-32 decides between an infinite t and a merely enormous one, so those numbers are not comparable between implementations and the flag says so.", "Pool sizes and ploidy come from the project's metadata as the frame resolved them, not from anything this module infers. Ploidy enters once, through n_chrom = ploidy * poolSize inside n_eff, so nothing here is limited to diploids.", "bin/cap_depth.awk keeps reads in coordinate order until the cap rather than subsampling, so at a capped position the retained reads are biased toward those mapping earliest. That is an allele-frequency bias and not only a depth truncation.", diff --git a/modules/association/test/association.sh b/modules/association/test/association.sh index 430c8d7..602e85c 100644 --- a/modules/association/test/association.sh +++ b/modules/association/test/association.sh @@ -5,6 +5,7 @@ # covers: modules/association/ modules/lib/ # covers: test/tools/freq_corpus.py # covers: analysis.nf modules/association/main.nf +# covers: dev/validation/agree.R dev/validation/lib.R modules/mds/mds.R bin/mask_depth.awk # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # @@ -44,6 +45,52 @@ association_direct() { --depths 'Test_snp_depth.tsv' --out "$dest" ) > "$dest/out.txt" 2>&1 } +# Empty one pool's cell at one site to the site's own arity, every count of it set to 0: a pool +# with no reads there, as step 7 publishes one. The pool is found by its header. +empty_cell() { + local table="$1" chrom="$2" pos="$3" pool="$4" + awk -F'\t' -v OFS='\t' -v c="$chrom" -v p="$pos" -v pool="$pool" ' + NR == 1 { + for (i = 1; i <= NF; i++) if ($i == pool) col = i + if (!col) { print "empty_cell: no column " pool > "/dev/stderr"; exit 2 } + print; next + } + $1 == c && $2 == p { + n = split($col, k, ","); zeros = "0" + for (i = 2; i <= n; i++) zeros = zeros ",0" + $col = zeros + } + { print }' "$table" > "$table.new" && mv "$table.new" "$table" +} + +# The corpus's six pools as units of the sizes given, in pool order, into $2: `2 2 2` is three +# units of two, a design with technical replication. A unit's pools take its first pool's +# phenotype, which the module otherwise refuses as a unit whose pools disagree. +grouped_design() { + local from="$1" into="$2"; shift 2 + python3 - "$from" "$into" "$@" <<'PY' +import json, sys +design = json.load(open(sys.argv[1])) +sizes = [int(size) for size in sys.argv[3:]] +pools = [entry["pool"] for entry in design["pools"]] +if sum(sizes) != len(pools): + sys.exit("grouped_design: %d pools and sizes %s" % (len(pools), sizes)) +starts = [sum(sizes[:u]) for u in range(len(sizes))] +design["units"] = [ + {"label": "U%d" % (u + 1), "key": {"exp_population": "Pop%d" % (u + 1)}, + "pools": pools[start:start + size], "members": pools[start:start + size]} + for u, (start, size) in enumerate(zip(starts, sizes))] +first = {i: start for start, size in zip(starts, sizes) for i in range(start, start + size)} +for entry in design["phenotypes"]: + for i, value in enumerate(entry["values"]): + value["value"] = entry["values"][first[i]]["value"] +json.dump(design, open(sys.argv[2], "w")) +PY +} + +# Three units of two. +paired_design() { grouped_design "$1" "$2" 2 2 2; } + # One cell of the site table, found by position rather than by column number. site_cell() { awk -F'\t' -v c="$2" -v p="$3" -v col="$4" ' @@ -51,6 +98,24 @@ site_cell() { $(h["chrom"]) == c && $(h["pos"]) == p { print $(h[col]); exit }' "$1" } +# One key of what freq_corpus.py --association-from wrote into $1. +oracle_value() { + awk -F'\t' -v k="$2" '$1 == k { print $2 }' "$1" +} + +# Every site row of the run in $1 against the oracle in $2, in each column named after them. +agrees_with_oracle() { + local run="$1" oracle="$2" chrom pos column; shift 2 + while IFS=$'\t' read -r chrom pos; do + for column in "$@"; do + assert_close "$(site_cell "$run/association.tsv" "$chrom" "$pos" "$column")" \ + "$(oracle_value "$oracle" "assoc.$chrom.$pos.$column")" \ + "$chrom:$pos: $column against the oracle" + done + done < <(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + { print $(h["chrom"]) "\t" $(h["pos"]) }' "$run/association.tsv") +} + # The options a case uses unless it is testing one of them. # # DISPERSION IS PINNED AT ZERO. The corpus is fourteen sites chosen by hand, so a theta estimated @@ -126,9 +191,9 @@ test_the_permutation_moves_residuals_and_not_labels() { # THE SET IS ENUMERATED WHILE IT FITS, AND THAT IS CORRECTNESS AND NOT SPEED. # # A sampled p is (1 + reached)/(1 + draws) and can land below the smallest value a design can -# support: four units allow 24 rearrangements and a floor of 2/24, and sampling 150 draws from -# them reaches under 0.05 by luck alone. Six units allow 720, so a budget above that must be -# spent enumerating and the reported count must be exactly 720. +# support: four units allow 24 rearrangements and a floor of 1/24, and 150 sampled draws can +# report 1/151. Six units allow 720, so a budget above that must be spent enumerating and the +# reported count must be exactly 720. test_a_rearrangement_set_that_fits_is_enumerated() { if ! have_analysis_r; then skip_case "no analysis environment"; return; fi local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-enumerate") @@ -140,19 +205,25 @@ test_a_rearrangement_set_that_fits_is_enumerated() { "and must say that it did" } -# The two floors, which are different numbers and are both owed to a reader, and the diagnostics -# beside them. +# The two floors, which part when the set is sampled and are both owed to a reader, and the +# diagnostics beside them. # # `floor` is the smallest p this RUN could report; `design_floor` is the smallest the DESIGN can -# reach by rearrangement at all - two over the units factorial, because reversing the phenotype -# negates every slope and leaves the statistic alone, so the reversal always ties. +# reach by rearrangement at all - one over the units factorial, the observed arrangement being +# the only one sure to tie with itself. Enumerated, the two are one number. design_floor was two +# over the units factorial until 2026-10-05, on the argument that reversing the phenotype always +# ties; it ties only where every unit carries the same weight and the phenotype is symmetric about +# its mean, and this corpus reaches 1/720. test_both_floors_and_the_diagnostics_are_published() { if ! have_analysis_r; then skip_case "no analysis environment"; return; fi local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-floors") association_corpus "$sb" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" assert_close "$(published_cell "$sb/run/permutations.tsv" pt_wingspan design_floor)" \ - "0.00277777777777778" "the design floor is 2/720 and not 1/720" + "0.00138888888888889" "the design floor is 1/720 and not 2/720" + assert_close "$(published_cell "$sb/run/permutations.tsv" pt_wingspan floor)" \ + "$(published_cell "$sb/run/permutations.tsv" pt_wingspan design_floor)" \ + "an enumerated run reaches the design's own floor" local column for column in dispersion depth_phenotype_cor lambda_gc arity_mean; do [ -n "$(published_cell "$sb/run/permutations.tsv" pt_wingspan "$column")" ] \ @@ -160,6 +231,37 @@ test_both_floors_and_the_diagnostics_are_published() { done } +# FOUR UNITS ARE A TEST AND THREE ARE A RANKING. The smallest p a design can reach is one over its +# units factorial: 1/24 at four, under 0.05, and 1/6 at three, which is not. So three are told +# the table ranks effect sizes and four are not. Under the two-over floor this replaced, four +# units sat at 0.083 and were told the same as three. +test_four_units_are_a_test_and_three_a_ranking() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-floor-units") + association_corpus "$sb" + grouped_design "$CORPUS_DIR/design.json" "$sb/four.json" 2 2 1 1 + paired_design "$CORPUS_DIR/design.json" "$sb/three.json" + association_direct "$sb/four" "$ASSOCIATION_OPTIONS" "$sb/four.json" + association_direct "$sb/three" "$ASSOCIATION_OPTIONS" "$sb/three.json" + local units + for units in four three; do + if [ ! -s "$sb/$units/permutations.tsv" ]; then + fail_case "the $units-unit run published nothing"$'\n'"$(cat "$sb/$units/out.txt")" + return + fi + done + assert_eq "4" "$(published_cell "$sb/four/permutations.tsv" pt_wingspan units)" \ + "the fixture must fit four units" + assert_close "$(published_cell "$sb/four/permutations.tsv" pt_wingspan design_floor)" \ + "0.0416666666666667" "four units: the design floor is 1/24" + grep -q "RANKING" "$sb/four/out.txt" \ + && fail_case "four units can reach 1/24, under 0.05, and were told they are a ranking" + assert_close "$(published_cell "$sb/three/permutations.tsv" pt_wingspan design_floor)" \ + "0.166666666666667" "three units: the design floor is 1/6" + grep -q "RANKING" "$sb/three/out.txt" \ + || fail_case "three units cannot reach 0.05 and must say the table is a ranking" +} + # A UNIT OF SEVERAL POOLS IS COLLAPSED BEFORE THE FIT, NOT AFTER. # # Re-reading a pool-level statistic against the unit count controls nothing - measured over- @@ -170,20 +272,7 @@ test_a_unit_of_several_pools_is_collapsed_before_the_fit() { if ! have_analysis_r; then skip_case "no analysis environment"; return; fi local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-rollup") association_corpus "$sb" - python3 - "$CORPUS_DIR/design.json" "$sb/paired.json" <<'PY' -import json, sys -design = json.load(open(sys.argv[1])) -pools = [entry["pool"] for entry in design["pools"]] -design["units"] = [ - {"label": "U%d" % (i + 1), "key": {"exp_population": "Pop%d" % (i + 1)}, - "pools": pools[2 * i:2 * i + 2], "members": pools[2 * i:2 * i + 2]} - for i in range(len(pools) // 2)] -# The phenotype has to agree inside a unit or the module refuses, which is the next case. -for entry in design["phenotypes"]: - for i, value in enumerate(entry["values"]): - value["value"] = entry["values"][2 * (i // 2)]["value"] -json.dump(design, open(sys.argv[2], "w")) -PY + paired_design "$CORPUS_DIR/design.json" "$sb/paired.json" association_direct "$sb/run" "$ASSOCIATION_OPTIONS" "$sb/paired.json" if [ ! -s "$sb/run/permutations.tsv" ]; then fail_case "the roll-up run published nothing"$'\n'"$(cat "$sb/run/out.txt")" @@ -195,6 +284,278 @@ PY "and permuted over three units, which is 6 rearrangements and not 720" } +# A SITE IS TESTED OVER THE UNITS IT WAS READ IN, and only if there are three. +# +# The defect this guards, measured before the fix: permutation_p() built its center from the raw +# weights, so one unread unit made every rearranged statistic NaN, the identity was never counted, +# and the site published perm_p 0 - exit 0, nothing said. chr1:250 went from 0.4 to 0, and the BH +# adjustment pulled sites with no unread unit down with it. +# +# The emptied sites are read in 5, 5, 4, 3, 2, 1 and 1 of the six pools; chr1:100 is the +# control, read in all six. The module rearranges a partly read site through the order its read +# units take inside each rearrangement of all six; freq_corpus.py --association-from enumerates +# the read units' own m! directly. Agreement between the two is the point. chr1:250 and chr2:250 +# are read in five DIFFERENT units, because the module groups sites by which units read them, and +# grouped by how many it would rearrange one of them over the other's units. +# +# The floor is 1/m!, not 2/m!. Only the identity is sure to tie the observed statistic; the +# reversal ties too only where every unit carries the same weight and the phenotype is symmetric +# about its mean. So chr2:100, read in 3, can never reach 0.05 - 3! is 6 - while chr1:700, read in +# 4, reaches 1/24 here, below it; the run says how many tested sites are like chr2:100. +# +# AN UNREAD UNIT HAS NO WEIGHT, NOT A WEIGHT OF 0. mean_weight and max_leverage are over the +# units read at the site, and depth_phenotype_cor over each unit's own read sites, all three +# against the oracle. The fix's first version let na.rm in roll_up() turn an unread unit into +# weight 0, and mean_weight at chr10:500 then averaged four zeros in. A site read in one unit has +# no slope for a unit to carry, so its max_leverage is NA: chr2:400 is read only in TestSample5, +# where the weighted mean of its own wingspan misses 11.2 by 2e-15 and a test on the sum of +# squares published 2; chr1:400, read only in TestSample6, cancels exactly and could not see it. +# +# fdr_p is BH over the TESTED sites, the oracle's own adjustment over its own p-values. +test_a_site_is_tested_over_the_units_it_was_read_in() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-unread") + association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" pool + empty_cell "$table" chr1 250 TestSample6 + empty_cell "$table" chr2 250 TestSample1 + for pool in TestSample1 TestSample2; do empty_cell "$table" chr1 700 "$pool"; done + for pool in TestSample1 TestSample2 TestSample3; do empty_cell "$table" chr2 100 "$pool"; done + for pool in TestSample1 TestSample2 TestSample3 TestSample4; do + empty_cell "$table" chr10 500 "$pool" + done + for pool in TestSample1 TestSample2 TestSample3 TestSample4 TestSample5; do + empty_cell "$table" chr1 400 "$pool" + done + for pool in TestSample1 TestSample2 TestSample3 TestSample4 TestSample6; do + empty_cell "$table" chr2 400 "$pool" + done + python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" + + association_direct "$sb/run" "$ASSOCIATION_OPTIONS" + if [ ! -s "$sb/run/association.tsv" ]; then + fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + + local site chrom pos read + for site in chr1:100:6 chr1:250:5 chr2:250:5 chr1:700:4 chr2:100:3 chr10:500:2 chr1:400:1 \ + chr2:400:1; do + IFS=: read -r chrom pos read <<< "$site" + assert_eq "$read" "$(site_cell "$sb/run/association.tsv" "$chrom" "$pos" n_observed)" \ + "$chrom:$pos is read in $read units" + done + agrees_with_oracle "$sb/run" "$sb/oracle.tsv" S perm_p fdr_p mean_weight max_leverage + assert_eq "NA" "$(site_cell "$sb/run/association.tsv" chr10 500 fdr_p)" \ + "a site read in 2 units is left out of the adjustment as well as the test" + assert_close "$(published_cell "$sb/run/permutations.tsv" pt_wingspan depth_phenotype_cor)" \ + "$(oracle_value "$sb/oracle.tsv" assoc.depth_phenotype_cor)" \ + "depth_phenotype_cor over each unit's mean at the sites it was read at" + grep -q "1 tested site(s) of pt_wingspan (snp) were read in too few units to reach 0.05" \ + "$sb/run/out.txt" \ + || fail_case "the run must say that one tested site, chr2:100, cannot reach 0.05" + grep -qi "warning" "$sb/run/out.txt" \ + && fail_case "the run warned:"$'\n'"$(grep -i -A2 warning "$sb/run/out.txt")" + + # The defect's own signature, over every row: an enumerated p is never 0, because the + # identity is one of the rearrangements counted. + assert_eq "" "$(awk -F'\t' 'NR == 1 { for (i = 1; i <= NF; i++) h[$i] = i; next } + $(h["perm_p"]) == "0" { print $(h["chrom"]) ":" $(h["pos"]) }' \ + "$sb/run/association.tsv")" \ + "no site may publish a permutation p of 0" +} + +# A UNIT KEEPS THE POOLS OF IT THAT WERE READ. Two pools of one unit are the same material read +# twice, so where one has no reads the unit is the other; roll_up() used to sum the unit's +# weights with no na.rm, which dropped the whole unit wherever either pool was unread. +# +# chr1:700 in the paired design, U1's second pool emptied: U1 must still be read, so the site +# stays read in all three units and is tested. Dropped, it would be read in two and untested. +# chr1:100 loses U3's second pool the same way, and chr2:550 both of U2's, which leaves U2 unread +# there and the site untested. Every site is checked against the oracle collapsing the same pools +# into the same units, so the weighting inside a unit is checked and not only its count. +test_a_unit_keeps_the_pools_of_it_that_were_read() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-rollup-unread") + association_corpus "$sb" + paired_design "$CORPUS_DIR/design.json" "$sb/paired.json" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" + empty_cell "$table" chr1 700 TestSample2 + empty_cell "$table" chr1 100 TestSample6 + empty_cell "$table" chr2 550 TestSample3 + empty_cell "$table" chr2 550 TestSample4 + python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" --units 2,2,2 \ + > "$sb/oracle.tsv" + + association_direct "$sb/run" "$ASSOCIATION_OPTIONS" "$sb/paired.json" + if [ ! -s "$sb/run/association.tsv" ]; then + fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + assert_eq "3" "$(site_cell "$sb/run/association.tsv" chr1 700 n_observed)" \ + "U1 is still read at chr1:700 through TestSample1" + assert_eq "2" "$(site_cell "$sb/run/association.tsv" chr2 550 n_observed)" \ + "U2 is not read at chr2:550, where both of its pools were emptied" + local stat; stat=$(site_cell "$sb/run/association.tsv" chr1 700 S) + case "$stat" in + ""|NA) fail_case "chr1:700 must be tested over its three units, and S is '$stat'" ;; + esac + agrees_with_oracle "$sb/run" "$sb/oracle.tsv" S perm_p fdr_p n_observed mean_weight \ + max_leverage +} + +# DISPERSION IS ESTIMATED FROM THE UNITS EACH SITE WAS READ IN. It used to take rowMeans() over +# every unit, so a site with any unread unit dropped out of the estimate, and with none fully read +# theta was NaN, every weight followed it, and the run published nothing tested at exit 0. +# +# TestSample6 emptied at every SNP site, so no site is fully read: a failed library. Dispersion is +# estimated rather than pinned, which every other case here does, and checked against the +# oracle's own estimate over the same units - noise on fourteen sites, but the same noise. A second +# run pins it and checks every site against the oracle, because one pattern of five read units +# covering the whole table is the shape the shortcut for a fully read bin must not take: taken, +# it rearranges the five over all six. +test_dispersion_is_estimated_from_the_units_each_site_was_read_in() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-dispersion-unread") + association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" chrom pos + while IFS=$'\t' read -r chrom pos; do + empty_cell "$table" "$chrom" "$pos" TestSample6 + done < <(awk -F'\t' 'NR > 1 { print $1 "\t" $2 }' "$table") + + association_direct "$sb/run" \ + '{"phenotypes":["pt_wingspan"],"permutations":5000,"dispersion":null,"fdr":"BH","reportBelow":1.0,"reportTop":100000,"chromosomes":[],"binSize":100000,"workers":1,"usecpp":false}' + if [ ! -s "$sb/run/permutations.tsv" ]; then + fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + local theta tested + theta=$(published_cell "$sb/run/permutations.tsv" pt_wingspan dispersion) + tested=$(published_cell "$sb/run/permutations.tsv" pt_wingspan tested) + awk -v v="$theta" 'BEGIN { exit !(v ~ /^[0-9.eE+-]+$/ && v + 0 >= 0) }' \ + || fail_case "theta must be estimated from the read units, and it is '$theta'" + [ "${tested:-0}" -gt 0 ] 2>/dev/null \ + || fail_case "the run must still test the sites read in five units, and tested is '$tested'" + + # A unit read nowhere has no mean weight and drops out of the correlation, which with that + # unit carried as NA came out NA itself. + local cor; cor=$(published_cell "$sb/run/permutations.tsv" pt_wingspan depth_phenotype_cor) + awk -v v="$cor" 'BEGIN { exit !(v ~ /^-?[0-9][0-9.eE+-]*$/) }' \ + || fail_case "a unit read nowhere must drop out of depth_phenotype_cor, and it is '$cor'" + + python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" + assert_close "$theta" "$(oracle_value "$sb/oracle.tsv" assoc.dispersion)" \ + "theta, estimated over the units each site was read in" + association_direct "$sb/pinned" "$ASSOCIATION_OPTIONS" + agrees_with_oracle "$sb/pinned" "$sb/oracle.tsv" S perm_p fdr_p mean_weight max_leverage + assert_close "$(published_cell "$sb/pinned/permutations.tsv" pt_wingspan depth_phenotype_cor)" \ + "$(oracle_value "$sb/oracle.tsv" assoc.depth_phenotype_cor)" \ + "depth_phenotype_cor over the five units that were read" +} + +# A RUN THAT TESTS NOTHING STILL PUBLISHES WHAT IT READ, qq.png included: the frame refuses a +# folder missing a declared output, and a table where no site is read in three units that differ +# in the phenotype is an ordinary outcome of the three-unit rule. +# +# The first run gives TestSample1-3 one wingspan and TestSample4-6 another, and empties 4-6 +# everywhere, so every site is read in three units that all carry one value: no slope to test. A +# weighted mean of one repeated value need not return it, so the module used to fit a slope to +# the difference - S near 1e-16, perm_p 1, counted as tested - and publish leverages above 1. +# +# The second empties all but TestSample1 under an estimated dispersion. No site has two units to +# estimate theta from, and the NA theta used to take every weight with it: n_observed published +# 0 where one unit was read, and mean_weight NA. The weights are now left as they are. +test_a_run_that_tests_nothing_still_publishes_what_it_read() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-nothing-tested") + association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" chrom pos pool sites + cp "$table" "$sb/full.tsv" + python3 - "$CORPUS_DIR/design.json" "$sb/two-values.json" <<'PY' +import json, sys +design = json.load(open(sys.argv[1])) +level = {"TestSample1": 12.4, "TestSample2": 12.4, "TestSample3": 12.4, + "TestSample4": 15.1, "TestSample5": 15.1, "TestSample6": 15.1} +for entry in design["phenotypes"]: + if entry["column"] == "pt_wingspan": + for value in entry["values"]: + value["value"] = level[value["pool"]] +json.dump(design, open(sys.argv[2], "w")) +PY + sites=$(awk -F'\t' 'NR > 1 { print $1 "\t" $2 }' "$table") + while IFS=$'\t' read -r chrom pos; do + for pool in TestSample4 TestSample5 TestSample6; do + empty_cell "$table" "$chrom" "$pos" "$pool" + done + done <<< "$sites" + association_direct "$sb/flat" "$ASSOCIATION_OPTIONS" "$sb/two-values.json" + + cp "$sb/full.tsv" "$table" + while IFS=$'\t' read -r chrom pos; do + for pool in TestSample2 TestSample3 TestSample4 TestSample5 TestSample6; do + empty_cell "$table" "$chrom" "$pos" "$pool" + done + done <<< "$sites" + python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" + association_direct "$sb/alone" \ + '{"phenotypes":["pt_wingspan"],"permutations":5000,"dispersion":null,"fdr":"BH","reportBelow":1.0,"reportTop":100000,"chromosomes":[],"binSize":100000,"workers":1,"usecpp":false}' + + local run column + for run in flat alone; do + if [ ! -s "$sb/$run/association.tsv" ]; then + fail_case "the $run run published nothing"$'\n'"$(cat "$sb/$run/out.txt")" + return + fi + assert_file "$sb/$run/qq.png" "the $run run must draw qq.png with nothing tested" + assert_eq "0" "$(published_cell "$sb/$run/permutations.tsv" pt_wingspan tested)" \ + "the $run run tests nothing" + grep -q "no site of pt_wingspan (snp) was tested" "$sb/$run/out.txt" \ + || fail_case "the $run run must say that it tested nothing" + done + while IFS=$'\t' read -r chrom pos; do + assert_eq "3" "$(site_cell "$sb/flat/association.tsv" "$chrom" "$pos" n_observed)" \ + "$chrom:$pos is read in three units that share a wingspan" + for column in S perm_p fdr_p max_leverage zero_variance; do + assert_eq "NA" "$(site_cell "$sb/flat/association.tsv" "$chrom" "$pos" "$column")" \ + "$chrom:$pos has no spread in the phenotype, so no $column" + done + assert_eq "1" "$(site_cell "$sb/alone/association.tsv" "$chrom" "$pos" n_observed)" \ + "$chrom:$pos is read in TestSample1 alone" + done <<< "$sites" + assert_eq "NA" "$(published_cell "$sb/alone/permutations.tsv" pt_wingspan dispersion)" \ + "no site read in two units leaves theta unestimated" + agrees_with_oracle "$sb/alone" "$sb/oracle.tsv" mean_weight +} + +# THE STRONGEST SITE STILL COUNTS ITSELF. A rearrangement reaches the observed statistic to within +# rounding, and the rounding grows with it: rebuilt, the identity lands up to 4e-14 of S away +# from S. Counted against an absolute 1e-12 it went uncounted above S of about 100, nothing else +# reached S, and the site published perm_p 0 and fdr_p 0 - the strongest sites in a table, +# reported as impossible ones. Each of these three rows, taken from a scan of 400 near-linear +# sites in which 15 did that, beats every other arrangement, so the answer is 1/720. +test_the_strongest_site_still_counts_itself() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-strongest") + association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" pos + printf '%s\n' \ + $'chr3\t213\tA\tT\t6035,1666\t915,184\t512,235\t1597,51\t291,296\t1720,204\t1000,696' \ + $'chr3\t341\tA\tT\t4564,1545\t452,98\t553,284\t1411,44\t588,697\t1214,151\t346,271' \ + $'chr3\t399\tA\tT\t5359,1647\t981,164\t566,203\t330,12\t873,623\t1689,169\t920,476' \ + >> "$table" + python3 "$REPO_ROOT/test/tools/freq_corpus.py" --association-from "$table" > "$sb/oracle.tsv" + association_direct "$sb/run" "$ASSOCIATION_OPTIONS" + if [ ! -s "$sb/run/association.tsv" ]; then + fail_case "the run published nothing"$'\n'"$(cat "$sb/run/out.txt")" + return + fi + for pos in 213 341 399; do + assert_close "$(site_cell "$sb/run/association.tsv" chr3 "$pos" perm_p)" \ + "0.00138888888888889" "chr3:$pos beats every rearrangement but its own" + done + agrees_with_oracle "$sb/run" "$sb/oracle.tsv" S perm_p fdr_p +} + # A unit takes one phenotype value. Two pools of one unit disagreeing is either metadata that # contradicts itself or a repeated measure, and the second is a mixed model this release does not # fit - so it refuses by name rather than averaging the two into something nobody measured. @@ -209,11 +570,17 @@ test_a_phenotype_that_disagrees_within_a_unit_refuses() { || fail_case "the refusal must name the column and the values"$'\n'"$(cat "$sb/run/out.txt")" } -# The compiled parse and the R one are the same table or one of them is wrong. +# The compiled parse and the R one are the same table or one of them is wrong - over the sites +# every pool read, and over two that some pools did not, where both parse a cell of zeros. test_both_paths_through_the_parse_agree() { if ! have_analysis_r; then skip_case "no analysis environment"; return; fi local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-paths") association_corpus "$sb" + local table="$CORPUS_DIR/Frequencies/Test_snp_depth.tsv" pool + empty_cell "$table" chr1 250 TestSample6 + for pool in TestSample1 TestSample2 TestSample3 TestSample4; do + empty_cell "$table" chr10 500 "$pool" + done association_direct "$sb/plain" "$ASSOCIATION_OPTIONS" association_direct "$sb/compiled" "${ASSOCIATION_OPTIONS/\"usecpp\":false/\"usecpp\":true}" # A FAILURE, NOT A SKIP. The analysis environment pins gcc_linux-64 and gxx_linux-64, so a @@ -230,6 +597,32 @@ test_both_paths_through_the_parse_agree() { || fail_case "the compiled parse and the R parse published different allele tables" } +# THE VALIDATION HARNESS DESCRIBES THIS MODULE, OR ITS NUMBERS DESCRIBE NOTHING. The missing_* +# sections of dev/validation/calibrate.R call association.R's own functions, but restate the +# fifteen lines of its main loop that put them in order -- the roll-up, theta, the untestable rule, +# BH over the tested sites -- and restate step 7's mask. dev/validation/agree.R runs the shipped +# association.R and mds.R and bin/mask_depth.awk on tables with unread cells and compares them with +# the harness site by site. It runs here because nothing else would make it run: a change to the +# main loop the harness did not follow would leave every missing_* number describing a computation +# no run makes, and calibrate.R would go on passing. Z agreed to put it in this suite on +# 2026-10-06. Three one-line mutants of the main loop -- untestable below two units instead of +# three, BH over every site, theta never applied -- each made it disagree. +test_the_validation_harness_agrees_with_the_module() { + if ! have_analysis_r; then skip_case "no analysis environment"; return; fi + local sb; sb=$(guard_path "$TEST_TMPDIR/assoc-agree") + rm -rf "$sb" + mkdir -p "$sb" + local status=0 + "$(analysis_rscript)" --vanilla "$REPO_ROOT/dev/validation/agree.R" "$REPO_ROOT/modules" \ + "$sb/work" > "$sb/agree.out" 2>&1 || status=$? + if [ "$status" != "0" ]; then + fail_case "agree.R exit $status: the harness and the shipped modules disagree"$'\n'"$( + grep -E 'DISAGREE|Error|FAILED' "$sb/agree.out" | head -20)" + return + fi + assert_contains "$(cat "$sb/agree.out")" "all agree" "agree.R must end by saying they agree" +} + # The module credits the statistics it computes and not the family they belong to. A reader who # follows an entry and finds nothing of it in the output is worse served than by no entry. test_association_cites_the_statistics_it_computes() { From e04215bbc10605697567a36b043b65f8a77616d2 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Tue, 6 Oct 2026 18:55:27 -0400 Subject: [PATCH 4/5] dropZeroDepth replaced with keepLowDepthAsZero, and minSamples is added to vcfFilter --- .../association-unread-units.md | 85 ++ .../depth-masking-proposal.md | 132 ++ .../missing-cell-validation.md | 66 + .../shell-and-nextflow-gotchas.md | 5 +- bin/config_migrate.sh | 20 +- bin/mask_depth.awk | 98 ++ dev/RELEASING.md | 2 +- dev/validation/README.md | 24 +- dev/validation/agree.R | 398 ++++++ dev/validation/calibrate.R | 1255 ++++++++++++++++- dev/validation/lib.R | 387 ++++- manual/PoolSeqFlow-manual.md | 136 +- modules/basicstats/basicstats.R | 2 +- modules/basicstats/manifest.json | 4 +- modules/basicstats/test/basicstats.sh | 12 +- nextflow.config | 13 +- parameters.config.template | 11 +- scripts/0_verify_environment.nf | 45 + scripts/7_vcf2freq.nf | 45 +- scripts/variants.nf | 3 +- test/README.md | 20 +- test/lib/analysis.sh | 12 + test/lib/scratch_envs.sh | 100 ++ test/run_tests.sh | 72 +- test/suites/00_static.sh | 85 ++ test/suites/01_migrate.sh | 40 +- test/suites/02_launcher.sh | 3 +- test/suites/03_helpers.sh | 208 ++- test/suites/04_pipeline.sh | 65 +- test/suites/05_guards.sh | 113 +- test/tools/freq_corpus.py | 266 +++- 31 files changed, 3440 insertions(+), 287 deletions(-) create mode 100644 .claude/development-notes/association-unread-units.md create mode 100644 .claude/development-notes/depth-masking-proposal.md create mode 100644 .claude/development-notes/missing-cell-validation.md create mode 100755 bin/mask_depth.awk create mode 100644 dev/validation/agree.R create mode 100644 test/lib/scratch_envs.sh diff --git a/.claude/development-notes/association-unread-units.md b/.claude/development-notes/association-unread-units.md new file mode 100644 index 0000000..0123bd1 --- /dev/null +++ b/.claude/development-notes/association-unread-units.md @@ -0,0 +1,85 @@ +# Association over the units a site was read in + +**Written 2026-10-05, against `dev` at `7419281` plus uncommitted work**: the change described here (`association` `20261005.001`), the moot `dropZeroDepth` off-branch fix, the `run_tests.sh` changes, and `depth-masking-proposal.md`, which records how the defect was found. + +## What was wrong + +One unit with no reads at a site made that site's enumerated `perm_p` exactly 0. `fit_alleles()` zeroed the NA weight, so the fit itself was sound, but `permutation_p()` built `center` from the raw weights: every rearranged statistic came out NaN, the identity was never counted, and `fdr_p` followed to 0 while lowering every other site's adjustment. The published `association-20260910.004` carries the same code. + +Two more places shared the blind spot. `roll_up()` summed a unit's pools without `na.rm`, so one unread pool blanked a unit of several. `dispersion_of()` took its moments over every unit, so a pool unread at every site, a failed library, made theta NA, and a run estimating dispersion then tested nothing. + +## The obvious fix is wrong + +Zeroing the unread units and keeping the one global permutation reports p-values below what the site supports. Every rearrangement moves residuals across all n units, so a read unit is handed an unread unit's zero in most of them, and the null is built from residuals the site never had. It reported p below 1/m! at sites read in m units. The comment over the per-site rearrangement said this during the build and moved here in the comment pass. + +## What it does now + +Each site is rearranged among the units it was read in. `tally()` groups a bin's sites by the set of units read. For each global rearrangement, a group read in M takes the residuals of M's units in the order they appear in that rearrangement, and an unread unit keeps its own zero. The order M's units take inside a uniform permutation of all n is a uniform permutation of M, and across the n! enumerated rearrangements each of M's m! orders appears exactly n!/m! times, so the enumerated p is the exact m-unit test. By the same argument a sampled p is an unbiased estimate of that test, and the verification below measured it so. A bin in which every site is read in every unit takes the original single expression. + +Around that, `permutation_p()` zeroes weights that are NA or not positive and the frequencies under them, as `fit_alleles()` does. `roll_up()` makes a unit the n_eff-weighted mean of its pools read at the site, with weight and frequency NA when none were. `dispersion_of()` estimates theta per site over the units read there, skips a site read in fewer than two, and is NA only when no site has two, which is a run with nothing testable anyway; the weights are then left as they are. A site whose read units all carry one phenotype value has no slope and is not tested, which `spread_of()` decides on the stored values. + +## Decided by Z, 2026-10-05 + +**At least three measured values to test a site.** A site read in fewer than three units is not tested. Its row in `association.tsv` stays, with `n_observed`, and `S`, `perm_p` and `fdr_p` NA; BH runs over the tested sites only, as it already did. Z first said "eliminate"; asked whether that meant dropping the row, Z answered "don't test them", and the row stays because the manifest documents the table as one row per site, always complete. + +**A unit of technical replicates is the mean of its lanes that were read**, by `rowSums(..., na.rm = TRUE)`. Z: "Rowsum looks correct." + +**NA for what does not exist.** The first build gave a unit with no read pool weight 0, which `na.rm` in `roll_up()` makes of an all-NA row, and `mean_weight` then averaged the 0 in: 33.4 at `chr10:500`, read in 2 of 6, where the two read units average 100.25. Z: "NA for non existant is correct." A unit with no read pool now has weight NA, and every aggregate over units leaves it out. `mean_weight` is the mean over the units read at the site. `max_leverage` is the largest over them, and NA where there is no slope to carry. A line that also dropped unread units from that maximum, on the reasoning that `pmax()` with `na.rm` would turn a one-unit site into 0, was reverted: run against the case, removing it changed nothing, because whenever the read unit's leverage is NaN every unread unit's is NaN too (0 times an infinite term), and when it is finite a 0 never wins. `depth_phenotype_cor` correlates the phenotype with each unit's mean weight over the sites it was read at, and a unit read nowhere drops out. That is the rule `basicstats` took for its depth summaries on 2026-09-29. + +## What an untested row shows + +Measured on the corpus with `chr10:500` read in 2 of 6 units: `S`, `perm_p` and `fdr_p` NA, `mean_weight` 100.25 over the two units read, and `max_leverage` 1 and `zero_variance` 1, both true of two points on a line. A site read in one unit, or in units that share one phenotype value, has no slope, and both are NA there. Its alleles do not appear in `association_alleles.tsv`, because selection by `reportBelow` and `reportTop` passes over an NA p, so the NA written into their `b1`, `se`, `t` and `p` is not visible in any published table today. + +## The floor is 1/n! + +The identity always ties, so a design of n units cannot fall below 1/n!, and a site read in m units cannot fall below 1/m!. Nothing guarantees that the reversal ties: it negates the slope exactly only under a symmetry of phenotype and depths that a real design rarely has. Measured on the corpus, `chr2:550`, read in all six units, reaches 1/720, and `chr1:700`, read in four, reached 1/24 = 0.042. I told Z 2/m! during the build and corrected it. + +The module said otherwise, and contradicted itself. The manifest gate that opened "THE SMALLEST P ANY DESIGN CAN REACH" and the comment over `limit` in `association.R` said 2/n! and that four units cannot reach 0.05. `design_floor` was 2/n! while `floor`, enumerated, was already 1/n!, so the comment over them, which says the two differ only when the set is sampled, was false. The manual's `#association-floor` opened with the 2/n! claim and a table to match, then read the two numbers as the floor at similar depths and the reach at differing ones. The per-site message added in this change used 1/m!, so a design of four units was told nothing could be significant while a site read in four of six was not flagged. + +**Ruled by Z, 2026-10-05: "1/n! is right. correct the statements."** `design_floor` is 1/n! and, enumerated, equals `floor`. The ranking sentence fires at three units and no longer at four. The manifest gate and the manual's floor section were rewritten: the table starts at three units, a site's own floor is stated, and a paragraph says what ties do. Two test comments and the floors case were corrected; the case now asserts 1/720 and that an enumerated `floor` equals `design_floor`, and a new case runs four units (1/24, tested, no ranking sentence) against three (1/6, the sentence). `dev/validation/calibrate.R` and `lib.R` keep 2/n!, which is right there: they move labels on an evenly spaced phenotype, where the reversal always ties, and their comments now say so. `calibration.md` is a dated note and is left as written. + +## Adversarial verification, 2026-10-05 + +Four agents attacked the change and a fifth checked what they had covered; the transcript is at `~/.claude/projects/-home-tholian-Nextcloud-GitHub-PoolSeqFlow/7de5ecd1-0266-4a1d-aeb2-69c4394fc0c5/subagents/workflows/wf_77aa51c3-634/`. The arithmetic held: partly read sites agree with independent oracles on thousands of values, fully read data reproduces HEAD, the compiled parse, two and three workers and bins of one to four sites publish byte-identical tables with unread cells, the sampled path sits within Monte Carlo error of the exact p (120 end-to-end runs and 1000 in-process calls at 400 draws), and 8000 simulated null sites with each pool unread at probability 0.3 rejected at nominal for sites read in four, five and six units and conservatively in three. The NA-weight ruling moved `mean_weight`, `depth_phenotype_cor`, `design_floor` and the ranking sentence and nothing else, over 127 compared inputs. + +What they broke, each fixed and given a case: + +- **A site read in three or more units that all carry one phenotype value was tested on rounding.** A weighted mean of one repeated value need not return it (15.1 came back 15.099999999999998), so `sxx` was 5e-28 rather than 0 and the slope over it published `S` near 1e-16 and `perm_p` 1, counted in BH's n and in `lambda_gc`. Unread units are what make such a site reachable. `spread_of()` tests the stored values; such a site is untestable, and its `max_leverage` and `zero_variance` are NA. +- **`max_leverage` 2 at a site read in one unit**, by the same mechanism, at 11.5% of unit and depth combinations on the corpus phenotype. HEAD published NA there, its NA weights propagating, so this was a regression of round 1. NA again, through the same test. +- **No `qq.png` when nothing was tested**, and the frame refuses a folder missing a declared output (`readmeShell()` in `analysis/lib/nf/outputs.nf`). The three-value rule makes such a run ordinary. The figure is drawn with a line saying nothing was tested, and the log says it too. +- **An NA theta took every weight with it.** With dispersion estimated and no site read in two units, `n_observed` was published as 0 where one unit was read. Pre-existing at HEAD. The weights are left as they are and `dispersion` is published NA. +- **The strongest sites published `perm_p` 0**, pre-existing since F2. The tie test was an absolute `seen - 1e-12`, and the identity, rebuilt, lands up to 4e-14 of S away from S, so above S of about 100 it went uncounted and nothing else reached S: 15 of 400 near-linear sites, 2 to 8% of sites with S over 100 in near-collinear tables, 0.38% of realistic fixed-difference binary sites. It contradicted the floor sentence Z had just ruled on. The tolerance is absolute below 1 and relative above it, in the module and in the oracle, and the 15 publish exactly 1/720. +- **Wording.** The new message said "tested site(s) ... ranked, not tested"; the manifest said an unread pool "carries weight zero"; the manual called `mean_weight` an effective sample size under an estimated dispersion too; and the ties sentence named equal depths and an evenly spaced phenotype, where the condition is equal weights and a phenotype symmetric about its mean, and missed the doubling a balanced categorical phenotype gets from the swap of its levels (3 against 3 at equal depths: 72 of 720 tie, so 0.1). +- **Stale floor statements predating today**: in `freq_corpus.py`, 2/20 at chr10:1600 where the corpus publishes 1/720, 4/720 where it publishes 9/720, and "capped at 0.1" where it publishes 6/720; in `calibrate.R`, 1/720 "whatever the phenotype's shape", false for its own flat rows. + +The suite could not fail on most of round 1, measured by running mutants of the module through its own cases: the new message, the shortcut for a fully read bin, grouping sites by how many units read them rather than which, the dispersion arithmetic, the weighting inside a unit, BH's n, and the warning the allele-row mask suppresses. Each now has an assertion. The oracle gained `--units`, `fdr_p` and theta, and the cases compare every site against it. + +Not fixed, and left to Z: + +- A depth table holding exactly one site stops `association.R` at `colnames<-`: the weight matrix collapses to a vector. Pre-existing and unrelated. +- Enumeration is capped at eight units whatever the budget, which the manual now says. With nine or more and `permutations` above n!, a sampled p can fall below 1/n!. +- Dropping the ranking sentence at four units leaves the default output less cautious than HEAD's. Under BH an `fdr_p` of 0.05 needs about 83% of tested sites at the floor at four units, 17% at five and 2.8% at six. +- The m-unit test assumes that which units were read at a site is unrelated to what they carry. A pool unread because of its own genotype breaks that, and no gate says so. + +## Verification + +`freq_corpus.py --association-from ` is the oracle: plain Python that leaves out a pool with depth 0 at each site and enumerates the read units' own m! rearrangements directly, so it shares neither code nor the induced-order construction with the R. The corpus files it writes, `expected.tsv` included, are byte-identical to HEAD. + +On the corpus with sites read in 6, 5, 4, 3 and 2 units, R and the oracle agree at all 14 sites. Each fix, reverted alone, fails its case: + +| reverted | what the case saw | +|---|---| +| the per-site rearrangement and the zeroing | `perm_p` 0 at the partly read sites | +| the three-value rule | the 2-unit site published `S` = 0 | +| `na.rm` in `roll_up()` | `n_observed` 2 and `S` NA where a unit kept one read pool of two | +| `dispersion_of()` | theta NA and nothing tested, with one pool unread everywhere | + +**`assert_close` in `test/lib/analysis.sh` could not fail on a missing value.** It compared through awk, which reads `NA`, `NaN` and `Inf` as 0, so "expected 0, got NA" passed, and the three-value revert passed under it. It now requires those tokens to match literally. Another entry for `gates-that-stopped-checking.md`. + +Suites, with `TEST_CONDA_ENV` at `PoolSeqFlow-3.2.0` and `TEST_ANALYSIS_ENV` at `PoolSeqFlow-3.1.1-analysis`: association, basicstats and mds in full, 39 passed; `--fast` 401 passed, 0 failed, 234 skipped, every skip `(--fast)`; `00_static` 52 passed; `check-analysis-versions.sh` clean. Those ran before the comment pass, which changed no code line: the non-comment lines of `association.R` are identical before and after it, and the file parses. + +After the rulings and the verification's fixes: association 15 passed, `00_static` 52 passed. The corpus files `freq_corpus.py` writes, in both of its modes, are still byte-identical to HEAD's, through the oracle's rewrite onto units. Each guard was reverted alone in a copy of the module and run through the suite's own cases from a stand-in `REPO_ROOT`, so the repository file never changed: 21 of them, and each failed the case meant to catch it, while round 1 as it stood fails all seven of the cases this work touched. A 22nd mutant passed, which proved a line redundant: the allele p-values computed over every site rather than the tested ones, since the t-ratio is already NA where untested and `pt()` returns NA for NA without warning. The line was simplified, and removing the mask itself now fails the case on its warning. + +## Not exercised + +The suite's cases run enumerated and one worker; `test_both_paths_through_the_parse_agree` now parses two partly read sites both ways. The sampled path, several workers and several bins with unread cells were run by the verification above and are not in the suite. Not run anywhere: the frame on a run that tests nothing, which was read from the code and not driven through Nextflow; real data with zero-coverage cells; a missing-cell scenario in `dev/validation/calibrate.R`; and more than 20,000 sites. diff --git a/.claude/development-notes/depth-masking-proposal.md b/.claude/development-notes/depth-masking-proposal.md new file mode 100644 index 0000000..9bd52f2 --- /dev/null +++ b/.claude/development-notes/depth-masking-proposal.md @@ -0,0 +1,132 @@ +# Masking shallow cells instead of dropping sites, and the association defect it exposed + +**Written 2026-10-05, against `dev` at `d76ba2f` plus uncommitted work** (the `dropZeroDepth` off-branch fix to `FMT/DP=minDP & FMT/DP>0)>=K'` reproduces every set above exactly, and `COUNT()` gives the same sets. Two traps, both silent: + +- **`&&` and `||` inside `N_PASS` are not per-pool.** At minDP 20, `N_PASS(FMT/DP>=20 && FMT/DP>0)` counts 4,2,4,3,4 where the true count is 4,2,2,1,3, so K=3 also keeps 300 and 400. The same holds for `||`: today's ON expression uses `||`, so a count ported from it would be wrong. A case asserting K=3 keeps exactly 100 and 500 on these depths catches it. +- **The floor alone counts an unread pool as measured at minDP 0.** `N_PASS(FMT/DP>=0)` is 4 everywhere. The `& FMT/DP>0` term is what the `|| FMT/DP==0` term does in today's expression. + +bcftools accepts any K without complaint: K=5 on four pools keeps nothing, K=0 and K=-1 keep everything, all exit 0. + +## Masking is new code, and it has to reach the depth table + +**Every module reads the DEPTH table, and nothing reads the frequency table's numbers.** The depth table is `FORMAT/AD` printed by `bin/createDepthFile.sh:48` before `bin/depth2freq.awk` runs. A site filter alone, with no mask, hands every module a 5-read pool as a measured frequency (site 300 publishes that pool at 0.6/0.4). So the mask has to rewrite `FORMAT/AD`, and `FORMAT/DP` with it. + +No bcftools route does that reliably. `filter -S` and `+setGT` change GT only, and `MajorAlleleToRef.py:84` has already set every GT to `./.`. `+fill-tags` rewrites `FORMAT/DP` but refuses `FORMAT/AD`, and a comparison inside its expression turns every cell missing. A `query -i ... -F '0,0'` followed by `annotate` reproduces the example but fails silently with exit 0 on two cases: a site where every pool is shallow stays unmasked, and a three-allele site gets a two-value AD that makes `MajorAlleleToRef.py` exit 1. A text pass works: an awk mask writing one 0 per allele (counted from the ALT column at the point of masking) and DP 0 reproduced the example exactly and ran cleanly through step 7's own scripts, publishing `0,0` in the depth table and NA on every allele row of the frequency table. + +## The association defect, which exists today + +**`modules/association/association.R` publishes a wrong p-value at every site where a unit is unmeasured, and says nothing.** `fit_alleles` zeroes an NA weight (`:111-112`), but `permutation_p` builds `center` from the raw weights (`:209-211`). A unit at depth 0 has an `n_eff` of NA, so every rearranged statistic is NaN, the identity is never counted, and an enumerated p comes out `reached / count = 0` (`:239`); a sampled one lands exactly on the floor. Reproduced: chr1:250 `perm_p` 0.4 to 0, `fdr_p` 0.627 to 0, an unaffected site's BH p 0.0542 to 0.0406, selected sites 2 to 4, exit 0. + +`association.R` is unchanged since `dbc522d` (2026-09-10), and the published `modules/repo/association-20260910.004.tar.gz` carries a byte-identical copy. `.claude/development-notes/unmeasured-cells.md:58` recorded association as safe "by the same route" as basicstats; that checked the fit, not the permutation. + +When it is reached: + +- committed `dev`: only with `minDP` at 0 and `dropZeroDepth` off +- the uncommitted off-branch fix: with `dropZeroDepth = false` alone, because the fix is what finally publishes zero-depth sites +- the proposal: by default + +**So the off-branch fix is correct as a filter and unsafe to ship while association is unfixed.** Before it, `false` dropped the sites it claimed to keep, and that bug was all that stood between a user and false association hits. + +Four more association behaviors that masking turns from rare into routine, each reproduced: + +- **No per-site minimum.** A site with 2 measured units publishes S 0, `perm_p` 0 and `fdr_p` 0; a site with 3 is tested on one degree of freedom (`df <- observed - 2`, `:134`). The refusal below three units (`:271-276`) counts the design's units, not the site's. +- **`roll_up()` sums without `na.rm`** (`:96-97`). One masked technical replicate drops its whole unit at that site, even when the other replicate was read. +- **Dispersion comes only from fully measured sites.** If none is, theta is NaN, every weight is NaN, and the run publishes all NA with exit 0. +- **`depth_phenotype_cor` turns NA from one masked cell** (`:445`), which is the diagnostic for depth lining up with phenotype. `lambda_gc` drifts with missingness. + +basicstats handles unmeasured cells correctly, and its compiled and R paths agree. mds handles them by pairwise deletion and stops loudly when two pools share no site, though its message points at a `distance.tsv` the failed run never writes. The shared libraries handle a depth-0 cell; `harmonic_mean()` returns 0 on any zero by design, and its only callers pass none. + +## What the evidence leaves for Z to decide + +- **Whether a pool below minDP counts as evidence anywhere.** The false-positive filter runs before the depth filter and counts any `dp > 0` as support (`bin/filterFalsePositives.sh:134`). Masking only at the depth filter published an allele with zero reads in every published pool, admitted on the support of pools then masked. Masking only at the start published a pool below minDP, because the FP filter dropped an allele and `MajorAlleleToRef.py:82` recomputed DP from 21 to 18. Masking at both points avoided both. QUAL has the same exposure, unmeasured: `bcftools call` computed it from every pool's reads. +- **How a masked cell is written.** Zeros work with every consumer but make a pool read 5 times identical to one never read, which reverses the premise of the 2026-09-29 NA work and changes what basicstats' `unmeasured` column counts. A per-allele missing value (`.,.`) keeps them apart and the libraries read it as unmeasured, but `MajorAlleleToRef.py:74` calls `int()` on every AD value, so it only works after the last re-sort, which is the placement that leaks. With today's scripts the two goals cannot both be had. +- **One parameter or two.** `true` is exactly K equal to the pool count; `false` differs from K=1 only at a site where every pool is masked, and only when the mask runs after the FP filter. Two parameters give 2 x N combinations for N behaviors, and the proposed default pair (true, 2) means either the count is ignored or it contradicts `true`. +- **K's default against what the modules need.** K counts VCF columns; association fits units built after `analysis.design.technicalRep`, and needs three per site. A default of 2 is below that, and two replicate pools of one unit satisfy K=2 while reaching association as one unit or none. +- **Whether missingness is informative.** If a pool's depth at a site depends on the allele it carries there (mapping loss on a divergent allele, an indel, copy number), masking removes that pool exactly where its frequency differs, where today's rule removes the site for everyone. Nothing measures what K=2 does to association's calibration or to where mds places a shallow pool, and `dev/validation/calibrate.R` has no missing-cell scenario. +- **The FP filter's threshold under masking.** `MINSAMPLES` is all columns times `sampleThreshold` (`filterFalsePositives.sh:67,73`). With the mask before it, the share of measured pools an allele must reach grows with missingness: ten pools at 0.2 need 2 of 2 at a site with two measured pools, 2 of 10 at a full one. + +## Decided by Z, 2026-10-05, later the same day + +The count called K above is the parameter Z named `minSamples`. + +- **`dropZeroDepth` is replaced by `keepLowDepthAsZero`, default `false`, with `minSamples` consulted only when it is `true`.** `false` is today's rule exactly -- every pool at or above `minDP` -- for every value of `minSamples`. `true` masks every cell below `minDP` and keeps a site with at least `minSamples` measured pools. Checked against the example: `false` keeps 100 at every `minSamples`; `true` reproduces the K sets above. The working tree's `FMT/DP=20)>=2"` as a one-line swap, which hands sub-minDP pools to every module as measured, and says per-sample blanking belongs in `bin/depth2freq.awk`, which writes only the frequency table no module reads. +- `modules/association/manifest.json:25`, `modules/mds/manifest.json:28` and both `main.nf` headers say a cell with no reads is published as 0; it has been NA since `772db1f`. +- Stale in the working tree after the off-branch fix: `modules/basicstats/test/basicstats.sh:262`, `test/suites/03_helpers.sh:1583-1587`, `test/tools/freq_corpus.py:65-66` and `test/README.md:222-223`, each saying a zero cell needs both minDP 0 and `dropZeroDepth` off. + +## Built, 2026-10-05, the same evening + +Built on top of the association fix in `association-unread-units.md`, which also records Z's confirmation of the replicate rule ("Rowsum looks correct"). Uncommitted. + +`vcffilter.keepLowDepthAsZero` (default `false`) and `vcffilter.minSamples` (default 2) replace `dropZeroDepth`. `false` runs today's expression unchanged, `FMT/DP = ` with no value; the mask is 7 to 13 times slower than `bcftools view -e` (48 pools by 20,000 records, 2.8 to 5.6 s against 0.42 s); mawk and BWK awk were never run; CRLF input loses its carriage return on masked records, which `MajorAlleleToRef.py` makes unreachable. diff --git a/.claude/development-notes/missing-cell-validation.md b/.claude/development-notes/missing-cell-validation.md new file mode 100644 index 0000000..c579de4 --- /dev/null +++ b/.claude/development-notes/missing-cell-validation.md @@ -0,0 +1,66 @@ +# Missing-cell scenarios in the validation harness + +**Written 2026-10-06, against `dev` at `7419281` plus uncommitted work**: the association fix for unread units (`association-unread-units.md`), the depth-mask redesign (`depth-masking-proposal.md`), and this change. Every number below was read from the harness as committed in this change, at its default seed 20260907 and default replicate count, under R 4.6.1. + +## What was asked + +Z decided on 2026-10-05 that missing-cell scenarios are to be added to the validation harness, alongside `vcffilter.keepLowDepthAsZero`. The proposal note had named the gap: nothing measured what masking does to association's calibration or to where mds places a shallow pool, and nothing tested whether missingness is informative. Z opened the work on 2026-10-06: "Let's move with the next task. missing cell scenarios?" + +## How it was designed + +A workflow, `wf_831b725c-90d` (transcript under `~/.claude/projects/-home-tholian-Nextcloud-GitHub-PoolSeqFlow/7de5ecd1-0266-4a1d-aeb2-69c4394fc0c5/subagents/workflows/`), every agent on Sonnet 5.5: three independent proposals, one each on calibration, on how cells really go missing in pool-seq, and on building it in this harness, then a synthesis that resolved their conflicts and wrote and ran a prototype of every piece. It took 150 minutes and 424 tool calls, far longer than expected, because the proposal agents ran the measurements themselves; Z chose to let it finish. The prototypes were then ported into the harness in its own style with the order of every random draw kept, and the port reproduces every number the prototypes printed at the default seed. + +The synthesis resolved four conflicts that are worth keeping. The estimator calls the module's own functions rather than a restatement of them, with a brute-force oracle as the independent check. Read sets are enumerated, every pattern of read units in every cohort, wherever a deterministic check or an exact stratum is wanted, because the module groups sites by which units were read and a grouping bug shows only on the patterns it gets wrong; realistic compositions are drawn only where the composition is itself the quantity. Rates over enumerated p are gated on four standard errors of the larger of the binomial and the across-cohort spread, and rates over shared nuisance on a t quantile of the spread alone, after a Monte Carlo of 400,000 nulls showed that four standard errors of a spread from four to eight cohorts alarm 0.2 to 1.1 percent per comparison. And OFF, step 7's default rule, is read as the six-unit stratum of the masked table for per-site rates, and run as a table of its own only where table-level numbers are compared. + +## Decided by Z, 2026-10-06 + +- **The `missing_*` estimators are the modules' own functions**, parsed out of `association.R` and `mds.R` by `module_functions()`, rather than a copy in `lib.R`. "1. yes." +- **`agree.R` runs as a case of association's suite**, so `--changed` runs it whenever the module, a library, `mds.R`, `bin/mask_depth.awk` or the harness's `lib.R` changes. It ends the README's "nothing here runs in run_tests.sh". "2 yes." +- **The manual's `lambda_gc` line is to be rewritten.** It says "If it is 3, nothing in the table is a p-value", and an exactly calibrated null crosses 3 at three-allele sites. "Definitely rewrite manual." The next item, not built here. +- **mds is to warn when a pair rests on few shared sites, at 30.** Z: "MDS warning sounds right. I feel like maybe we should not even entertain MDS below 100 shared sites. But 30 sounds reasonable and not arbitrary." Taken as a warning at 30, with 100 a thought rather than a decision. Not built here. + +## What the sections measured + +**`missing_null`: the masked p is the exact m-unit test.** Over every pattern of read units in seven cells, 18,605 tested sites, no p fell below 1/m!, every n_observed was the units the generator read, every site that must carry no statistic carried none, and every p equaled a brute-force enumeration of the m! orders to 1e-9 (largest gap 0). The rejected scheme, unread units zeroed under one rearrangement of all six, put 47 of 767 partly read sites below 1/m!, disagreed with the oracle at 98 percent of them, and rejected 0.077 of the three-unit sites against a bound of 0, so the gates can see it. Pooled rates at four, five and six units read 0.0319, 0.0471 and 0.0493 against bounds of 0.0417, 0.05 and 0.05. At resolution, 8,000 sites per stratum, flat depth read 0.0286, 0.0524 and 0.0494, the last two the two-sided control, and aligned depth 0.0366, 0.0501 and 0.0527. Four units sit 20 to 30 percent under their bound from count ties alone: any tie among four frequencies makes 1/24 unreachable. The sampled path at eight units took 499 rearrangements, its smallest p was 1/(1+B), and its p less the exact p less (1 - p)/(1 + B) averaged -0.0011 against a margin of 0.0026. Units of two lanes of one pool, 57 percent of them read through one lane, kept their n_observed and read 0.0363, 0.0466 and 0.0514. + +**`missing_published`: what a run says about itself.** Masking kept 99 percent of raw sites where the default rule kept 38, 12 and 0.7 percent at 15 percent, 30 percent and a ladder of 94 to 7 percent of cells under the floor; 0.5, 6 and 12 percent of the masked rule's sites were read in fewer than three units and untested. `lambda_gc` on a perfectly null table read 1.296, 1.485 and 1.657, each within 2 percent of the null of its own mix of units (ratios 0.994, 1.019, 0.995, and 0.993 on the aligned ladder). The null itself is 2.198, 1.465, 1.286 and 1.206 for sites read in three to six units, and 3.81 at three alleles and 6.03 at four over six units, so the published value is not near 1 on a clean table and crosses 3 on an exactly calibrated one. Theta, each cohort analyzed fully read and masked, moved by 2.1 percent of its true value at most under random, depth-floor and phenotype-aligned masks; it stayed finite with a library unread at every site and with 85 percent of cells masked, and fell 4 percent under allele-linked loss, which is reported only. With missingness lined up with the phenotype at equal depth, `depth_phenotype_cor` is NA: every unit carries the same weight, and the published diagnostic cannot see it. + +**`missing_informative`: allele-linked read loss.** The ALT-carrying reads were kept with probability r, pools' true frequencies spread by F, coverage negative binomial. With depth graded 22 to 77 in phenotype order and a floor of 20, the closed-form rate at 0.05 under the default rule read 0.0557, 0.0626 and 0.0733 at r 0.4 and F 0.1, 0.3 and 0.5, against an unfloored 0.0483, 0.0486 and 0.0504; at r 0.7 and at r 1 it did not move. The same floor applied to the coverage before the loss, which no real run can do, never moved the rate by more than 0.0008, so the floor acting on a depth that depends on the allele is the mechanism. One shallow pool at one end was never inflated, only conservative. More variable coverage, size 10 against 50, roughly halved the excess. Through the module at r 0.4 and F 0.3, a site read in every unit read 0.0702 under either rule, sites read in five and four units under the mask 0.0567 and 0.0442, and the excess false calls per 1,000 raw sites were 2.88 under the default rule and 6.80 under the mask, which kept every raw site where the default kept 14 percent. The module published `depth_phenotype_cor` 0.993 and `lambda_gc` 1.43 for that table, and `depth_phenotype_cor` reads 0.97 on the harmless r = 1 ladder too. By the frequency of the allele that loses reads, the default rule read 0.0399, 0.0554, 0.0997 and 0.1647 in four bins from 0.05 to 0.95, the last two holding 10 percent of its sites. + +**`missing_yield`: what masking buys.** With 10 percent of 2,000 sites planted at a slope of 0.14 and cells masked independently, the default rule kept 560 sites and found 20 percent of the planted ones; the mask kept 2,000 and found 52 percent, at null rates of 0.0439 and 0.0430. When one gamma factor made the same sites shallow in every pool the gain fell to 26 against 40 percent. Under allele-linked loss the unfloored table found 72 percent, the default rule 18 at a null rate of 0.0629, and the mask 49. `minSamples` 2 and 3 found the same sites, 4 lost none, and 5 lost 13 percent. BH selected nothing in any arm: at six units the smallest p is 1/720, so BH needs one site at that floor for every 36 tested, and these tables held 2.8 to 15 against 9 to 56 needed. + +**`missing_mds`: pairwise deletion.** Over 40 studies of two clades by drift, with one pool 85 percent masked, no pair's corrected distance was biased beyond its margin under either rule (largest z 2.70 of 30 comparisons), and the mask cut the error of the pairs that do not involve the shallow pool 3.3 to 4.6 times against the default rule. The error of one distance went as 0.048 over the root of the sites it rests on, at a distance near 0.018: 133 percent of the distance at 4 shared sites, 61 percent at 17, 34 percent at 58 and 20 percent at 172. That is what the warning at 30 rests on, and the sites here are independent, so linked real data are noisier. With every pool one population, a pool 94 percent masked was the outermost point in the plot in 50 percent of studies under the mask and 46 percent unfloored, against 1/6 if nothing singled it out: noise, not bias. Allele-linked loss at r 0.4 shrank a divergent pool's distances to 0.35 of the truth under the default rule and 0.49 under the mask, and turned a star into a depth gradient in the plot (Spearman of depth with distance from the center 0.68 under the mask). + +## Bite proofs + +Every gate was run against a mutant meant to break it, from scratch copies and in-memory parses, never the repository. In `missing_null`: grouping sites by how many units were read instead of which failed H4 at 4,942 sites; a +1 on the enumerated p failed H4 at 8,197; `roll_up()` without `na.rm` passed the patterns, which hold one pool per unit, and failed the lanes arm at 4,885 sites; an absolute tie tolerance failed H1 and H4 at 10 sites, all in the near-collinear cell; the unmutated module passed both. A dispersion count of one instead of two cannot be caught by any table and is not a gap. In `agree.R`: three one-line mutants of the main loop -- untestable below two units instead of three, BH over every site instead of the tested, theta never applied to the weights -- each disagreed, at 20, 4 and 10 comparisons; the unmutated copy agreed. Through the suite's own case, from a stand-in `REPO_ROOT`, the theta mutant failed it and the unmutated tree passed. The lanes mutant first stopped the section with an R error rather than a FAIL, because the port had dropped an `isTRUE` the prototype carried; every rate gate now has one. + +## A correction to an earlier note + +`association-unread-units.md` says a sampled p "is an unbiased estimate" of the m-unit test. It is valid and not unbiased: with reached ~ Binomial(B, p), the mean of (1 + reached)/(1 + B) is p + (1 - p)/(1 + B). The sampled arm of `missing_null` subtracts exactly that and gates what is left at zero. + +## Where the read loss happens, ruled by Z the same day + +Asked when reads carrying one allele get lost, the answer was: alignment scores them (`bwa mem -T 30`), and the filters after it remove them -- the BAM cleanup's `samtools view -q 30 -f 0x2 -F 0xF0C`, and at variant calling `bcftools mpileup -C 100 -q 30 -Q 30`, whose survivors are what `FORMAT/DP` counts -- with a deletion haplotype contributing no reads at all. Z: "All of those are correct calls. Nothing to change." The reason Z gave: the pipeline promises the user a very conservative number, so every read it counts must be of high quality, which is why only properly paired reads are kept. The manual already says so, in the cleanup walkthrough ("deliberately strict") and in the account of the design. The filters stay as they are, and allele-linked loss is not a reason to revisit them: it is the price of that promise, paid in depth. A check offered at the same time, a pool's `FORMAT/DP` against its non-reference frequency in the step-6 call set, was not taken up. + +## The mds warning at 30 shared sites, built the same day + +`mds` `20261004.001` became `20261006.001`. A pair averaged over fewer than 30 sites is flagged in three places: `few_sites` in `distance.tsv`, 1 below 30; a line on the run's console, under the target's name, naming up to ten such pairs; and a count under `mds.png`. There is no setting for it, because a warning changes no number. The same version drops a clause from a manifest gate and from a comment in `main.nf` that said a cell with no reads is published as 0 in the frequency table; it has been NA since `772db1f`. + +**Why stdout, and why `debug true` on the module's process.** Measured with a two-process Nextflow script: a task that succeeds shows nothing on the console unless its process sets `debug`, and then only its stdout, never its stderr. So `mds.R` writes the warning with `cat()` and `main.nf` prints it, prefixed `MDS