diff --git a/.claude/development-notes/association-unread-units.md b/.claude/development-notes/association-unread-units.md new file mode 100644 index 0000000..0123bd1 --- /dev/null +++ b/.claude/development-notes/association-unread-units.md @@ -0,0 +1,85 @@ +# Association over the units a site was read in + +**Written 2026-10-05, against `dev` at `7419281` plus uncommitted work**: the change described here (`association` `20261005.001`), the moot `dropZeroDepth` off-branch fix, the `run_tests.sh` changes, and `depth-masking-proposal.md`, which records how the defect was found. + +## What was wrong + +One unit with no reads at a site made that site's enumerated `perm_p` exactly 0. `fit_alleles()` zeroed the NA weight, so the fit itself was sound, but `permutation_p()` built `center` from the raw weights: every rearranged statistic came out NaN, the identity was never counted, and `fdr_p` followed to 0 while lowering every other site's adjustment. The published `association-20260910.004` carries the same code. + +Two more places shared the blind spot. `roll_up()` summed a unit's pools without `na.rm`, so one unread pool blanked a unit of several. `dispersion_of()` took its moments over every unit, so a pool unread at every site, a failed library, made theta NA, and a run estimating dispersion then tested nothing. + +## The obvious fix is wrong + +Zeroing the unread units and keeping the one global permutation reports p-values below what the site supports. Every rearrangement moves residuals across all n units, so a read unit is handed an unread unit's zero in most of them, and the null is built from residuals the site never had. It reported p below 1/m! at sites read in m units. The comment over the per-site rearrangement said this during the build and moved here in the comment pass. + +## What it does now + +Each site is rearranged among the units it was read in. `tally()` groups a bin's sites by the set of units read. For each global rearrangement, a group read in M takes the residuals of M's units in the order they appear in that rearrangement, and an unread unit keeps its own zero. The order M's units take inside a uniform permutation of all n is a uniform permutation of M, and across the n! enumerated rearrangements each of M's m! orders appears exactly n!/m! times, so the enumerated p is the exact m-unit test. By the same argument a sampled p is an unbiased estimate of that test, and the verification below measured it so. A bin in which every site is read in every unit takes the original single expression. + +Around that, `permutation_p()` zeroes weights that are NA or not positive and the frequencies under them, as `fit_alleles()` does. `roll_up()` makes a unit the n_eff-weighted mean of its pools read at the site, with weight and frequency NA when none were. `dispersion_of()` estimates theta per site over the units read there, skips a site read in fewer than two, and is NA only when no site has two, which is a run with nothing testable anyway; the weights are then left as they are. A site whose read units all carry one phenotype value has no slope and is not tested, which `spread_of()` decides on the stored values. + +## Decided by Z, 2026-10-05 + +**At least three measured values to test a site.** A site read in fewer than three units is not tested. Its row in `association.tsv` stays, with `n_observed`, and `S`, `perm_p` and `fdr_p` NA; BH runs over the tested sites only, as it already did. Z first said "eliminate"; asked whether that meant dropping the row, Z answered "don't test them", and the row stays because the manifest documents the table as one row per site, always complete. + +**A unit of technical replicates is the mean of its lanes that were read**, by `rowSums(..., na.rm = TRUE)`. Z: "Rowsum looks correct." + +**NA for what does not exist.** The first build gave a unit with no read pool weight 0, which `na.rm` in `roll_up()` makes of an all-NA row, and `mean_weight` then averaged the 0 in: 33.4 at `chr10:500`, read in 2 of 6, where the two read units average 100.25. Z: "NA for non existant is correct." A unit with no read pool now has weight NA, and every aggregate over units leaves it out. `mean_weight` is the mean over the units read at the site. `max_leverage` is the largest over them, and NA where there is no slope to carry. A line that also dropped unread units from that maximum, on the reasoning that `pmax()` with `na.rm` would turn a one-unit site into 0, was reverted: run against the case, removing it changed nothing, because whenever the read unit's leverage is NaN every unread unit's is NaN too (0 times an infinite term), and when it is finite a 0 never wins. `depth_phenotype_cor` correlates the phenotype with each unit's mean weight over the sites it was read at, and a unit read nowhere drops out. That is the rule `basicstats` took for its depth summaries on 2026-09-29. + +## What an untested row shows + +Measured on the corpus with `chr10:500` read in 2 of 6 units: `S`, `perm_p` and `fdr_p` NA, `mean_weight` 100.25 over the two units read, and `max_leverage` 1 and `zero_variance` 1, both true of two points on a line. A site read in one unit, or in units that share one phenotype value, has no slope, and both are NA there. Its alleles do not appear in `association_alleles.tsv`, because selection by `reportBelow` and `reportTop` passes over an NA p, so the NA written into their `b1`, `se`, `t` and `p` is not visible in any published table today. + +## The floor is 1/n! + +The identity always ties, so a design of n units cannot fall below 1/n!, and a site read in m units cannot fall below 1/m!. Nothing guarantees that the reversal ties: it negates the slope exactly only under a symmetry of phenotype and depths that a real design rarely has. Measured on the corpus, `chr2:550`, read in all six units, reaches 1/720, and `chr1:700`, read in four, reached 1/24 = 0.042. I told Z 2/m! during the build and corrected it. + +The module said otherwise, and contradicted itself. The manifest gate that opened "THE SMALLEST P ANY DESIGN CAN REACH" and the comment over `limit` in `association.R` said 2/n! and that four units cannot reach 0.05. `design_floor` was 2/n! while `floor`, enumerated, was already 1/n!, so the comment over them, which says the two differ only when the set is sampled, was false. The manual's `#association-floor` opened with the 2/n! claim and a table to match, then read the two numbers as the floor at similar depths and the reach at differing ones. The per-site message added in this change used 1/m!, so a design of four units was told nothing could be significant while a site read in four of six was not flagged. + +**Ruled by Z, 2026-10-05: "1/n! is right. correct the statements."** `design_floor` is 1/n! and, enumerated, equals `floor`. The ranking sentence fires at three units and no longer at four. The manifest gate and the manual's floor section were rewritten: the table starts at three units, a site's own floor is stated, and a paragraph says what ties do. Two test comments and the floors case were corrected; the case now asserts 1/720 and that an enumerated `floor` equals `design_floor`, and a new case runs four units (1/24, tested, no ranking sentence) against three (1/6, the sentence). `dev/validation/calibrate.R` and `lib.R` keep 2/n!, which is right there: they move labels on an evenly spaced phenotype, where the reversal always ties, and their comments now say so. `calibration.md` is a dated note and is left as written. + +## Adversarial verification, 2026-10-05 + +Four agents attacked the change and a fifth checked what they had covered; the transcript is at `~/.claude/projects/-home-tholian-Nextcloud-GitHub-PoolSeqFlow/7de5ecd1-0266-4a1d-aeb2-69c4394fc0c5/subagents/workflows/wf_77aa51c3-634/`. The arithmetic held: partly read sites agree with independent oracles on thousands of values, fully read data reproduces HEAD, the compiled parse, two and three workers and bins of one to four sites publish byte-identical tables with unread cells, the sampled path sits within Monte Carlo error of the exact p (120 end-to-end runs and 1000 in-process calls at 400 draws), and 8000 simulated null sites with each pool unread at probability 0.3 rejected at nominal for sites read in four, five and six units and conservatively in three. The NA-weight ruling moved `mean_weight`, `depth_phenotype_cor`, `design_floor` and the ranking sentence and nothing else, over 127 compared inputs. + +What they broke, each fixed and given a case: + +- **A site read in three or more units that all carry one phenotype value was tested on rounding.** A weighted mean of one repeated value need not return it (15.1 came back 15.099999999999998), so `sxx` was 5e-28 rather than 0 and the slope over it published `S` near 1e-16 and `perm_p` 1, counted in BH's n and in `lambda_gc`. Unread units are what make such a site reachable. `spread_of()` tests the stored values; such a site is untestable, and its `max_leverage` and `zero_variance` are NA. +- **`max_leverage` 2 at a site read in one unit**, by the same mechanism, at 11.5% of unit and depth combinations on the corpus phenotype. HEAD published NA there, its NA weights propagating, so this was a regression of round 1. NA again, through the same test. +- **No `qq.png` when nothing was tested**, and the frame refuses a folder missing a declared output (`readmeShell()` in `analysis/lib/nf/outputs.nf`). The three-value rule makes such a run ordinary. The figure is drawn with a line saying nothing was tested, and the log says it too. +- **An NA theta took every weight with it.** With dispersion estimated and no site read in two units, `n_observed` was published as 0 where one unit was read. Pre-existing at HEAD. The weights are left as they are and `dispersion` is published NA. +- **The strongest sites published `perm_p` 0**, pre-existing since F2. The tie test was an absolute `seen - 1e-12`, and the identity, rebuilt, lands up to 4e-14 of S away from S, so above S of about 100 it went uncounted and nothing else reached S: 15 of 400 near-linear sites, 2 to 8% of sites with S over 100 in near-collinear tables, 0.38% of realistic fixed-difference binary sites. It contradicted the floor sentence Z had just ruled on. The tolerance is absolute below 1 and relative above it, in the module and in the oracle, and the 15 publish exactly 1/720. +- **Wording.** The new message said "tested site(s) ... ranked, not tested"; the manifest said an unread pool "carries weight zero"; the manual called `mean_weight` an effective sample size under an estimated dispersion too; and the ties sentence named equal depths and an evenly spaced phenotype, where the condition is equal weights and a phenotype symmetric about its mean, and missed the doubling a balanced categorical phenotype gets from the swap of its levels (3 against 3 at equal depths: 72 of 720 tie, so 0.1). +- **Stale floor statements predating today**: in `freq_corpus.py`, 2/20 at chr10:1600 where the corpus publishes 1/720, 4/720 where it publishes 9/720, and "capped at 0.1" where it publishes 6/720; in `calibrate.R`, 1/720 "whatever the phenotype's shape", false for its own flat rows. + +The suite could not fail on most of round 1, measured by running mutants of the module through its own cases: the new message, the shortcut for a fully read bin, grouping sites by how many units read them rather than which, the dispersion arithmetic, the weighting inside a unit, BH's n, and the warning the allele-row mask suppresses. Each now has an assertion. The oracle gained `--units`, `fdr_p` and theta, and the cases compare every site against it. + +Not fixed, and left to Z: + +- A depth table holding exactly one site stops `association.R` at `colnames<-`: the weight matrix collapses to a vector. Pre-existing and unrelated. +- Enumeration is capped at eight units whatever the budget, which the manual now says. With nine or more and `permutations` above n!, a sampled p can fall below 1/n!. +- Dropping the ranking sentence at four units leaves the default output less cautious than HEAD's. Under BH an `fdr_p` of 0.05 needs about 83% of tested sites at the floor at four units, 17% at five and 2.8% at six. +- The m-unit test assumes that which units were read at a site is unrelated to what they carry. A pool unread because of its own genotype breaks that, and no gate says so. + +## Verification + +`freq_corpus.py --association-from ` is the oracle: plain Python that leaves out a pool with depth 0 at each site and enumerates the read units' own m! rearrangements directly, so it shares neither code nor the induced-order construction with the R. The corpus files it writes, `expected.tsv` included, are byte-identical to HEAD. + +On the corpus with sites read in 6, 5, 4, 3 and 2 units, R and the oracle agree at all 14 sites. Each fix, reverted alone, fails its case: + +| reverted | what the case saw | +|---|---| +| the per-site rearrangement and the zeroing | `perm_p` 0 at the partly read sites | +| the three-value rule | the 2-unit site published `S` = 0 | +| `na.rm` in `roll_up()` | `n_observed` 2 and `S` NA where a unit kept one read pool of two | +| `dispersion_of()` | theta NA and nothing tested, with one pool unread everywhere | + +**`assert_close` in `test/lib/analysis.sh` could not fail on a missing value.** It compared through awk, which reads `NA`, `NaN` and `Inf` as 0, so "expected 0, got NA" passed, and the three-value revert passed under it. It now requires those tokens to match literally. Another entry for `gates-that-stopped-checking.md`. + +Suites, with `TEST_CONDA_ENV` at `PoolSeqFlow-3.2.0` and `TEST_ANALYSIS_ENV` at `PoolSeqFlow-3.1.1-analysis`: association, basicstats and mds in full, 39 passed; `--fast` 401 passed, 0 failed, 234 skipped, every skip `(--fast)`; `00_static` 52 passed; `check-analysis-versions.sh` clean. Those ran before the comment pass, which changed no code line: the non-comment lines of `association.R` are identical before and after it, and the file parses. + +After the rulings and the verification's fixes: association 15 passed, `00_static` 52 passed. The corpus files `freq_corpus.py` writes, in both of its modes, are still byte-identical to HEAD's, through the oracle's rewrite onto units. Each guard was reverted alone in a copy of the module and run through the suite's own cases from a stand-in `REPO_ROOT`, so the repository file never changed: 21 of them, and each failed the case meant to catch it, while round 1 as it stood fails all seven of the cases this work touched. A 22nd mutant passed, which proved a line redundant: the allele p-values computed over every site rather than the tested ones, since the t-ratio is already NA where untested and `pt()` returns NA for NA without warning. The line was simplified, and removing the mask itself now fails the case on its warning. + +## Not exercised + +The suite's cases run enumerated and one worker; `test_both_paths_through_the_parse_agree` now parses two partly read sites both ways. The sampled path, several workers and several bins with unread cells were run by the verification above and are not in the suite. Not run anywhere: the frame on a run that tests nothing, which was read from the code and not driven through Nextflow; real data with zero-coverage cells; a missing-cell scenario in `dev/validation/calibrate.R`; and more than 20,000 sites. diff --git a/.claude/development-notes/depth-masking-proposal.md b/.claude/development-notes/depth-masking-proposal.md new file mode 100644 index 0000000..9bd52f2 --- /dev/null +++ b/.claude/development-notes/depth-masking-proposal.md @@ -0,0 +1,132 @@ +# Masking shallow cells instead of dropping sites, and the association defect it exposed + +**Written 2026-10-05, against `dev` at `d76ba2f` plus uncommitted work** (the `dropZeroDepth` off-branch fix to `FMT/DP=minDP & FMT/DP>0)>=K'` reproduces every set above exactly, and `COUNT()` gives the same sets. Two traps, both silent: + +- **`&&` and `||` inside `N_PASS` are not per-pool.** At minDP 20, `N_PASS(FMT/DP>=20 && FMT/DP>0)` counts 4,2,4,3,4 where the true count is 4,2,2,1,3, so K=3 also keeps 300 and 400. The same holds for `||`: today's ON expression uses `||`, so a count ported from it would be wrong. A case asserting K=3 keeps exactly 100 and 500 on these depths catches it. +- **The floor alone counts an unread pool as measured at minDP 0.** `N_PASS(FMT/DP>=0)` is 4 everywhere. The `& FMT/DP>0` term is what the `|| FMT/DP==0` term does in today's expression. + +bcftools accepts any K without complaint: K=5 on four pools keeps nothing, K=0 and K=-1 keep everything, all exit 0. + +## Masking is new code, and it has to reach the depth table + +**Every module reads the DEPTH table, and nothing reads the frequency table's numbers.** The depth table is `FORMAT/AD` printed by `bin/createDepthFile.sh:48` before `bin/depth2freq.awk` runs. A site filter alone, with no mask, hands every module a 5-read pool as a measured frequency (site 300 publishes that pool at 0.6/0.4). So the mask has to rewrite `FORMAT/AD`, and `FORMAT/DP` with it. + +No bcftools route does that reliably. `filter -S` and `+setGT` change GT only, and `MajorAlleleToRef.py:84` has already set every GT to `./.`. `+fill-tags` rewrites `FORMAT/DP` but refuses `FORMAT/AD`, and a comparison inside its expression turns every cell missing. A `query -i ... -F '0,0'` followed by `annotate` reproduces the example but fails silently with exit 0 on two cases: a site where every pool is shallow stays unmasked, and a three-allele site gets a two-value AD that makes `MajorAlleleToRef.py` exit 1. A text pass works: an awk mask writing one 0 per allele (counted from the ALT column at the point of masking) and DP 0 reproduced the example exactly and ran cleanly through step 7's own scripts, publishing `0,0` in the depth table and NA on every allele row of the frequency table. + +## The association defect, which exists today + +**`modules/association/association.R` publishes a wrong p-value at every site where a unit is unmeasured, and says nothing.** `fit_alleles` zeroes an NA weight (`:111-112`), but `permutation_p` builds `center` from the raw weights (`:209-211`). A unit at depth 0 has an `n_eff` of NA, so every rearranged statistic is NaN, the identity is never counted, and an enumerated p comes out `reached / count = 0` (`:239`); a sampled one lands exactly on the floor. Reproduced: chr1:250 `perm_p` 0.4 to 0, `fdr_p` 0.627 to 0, an unaffected site's BH p 0.0542 to 0.0406, selected sites 2 to 4, exit 0. + +`association.R` is unchanged since `dbc522d` (2026-09-10), and the published `modules/repo/association-20260910.004.tar.gz` carries a byte-identical copy. `.claude/development-notes/unmeasured-cells.md:58` recorded association as safe "by the same route" as basicstats; that checked the fit, not the permutation. + +When it is reached: + +- committed `dev`: only with `minDP` at 0 and `dropZeroDepth` off +- the uncommitted off-branch fix: with `dropZeroDepth = false` alone, because the fix is what finally publishes zero-depth sites +- the proposal: by default + +**So the off-branch fix is correct as a filter and unsafe to ship while association is unfixed.** Before it, `false` dropped the sites it claimed to keep, and that bug was all that stood between a user and false association hits. + +Four more association behaviors that masking turns from rare into routine, each reproduced: + +- **No per-site minimum.** A site with 2 measured units publishes S 0, `perm_p` 0 and `fdr_p` 0; a site with 3 is tested on one degree of freedom (`df <- observed - 2`, `:134`). The refusal below three units (`:271-276`) counts the design's units, not the site's. +- **`roll_up()` sums without `na.rm`** (`:96-97`). One masked technical replicate drops its whole unit at that site, even when the other replicate was read. +- **Dispersion comes only from fully measured sites.** If none is, theta is NaN, every weight is NaN, and the run publishes all NA with exit 0. +- **`depth_phenotype_cor` turns NA from one masked cell** (`:445`), which is the diagnostic for depth lining up with phenotype. `lambda_gc` drifts with missingness. + +basicstats handles unmeasured cells correctly, and its compiled and R paths agree. mds handles them by pairwise deletion and stops loudly when two pools share no site, though its message points at a `distance.tsv` the failed run never writes. The shared libraries handle a depth-0 cell; `harmonic_mean()` returns 0 on any zero by design, and its only callers pass none. + +## What the evidence leaves for Z to decide + +- **Whether a pool below minDP counts as evidence anywhere.** The false-positive filter runs before the depth filter and counts any `dp > 0` as support (`bin/filterFalsePositives.sh:134`). Masking only at the depth filter published an allele with zero reads in every published pool, admitted on the support of pools then masked. Masking only at the start published a pool below minDP, because the FP filter dropped an allele and `MajorAlleleToRef.py:82` recomputed DP from 21 to 18. Masking at both points avoided both. QUAL has the same exposure, unmeasured: `bcftools call` computed it from every pool's reads. +- **How a masked cell is written.** Zeros work with every consumer but make a pool read 5 times identical to one never read, which reverses the premise of the 2026-09-29 NA work and changes what basicstats' `unmeasured` column counts. A per-allele missing value (`.,.`) keeps them apart and the libraries read it as unmeasured, but `MajorAlleleToRef.py:74` calls `int()` on every AD value, so it only works after the last re-sort, which is the placement that leaks. With today's scripts the two goals cannot both be had. +- **One parameter or two.** `true` is exactly K equal to the pool count; `false` differs from K=1 only at a site where every pool is masked, and only when the mask runs after the FP filter. Two parameters give 2 x N combinations for N behaviors, and the proposed default pair (true, 2) means either the count is ignored or it contradicts `true`. +- **K's default against what the modules need.** K counts VCF columns; association fits units built after `analysis.design.technicalRep`, and needs three per site. A default of 2 is below that, and two replicate pools of one unit satisfy K=2 while reaching association as one unit or none. +- **Whether missingness is informative.** If a pool's depth at a site depends on the allele it carries there (mapping loss on a divergent allele, an indel, copy number), masking removes that pool exactly where its frequency differs, where today's rule removes the site for everyone. Nothing measures what K=2 does to association's calibration or to where mds places a shallow pool, and `dev/validation/calibrate.R` has no missing-cell scenario. +- **The FP filter's threshold under masking.** `MINSAMPLES` is all columns times `sampleThreshold` (`filterFalsePositives.sh:67,73`). With the mask before it, the share of measured pools an allele must reach grows with missingness: ten pools at 0.2 need 2 of 2 at a site with two measured pools, 2 of 10 at a full one. + +## Decided by Z, 2026-10-05, later the same day + +The count called K above is the parameter Z named `minSamples`. + +- **`dropZeroDepth` is replaced by `keepLowDepthAsZero`, default `false`, with `minSamples` consulted only when it is `true`.** `false` is today's rule exactly -- every pool at or above `minDP` -- for every value of `minSamples`. `true` masks every cell below `minDP` and keeps a site with at least `minSamples` measured pools. Checked against the example: `false` keeps 100 at every `minSamples`; `true` reproduces the K sets above. The working tree's `FMT/DP=20)>=2"` as a one-line swap, which hands sub-minDP pools to every module as measured, and says per-sample blanking belongs in `bin/depth2freq.awk`, which writes only the frequency table no module reads. +- `modules/association/manifest.json:25`, `modules/mds/manifest.json:28` and both `main.nf` headers say a cell with no reads is published as 0; it has been NA since `772db1f`. +- Stale in the working tree after the off-branch fix: `modules/basicstats/test/basicstats.sh:262`, `test/suites/03_helpers.sh:1583-1587`, `test/tools/freq_corpus.py:65-66` and `test/README.md:222-223`, each saying a zero cell needs both minDP 0 and `dropZeroDepth` off. + +## Built, 2026-10-05, the same evening + +Built on top of the association fix in `association-unread-units.md`, which also records Z's confirmation of the replicate rule ("Rowsum looks correct"). Uncommitted. + +`vcffilter.keepLowDepthAsZero` (default `false`) and `vcffilter.minSamples` (default 2) replace `dropZeroDepth`. `false` runs today's expression unchanged, `FMT/DP = ` with no value; the mask is 7 to 13 times slower than `bcftools view -e` (48 pools by 20,000 records, 2.8 to 5.6 s against 0.42 s); mawk and BWK awk were never run; CRLF input loses its carriage return on masked records, which `MajorAlleleToRef.py` makes unreachable. diff --git a/.claude/development-notes/missing-cell-validation.md b/.claude/development-notes/missing-cell-validation.md new file mode 100644 index 0000000..c579de4 --- /dev/null +++ b/.claude/development-notes/missing-cell-validation.md @@ -0,0 +1,66 @@ +# Missing-cell scenarios in the validation harness + +**Written 2026-10-06, against `dev` at `7419281` plus uncommitted work**: the association fix for unread units (`association-unread-units.md`), the depth-mask redesign (`depth-masking-proposal.md`), and this change. Every number below was read from the harness as committed in this change, at its default seed 20260907 and default replicate count, under R 4.6.1. + +## What was asked + +Z decided on 2026-10-05 that missing-cell scenarios are to be added to the validation harness, alongside `vcffilter.keepLowDepthAsZero`. The proposal note had named the gap: nothing measured what masking does to association's calibration or to where mds places a shallow pool, and nothing tested whether missingness is informative. Z opened the work on 2026-10-06: "Let's move with the next task. missing cell scenarios?" + +## How it was designed + +A workflow, `wf_831b725c-90d` (transcript under `~/.claude/projects/-home-tholian-Nextcloud-GitHub-PoolSeqFlow/7de5ecd1-0266-4a1d-aeb2-69c4394fc0c5/subagents/workflows/`), every agent on Sonnet 5.5: three independent proposals, one each on calibration, on how cells really go missing in pool-seq, and on building it in this harness, then a synthesis that resolved their conflicts and wrote and ran a prototype of every piece. It took 150 minutes and 424 tool calls, far longer than expected, because the proposal agents ran the measurements themselves; Z chose to let it finish. The prototypes were then ported into the harness in its own style with the order of every random draw kept, and the port reproduces every number the prototypes printed at the default seed. + +The synthesis resolved four conflicts that are worth keeping. The estimator calls the module's own functions rather than a restatement of them, with a brute-force oracle as the independent check. Read sets are enumerated, every pattern of read units in every cohort, wherever a deterministic check or an exact stratum is wanted, because the module groups sites by which units were read and a grouping bug shows only on the patterns it gets wrong; realistic compositions are drawn only where the composition is itself the quantity. Rates over enumerated p are gated on four standard errors of the larger of the binomial and the across-cohort spread, and rates over shared nuisance on a t quantile of the spread alone, after a Monte Carlo of 400,000 nulls showed that four standard errors of a spread from four to eight cohorts alarm 0.2 to 1.1 percent per comparison. And OFF, step 7's default rule, is read as the six-unit stratum of the masked table for per-site rates, and run as a table of its own only where table-level numbers are compared. + +## Decided by Z, 2026-10-06 + +- **The `missing_*` estimators are the modules' own functions**, parsed out of `association.R` and `mds.R` by `module_functions()`, rather than a copy in `lib.R`. "1. yes." +- **`agree.R` runs as a case of association's suite**, so `--changed` runs it whenever the module, a library, `mds.R`, `bin/mask_depth.awk` or the harness's `lib.R` changes. It ends the README's "nothing here runs in run_tests.sh". "2 yes." +- **The manual's `lambda_gc` line is to be rewritten.** It says "If it is 3, nothing in the table is a p-value", and an exactly calibrated null crosses 3 at three-allele sites. "Definitely rewrite manual." The next item, not built here. +- **mds is to warn when a pair rests on few shared sites, at 30.** Z: "MDS warning sounds right. I feel like maybe we should not even entertain MDS below 100 shared sites. But 30 sounds reasonable and not arbitrary." Taken as a warning at 30, with 100 a thought rather than a decision. Not built here. + +## What the sections measured + +**`missing_null`: the masked p is the exact m-unit test.** Over every pattern of read units in seven cells, 18,605 tested sites, no p fell below 1/m!, every n_observed was the units the generator read, every site that must carry no statistic carried none, and every p equaled a brute-force enumeration of the m! orders to 1e-9 (largest gap 0). The rejected scheme, unread units zeroed under one rearrangement of all six, put 47 of 767 partly read sites below 1/m!, disagreed with the oracle at 98 percent of them, and rejected 0.077 of the three-unit sites against a bound of 0, so the gates can see it. Pooled rates at four, five and six units read 0.0319, 0.0471 and 0.0493 against bounds of 0.0417, 0.05 and 0.05. At resolution, 8,000 sites per stratum, flat depth read 0.0286, 0.0524 and 0.0494, the last two the two-sided control, and aligned depth 0.0366, 0.0501 and 0.0527. Four units sit 20 to 30 percent under their bound from count ties alone: any tie among four frequencies makes 1/24 unreachable. The sampled path at eight units took 499 rearrangements, its smallest p was 1/(1+B), and its p less the exact p less (1 - p)/(1 + B) averaged -0.0011 against a margin of 0.0026. Units of two lanes of one pool, 57 percent of them read through one lane, kept their n_observed and read 0.0363, 0.0466 and 0.0514. + +**`missing_published`: what a run says about itself.** Masking kept 99 percent of raw sites where the default rule kept 38, 12 and 0.7 percent at 15 percent, 30 percent and a ladder of 94 to 7 percent of cells under the floor; 0.5, 6 and 12 percent of the masked rule's sites were read in fewer than three units and untested. `lambda_gc` on a perfectly null table read 1.296, 1.485 and 1.657, each within 2 percent of the null of its own mix of units (ratios 0.994, 1.019, 0.995, and 0.993 on the aligned ladder). The null itself is 2.198, 1.465, 1.286 and 1.206 for sites read in three to six units, and 3.81 at three alleles and 6.03 at four over six units, so the published value is not near 1 on a clean table and crosses 3 on an exactly calibrated one. Theta, each cohort analyzed fully read and masked, moved by 2.1 percent of its true value at most under random, depth-floor and phenotype-aligned masks; it stayed finite with a library unread at every site and with 85 percent of cells masked, and fell 4 percent under allele-linked loss, which is reported only. With missingness lined up with the phenotype at equal depth, `depth_phenotype_cor` is NA: every unit carries the same weight, and the published diagnostic cannot see it. + +**`missing_informative`: allele-linked read loss.** The ALT-carrying reads were kept with probability r, pools' true frequencies spread by F, coverage negative binomial. With depth graded 22 to 77 in phenotype order and a floor of 20, the closed-form rate at 0.05 under the default rule read 0.0557, 0.0626 and 0.0733 at r 0.4 and F 0.1, 0.3 and 0.5, against an unfloored 0.0483, 0.0486 and 0.0504; at r 0.7 and at r 1 it did not move. The same floor applied to the coverage before the loss, which no real run can do, never moved the rate by more than 0.0008, so the floor acting on a depth that depends on the allele is the mechanism. One shallow pool at one end was never inflated, only conservative. More variable coverage, size 10 against 50, roughly halved the excess. Through the module at r 0.4 and F 0.3, a site read in every unit read 0.0702 under either rule, sites read in five and four units under the mask 0.0567 and 0.0442, and the excess false calls per 1,000 raw sites were 2.88 under the default rule and 6.80 under the mask, which kept every raw site where the default kept 14 percent. The module published `depth_phenotype_cor` 0.993 and `lambda_gc` 1.43 for that table, and `depth_phenotype_cor` reads 0.97 on the harmless r = 1 ladder too. By the frequency of the allele that loses reads, the default rule read 0.0399, 0.0554, 0.0997 and 0.1647 in four bins from 0.05 to 0.95, the last two holding 10 percent of its sites. + +**`missing_yield`: what masking buys.** With 10 percent of 2,000 sites planted at a slope of 0.14 and cells masked independently, the default rule kept 560 sites and found 20 percent of the planted ones; the mask kept 2,000 and found 52 percent, at null rates of 0.0439 and 0.0430. When one gamma factor made the same sites shallow in every pool the gain fell to 26 against 40 percent. Under allele-linked loss the unfloored table found 72 percent, the default rule 18 at a null rate of 0.0629, and the mask 49. `minSamples` 2 and 3 found the same sites, 4 lost none, and 5 lost 13 percent. BH selected nothing in any arm: at six units the smallest p is 1/720, so BH needs one site at that floor for every 36 tested, and these tables held 2.8 to 15 against 9 to 56 needed. + +**`missing_mds`: pairwise deletion.** Over 40 studies of two clades by drift, with one pool 85 percent masked, no pair's corrected distance was biased beyond its margin under either rule (largest z 2.70 of 30 comparisons), and the mask cut the error of the pairs that do not involve the shallow pool 3.3 to 4.6 times against the default rule. The error of one distance went as 0.048 over the root of the sites it rests on, at a distance near 0.018: 133 percent of the distance at 4 shared sites, 61 percent at 17, 34 percent at 58 and 20 percent at 172. That is what the warning at 30 rests on, and the sites here are independent, so linked real data are noisier. With every pool one population, a pool 94 percent masked was the outermost point in the plot in 50 percent of studies under the mask and 46 percent unfloored, against 1/6 if nothing singled it out: noise, not bias. Allele-linked loss at r 0.4 shrank a divergent pool's distances to 0.35 of the truth under the default rule and 0.49 under the mask, and turned a star into a depth gradient in the plot (Spearman of depth with distance from the center 0.68 under the mask). + +## Bite proofs + +Every gate was run against a mutant meant to break it, from scratch copies and in-memory parses, never the repository. In `missing_null`: grouping sites by how many units were read instead of which failed H4 at 4,942 sites; a +1 on the enumerated p failed H4 at 8,197; `roll_up()` without `na.rm` passed the patterns, which hold one pool per unit, and failed the lanes arm at 4,885 sites; an absolute tie tolerance failed H1 and H4 at 10 sites, all in the near-collinear cell; the unmutated module passed both. A dispersion count of one instead of two cannot be caught by any table and is not a gap. In `agree.R`: three one-line mutants of the main loop -- untestable below two units instead of three, BH over every site instead of the tested, theta never applied to the weights -- each disagreed, at 20, 4 and 10 comparisons; the unmutated copy agreed. Through the suite's own case, from a stand-in `REPO_ROOT`, the theta mutant failed it and the unmutated tree passed. The lanes mutant first stopped the section with an R error rather than a FAIL, because the port had dropped an `isTRUE` the prototype carried; every rate gate now has one. + +## A correction to an earlier note + +`association-unread-units.md` says a sampled p "is an unbiased estimate" of the m-unit test. It is valid and not unbiased: with reached ~ Binomial(B, p), the mean of (1 + reached)/(1 + B) is p + (1 - p)/(1 + B). The sampled arm of `missing_null` subtracts exactly that and gates what is left at zero. + +## Where the read loss happens, ruled by Z the same day + +Asked when reads carrying one allele get lost, the answer was: alignment scores them (`bwa mem -T 30`), and the filters after it remove them -- the BAM cleanup's `samtools view -q 30 -f 0x2 -F 0xF0C`, and at variant calling `bcftools mpileup -C 100 -q 30 -Q 30`, whose survivors are what `FORMAT/DP` counts -- with a deletion haplotype contributing no reads at all. Z: "All of those are correct calls. Nothing to change." The reason Z gave: the pipeline promises the user a very conservative number, so every read it counts must be of high quality, which is why only properly paired reads are kept. The manual already says so, in the cleanup walkthrough ("deliberately strict") and in the account of the design. The filters stay as they are, and allele-linked loss is not a reason to revisit them: it is the price of that promise, paid in depth. A check offered at the same time, a pool's `FORMAT/DP` against its non-reference frequency in the step-6 call set, was not taken up. + +## The mds warning at 30 shared sites, built the same day + +`mds` `20261004.001` became `20261006.001`. A pair averaged over fewer than 30 sites is flagged in three places: `few_sites` in `distance.tsv`, 1 below 30; a line on the run's console, under the target's name, naming up to ten such pairs; and a count under `mds.png`. There is no setting for it, because a warning changes no number. The same version drops a clause from a manifest gate and from a comment in `main.nf` that said a cell with no reads is published as 0 in the frequency table; it has been NA since `772db1f`. + +**Why stdout, and why `debug true` on the module's process.** Measured with a two-process Nextflow script: a task that succeeds shows nothing on the console unless its process sets `debug`, and then only its stdout, never its stderr. So `mds.R` writes the warning with `cat()` and `main.nf` prints it, prefixed `MDS