Skip to content

fix(setup): write genetic maps atomically so a truncated map is rebuilt - #106

Closed
geiresx wants to merge 135 commits into
mainfrom
pgp/rust-p5
Closed

geiresx wants to merge 135 commits into
mainfrom
pgp/rust-p5

Conversation

@geiresx

@geiresx geiresx commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

GeiserX#377

Summary by CodeRabbit

  • New Features
    • Added setup commands to build STRchive catalogs, extract Exomiser archives, and prepare genetic maps, including optional RFMix-formatted maps.
    • Catalog setup supports optional extra loci and repeat-range output.
    • Archive extraction refuses absolute paths and paths containing ...
  • Bug Fixes
    • Genetic-map setup reports malformed or missing chromosome data and preserves existing maps when processing fails.
    • Fine-ancestry maps are regenerated when their completion stamp no longer matches the current inputs.
    • Invalid catalog or locus data is reported before output files are written.
  • Documentation
    • Updated setup guides to describe the commands and their use.

og pgx cpic-report and og pgx step36-outside, in the new crate og-pgx:
the gene status as one typed classifier, the drug guidance with its
sources as an enum, the step 36, pypgx, phenoconversion and
contraindication sections as functions, and Python's own error texts
where the report quotes them. --date sets the Date line (default today
in UTC); the tables are written atomically.

Bead: pgp-bs9u.19
Every report-level fixture of tests/test_cpic_parser.py and of the
step 36 checks, the real 3.2.0 and 3.4.0 reports, the four layouts and
the error paths, compared item by item with the Date line masked.

Bead: pgp-bs9u.19
…e.py

Step 27, the CPIC_LOOKUP module, the demo sample builder and the unit
tests now run og pgx. tests/og_binary.py finds the og a test runs; Guard
builds it once for the unit-test shards, and the docs and demo-pictures
builds build it too. The function-level checks of test_cpic_parser.py
moved to Rust tests; the report-level ones run og as a subprocess.

Bead: pgp-bs9u.19
Red control, reverted in the next commit: a gene counts as ambiguous
only with three or more different phenotypes.

Bead: pgp-bs9u.19
The red control did its job; the rule is two or more different
phenotypes again.

Bead: pgp-bs9u.19
The e2e scope covers bin/ but not rust/, so the cases that run step 27
list the og shim; a change to og-pgx is held by the parity cases.

Bead: pgp-bs9u.19
The Rust port of step 40's tool: the site table from the 23andMe v5 list
and dbSNP read in parallel, the gVCF walk that genotypes every chip site,
both chip layouts zipped with a fixed date, and the strict reader.

Bead: pgp-bs9u.14
A case that writes a gzip table and a plain TSV needs gz-content for one
and bytes for the other. The kinds column now takes GLOB=KIND pairs and a
default; the self-test plants a difference the per-file kind must catch.

Bead: pgp-bs9u.14
Synthetic gVCFs, site tables, dbSNP slices and chip files, plus a
generator for a whole-genome-sized synthetic case. The goldens are the
deliberate fixes: ALTs of one rsID at one position are joined, and a call
of three alleles stops the export.

Bead: pgp-bs9u.14
The step script, the Nextflow module, setup.sh --consumer-sites and the
e2e case call og consumer; bin/export_consumer.py and its unit test go,
their cases now live in tests/parity and the crate's tests. build-sites
reads dbSNP itself on 4 cores, so the grep prefilter goes too.

Bead: pgp-bs9u.14
PharmCAT's example quotes genotypes in its guideline text, which the
personal-data scan refuses outside tests/fixtures, so the case builds
its report from the fixture instead of committing a copy.

Bead: pgp-bs9u.19
A new crate, og-qc, holds the estimate: one streaming pass over
mosdepth's windows (BGZF inflated on every core) and the VCF, with
CPython's medians, fmean and exact variance from pycompat::stats and
every int quotient correctly rounded. 32 parity cases check it against
bin/mosaic_sex_loss.py in PYTHON_IMAGE; four goldens hold the deliberate
fixes (a nan depth, an AD past 32 bits, non-UTF-8 input, og's usage line).

Bead: pgp-bs9u.22
…gone

scripts/33-sample-qc.sh, the SAMPLE_QC module, the e2e case and the demo
sample builder call og qc sex-loss. Guard builds og once for the unit
tests (tests/og_binary.py), and the docs and demo-picture builds build it
for the demo's mosaic rows. The Python unit test moved into the crate.

Bead: pgp-bs9u.22
… oracles

Four oracle.ref rows (the three cyp2d6 subcommands and the motif count)
at 81567aa, and six cyp2d6-check cases on the depth fixtures.

Bead: pgp-bs9u.23
Steps 21, 32 and 9b and their modules call og pgx cyp2d6-* and og str
motif; the bash steps run og from their output directory with relative
outputs and pass OG_SHADOW, so CI can shadow them. The Python is deleted.

Bead: pgp-bs9u.23
…he tests onto og

OG_SHADOW names the four tools; the fake-docker cases require og and
refuse the Python; the unit tests check the meaning of the goldens og is
held to.

Bead: pgp-bs9u.23
# Conflicts:
#	CHANGELOG.md
#	rust/Cargo.lock
#	rust/og/src/main.rs
…cle runs measure

Planted on purpose, reverted in the next commit: Guard's fake-docker
case and parity job must go red.

Bead: pgp-bs9u.23
The repo ignores *.vcf, so the synthetic input stayed out of the commit
and the parity job stopped on the missing file.

Bead: pgp-bs9u.22
This reverts commit e917d9b. The naive variance was planted to show
that the bit-exact estimate test fails on it.

Bead: pgp-bs9u.22
# Conflicts:
#	rust/Cargo.lock
#	rust/og/src/main.rs
…one star table

og-clinvar holds the review-star table, the genotype and combined-allele
wording and the call floor once. og clinvar join-adjacent --mode build reads
the reference itself, so the prepare, fill-from-fasta, finish pipe is one
process. The parity cases compare every mode with the awk it replaces.

Bead: pgp-bs9u.13
GeiserX and others added 29 commits November 11, 2026 17:14
# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	docs/rust.md
#	rust/og/src/main.rs
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
# Conflicts:
#	CHANGELOG.md
#	docs/rust.md
#	rust/Cargo.lock
#	rust/Cargo.toml
#	rust/og/src/main.rs
#	scripts/ci/e2e-case-paths.tsv
# Conflicts:
#	CHANGELOG.md
#	docs/rust.md
#	rust/og/Cargo.toml
#	rust/og/src/main.rs
#	scripts/ci/e2e-case-paths.tsv
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…-z2a

# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	docs/rust.md
#	rust/og/src/main.rs
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…-r0-switch

# Conflicts:
#	.github/workflows/guard.yml
# Conflicts:
#	.github/workflows/guard.yml
#	docs/rust.md
#	rust/Cargo.toml
#	rust/og/Cargo.toml
#	rust/og/src/main.rs
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
install_fine_maps now holds step 26b's map block, so a test can run it.
tests/test_setup_fine_maps.sh plants a cut-short chr22.rfmix.map with no
maps/DONE and checks it is written again, that maps/DONE then stops a
rerun, and that a stamp of other chromosomes does not. Its control runs
the rule of before and sees the cut-short map kept.
# Conflicts:
#	.github/workflows/demo-pictures.yml
#	.github/workflows/docs.yml
#	CHANGELOG.md
#	docs/nextflow.md
#	docs/rust.md
#	rust/Cargo.lock
#	rust/Cargo.toml
#	rust/README.md
#	rust/og.sh
#	rust/og/Cargo.toml
#	rust/og/src/main.rs
#	scripts/ci/e2e-case-paths.tsv
#	scripts/setup.sh
#	tests/demo/make_demo_sample.py
#	tests/og_binary.py
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…-c6c

# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	docs/rust.md
#	rust/Cargo.toml
#	rust/og/src/main.rs
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…atalog og cannot replace

A damaged zip whose end record claims more than the file holds, or a
member that inflates past its recorded size, now stops with exit 2
before memory or disk fills. og str catalog takes -h and --help and
refuses an extra locus whose names are not text. setup.sh builds og
before it removes an outdated STRchive catalog, and rewrites step 26b's
maps when a map file is missing under maps/DONE.
…-b1-switch

# Conflicts:
#	.github/workflows/e2e.yml
#	CHANGELOG.md
#	docs/nextflow.md
#	docs/rust.md
#	docs/testing.md
#	rust/og.sh
#	scripts/ci/e2e-case-paths.tsv
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…ilds without it

og str catalog used og_bam::motif::forms, which made og-setup, and so
every og build, depend on og-bam and htslib, the C library the macOS
build leaves out. og-setup now holds the same 20-line rule, and a test
checks it against og-bam's on every motif of up to six bases.
# Conflicts:
#	CHANGELOG.md
#	docs/rust.md
#	rust/Cargo.lock
#	rust/crates/og-steps/Cargo.toml
#	rust/crates/og-steps/src/lib.rs
#	rust/og/src/main.rs
#	scripts/ci/e2e-case-paths.tsv
#	scripts/ci/oracle-diff.sh
#	tests/parity/.gitignore
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…require it in the rust job

unit-tests now has a host og in OG for the og dev checks, so OG being set no
longer means bin/.og is built. The two tests that run the real step 06 ask
bin/og itself, as the step 05 tests do; the rust job sets OG_REQUIRED so a
skip there is a failure.
# Conflicts:
#	.github/workflows/e2e.yml
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	bin/rfc1_motif.py
#	docs/rust.md
#	rust/Cargo.toml
#	rust/README.md
#	rust/og/src/main.rs
#	scripts/ci/e2e-case-paths.tsv
#	tests/fake-docker/str-catalog-setup-and-steps.sh
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
#	tests/test_str_catalog.py
…Python does

os.walk's relpath depth counts the root and its children both as 0, so the
Python lists files four folders below the sample and stops there. og stopped
one folder sooner and dropped files such as mito/eklipse/<dir>/<dir>/*, which
made the dev-contract shadow fail on the HG002 tree. A unit test builds a
five-level tree and expects the oracle's listing.
…roducibly (GeiserX#371)

## The problem

Step 40 writes a person's genome as the raw-data file a chip company gives its customers, in the 23andMe layout and the AncestryDNA layout. A Python program did the work. It read the whole gVCF in one process, held every row it checked in memory, and wrote zips whose bytes changed with the clock. On a synthetic gVCF of 19 million lines with a table of 611,936 sites, it took 17 s and 667 MB in a Linux container. Building the site table also needed a grep pass over the 30 GB dbSNP file before Python could start.

## The fix

**The port.** `og consumer` has three verbs in a new crate, `rust/crates/og-steps`:

- `og consumer build-sites` reads dbSNP's BGZF file on 4 cores. It looks at the ID column of a line before it parses anything else, so `setup.sh --consumer-sites` drops the grep prefilter.
- `og consumer export` reads the gVCF on up to 4 cores and runs the same walk as the Python: the pending records at one position, and the spans of called deletions. It writes one layout at a time. Each line goes to the strict reader and to the zip member as it is made, so no whole text is held in memory.
- `og consumer check` is the strict reader of both layouts, on a `.txt` or a `.zip`.

The step script, the Nextflow module, `setup.sh --consumer-sites` and the e2e case `consumer-export-1` call `og`. `bin/export_consumer.py` and `tests/test_export_consumer.py` are deleted. Their cases moved to `tests/parity` and to the crate's own tests.

**What changes on purpose:**

- The zips are reproducible. Each member is dated with the day of the export (`--date YYYY-MM-DD`, else `SOURCE_DATE_EPOCH`, else today in UTC) at midnight. The same gVCF, table and day give the same bytes.
- A successful export leaves no `.txt` files, and it removes the two a failed run left behind, so no one uploads a file the strict reader refused. When the strict reader fails, both `.txt` files are still left unzipped for a look, as before.
- A call of three or more alleles at a site stops the export with the site and its position. The Python wrote a three-letter genotype that its own strict reader then refused. A female's chrY site stays `--`, as before.
- The site table reader is stricter. A position must be a whole number that fits in 32 bits, and the GRCh38 reference must be one base. The Python failed later, with a traceback.
- `build-sites` joins the ALT alleles of two dbSNP SNV records of one rsID at one position before the strand test. The Python kept the first record's ALTs only, so such a site could be dropped or get the wrong strand.
- `og consumer check` on a zip it cannot open reports the problem on stdout, like every other problem. The Python ended with a traceback.

**The harness.** The compare-kind column of `tests/parity/cases.tsv` now takes a kind per output file: `GLOB=KIND ... DEFAULT`. A `build-sites` case writes a gzip table and a plain TSV, so it needs `gz-content` for one and `bytes` for the other. The self-test plants three new differences for it.

**Decisions:**

- The consumer tool is not in `OG_SHADOW`. Shadow mode compares bytes, and the zip and gzip bytes differ from the Python's on purpose. `consumer-export-1` checks the output on real calls instead.
- `rust/og.sh` keeps `REQUIRED=false`. Step 40 is opt-in, so a checkout without `og` still runs every default step. Step 40 itself runs `rust/og.sh check --required` first and stops with the setup command when `og` is missing.
- `CHANGELOG.md` has one entry under Unreleased for the changes above.
- Files outside the package list, changed only because the Python is deleted: `scripts/ci/oracle-diff.sh` and `tests/parity/**` (the cases), `tests/e2e/consumer-export-1.sh` and `scripts/ci/e2e-case-paths.tsv` (the e2e case), `docs/40-export-consumer.md` and `docs/updating.md`, and comments in `bin/genome_lookup.py`, `tests/test_genome_lookup.py`, `tests/demo/make_demo_sample.py`, `tests/fixtures/consumer_export/README.md` and `scripts/chip-to-vcf.sh`.
- The comment above `CONSUMER_SITES_TABLE_SHA256` in `versions.env` now names `og consumer build-sites`. A `versions.env` change runs every e2e case, which is fine for this PR.
- The musl builds of `og` (the release binaries) link the `mimalloc` allocator. With musl's own allocator, the multi-threaded gVCF read held 160 to 350 MB on a 552 MB gVCF, several times its live data. A glibc build does not need it, so only the musl target links it.
- `rust/` is outside the e2e scope, so a later PR that changes only `rust/crates/og-steps` runs Guard's parity cases but not `consumer-export-1`. We accept that here, as for the other ported tools; parity checks the output against the Python oracle on every PR.

## How it was checked

- Parity: 37 consumer cases, 155 items, all agree with the Python or with a golden. The cases cover both layouts, both sexes and no sex, a missing PAR file, BGZF input, a hole in the gVCF, a run of heterozygous calls, a table shifted by one, an unsorted gVCF, a contig listed twice, three alleles, a bad site row, and a synthetic case of about 1.9 million gVCF records and 180,578 sites from `tests/parity/gen/consumer-synthetic.sh`.
- Speed and memory, with the aarch64 musl release build of this head in a Linux container with 4 CPUs, on a synthetic gVCF of the size a 30x genome gives (552 MB compressed, 32 million records over every chromosome) with a table of 617,044 sites: `og` took 2.2 to 2.3 s and at most 43 MB resident, three runs. The Python took 30 s and 660 MB on the same files. The decoded 23andMe and AncestryDNA files hold the same 617,044 and 617,045 data rows as the Python's. Not measured here: a real genome's gVCF.
- The ALT join of `build-sites` on the real dbSNP b157 file (md5 `6a6f313e92a39c337571174dad12cfe1`, the pin): no site on the 23andMe v5 list has two SNV records at one GRCh38 position, so the join changes 0 sites. A full build with a counting copy of `build-sites` took 2 min 7 s, kept 614,138 sites and dropped 24,325, and the table's content sha256 is the pinned `CONSUMER_SITES_TABLE_SHA256` exactly. The counter was shown to fire on a planted two-record input first.
- Unit tests in the crate, including a test that a good export removes the `.txt` files a failed one left (red when the removal is skipped), a test that compares every site with the genotype the gVCF holds, and a test that a table off by one or an unsorted gVCF stops the export.
- Red control: commit f2df1d4 plants a flip in the rule "at a record's start, a variant wins over a block". On that head Guard went red in the `parity` job with `oracle-diff: 30 of 203 items in 49 cases differ`, for example `consumer/export-male out/S_23andme_v5.zip: member S_23andme_v5.txt: byte 726 (line 18): oracle 'C', new 'G'`: https://github.com/GeiserX/OwnGenome/actions/runs/42902437621 . The crate's own test of every site against the gVCF fails on the same flip. Commit 5f67519 takes it out again.
- Head 126c88d (main merged in, with the og-chip crate of step 49 beside this one; the musl allocator, the line-by-line check and the removal of stale `.txt` files are unchanged since 161e26b). Locally: `cargo fmt --check`, `cargo build --locked`, `cargo clippy --locked --workspace --all-targets -- -D warnings` with and without default features, `cargo test --locked --workspace` and `cargo doc` with `-D warnings` all pass. On GitHub: Guard parity, self-tests, rust and unit tests https://github.com/GeiserX/OwnGenome/actions/runs/43057000725 . Rust, which builds the musl release binaries on amd64 and arm64 with `mimalloc`: https://github.com/GeiserX/OwnGenome/actions/runs/43057000812 . Container Test: https://github.com/GeiserX/OwnGenome/actions/runs/43057001060 . Lint, Nextflow, Docs, Personal data, Renovate dry run, Release and Demo pictures passed too. Demo pictures failed once because the runner could not resolve the Ubuntu mirrors, and passed on its one rerun.
- e2e on the fork, head 161e26b: https://github.com/geiresx/OwnGenome/actions/runs/43027801433 . `consumer-export-1` passed all 34 checks (shard 4) and `chip-mt-1-roundtrip` all 19 (shard 3). Shard 2 failed in `reports-and-pgx-correctness-revel`, which could not install `tabix` because the runner could not resolve security.ubuntu.com. That case does not touch step 40. The e2e run of the final head is still to come.
- Earlier e2e on the fork, head 5f67519 (before the memory work), all legs green: https://github.com/geiresx/OwnGenome/actions/runs/42904256807 . `consumer-export-1` passed all 34 checks, including the strict reader through `og consumer check`, every site against an independent reading of the gVCF, the round trip through `chip-to-vcf.sh`, both shifted-table red controls and the Nextflow module. `chip-mt-1-roundtrip` passed all 19. The e2e run on 5861eb8 is green too, all legs: https://github.com/geiresx/OwnGenome/actions/runs/42919412337 .

Bead: pgp-bs9u.14

E2E evidence: https://github.com/geiresx/OwnGenome/actions/runs/43113261796
@geiresx geiresx closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants