Repository navigation
Conversation
og pgx cpic-report and og pgx step36-outside, in the new crate og-pgx: the gene status as one typed classifier, the drug guidance with its sources as an enum, the step 36, pypgx, phenoconversion and contraindication sections as functions, and Python's own error texts where the report quotes them. --date sets the Date line (default today in UTC); the tables are written atomically. Bead: pgp-bs9u.19
Every report-level fixture of tests/test_cpic_parser.py and of the step 36 checks, the real 3.2.0 and 3.4.0 reports, the four layouts and the error paths, compared item by item with the Date line masked. Bead: pgp-bs9u.19
…e.py Step 27, the CPIC_LOOKUP module, the demo sample builder and the unit tests now run og pgx. tests/og_binary.py finds the og a test runs; Guard builds it once for the unit-test shards, and the docs and demo-pictures builds build it too. The function-level checks of test_cpic_parser.py moved to Rust tests; the report-level ones run og as a subprocess. Bead: pgp-bs9u.19
Red control, reverted in the next commit: a gene counts as ambiguous only with three or more different phenotypes. Bead: pgp-bs9u.19
The red control did its job; the rule is two or more different phenotypes again. Bead: pgp-bs9u.19
The e2e scope covers bin/ but not rust/, so the cases that run step 27 list the og shim; a change to og-pgx is held by the parity cases. Bead: pgp-bs9u.19
The Rust port of step 40's tool: the site table from the 23andMe v5 list and dbSNP read in parallel, the gVCF walk that genotypes every chip site, both chip layouts zipped with a fixed date, and the strict reader. Bead: pgp-bs9u.14
A case that writes a gzip table and a plain TSV needs gz-content for one and bytes for the other. The kinds column now takes GLOB=KIND pairs and a default; the self-test plants a difference the per-file kind must catch. Bead: pgp-bs9u.14
Synthetic gVCFs, site tables, dbSNP slices and chip files, plus a generator for a whole-genome-sized synthetic case. The goldens are the deliberate fixes: ALTs of one rsID at one position are joined, and a call of three alleles stops the export. Bead: pgp-bs9u.14
The step script, the Nextflow module, setup.sh --consumer-sites and the e2e case call og consumer; bin/export_consumer.py and its unit test go, their cases now live in tests/parity and the crate's tests. build-sites reads dbSNP itself on 4 cores, so the grep prefilter goes too. Bead: pgp-bs9u.14
Bead: pgp-bs9u.14
PharmCAT's example quotes genotypes in its guideline text, which the personal-data scan refuses outside tests/fixtures, so the case builds its report from the fixture instead of committing a copy. Bead: pgp-bs9u.19
Bead: pgp-bs9u.14
A new crate, og-qc, holds the estimate: one streaming pass over mosdepth's windows (BGZF inflated on every core) and the VCF, with CPython's medians, fmean and exact variance from pycompat::stats and every int quotient correctly rounded. 32 parity cases check it against bin/mosaic_sex_loss.py in PYTHON_IMAGE; four goldens hold the deliberate fixes (a nan depth, an AD past 32 bits, non-UTF-8 input, og's usage line). Bead: pgp-bs9u.22
…gone scripts/33-sample-qc.sh, the SAMPLE_QC module, the e2e case and the demo sample builder call og qc sex-loss. Guard builds og once for the unit tests (tests/og_binary.py), and the docs and demo-picture builds build it for the demo's mosaic rows. The Python unit test moved into the crate. Bead: pgp-bs9u.22
… oracles Four oracle.ref rows (the three cyp2d6 subcommands and the motif count) at 81567aa, and six cyp2d6-check cases on the depth fixtures. Bead: pgp-bs9u.23
Steps 21, 32 and 9b and their modules call og pgx cyp2d6-* and og str motif; the bash steps run og from their output directory with relative outputs and pass OG_SHADOW, so CI can shadow them. The Python is deleted. Bead: pgp-bs9u.23
…he tests onto og OG_SHADOW names the four tools; the fake-docker cases require og and refuse the Python; the unit tests check the meaning of the goldens og is held to. Bead: pgp-bs9u.23
Bead: pgp-bs9u.23
…e scope Bead: pgp-bs9u.23
# Conflicts: # CHANGELOG.md # rust/Cargo.lock # rust/og/src/main.rs
…ut file Bead: pgp-bs9u.23
…cle runs measure Planted on purpose, reverted in the next commit: Guard's fake-docker case and parity job must go red. Bead: pgp-bs9u.23
The repo ignores *.vcf, so the synthetic input stayed out of the commit and the parity job stopped on the missing file. Bead: pgp-bs9u.22
This reverts commit e917d9b. The naive variance was planted to show that the bit-exact estimate test fails on it. Bead: pgp-bs9u.22
# Conflicts: # rust/Cargo.lock # rust/og/src/main.rs
Bead: pgp-bs9u.14
Bead: pgp-bs9u.23
…one star table og-clinvar holds the review-star table, the genotype and combined-allele wording and the call floor once. og clinvar join-adjacent --mode build reads the reference itself, so the prepare, fill-from-fasta, finish pipe is one process. The parity cases compare every mode with the awk it replaces. Bead: pgp-bs9u.13
# Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # docs/rust.md # rust/og/src/main.rs # tests/parity/cases.tsv # tests/parity/oracle.ref
# Conflicts: # CHANGELOG.md # docs/rust.md # rust/Cargo.lock # rust/Cargo.toml # rust/og/src/main.rs # scripts/ci/e2e-case-paths.tsv
# Conflicts: # CHANGELOG.md # docs/rust.md # rust/og/Cargo.toml # rust/og/src/main.rs # scripts/ci/e2e-case-paths.tsv # tests/parity/cases.tsv # tests/parity/oracle.ref
…-z2a # Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # docs/rust.md # rust/og/src/main.rs # tests/parity/cases.tsv # tests/parity/oracle.ref
…-r0-switch # Conflicts: # .github/workflows/guard.yml
# Conflicts: # .github/workflows/guard.yml # docs/rust.md # rust/Cargo.toml # rust/og/Cargo.toml # rust/og/src/main.rs # tests/parity/cases.tsv # tests/parity/oracle.ref
install_fine_maps now holds step 26b's map block, so a test can run it. tests/test_setup_fine_maps.sh plants a cut-short chr22.rfmix.map with no maps/DONE and checks it is written again, that maps/DONE then stops a rerun, and that a stamp of other chromosomes does not. Its control runs the rule of before and sees the cut-short map kept.
# Conflicts: # .github/workflows/demo-pictures.yml # .github/workflows/docs.yml # CHANGELOG.md # docs/nextflow.md # docs/rust.md # rust/Cargo.lock # rust/Cargo.toml # rust/README.md # rust/og.sh # rust/og/Cargo.toml # rust/og/src/main.rs # scripts/ci/e2e-case-paths.tsv # scripts/setup.sh # tests/demo/make_demo_sample.py # tests/og_binary.py # tests/parity/cases.tsv # tests/parity/oracle.ref
…-c6c # Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # docs/rust.md # rust/Cargo.toml # rust/og/src/main.rs # tests/parity/cases.tsv # tests/parity/oracle.ref
…atalog og cannot replace A damaged zip whose end record claims more than the file holds, or a member that inflates past its recorded size, now stops with exit 2 before memory or disk fills. og str catalog takes -h and --help and refuses an extra locus whose names are not text. setup.sh builds og before it removes an outdated STRchive catalog, and rewrites step 26b's maps when a map file is missing under maps/DONE.
…-b1-switch # Conflicts: # .github/workflows/e2e.yml # CHANGELOG.md # docs/nextflow.md # docs/rust.md # docs/testing.md # rust/og.sh # scripts/ci/e2e-case-paths.tsv # tests/parity/cases.tsv # tests/parity/oracle.ref
…ilds without it og str catalog used og_bam::motif::forms, which made og-setup, and so every og build, depend on og-bam and htslib, the C library the macOS build leaves out. og-setup now holds the same 20-line rule, and a test checks it against og-bam's on every motif of up to six bases.
# Conflicts: # CHANGELOG.md # docs/rust.md # rust/Cargo.lock # rust/crates/og-steps/Cargo.toml # rust/crates/og-steps/src/lib.rs # rust/og/src/main.rs # scripts/ci/e2e-case-paths.tsv # scripts/ci/oracle-diff.sh # tests/parity/.gitignore # tests/parity/cases.tsv # tests/parity/oracle.ref
…require it in the rust job unit-tests now has a host og in OG for the og dev checks, so OG being set no longer means bin/.og is built. The two tests that run the real step 06 ask bin/og itself, as the step 05 tests do; the rust job sets OG_REQUIRED so a skip there is a failure.
# Conflicts: # .github/workflows/e2e.yml # .github/workflows/guard.yml # CHANGELOG.md # bin/rfc1_motif.py # docs/rust.md # rust/Cargo.toml # rust/README.md # rust/og/src/main.rs # scripts/ci/e2e-case-paths.tsv # tests/fake-docker/str-catalog-setup-and-steps.sh # tests/parity/cases.tsv # tests/parity/oracle.ref # tests/test_str_catalog.py
…Python does os.walk's relpath depth counts the root and its children both as 0, so the Python lists files four folders below the sample and stops there. og stopped one folder sooner and dropped files such as mito/eklipse/<dir>/<dir>/*, which made the dev-contract shadow fail on the HG002 tree. A unit test builds a five-level tree and expects the oracle's listing.
…roducibly (GeiserX#371) ## The problem Step 40 writes a person's genome as the raw-data file a chip company gives its customers, in the 23andMe layout and the AncestryDNA layout. A Python program did the work. It read the whole gVCF in one process, held every row it checked in memory, and wrote zips whose bytes changed with the clock. On a synthetic gVCF of 19 million lines with a table of 611,936 sites, it took 17 s and 667 MB in a Linux container. Building the site table also needed a grep pass over the 30 GB dbSNP file before Python could start. ## The fix **The port.** `og consumer` has three verbs in a new crate, `rust/crates/og-steps`: - `og consumer build-sites` reads dbSNP's BGZF file on 4 cores. It looks at the ID column of a line before it parses anything else, so `setup.sh --consumer-sites` drops the grep prefilter. - `og consumer export` reads the gVCF on up to 4 cores and runs the same walk as the Python: the pending records at one position, and the spans of called deletions. It writes one layout at a time. Each line goes to the strict reader and to the zip member as it is made, so no whole text is held in memory. - `og consumer check` is the strict reader of both layouts, on a `.txt` or a `.zip`. The step script, the Nextflow module, `setup.sh --consumer-sites` and the e2e case `consumer-export-1` call `og`. `bin/export_consumer.py` and `tests/test_export_consumer.py` are deleted. Their cases moved to `tests/parity` and to the crate's own tests. **What changes on purpose:** - The zips are reproducible. Each member is dated with the day of the export (`--date YYYY-MM-DD`, else `SOURCE_DATE_EPOCH`, else today in UTC) at midnight. The same gVCF, table and day give the same bytes. - A successful export leaves no `.txt` files, and it removes the two a failed run left behind, so no one uploads a file the strict reader refused. When the strict reader fails, both `.txt` files are still left unzipped for a look, as before. - A call of three or more alleles at a site stops the export with the site and its position. The Python wrote a three-letter genotype that its own strict reader then refused. A female's chrY site stays `--`, as before. - The site table reader is stricter. A position must be a whole number that fits in 32 bits, and the GRCh38 reference must be one base. The Python failed later, with a traceback. - `build-sites` joins the ALT alleles of two dbSNP SNV records of one rsID at one position before the strand test. The Python kept the first record's ALTs only, so such a site could be dropped or get the wrong strand. - `og consumer check` on a zip it cannot open reports the problem on stdout, like every other problem. The Python ended with a traceback. **The harness.** The compare-kind column of `tests/parity/cases.tsv` now takes a kind per output file: `GLOB=KIND ... DEFAULT`. A `build-sites` case writes a gzip table and a plain TSV, so it needs `gz-content` for one and `bytes` for the other. The self-test plants three new differences for it. **Decisions:** - The consumer tool is not in `OG_SHADOW`. Shadow mode compares bytes, and the zip and gzip bytes differ from the Python's on purpose. `consumer-export-1` checks the output on real calls instead. - `rust/og.sh` keeps `REQUIRED=false`. Step 40 is opt-in, so a checkout without `og` still runs every default step. Step 40 itself runs `rust/og.sh check --required` first and stops with the setup command when `og` is missing. - `CHANGELOG.md` has one entry under Unreleased for the changes above. - Files outside the package list, changed only because the Python is deleted: `scripts/ci/oracle-diff.sh` and `tests/parity/**` (the cases), `tests/e2e/consumer-export-1.sh` and `scripts/ci/e2e-case-paths.tsv` (the e2e case), `docs/40-export-consumer.md` and `docs/updating.md`, and comments in `bin/genome_lookup.py`, `tests/test_genome_lookup.py`, `tests/demo/make_demo_sample.py`, `tests/fixtures/consumer_export/README.md` and `scripts/chip-to-vcf.sh`. - The comment above `CONSUMER_SITES_TABLE_SHA256` in `versions.env` now names `og consumer build-sites`. A `versions.env` change runs every e2e case, which is fine for this PR. - The musl builds of `og` (the release binaries) link the `mimalloc` allocator. With musl's own allocator, the multi-threaded gVCF read held 160 to 350 MB on a 552 MB gVCF, several times its live data. A glibc build does not need it, so only the musl target links it. - `rust/` is outside the e2e scope, so a later PR that changes only `rust/crates/og-steps` runs Guard's parity cases but not `consumer-export-1`. We accept that here, as for the other ported tools; parity checks the output against the Python oracle on every PR. ## How it was checked - Parity: 37 consumer cases, 155 items, all agree with the Python or with a golden. The cases cover both layouts, both sexes and no sex, a missing PAR file, BGZF input, a hole in the gVCF, a run of heterozygous calls, a table shifted by one, an unsorted gVCF, a contig listed twice, three alleles, a bad site row, and a synthetic case of about 1.9 million gVCF records and 180,578 sites from `tests/parity/gen/consumer-synthetic.sh`. - Speed and memory, with the aarch64 musl release build of this head in a Linux container with 4 CPUs, on a synthetic gVCF of the size a 30x genome gives (552 MB compressed, 32 million records over every chromosome) with a table of 617,044 sites: `og` took 2.2 to 2.3 s and at most 43 MB resident, three runs. The Python took 30 s and 660 MB on the same files. The decoded 23andMe and AncestryDNA files hold the same 617,044 and 617,045 data rows as the Python's. Not measured here: a real genome's gVCF. - The ALT join of `build-sites` on the real dbSNP b157 file (md5 `6a6f313e92a39c337571174dad12cfe1`, the pin): no site on the 23andMe v5 list has two SNV records at one GRCh38 position, so the join changes 0 sites. A full build with a counting copy of `build-sites` took 2 min 7 s, kept 614,138 sites and dropped 24,325, and the table's content sha256 is the pinned `CONSUMER_SITES_TABLE_SHA256` exactly. The counter was shown to fire on a planted two-record input first. - Unit tests in the crate, including a test that a good export removes the `.txt` files a failed one left (red when the removal is skipped), a test that compares every site with the genotype the gVCF holds, and a test that a table off by one or an unsorted gVCF stops the export. - Red control: commit f2df1d4 plants a flip in the rule "at a record's start, a variant wins over a block". On that head Guard went red in the `parity` job with `oracle-diff: 30 of 203 items in 49 cases differ`, for example `consumer/export-male out/S_23andme_v5.zip: member S_23andme_v5.txt: byte 726 (line 18): oracle 'C', new 'G'`: https://github.com/GeiserX/OwnGenome/actions/runs/42902437621 . The crate's own test of every site against the gVCF fails on the same flip. Commit 5f67519 takes it out again. - Head 126c88d (main merged in, with the og-chip crate of step 49 beside this one; the musl allocator, the line-by-line check and the removal of stale `.txt` files are unchanged since 161e26b). Locally: `cargo fmt --check`, `cargo build --locked`, `cargo clippy --locked --workspace --all-targets -- -D warnings` with and without default features, `cargo test --locked --workspace` and `cargo doc` with `-D warnings` all pass. On GitHub: Guard parity, self-tests, rust and unit tests https://github.com/GeiserX/OwnGenome/actions/runs/43057000725 . Rust, which builds the musl release binaries on amd64 and arm64 with `mimalloc`: https://github.com/GeiserX/OwnGenome/actions/runs/43057000812 . Container Test: https://github.com/GeiserX/OwnGenome/actions/runs/43057001060 . Lint, Nextflow, Docs, Personal data, Renovate dry run, Release and Demo pictures passed too. Demo pictures failed once because the runner could not resolve the Ubuntu mirrors, and passed on its one rerun. - e2e on the fork, head 161e26b: https://github.com/geiresx/OwnGenome/actions/runs/43027801433 . `consumer-export-1` passed all 34 checks (shard 4) and `chip-mt-1-roundtrip` all 19 (shard 3). Shard 2 failed in `reports-and-pgx-correctness-revel`, which could not install `tabix` because the runner could not resolve security.ubuntu.com. That case does not touch step 40. The e2e run of the final head is still to come. - Earlier e2e on the fork, head 5f67519 (before the memory work), all legs green: https://github.com/geiresx/OwnGenome/actions/runs/42904256807 . `consumer-export-1` passed all 34 checks, including the strict reader through `og consumer check`, every site against an independent reading of the gVCF, the round trip through `chip-to-vcf.sh`, both shifted-table red controls and the Nextflow module. `chip-mt-1-roundtrip` passed all 19. The e2e run on 5861eb8 is green too, all legs: https://github.com/geiresx/OwnGenome/actions/runs/42919412337 . Bead: pgp-bs9u.14 E2E evidence: https://github.com/geiresx/OwnGenome/actions/runs/43113261796
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GeiserX#377
Summary by CodeRabbit
...