diff --git a/.github/workflows/container-test.yml b/.github/workflows/container-test.yml index a91c5f7c2..bd1b06479 100644 --- a/.github/workflows/container-test.yml +++ b/.github/workflows/container-test.yml @@ -175,6 +175,13 @@ jobs: fi echo "SMOKE_WORK=${work}" >> "$GITHUB_ENV" + # The LOCITYPER_IMAGE row runs /pgp-bin/og locityper report in the image. + - name: Build og as scripts/setup.sh --og does + if: matrix.var == 'LOCITYPER_IMAGE' + run: | + scripts/setup.sh --og + rust/og.sh check --required + - name: Run ${{ matrix.var }} on the fixture env: GH_TOKEN: ${{ github.token }} diff --git a/.github/workflows/guard.yml b/.github/workflows/guard.yml index 273284a47..091f86063 100644 --- a/.github/workflows/guard.yml +++ b/.github/workflows/guard.yml @@ -313,7 +313,8 @@ jobs: grep -q 'summary.json validates (og dev schema)' <<< "$out" \ || { echo "ERROR: reports-stale-step did not run og dev schema"; exit 1; } - # The tests of the og filters steps 04, 05 and 07 run, og chip's (step 49), og family's (step 38), og run delta's, step 26b's og ancestry-fine, the + # The tests of the og filters steps 04, 05 and 07 run, og chip's (step 49), og cn's (steps 35, 41 and 46), og family's (step 38), og run delta's, step 26b's og ancestry-fine, + # the opt-in step reports' (45, 47, 48, 50 to 53), the # og tools setup.sh runs (og str catalog, og setup unzip and genetic-map), and the step 25 # tests that compare og prs format with the folders they write by hand: # unit-tests has no og, so there they skip; here OG_REQUIRED turns a @@ -330,7 +331,12 @@ jobs: tests/test_annotsv_sv_inputs.sh tests/test_annotsv_postfilter.sh tests/test_annotsv_benign_mane.sh \ tests/test_str_catalog.py tests/test_str_ranges.py tests/test_og_setup_zip.py tests/test_setup_fine_maps.sh \ tests/test_render_report.py tests/test_collect_summary.py tests/test_chip_report.py \ - tests/test_family_report.py tests/test_family_segments.py; do + tests/test_family_report.py tests/test_family_segments.py \ + tests/test_vntyper_report.py tests/test_locityper_report.py tests/test_mocha_report.py \ + tests/test_mei_report.py tests/test_viral_report.py tests/test_ystr_report.py \ + tests/test_exomiser_report.py \ + tests/test_copy_number.py tests/test_hba_copy_number.py \ + tests/test_lpa_kiv2.py tests/test_traits_report.py tests/test_callability.py; do rc=0 echo "::group::${t}" case "$t" in @@ -342,6 +348,15 @@ jobs: done exit "$fail" + # The fake-docker job builds no og, so its cn-engine-calls case says it + # is skipped there; here bin/og is built and the case must run. + - name: The fake-docker case cn-engine-calls (steps 35, 41 and 46 through og cn) with og + working-directory: . + run: | + out=$(scripts/ci/run-fake-docker-suite.sh cn-engine-calls 2>&1) || { echo "$out"; exit 1; } + echo "$out" + if grep -q '^skip: bin/og is not built' <<<"$out"; then echo "ERROR: the case was skipped, with og built"; exit 1; fi + # A Python file kept only for what other Python imports from it holds # those names and nothing else (tests/parity/remnants.tsv). - name: og dev remnants diff --git a/.gitleaksignore b/.gitleaksignore index 3bc7bbfcc..cb0ae8e24 100644 --- a/.gitleaksignore +++ b/.gitleaksignore @@ -4,3 +4,8 @@ # A test string in the Parascopy freshness self-test ("GRCh37_v9.9.tar.gz"), # not a key; the line was rewritten in 7d6dad4. 63e12d6c2b29078a49e14de8ef4c4e57a9ebd8b9:scripts/ci/freshness.py:generic-api-key:940 +# A Rust constant name compared with a marker's name in og-cn, not a key; the +# comparison was turned round in the same pull request. The second line is this +# file's own first comment about it, which quoted the expression. +0bc60efda244e5361c54fe411f46db9514e5e8c3:rust/crates/og-cn/src/cli.rs:generic-api-key:590 +63b5a9273c444970e469d04c8da52aa8890726b5:.gitleaksignore:generic-api-key:7 diff --git a/CHANGELOG.md b/CHANGELOG.md index c6e64c3c9..5fab1af08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,26 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The copy-number engine now runs in the Rust `og` binary, as `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions` and `og cn marker-read`. That covers step 35's depth counts (HBA, the gene-dosage rows, CYP21A2, the interval rows) and its SMN1 marker, step 46's KIV-2 count and 4925G>A reading, the RHD regions of step 41 and the ALT-contig check of steps 32 and 35. They replace `bin/hba_copy_number.py` and `bin/lpa_kiv2.py`, which are gone, the commands of `bin/copy_number.py` and the marker reader of `bin/traits_report.py`. Two parts stay in Python for now: step 41's report makes the RhD call with the catalog reader and `assess()` of `bin/copy_number.py`, and step 32's CYP2D6 depth check is its own tool. `bin/traits_report.py` keeps `MARKERS` for the summary. Every table, file name and option a step uses stays the same; the parity cases in `tests/parity` hold every output that changed on purpose: + - Step 46: the KIV-2 4925G>A reading is made in the same call as the count and judges the aligner on the count's own MAPQ 0 share. Before, it read the share the count table prints, rounded to three places, so at a share just under 0.20 the count said not assessed while the marker still read `present`. Both now say not assessed. + - Step 35: the SMN1 marker message takes the SMN1 psCN from the first row of the SMN table that covers exon 7, the rule of the Paraphase comparison. It took the last such row, so a table with two rows over exon 7 made the two read different counts. + - Steps 35 and 46: a status the HBA or LPA table layout did not name (a catalog edit could give one) now writes `not assessed: `. The HBA table wrote `called` with no count, and the LPA table stopped with an error. + - A catalog row with a `more=` reading and no numbered reading is refused when the catalog loads, naming the row. Before, a call that reached it failed without naming the row. + - `og cn call --sex` reads a gene row on chrX or chrY against the sample's count there. The Python call had no sex, so such a row could only read `sex_unknown`. No row of the shipped catalog is a gene row on chrX or chrY. + - The tables are written to a temporary file and renamed, so a failed run leaves no half-written table. + - `og cn bed` and `og cn marker-regions` exit 2 when they cannot write the region list, so a step never reads a truncated list. + - `copy_number.py call --json` is gone with no replacement: no step passed it. + - Error messages about a wrong regions table say to make it with `og cn bed`. + +- The opt-in steps 45, 47, 48 and 50 to 53 now read their tool's output with the Rust `og` binary: `og y str-bed` and `og y str-report` (step 45), `og viral report` (47), `og exomiser prepare` and `og exomiser report` (48), `og mei report` and `og mei annotsv-input` (50), `og mocha report` (51), `og vntyper report` (52) and `og locityper report` (53). They replace `bin/ystr_report.py`, `bin/viral_report.py`, `bin/exomiser_report.py`, `bin/mei_report.py`, `bin/mocha_report.py`, `bin/vntyper_report.py` and `bin/locityper_report.py`, which are gone. og is static, so each step still runs in the image it ran in. The tables are the same, byte for byte, on the 123 parity cases, except where a line below says otherwise. Every output is written whole or not at all, and an input a step cannot read stops it with one `ERROR:` line, not a Python traceback. + - Step 48: the summary printed at the end of `og exomiser report` has one line per rank. The Python cut it by variant count, so a rank with several variants could push the closing note in or out. The text file is unchanged. + - Step 50: the depth of each catalog window is its own. The Python looked it up by gene and chromosome, so two windows of one gene on one chromosome both got the last one's depth. The shipped catalog has no such pair, so its results do not change. + - Step 51: `calls_mocha` in the summary counts this sample's calls only. The Python counted every row of MoChA's call table, so a table of several samples gave a wrong count. A call row shorter than the header now stops the step with exit 2 and the file and line. + - Step 52: a coverage table with a negative mean depth stops the step with exit 2 and says so. The Python crashed. + - Step 53: a `res.json.gz` that holds JSON but not an object, or a genotype that is not text, now reads "not called" with the reason. The Python crashed and wrote no table. A loci BED start written in non-ASCII digits is refused with its line. + - Step 45: a panel region with a motif length under 1, or a negative copy number, is refused with the marker's name. The Python crashed on the first and used the second. + - Steps 47, 50 and 51: an unreadable input stops with `ERROR: ` on stderr, where the Python wrote `