Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
- Gene callability (step 16b and the Nextflow `MOSDEPTH` process) no longer counts a fragment twice. The gene pass ran mosdepth with `--fast-mode`, which counts every base from a read's start to its end: where the two mates of a pair overlap, a base counted two reads, and the bases of a deletion in a read counted as read. A gene that fewer than 10 fragments covered could reach 10x and read as read. The gene pass now runs without `--fast-mode`; the genome-wide 500 bp pass keeps it. The file name and columns of `<sample>_callability.bed.gz` do not change; its `10X` counts can be lower, so a gene near 90% can move to "not read". Step 16b skips a sample it already ran, so a sample from before this change keeps its old counts: delete `<sample>/mosdepth/<sample>_callability.bed.gz` and run step 16b again, and only the gene pass runs. On public HG002 30x, 4 of 1,553 genes moved (DHX40, SPATA31C2, GOLGA6L10 and UGT2B17), and the pass takes about 37% more CPU time, 3.5-4 min on a 10-core Mac mini. The step doc and `docs/lessons-learned.md` said `--fast-mode` drops the per-base output; `--no-per-base` does that, and they now say so. A new e2e case, `i411-callability-overlap-1-mates`, runs both paths on a small BAM with overlapping pairs and a deletion.
- The output contract check now compares the contract with the last release, so a stable output cannot disappear between releases. Before, `og dev contract --static` compared the code with the contract of the same checkout only: a change that renamed a stable output in the code and in its row, removed a stable param with its row, or made a stable row experimental passed. `og dev contract --baseline FILE` checks every stable row of a release's contract against this one: still listed, still stable, the same format, every leg that wrote the file and every required field. New rows and changes to experimental and internal rows pass. Guard's `rust` job runs it against the contract of the newest `v1.x.y` tag; until 1.0.0 is tagged there is none, and the step says so. The HG002 e2e case now also checks 31 stable files the base run writes whose `e2e` cell was `-` (the clinical summary, the slivar, mitochondrial and pypgx tables, the ClinVar cDNA and deep-intronic tables, the gVCF, the read count, the charts and search index, the PharmCAT JSON and HTML, the coverage beds, the sex check, indexcov, the Stranger VCF of ExpansionHunter, the indexes of the clinical, slivar and mitochondrial VCFs, and the Mutect2 stats of chrM). `run_manifest.tsv`, those three indexes and the Mutect2 stats are now listed on both legs, since the step scripts write them too. No output changes. `.github/CODEOWNERS` now says it routes review requests: main requires its status checks, not an approving review.
- Step 50 marks a known disease insertion (the FKTN, TAF1 and BBS1 SVAs of `assets/mei_sites.tsv`) seen only when MEGAnE's call there passed its own filters (FILTER `PASS`). Before, a call MEGAnE had flagged, such as `LC` for low confidence, also made the site read seen, so a call MEGAnE did not trust could read as carrier status. Now a site whose only calls are flagged reads `call filtered out (LC)` (with the call's flags) in the `status` column of `mei/<sample>_mei_sites.tsv`, and `call` names the flagged call. A `PASS` call in the same window still makes the site seen. That status value is new; the columns and the other values (seen, not seen, not callable) do not change. A window with zero depth and no call reads not callable, never not seen, as before; a new test now holds that. The calls table `mei/<sample>_mei.tsv` does not change: it lists flagged calls with their `filter`, as before. Tests: `rust/crates/og-reports/tests/i411_mei_seen_pass.rs`. Part of #411.
- The MEI section of both reports now says what `call filtered out (LC)` means. Step 50 writes that status when MEGAnE's only calls at a catalog site are ones it flagged, but the note under the section explained not callable and not seen only. The note now adds one sentence: MEGAnE called an insertion there but flagged it, so it is not a finding, and not proof of absence either. The text and HTML reports change by that sentence; no column or file changes. `docs/pipeline-overview.md` and `docs/50-mei.md` list the status too. Two parity cases (`mei-report/filtered`, `mei-report/filtered-pass`) record with a golden that `og mei report` reads FILTER where the Python it replaced did not, and the report-render parity oracle moves to the commit with the new note, so the Rust and Python reports still compare byte for byte. The status `not callable` keeps its name: renaming it to `not assessed` would change an output value of step 50 for no gain in meaning. Tests: `rust/crates/og-render/tests/i411_mei_filtered_status_docs.rs`. Part of #411.
- Step 53 (Locityper) no longer shows a pass at CYP2D6 and FCGR. The authors set their quality filter (no warning, a weighted distance of 30 or less, few unexplained reads) on HLA and KIR genes only, but `og locityper report` applied it as pass or fail at every locus, so a CYP2D6 or FCGR call could read as a green pass that nobody had measured. The `filter` column now reads `pass` or `fail` only at a locus whose name starts with `HLA-` or `KIR`; at any other locus it reads `uncalibrated`, and `reason` says what the rule alone gives there (`... by its rule alone it passes`, or `... it fails:` with the reasons). Both reports show `uncalibrated` in grey and count only `pass` rows as passing. The step's last line adds the count of uncalibrated loci. `docs/53-locityper.md` also says that the cut of 30 comes from Locityper 0.17.3, whose weighted distance was a minimizer Jaccard distance, while the 1.7.14 the step runs gives an edit distance. No column or file name changes; the `filter` column has one new value. A cargo test, `rust/crates/og-reports/tests/i411_locityper_uncalibrated.rs`, checks all five shipped loci; 12 parity goldens of `locityper-report` record the new rows. The docs demo sample (`og dev demo-sample`) now writes its step 53 table with `og locityper report` too, so the demo report shows CYP2D6 and FCGR as `uncalibrated` like a real run; `rust/og/tests/port_demo_sample.rs` checks it.
- Step 45's Y-STR counts are on FamilyTreeDNA's scale at the 24 markers with a published counting rule. The marker set's regions did not always cover the repeat blocks a lab counts, so DYS19 read 16 where FamilyTreeDNA reads 15 on the same DNA, DYS449 read 36 for 29 and Y-GATA-H4 31 for 11. `assets/ystr_panel.tsv` now carries each marker's offset, from the GRCh38 sequence counted by NIST STRBase's repeat structure, and `og y str-report` adds it. A called marker with no published rule says in its note that its count is on the reference's scale.
- `high` confidence now also needs at least 70% of the reads that span the repeat to show the called length (HipSTR's `MALLREADS`). Before, it meant read quality only: a call could be high while most reads showed another length. A record without `MALLREADS` is not high.
Expand Down
4 changes: 3 additions & 1 deletion bin/render_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -2265,7 +2265,9 @@ def viral_body(d):
MEI_NOTE = ("Alu, L1 and SVA copies the genome carries and the reference does not, found and genotyped by "
"MEGAnE (Kojima et al. 2023) from short read pairs. Most are common and harmless; the catalog rows are "
"insertions known to cause disease. Not callable: too few reads in that window to tell. Not seen is "
"not proof of absence: short reads find an SVA's edges, not its body. Every call: mei/<sample>_mei.tsv.")
"not proof of absence: short reads find an SVA's edges, not its body. Call filtered out: MEGAnE called "
"an insertion there but flagged it (LC is low confidence), so it is not a finding, and not proof of "
"absence either. Every call: mei/<sample>_mei.tsv.")


def mei_lines(d):
Expand Down
4 changes: 2 additions & 2 deletions docs/50-mei.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ With `run-all.sh`: `TOOLS=...,mei`; a run lists step 50 only when `TOOLS` names
| `mei/${SAMPLE}_mei_depth.tsv` | `samtools bedcov -c` over the catalog windows |
| `mei/${SAMPLE}_megane.log` | MEGAnE's log |

Both reports show the count per element and the catalog in their "Mobile element insertions" section; the AnnotSV columns are in the calls table only.
Both reports show the count per element and the catalog in their "Mobile element insertions" section; the AnnotSV columns are in the calls table only. The note under the section says what each status means: not callable is too few reads to tell, not seen is not proof of absence, and call filtered out is an insertion MEGAnE called but flagged (`LC` is low confidence), so it is not a finding and not proof of absence either.

AnnotSV classifies deletions and duplications only. In AnnotSV 3.5 the ranking of an insertion is empty (`SVrankingINS` in its `AnnotSV-ranking.tcl` scores nothing), so `acmg_class` is `NA` for every insertion. Read an insertion by its gene and location instead: one inside an exon can break the gene; most in introns or between genes do nothing, though some do (the TAF1 SVA above is intronic). A `benign_af` above a few percent means insertions at that place are common. Without step 50, the report lists mobile element insertions under what the run did not assess.

Expand All @@ -84,7 +84,7 @@ On HG002's chr20 at 30x (13 million mapped reads) on a 4-CPU GitHub runner, MEGA

## Validation

The e2e case `tests/e2e/mei-1.sh` runs the step on HG002's chr20: the reads of chr20 in GIAB's 60x novoalign BAM, half the pairs (30x), realigned with minimap2 to chr20 alone, the unmapped reads dropped (so the case fits a runner's time), and the k-mer set of that chromosome. Its Alu calls are scored against the GIAB v5.0q insertions of HG002 on chr20 (RepeatMasker SINE/Alu, interspersed, 250 to 400 bp, inside the benchmark regions): recall must be at least 0.70 and precision at least 0.80, within 100 bp. On 2026-10-27 it found 12 of the 15 insertions (0.80), and 13 of its 15 Alu calls inside the benchmark regions matched a GIAB insertion (0.87); with the unmapped reads kept, the same 12 of 15 and 13 of 15. With every read pair within 1 kb of a GIAB Alu insertion removed, it found 0 of 15 and the recall check failed. The same case then runs AnnotSV on the calls with made-up AnnotSV data (the bundle of `tests/fixtures/annotsv/class45.sh` plus one made-up gene over all of chr20) and checks that every call comes back inside that gene, with its location there and class `NA`; the same VCF read without AnnotSV's table fails that check. It also runs the Nextflow pipeline with `--tools mei` and `--annotsv_annotations` on the same BAM and checks that its calls table is the bash step's, byte for byte. `rust/crates/og-reports/tests/optin_mei.rs` plants an SVA in the FKTN window of a synthetic MEGAnE VCF and checks it is seen, and not seen without it, and joins a table in AnnotSV's columns back to the calls. `rust/crates/og-reports/tests/i411_mei_seen_pass.rs` checks that an SVA there that MEGAnE flagged `LC` reads `call filtered out (LC)`, not seen, that a `PASS` call in the same window wins over a flagged one, and that a window with zero depth and no call reads not callable, never not seen.
The e2e case `tests/e2e/mei-1.sh` runs the step on HG002's chr20: the reads of chr20 in GIAB's 60x novoalign BAM, half the pairs (30x), realigned with minimap2 to chr20 alone, the unmapped reads dropped (so the case fits a runner's time), and the k-mer set of that chromosome. Its Alu calls are scored against the GIAB v5.0q insertions of HG002 on chr20 (RepeatMasker SINE/Alu, interspersed, 250 to 400 bp, inside the benchmark regions): recall must be at least 0.70 and precision at least 0.80, within 100 bp. On 2026-10-27 it found 12 of the 15 insertions (0.80), and 13 of its 15 Alu calls inside the benchmark regions matched a GIAB insertion (0.87); with the unmapped reads kept, the same 12 of 15 and 13 of 15. With every read pair within 1 kb of a GIAB Alu insertion removed, it found 0 of 15 and the recall check failed. The same case then runs AnnotSV on the calls with made-up AnnotSV data (the bundle of `tests/fixtures/annotsv/class45.sh` plus one made-up gene over all of chr20) and checks that every call comes back inside that gene, with its location there and class `NA`; the same VCF read without AnnotSV's table fails that check. It also runs the Nextflow pipeline with `--tools mei` and `--annotsv_annotations` on the same BAM and checks that its calls table is the bash step's, byte for byte. `rust/crates/og-reports/tests/optin_mei.rs` plants an SVA in the FKTN window of a synthetic MEGAnE VCF and checks it is seen, and not seen without it, and joins a table in AnnotSV's columns back to the calls. `rust/crates/og-reports/tests/i411_mei_seen_pass.rs` checks that an SVA there that MEGAnE flagged `LC` reads `call filtered out (LC)`, not seen, that a `PASS` call in the same window wins over a flagged one, and that a window with zero depth and no call reads not callable, never not seen. The parity cases `mei-report/filtered` and `mei-report/filtered-pass` (`tests/parity/cases.tsv`) record the same two readings against the Python that `og mei report` replaced, which read any call in the window as seen: each has a golden sites table under `tests/parity/golden/mei-report/`. `rust/crates/og-render/tests/i411_mei_filtered_status_docs.rs` checks that both reports show a `call filtered out (LC)` row with the note that explains it.

## Notes

Expand Down
2 changes: 1 addition & 1 deletion docs/design/journal/BRIEF.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ The site has to show the result families below. The report already groups them i
| Variant prioritisation (31 slivar) | 4 groups; compound-het pairs with phase (trans/cis/unknown) | expert; most pairs are noise | info | phase is mostly unknown |
| Symptom ranking (48 Exomiser, opt-in) | top 10: rank, gene, score, mode of inheritance, variants, ACMG class; the HPO terms used | a ranking to take to a geneticist | review | not a diagnosis; an empty list rules nothing out |
| Structural variants (04/15/18/19/22/05) | type (DEL, DUP, INV, BND, INS), position, length, genotype, supporting callers, duphold DHFFC/DHBFC, AnnotSV class 1-5, genes, CDS overlap %, gnomAD-SV AF; counts per caller | 5-10k SVs is normal. **Only class 4-5 in a disease gene matters.** Single-caller BND calls are mostly artefacts | act (class 4-5) / info | short reads miss insertions and inversions |
| Mobile elements (50, opt-in) | 3 catalogued sites (FKTN, TAF1, BBS1): seen / not seen / not callable; about 1,200 Alu/L1/SVA calls | "not seen" is not proof of absence | carrier / review | none |
| Mobile elements (50, opt-in) | 3 catalogued sites (FKTN, TAF1, BBS1): seen / call filtered out / not seen / not callable; about 1,200 Alu/L1/SVA calls | "not seen" is not proof of absence; "call filtered out" is a call MEGAnE flagged, not a finding | carrier / review | none |
| Repeat expansions (09/09b) | locus, gene, two allele counts with 95% CI, Stranger status (normal, pre_mutation, full_mutation), normal max, pathogenic min, sized or flanking-only | HTT, FMR1, C9orf72, DMPK and ATXN1 first. **Intermediate = no disease, may expand in children. Full mutation = see a clinical geneticist.** An unsized locus is never shown as normal | act (full) / review (pre or intermediate) / info | short-read sizing fails on long expansions; C9orf72 thresholds vary between labs |
| MUC1 VNTR (52, opt-in) | positive / possible / negative / not assessed, sensitivity at this depth | ADTKD-MUC1, a kidney disease | act (positive) / review (possible) | needs confirmation (SNaPshot) |
| Mito variants (20) | site, allele fraction (heteroplasmy), ClinVar class, gnomAD AF; poly-C length variants listed apart | above 95% = haplogroup marker; 10-95% = look it up; m.3243A>G or m.8344A>G = specialist | act / review / info | blood under-represents heteroplasmy in other tissues |
Expand Down
2 changes: 1 addition & 1 deletion docs/pipeline-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
| **Traits** | Lactase persistence, caffeine, alcohol flush, bitter taste, earwax, pigmentation, APOE and other replicated variants, each row citing its primary paper | 42 |
| **Archaic Ancestry** | The Neanderthal and Denisovan shares and the segments carried, with IBDmix against the Altai and Vindija Neanderthals and the Altai Denisovan (opt-in) | 43 |
| **Viral DNA** | Inherited integrated HHV-6 and the DNA of 23 human viruses (herpesviruses, HPV, HBV, HIV-1, polyomaviruses and others) in the reads that do not map to the human genome, with phiX as the lab control (opt-in) | 47 |
| **Mobile Elements** | Alu, L1 and SVA insertions the reference lacks, each genotyped as one or two copies, and three SVA insertions known to cause disease (FKTN, TAF1, BBS1) read as seen, not seen or not callable (opt-in; needs the BAM) | 50 |
| **Mobile Elements** | Alu, L1 and SVA insertions the reference lacks, each genotyped as one or two copies, and three SVA insertions known to cause disease (FKTN, TAF1, BBS1) read as seen, call filtered out (a call MEGAnE flagged), not seen or not callable (opt-in; needs the BAM) | 50 |
| **Mosaic Alterations** | Large gains, losses and copy-neutral loss of heterozygosity on chr1 to chr22 in a share of the blood cells, with the cell fraction (opt-in; mosaic loss of Y and X are step 33's) | 51 |

## Pipeline Overview
Expand Down
4 changes: 3 additions & 1 deletion rust/crates/og-render/src/notes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,9 @@ pub const MEI_NOTE: &str = "Alu, L1 and SVA copies the genome carries and the re
genotyped by MEGAnE (Kojima et al. 2023) from short read pairs. Most are common and \
harmless; the catalog rows are insertions known to cause disease. Not callable: too few \
reads in that window to tell. Not seen is not proof of absence: short reads find an \
SVA's edges, not its body. Every call: mei/<sample>_mei.tsv.";
SVA's edges, not its body. Call filtered out: MEGAnE called an insertion there but \
flagged it (LC is low confidence), so it is not a finding, and not proof of absence \
either. Every call: mei/<sample>_mei.tsv.";
pub const MCA_NOTE: &str = "Large gains, losses and copy-neutral losses of heterozygosity (CN-LOH) carried by a \
share of the blood cells, from MoChA on the phased heterozygous sites of chr1 to chr22. \
In a UK Biobank study they became more common with age and went with a higher risk of a \
Expand Down
Loading
Loading