Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- summary.json has a new section, `acmg_sf`: the ACMG secondary-findings card, built from step 06's ClinVar hits on every run, with or without the VEP cache. Before, the card existed only inside step 23's clinical section, so a run without the VEP cache had no card at all. The new section's state follows step 06. Without step 23's clinical VCF it has `high_impact_checked: false` and a `high_impact_note` that says why and what it can miss (a predicted loss-of-function variant not in ClinVar). A clinical VCF that cannot be read no longer costs the ClinVar rows of this card. `sections.clinical.data.acmg_sf` stays as it was, and the reports still read the card there, so they show it only when step 23 ran. The first run after the update lists the new section under "New since" once.
- A third-party outage no longer fails the Freshness check of a pull request. Since 2026-11-18 rest.ensembl.org has not answered, and the pull-request job's dry run of `og dev freshness` failed on every pull request that touched `versions.env` or `og-fresh`. `og dev freshness --unreachable-warns` now lists a lookup whose upstream did not answer (no connection, a timeout, HTTP 5xx or 429 after the retries, or HTTP 403) under "Upstreams that did not answer" as a warning, and its database row reads `NO ANSWER`; it does not change the exit. The 403 counts because rest.ensembl.org came back on 2026-11-19 answering 403 to GitHub's runners and 200 elsewhere, which still failed the pull-request job. A 404, an empty answer or an answer in an odd shape still exits 1. `freshness.yml` passes the flag only in the pull-request dry run. Without the flag the report and the exit are as before, and the monthly report on main still counts an outage as an error in the issue.
- Three statements in the docs were wrong. Troubleshooting said exit code 137 confirms a kill for memory, and checked it with `docker ps -a --latest`. 137 is any SIGKILL, and the step scripts run their containers with `--rm`, so that command shows an older, unrelated container. The page now says how to confirm a kill for memory: `docker events --filter event=oom`, the kernel log, or `.State.OOMKilled` for a container kept without `--rm`. The run-state and troubleshooting pages said `bin/og` does not run on a Mac and told Mac users to build `og` with cargo or run it in a container. `bin/og` runs the native macOS build that `scripts/setup.sh --og` installs, and the pages now say so; on a Mac, `og run headroom` reports the machine's memory as unknown. Troubleshooting, the FAQ and the lessons-learned page said the step scripts run their containers as root, and lessons-learned gave `:latest` images and `chmod 777` as current fixes. The step scripts run their containers as you; those entries are now marked as past fixes and name what the pipeline does today. Documentation only: no option, output or behaviour changes.
- Lint's ShellCheck job now checks every tracked shell script, not only `scripts/`: the e2e cases and their library, the fake-docker and parity tests, `tests/*.sh`, `bin/og` and the fake tools, 482 files in all, at every ShellCheck level. The 265 findings these files hold today are listed in `tests/shellcheck-baseline.txt` and still pass; any other finding fails lint. A listed finding is matched by its file, its code and the text of its line, so it keeps matching when lines above it move, and a new line with the same mistake still fails. Each run plants an unquoted `$1` in a new test script and on a new line of a listed script, and fails if either passes or if the file search misses the new script or `bin/og`. The `scripts/` check at warning level stays as it was. Part of #411 (plat-10).
Expand Down
2 changes: 2 additions & 0 deletions docs/06-clinvar-screen.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ The [runtime table](hardware-requirements.md#runtime-per-step) gives the measure

Both reports (step 24 and `generate-report.sh`) read the hits file and show gene, genotype (het, hom, or "one copy, other copy not called ref" for the half of a split `1/2` call), significance, review status, stars and the combined-allele note ("combined allele of ...", "part of a combined allele: ...", both followed by "; unphased, so the parts may sit on different haplotypes" on an unphased het call, or "possibly one allele with ...: check ClinVar for a combined record") for each hit, the best-reviewed first, with the ClinVar file's release date, the gnomAD frequency and the low-confidence mark. The headline count leaves the low-confidence calls out and a line beside it counts them. When the cDNA check table is there, both print its result beside each hit (the HTML hits table has a "cDNA read check" column) and the ClinVar card counts the flagged hits. Older versions of this step wrote `clinvar/isec/0002.vcf`; that file holds only the sample's side of the intersection, with no gene or significance, and nothing reads it any more. Rerun step 6 to get the hits file.

summary.json also has an ACMG secondary-findings card built from these hits, `sections.acmg_sf`. It follows this step, not step 23: it is there whenever the hits file is, also in a run without the VEP cache, and is not run (or failed) with this step's reason otherwise. It holds the hits in the 84 genes of the ACMG SF v3.3 list with each gene's rule applied, step 05's class 4 and 5 SVs in the genes whose rule allows them, and the deletions that count as a second hit. Step 23's rare HIGH-impact records join it when step 23 left its clinical VCF. Without that VCF the card says `high_impact_checked: false`, and `high_impact_note` names why (step 23 skipped, with the run's reason, failed, or its VCF unreadable) and what that can miss: a predicted loss-of-function variant that is not in ClinVar, alone or as the second hit of a recessive gene. [Step 23](23-clinical-filter.md#secondary-findings-genes-acmg-sf-v33) gives the rules. The reports still show the card from step 23's section, so they show it only when step 23 ran.

`${SAMPLE}_clinvar_hits.tsv` is the table to read or load elsewhere. To list the hits from the VCF yourself:

```bash
Expand Down
2 changes: 2 additions & 0 deletions docs/23-clinical-filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ The Nextflow `CLINICAL_FILTER` module applies the same tiers, ranking, frequency

Both reports list, beside the clinical filter's counts, the secondary findings in the 84 genes of the ACMG SF v3.3 list (Lee et al., Genet Med 2025;27(8):101454; the genes in which the ACMG recommends reporting pathogenic variants found by chance, such as BRCA1, LDLR, MYBPC3 or TTN): the step 6 ClinVar hits in those genes, the clinical filter's records with a HIGH-impact consequence on the ranked transcript in them, and step 5's AnnotSV class 4 and 5 structural variants in the genes whose rule allows them. Each gene's rule from Table 1 of the list is a row of `assets/acmg_sf.tsv`: one variant in a dominant gene, two in a recessive one (MUTYH, BTD, ATP7B and others; a homozygote, or two variants of unknown phase), HFE only as homozygous p.C282Y, GLA and OTC also in a heterozygous female, ABCD1 hemizygous, homozygous or with two heterozygous variants. A variant the rule does not report is listed apart with the reason; a pseudodeficiency allele is not counted, and a BTD finding with the p.Asp444His hypomorph says partial, not profound. Of the reported HIGH-impact records the reports count as "rare HIGH impact" only the rare ones with a sound call; a weak call or an unknown frequency is listed with its mark and counted apart, and a common one is left out. The list version is `ACMG_SF_VERSION` in `og summary collect` (`rust/crates/og-summary`). It is a list to review with a clinician: a rare HIGH-impact variant is not a ClinVar classification. [Interpreting results](interpreting-results.md#carrier-status-is-not-disease) gives every rule. Step 17 (CPSR) reports the ACMG secondary findings with its own classification.

summary.json holds the card twice. `sections.clinical.data.acmg_sf` is the card as both reports read it. It is there whenever this step's summary table or clinical VCF can be read, also one left by an earlier run (then, when this step did not run ok, the clinical section is stale). `sections.acmg_sf` is the same card as a section of step 6: it is written whenever step 6's hits are there, so a run without the VEP cache still gets the ClinVar and SV findings. Without this step's clinical VCF it has `high_impact_checked: false` and a `high_impact_note` that names why (this step skipped, with the run's reason, failed, or its VCF unreadable) and what that can miss: a predicted loss-of-function variant that is not in ClinVar, alone or as the second hit of a recessive gene. With the clinical VCF the two cards are equal, except when that VCF is from an earlier run (this step not ok in this run): then `sections.acmg_sf` adds a `high_impact_note` with the earlier VCF's date. A clinical VCF that cannot be read, or whose HIGH-impact records cannot be parsed, makes the clinical section's card unavailable, while `sections.acmg_sf` keeps its ClinVar rows with `high_impact_checked: false` and says the VCF could not be read. The two cards then differ.

The `table_text` column of `assets/acmg_sf.tsv` quotes each gene's rule from Table 1 of the paper (doi:10.1016/j.gim.2025.101454), such as "All P and LP" or "P and LP (2 variants)". summary.json gives that text as the rule of each finding, and the reports use it in the reason a variant is not reported. These are short cited quotations of a factual reporting rule: six phrases, the longest 44 characters. "(See text)" after a rule points to the paper's section on the dilated cardiomyopathy (DCM) genes. The `note` column says what the table's footnotes add, in our own words.

## Runtime
Expand Down
2 changes: 1 addition & 1 deletion docs/24-html-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ The cards, by chapter (cards not listed here are named in the chapter table abov

- **ClinVar Screening** — hit count by review stars, the ClinVar file date, and the hits table best-reviewed first with a Stars column. The count's badge is yellow when there is a hit and gray when there is none, never green. Each hit shows its gnomAD allele frequency from step 13's VEP output, and a line says where the frequencies came from and how many hits are above 1%, or why every frequency is unknown: a common allele with a P/LP record is often low-penetrance, so read every hit with the frequency the card shows, and look up a hit whose frequency is unknown. A low-confidence line counts the hits whose call is under step 6's GQ or DP floor: they are listed but not counted in the total. When step 6's cDNA read check ran, the card also counts the hits it flagged
- **Clinical Filter** and **slivar** — variant counts from steps 23 and 31
- **Secondary-Findings Genes (ACMG SF v3.3)** — the step 06 ClinVar hits, the clinical filter's rare HIGH-impact records and step 05's class 4 and 5 SVs in the 84 genes of the ACMG list, each gene's reporting rule applied (`assets/acmg_sf.tsv`, for example HFE homozygous C282Y only), and the variants a rule does not report listed apart with the reason
- **Secondary-Findings Genes (ACMG SF v3.3)** — the step 06 ClinVar hits, the clinical filter's rare HIGH-impact records and step 05's class 4 and 5 SVs in the 84 genes of the ACMG list, each gene's reporting rule applied (`assets/acmg_sf.tsv`, for example HFE homozygous C282Y only), and the variants a rule does not report listed apart with the reason. The card is shown when step 23 ran. summary.json also writes it as its own section, `acmg_sf`, from step 06's hits, with or without step 23 ([step 06](06-clinvar-screen.md#output))
- **Cancer Predisposition** — CPSR status and the classification breakdown (read by column name from `${SAMPLE}.cpsr.grch38.classification.tsv.gz`)
- **Structural Variants** — Manta, Delly, CNVpytor and the SURVIVOR consensus (2+ callers) counts; then step 05's class 4 and 5 SVs (AnnotSV) with their caller support, exon-edge and frequency checks. A call step 05's rules moved to class 3 (a coding tandem repeat, an immunoglobulin or T-cell receptor locus, the MHC, a pseudogene named by a disease gene's alias, or read depth that does not show the change) is listed apart with AnnotSV's class, the rule and the reason, and the class 4-5 count leaves it out
- **Repeat Expansions** — a table of every locus, the five key loci (HTT, FMR1, C9orf72, ATXN1, DMPK) first, with its repeat count, Stranger's status and notes; and counts of the loci outside the normal range (step 9b, Stranger), of those short reads could not size (not counted as normal), and the loci in an intermediate range (`bin/str_intermediate_ranges.tsv`). When Stranger flags RFC1, or another locus where the motif decides (`expansion_hunter/${SAMPLE}_str_motifs.tsv`), the card adds the verdict of the read-level motif count (step 9b). A locus the repeat catalog gives no thresholds (its `NoThresholds` field in `expansion_hunter/${SAMPLE}_eh_catalog.json`: MUC1, CEL, TYMS and PLIN4 with the STRchive catalog, among others) is marked not graded, with the catalog's reason, never normal. Any other locus outside Stranger's catalog has no status and is counted apart. Without step 9b the card says Stranger did not run; a Stranger file older than the ExpansionHunter result it came from is not read, and the card says to rerun step 9b. Repeat-expansion calls from short reads can be wrong at some loci
Expand Down
4 changes: 3 additions & 1 deletion docs/results/secondary-findings.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The most common "pathogenic" finding in any genome is **heterozygous carrier sta

**Your own carrier list (step 24 report).** The ClinVar card of both reports is followed by a carrier list for this one sample, with no partner needed: every step 6 ClinVar hit in the 3,292 recessive and X-linked genes of `assets/carrier_genes.tsv` (the list of the [couple screen](../39-couple.md#the-carrier-list)). An allele that list marks as a hypomorph (for example BTD p.Asp444His or the GALT Duarte variants) or a pseudodeficiency allele (HEXA c.739C>T and c.745C>T, which lower the enzyme test result but cause no Tay-Sachs disease) is listed as set aside, never as a carrier finding on its own.

**Secondary findings follow each gene's ACMG rule.** The secondary-findings card (step 23's section of the report) applies the reporting rule of each of the 84 ACMG SF v3.3 genes (Lee et al., Genet Med 2025;27(8):101454, Table 1), written gene by gene in `assets/acmg_sf.tsv`:
**Secondary findings follow each gene's ACMG rule.** The secondary-findings card (step 23's section of the report; in summary.json also `sections.acmg_sf`, written from step 6's hits on every run, VEP cache or not) applies the reporting rule of each of the 84 ACMG SF v3.3 genes (Lee et al., Genet Med 2025;27(8):101454, Table 1), written gene by gene in `assets/acmg_sf.tsv`:

- a dominant gene (BRCA1, LDLR, MYH7 and most others): one pathogenic or likely pathogenic variant is reported;
- a recessive gene (MUTYH, ATP7B, BTD, CYP27A1, GAA, RPE65, CASQ2, TRDN): two variants are needed, a homozygote or two different ones even when their phase is unknown. One heterozygous MUTYH variant is listed apart as not reported, with the reason;
Expand All @@ -31,6 +31,8 @@ The most common "pathogenic" finding in any genome is **heterozygous carrier sta

Examples: an exon deletion of MUTYH plus a pathogenic MUTYH variant; three alpha-globin genes plus an HBA2 point variant; one SMN1 copy plus an SMN1 variant (short reads place few SMN1 variants reliably, so check with a targeted test); and the 1q21.1 deletion plus one of the two RBM8A low-expression alleles (c.-21G>A, rs139428292, or c.67+32G>C, rs201779890), the cause of TAR syndrome (thrombocytopenia with absent radius; Albers et al., Nat Genet 2012;44(4):435-439, PMID 22366785). On its own an RBM8A low-expression allele stays set aside. The [couple screen](../39-couple.md) (step 39) applies the same pairing between partners: a hypomorph allele of the list in one partner and a full loss of function in the same gene in the other (a ClinVar pathogenic variant, a Delly deletion) is a shared gene with its risk for a child; the allele in both partners, or in one alone, stays set aside. In the secondary-findings card a called deletion counts as one of the two variants a recessive gene needs, unless read-backed phase puts it on the variant's copy. The line under the carrier list names the deletion callers it read and the ones it could not read. A deletion AnnotSV ranks below class 4 is not counted.

Without a readable clinical VCF from step 23 (no VEP cache, step 23 skipped or failed with no VCF left, or a VCF that cannot be read) the card in `sections.acmg_sf` holds the ClinVar hits and the SVs only, and its `high_impact_note` says so: a predicted loss-of-function variant that is not in ClinVar is not screened, alone or as the second hit of a recessive gene. A clinical VCF left by an earlier run is still read when step 23 failed in this run, as the clinical section reads it; that section then shows as stale, and the card's `high_impact_note` gives the date of that earlier VCF and step 23's result in this run.

The TTN rule (truncating variants only) and the LMNA rule (missense only with cardiac evidence) are printed beside the finding, not applied. It is still a list to review with a clinician.

---
Expand Down
24 changes: 22 additions & 2 deletions rust/crates/og-render/src/chapters.rs
Original file line number Diff line number Diff line change
Expand Up @@ -266,8 +266,21 @@ pub fn step_of(key: &str) -> &'static str {
.map_or("?", |s| s.step)
}

/// Summary keys no frame section lists, because the section of another key
/// already shows what they hold: `(key, the key whose section shows it)`.
/// `acmg_sf` is the secondary-findings card the clinical section also holds
/// as `clinical.data.acmg_sf`, and the reports read it there. Listing it in
/// the clinical-filter frame would change that section's step, state and
/// file date before any reader shows it. It moves into the frame when the
/// readers switch to `sections.acmg_sf` (i411-display-sf).
pub const SHOWN_WITH: &[(&str, &str)] = &[("acmg_sf", "clinical")];

/// The section that shows summary key `key`.
pub fn section_of(key: &str) -> Option<&'static Sec> {
let key = SHOWN_WITH
.iter()
.find(|(k, _)| *k == key)
.map_or(key, |(_, with)| *with);
CHAPTERS
.iter()
.flat_map(|c| c.sections.iter())
Expand All @@ -283,12 +296,19 @@ pub fn section(sid: &str) -> Option<&'static Sec> {
}

/// What is wrong with a chapter table against the summary keys: a key no
/// section shows, a key two sections show, a key no collector writes.
/// section shows, a key two sections show, a key no collector writes. A key
/// of `SHOWN_WITH` counts as shown where its partner key is.
pub fn key_problems(chapters: &[Chapter], keys: &[&str]) -> Vec<String> {
let seen: Vec<&str> = chapters
let mut seen: Vec<&str> = chapters
.iter()
.flat_map(|c| c.sections.iter().flat_map(|s| s.keys.iter().copied()))
.collect();
let carried: Vec<&str> = SHOWN_WITH
.iter()
.filter(|(_, with)| seen.contains(with))
.map(|(k, _)| *k)
.collect();
seen.extend(carried);
let mut out: Vec<String> = keys
.iter()
.filter(|k| !seen.contains(k))
Expand Down
7 changes: 7 additions & 0 deletions rust/crates/og-render/src/index.rs
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,13 @@ pub fn build(
if !matches!(s.get("state"), Some(Value::Str(st)) if st == "ok" || st == "stale") {
continue;
}
// The ACMG card of its own section is the card the clinical section
// also holds (`clinical.data.acmg_sf`), which the index already reads.
// Skip it so a gene is not listed twice, until the report readers
// move from the nested card to this section (i411-display-sf).
if key == "acmg_sf" {
continue;
}
let field = |k: &str| s.get(k).cloned().unwrap_or(Value::Null);
let mut add = |term: &str, txt: &str, locus: Option<&str>| {
let items = terms.entry(term.to_owned()).or_default();
Expand Down
Loading
Loading