Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- The callability section of `summary.json` now says, for each gene a step of its own calls, whether that step ran. A gene such as SMN1, CYP2D6, HBA1, LPA or RHD that step 16b did not read is listed under `not_read_owned` with its owner, and it read as handed to that step even when the step never ran for the sample, so its coverage stood alone as the answer. Each entry gains two fields: `owner_step` (`35`, `32`, `46` or `41`) and `owner_status` (`ran`; `not run` when that step wrote no file of its own for the gene in the sample folder, a Parascopy gene being judged by its own table in `assets/paralog_loci.tsv`; or `from an earlier run` when that file is older than the last run and the step was not `ok` in it, the rule that makes a section stale). Fields are only added; none is renamed or removed, and the reports do not show them yet. The share keeps its four decimals (0.8988 stays 0.8988). Part of #411 (clin-25).
- Step 10 (telomere content) always reads the GRCh38 chromosome bands now. When `reference/cytoBand.hg38.txt` was missing, `scripts/10-telomere-hunter.sh` warned and ran TelomereHunter without `-b`, so it sorted reads by its own hg19 band ends on GRCh38 positions; the Nextflow module did the same whenever `--cytoband` was not set, which is the default. The script now installs the copy the repository ships (`assets/reference/cytoBand.hg38.txt`) before it runs, and the pipeline passes that copy when `--cytoband` is not set. The step also moves from TelomereHunter 1.1.0, Python 2.7 code, to its maintained successor TelomereHunter2 1.0.12 (`TELOMEREHUNTER_IMAGE`, `quay.io/biocontainers/telomerehunter2`). On the image test's input both give the same `tel_content` (5347.255643 with 1,100 planted telomeric reads, 5337.585253 with 1,098), so earlier values stay comparable. THREADS now sets its worker count (`-c`), and the container gets a higher open-file limit, which it needs on a BAM with many contigs. Output change: `telomere/<sample>/<sample>/<sample>_summary.tsv` keeps its name, but its columns follow TelomereHunter2 (`tel_content` third, `intratelomeric_reads`, `tel_read_count`, `read_lengths`, plus three read-class counts). `summary.json` and the reports read both layouts by name and do not change. The docs no longer describe the value as an age measure: `tests/test_telomere_wording.sh` fails when a file that describes step 10 does. Part of #411 (trait-46, trait-47, trait-48, trait-49).
- Step 25 and the Nextflow `PRS_SUMMARY` withhold a score's percentile when the genotypes pgsc_calc scored list variants only. Every site such a VCF leaves out adds nothing to the sum, so the sum leans away from the panel's sums, which come from complete genotypes, and the percentile was not comparable. Each score keeps its raw sum. Its `Percentile` is `NA`, its `Effect` is empty, and a new column before `Input`, `Percentile_Note`, says `percentile withheld: the input lists variants only`. The decision reads the genotypes, not the kind of input: at least one genotyped position in five must be a reference call (0/0), the rule step 26b applies to the panel's sites. Genotypes from a gVCF and chip VCFs from `scripts/chip-to-vcf.sh`, which list their reference calls, keep their percentiles. `og prs table` takes the genotypes with a new option, `--target VCF`, and prints the NOTE that step 25 printed before; the NOTE now follows the genotypes too, so a gVCF without reference blocks at the score positions gets it. `og summary collect` passes the note on as `percentile_note`. When every percentile was withheld, both reports say "Percentile withheld: the input lists variants only" and why, and the summary's `not_assessed` list says the percentiles were withheld, in place of the lines for a run without the panel, which told the user to install a panel the run had used. The output gains one column; no column, option or file name changes.
- Step 41 (blood groups) no longer reads a VCF that does not cover a gene as the reference allele. Without a BAM, a system with no VCF record at any of its sites now reads `not assessed` (phenotype and confidence) instead of the reference phenotype at confidence medium, so a VCF with no chr9 record no longer reads ABO as O. A call with a genotype quality (`GQ`) under 20 or a read depth (`DP`, else `MIN_DP`) under 8 keeps its genotype but gives its system confidence low, and the note and the evidence file name the values. An ABO A allele now reads at confidence low, because no variant shows it: it is what is left of a functional allele with neither c.802G>A nor c.803G>C. Duffy reads the VCF's phase: a GATA box phased with one Fyb in one phase set goes on the chromosome the VCF gives (on Fya that is Fy(a-b+)). When H reads H- and ABO is `not assessed` or `not determined`, the ABO row no longer carries the Bombay note about its alleles. The notes now say what the DNA cannot check: the ABO alleles outside the five variants, RHD null and DEL alleles behind a D+, and null alleles the table does not hold. The table keeps its columns; `not assessed` is a new value in the phenotype and confidence columns.
Expand Down
2 changes: 1 addition & 1 deletion docs/16b-mosdepth.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Shows what fraction of each region is covered at key depths:

A gene can have several reasons. Gene positions are GENCODE 50 gene spans. The `owner` column names the step that calls the gene with a method of its own: `parascopy` (step 35, the genes of `assets/paralog_loci.tsv`), `pypgx` (step 32, the bam genes of `assets/pgx_genes.tsv` it can call on the no-ALT reference, so not GSTT1) or `cn_loci:<locus>` (the depth engine, `bin/cn_loci.tsv`: the genes a row names, HBA1, HBA2, LPA and RHD, not a gene that only holds one of its targets); `.` for none.

A gene counts as **read** when 90% or more of its bases have 10 or more reads of MAPQ 20 or more. The gene pass runs mosdepth without `--fast-mode`, so a base where the two mates of one pair overlap counts one read, not two, and a base inside a deletion of a read's CIGAR counts no read. With `--fast-mode` mosdepth counts every base from a read's start to its end once per read: short-insert pairs, whose mates overlap, and small deletions lift a base to 10x when fewer than 10 fragments read it, and the gene reads as read. The genome-wide pass (the 500 bp windows and their thresholds) keeps `--fast-mode`: it describes the run's depth, and nothing is decided per gene from it. Both reports have a section "Genes this genome could not read reliably" in the Quality chapter. It lists every gene with no owner that was not read, with the share of its bases that reached that depth, and every ALT-only gene with no owner as "not on this reference". Genes with an owner that were not read are listed apart: their own step's section says what it called. On a partial BAM (the CI fixture, an exome) most genes are not read, because they have no reads at all.
A gene counts as **read** when 90% or more of its bases have 10 or more reads of MAPQ 20 or more. The gene pass runs mosdepth without `--fast-mode`, so a base where the two mates of one pair overlap counts one read, not two, and a base inside a deletion of a read's CIGAR counts no read. With `--fast-mode` mosdepth counts every base from a read's start to its end once per read: short-insert pairs, whose mates overlap, and small deletions lift a base to 10x when fewer than 10 fragments read it, and the gene reads as read. The genome-wide pass (the 500 bp windows and their thresholds) keeps `--fast-mode`: it describes the run's depth, and nothing is decided per gene from it. Both reports have a section "Genes this genome could not read reliably" in the Quality chapter. It lists every gene with no owner that was not read, with the share of its bases that reached that depth, and every ALT-only gene with no owner as "not on this reference". Genes with an owner that were not read are listed apart: their own step's section says what it called. In `summary.json` each of them (`not_read_owned`) also names that step (`owner_step`: `35` for Parascopy and the HBA, CYP21A2, HP and CCL3L1 rows of the depth engine, `32` for pypgx, `46` for LPA, `41` for RHD) and says whether it ran for this sample (`owner_status`: `ran`, `not run` or `from an earlier run`). A step ran when it wrote the gene's file in the sample folder: for a Parascopy gene its own table, `paralogs/<sample>_<table>_copy_number.tsv` with the `table` of its row in `assets/paralog_loci.tsv` (the `smn` table for SMN1 and SMN2), `pypgx/<sample>_pypgx_summary.tsv`, `paralogs/<sample>_hba_copy_number.tsv`, `_cyp21a2_psv.tsv` or `_cn_dosage.tsv`, `lpa/<sample>_lpa_kiv2.tsv`, or `bloodtype/<sample>_bloodtype.tsv`. A file older than the start of the last run, from a step that was not `ok` in that run (`logs/run_status.tsv`), is `from an earlier run`: the rule that makes a section of `summary.json` stale. A gene whose own step did not run has no answer from it, only its coverage here. The share is the share of bases at the depth, to four decimals. On a partial BAM (the CI fixture, an exome) most genes are not read, because they have no reads at all.

`CALLABILITY_GENES` names another table with the same columns (the CI case uses a two-gene one). A table from an older catalog, one that lacks a gene of the current catalog, makes the section unreadable: run step 16b again. When only `<sample>_callability.bed.gz` is missing, step 16b runs the gene pass alone.

Expand Down
139 changes: 137 additions & 2 deletions rust/crates/og-summary/src/collect/sections.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
//! counts. Each returns the file it read and its typed data, `None` when
//! the step wrote nothing, or the note of an unreadable section.

use super::model::opt_str;
use super::model::{RUN_ALL_STEPS, RunStatus, is_stale, opt_str};
use crate::{basename, float, int, text_lines};
use indexmap::IndexMap;
use pycompat::csv::{DictReader, DictRow, Quoting};
Expand Down Expand Up @@ -200,9 +200,99 @@ pub struct Gene {
pub hgnc_id: String,
pub reasons: Vec<String>,
pub owner: Option<String>,
/// The owner's step and whether it ran, for an owner [`owner_step`] knows.
pub owner_run: Option<OwnerRun>,
pub reading: Reading,
}

/// The step that calls an owned gene, and what it left for this sample.
#[derive(Debug, Clone, PartialEq)]
pub struct OwnerRun {
pub step: &'static str,
pub status: OwnerStatus,
}

/// What the owner's step left in the sample folder.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum OwnerStatus {
/// Its file is there, from this run.
Ran,
/// Its file is not there.
NotRun,
/// Its file is older than the last run and the step was not ok in that
/// run: the rule that makes a section stale.
Earlier,
}

impl OwnerStatus {
fn word(self) -> &'static str {
match self {
OwnerStatus::Ran => "ran",
OwnerStatus::NotRun => "not run",
OwnerStatus::Earlier => "from an earlier run",
}
}
}

/// The step behind an owner of `assets/callability_genes.tsv` and the file
/// that step writes in the sample folder (`{s}` the sample, `{t}` the
/// gene's table in `assets/paralog_loci.tsv`) when it ran. `None` for an
/// owner this list does not know.
fn owner_step(owner: &str) -> Option<(&'static str, &'static str)> {
Some(match owner {
"parascopy" => ("35", "paralogs/{s}_{t}_copy_number.tsv"),
"pypgx" => ("32", "pypgx/{s}_pypgx_summary.tsv"),
"cn_loci:HBA" => ("35", "paralogs/{s}_hba_copy_number.tsv"),
"cn_loci:CYP21A2" => ("35", "paralogs/{s}_cyp21a2_psv.tsv"),
"cn_loci:HP" | "cn_loci:CCL3L1" | "cn_loci:LCE3BC" => ("35", "paralogs/{s}_cn_dosage.tsv"),
"cn_loci:LPA_KIV2" => ("46", "lpa/{s}_lpa_kiv2.tsv"),
"cn_loci:RHD" => ("41", "bloodtype/{s}_bloodtype.tsv"),
_ => return None,
})
}

/// The file the owner's step writes for this gene in `dir`, the sample
/// folder, and its step. A Parascopy gene without a table has no file:
/// step 35 never answers for it.
fn owner_file(
r: &CatalogRow,
dir: &std::path::Path,
s: &str,
) -> Option<(&'static str, Option<std::path::PathBuf>)> {
let (step, file) = owner_step(&r.owner)?;
let file = if r.owner == "parascopy" {
r.table.as_deref().map(|t| file.replace("{t}", t))
} else {
Some(file.to_owned())
};
Some((step, file.map(|f| dir.join(f.replace("{s}", s)))))
}

/// What the owner's step left at `file`, judged as a section is
/// ([`super::model::Section::new`]): a file older than the run's start, from
/// a step that was not ok in that run, is from an earlier run.
fn owner_status(
step: &str,
file: Option<&std::path::Path>,
run: Option<&RunStatus>,
) -> OwnerStatus {
let Some(mtime) = file
.filter(|f| f.is_file())
.and_then(|f| pycompat::mtime::mtime(f).ok())
else {
return OwnerStatus::NotRun;
};
if let Some(run) = run
&& let Some(started) = run.started_epoch
&& RUN_ALL_STEPS.contains(&step)
&& run.steps.get(step).map(String::as_str) != Some("ok")
&& is_stale(mtime, started)
{
return OwnerStatus::Earlier;
}
OwnerStatus::Ran
}

#[derive(Debug, Clone, PartialEq)]
pub struct Callability {
pub min_depth: i128,
Expand All @@ -217,6 +307,8 @@ pub struct CatalogRow {
pub start: String,
pub reasons: String,
pub owner: String,
/// A Parascopy gene's table in `assets/paralog_loci.tsv`.
pub table: Option<String>,
}

/// Reads the catalog: its comment lines left out, cells as written.
Expand Down Expand Up @@ -251,8 +343,25 @@ pub fn read_catalog(assets: &str) -> Result<Vec<CatalogRow>, String> {
start: cell("start")?,
reasons: cell("reasons")?,
owner: cell("owner")?,
table: None,
});
}
// Step 35 writes one Parascopy table per row of paralog_loci.tsv, for
// the genes the row names.
if out.iter().any(|r| r.owner == "parascopy")
&& crate::is_file(crate::join(assets, "paralog_loci.tsv"))
{
let mut tables = std::collections::HashMap::new();
for r in super::assets::asset_rows(assets, "paralog_loci.tsv")? {
let cell = |k: &str| r.get(k).flatten().unwrap_or_default().to_owned();
for g in cell("genes").split(',') {
tables.entry(g.to_owned()).or_insert_with(|| cell("table"));
}
}
for r in out.iter_mut().filter(|r| r.owner == "parascopy") {
r.table = tables.get(&r.gene).cloned();
}
}
Ok(out)
}

Expand Down Expand Up @@ -307,6 +416,18 @@ impl Callability {
// a chromosome that has rows means another catalog wrote the table.
let contigs: std::collections::HashSet<&str> =
rows[1..].iter().map(|c| c[0].as_str()).collect();
// The table is <sample folder>/mosdepth/ (bash) or coverage/ (Nextflow)
// <s>_callability.bed.gz: the owner steps write into the same folder.
let folder = std::path::Path::new(table)
.parent()
.and_then(|p| p.parent());
let sample = name.strip_suffix("_callability.bed.gz");
// run-all.sh's run status, which tells a file of this run from one an
// earlier run left (Folder::open has read it already; a run status it
// could not read stopped the summary before this reader).
let run = folder
.and_then(|d| std::fs::read_to_string(d.join("logs").join("run_status.tsv")).ok())
.map(|t| RunStatus::parse(&t));
let mut genes = Vec::with_capacity(catalog.len());
for r in catalog {
let reading = if r.start == "." || !contigs.contains(r.chrom.as_str()) {
Expand All @@ -330,11 +451,20 @@ impl Callability {
Reading::NotRead(share)
}
};
let owner = (r.owner != ".").then(|| r.owner.clone());
let owner_run = match (folder, sample) {
(Some(dir), Some(s)) => owner_file(r, dir, s).map(|(step, file)| OwnerRun {
step,
status: owner_status(step, file.as_deref(), run.as_ref()),
}),
_ => None,
};
genes.push(Gene {
gene: r.gene.clone(),
hgnc_id: r.hgnc_id.clone(),
reasons: r.reasons.split(',').map(str::to_owned).collect(),
owner: (r.owner != ".").then(|| r.owner.clone()),
owner,
owner_run,
reading,
});
}
Expand All @@ -356,6 +486,11 @@ impl Callability {
("hgnc_id", Value::Str(g.hgnc_id.clone()))
};
let mut m = vec![("gene", Value::Str(g.gene.clone())), who];
if owned {
let run = g.owner_run.as_ref();
m.push(("owner_step", opt_str(run.map(|r| r.step))));
m.push(("owner_status", opt_str(run.map(|r| r.status.word()))));
}
if !owned {
m.push((
"reasons",
Expand Down
2 changes: 1 addition & 1 deletion rust/crates/og-summary/src/collect/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ fn callability_against_the_catalog() {
let want = loads(
r#"{"not_read": [{"gene": "C3", "hgnc_id": "HGNC:3", "reasons": ["segdup"], "share": null, "status": "not on this reference"},
{"gene": "D4", "hgnc_id": "HGNC:4", "reasons": ["ALT-only"], "share": null, "status": "not on this reference"}],
"not_read_owned": [{"gene": "B2", "owner": "parascopy", "share": 0.5, "status": "not read"}],
"not_read_owned": [{"gene": "B2", "owner": "parascopy", "owner_step": "35", "owner_status": "not run", "share": 0.5, "status": "not read"}],
"not_on_reference": ["C3", "D4"]}"#,
)
.unwrap();
Expand Down
Loading
Loading