Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
- A pull request that changes only Rust code now runs the e2e cases that run `og`. Before, `rust/` was outside the e2e scope, but for two files of `og-dev`, so such a pull request ran no case, and a merge of it started no run on main: a change in what `og` writes reached main with no real-tool run. A change under `rust/` now picks every case whose row in `scripts/ci/e2e-case-paths.tsv` has a glob that matches `bin/og`, as a change to `bin/og` does, and main runs every case after such a merge. 20 case rows that run `og` through a step, helper or module now list `bin/og`. `tests/test_e2e_scope.sh` checks the pick.
- The required `e2e` check reads cancelled, not failed, when its run was cancelled, by a newer push or by hand. Before, it ran anyway and failed, which read as a broken case when the cases had only been stopped. A cancelled check still blocks a merge.
- The testing guide says what a green `container-ok` means on a pull request that changes no image: no smoke row ran. It also says why the AnnotSV row with its 5.3 GB of full data stays monthly and on dispatch: no image pull request waits on that download.
- `summary.json` says what part of the genome a run covered. A run on part of the genome (`--intervals`, or a VCF cut down to chr20 and chr22) gave callability, polygenic score, ancestry, telomere, ROH and archaic results that read as whole-genome results. `og summary collect` now reads the chromosomes of the sample VCF's records and writes a new top-level `scope` object: `autosomes_covered`, `autosomes`, `chrx`, `intervals` (a `run intervals` row of `run_manifest.tsv`, else `null`), `partial` and `mark`. No step writes the `run intervals` row yet, so today `intervals` is `null`, and a run limited with `--intervals` is caught only by the chromosomes its calls are on. When fewer than 22 autosomes have calls, or the run was limited to intervals, each of those six sections that has a result of this run gets a new `partial` field, such as `partial: covers 2 of 22 autosomes`. A stale one of the six, a result from an earlier run whose scope was not recorded, gets `partial: not known, from an earlier run` whatever this run covered. Without a readable sample VCF there is no `scope` and nothing is marked. These are new keys; no key, file or format changes or goes away. The reports and the live site do not show the mark yet. `docs/results/quality.md` describes it, and `tests/schema/summary.schema.json` has both fields.
- `summary.json` says what part of the genome a run covered. A run on part of the genome (`--intervals`, or a VCF cut down to chr20 and chr22) gave callability, polygenic score, ancestry, telomere, ROH and archaic results that read as whole-genome results. `og summary collect` now reads the chromosomes of the sample VCF's records and writes a new top-level `scope` object: `autosomes_covered`, `autosomes`, `chrx`, `intervals` (a `run intervals` row of `run_manifest.tsv`, else `null`), `partial` and `mark`. `run_manifest.tsv` has that new row when the run was limited with `--intervals` (`INTERVALS` for `run-all.sh`): `bin/write_manifest.sh` writes it from `INTERVALS`, and the Nextflow `HTML_REPORT` from `--intervals`, the regions on one line with single spaces. A whole-genome run has no such row. So a run limited to parts of all 22 autosomes is marked too, not only one whose calls miss a chromosome. When fewer than 22 autosomes have calls, or the run was limited to intervals, each of those six sections that has a result of this run gets a new `partial` field, such as `partial: covers 2 of 22 autosomes`. A stale one of the six, a result from an earlier run whose scope was not recorded, gets `partial: not known, from an earlier run` whatever this run covered. Without a readable sample VCF there is no `scope` and nothing is marked. These are new keys; no key, file or format changes or goes away. The reports and the live site do not show the mark yet. `docs/results/quality.md` describes it, and `tests/schema/summary.schema.json` has both fields.
- Gene callability (step 16b and the Nextflow `MOSDEPTH` process) no longer counts a fragment twice. The gene pass ran mosdepth with `--fast-mode`, which counts every base from a read's start to its end: where the two mates of a pair overlap, a base counted two reads, and the bases of a deletion in a read counted as read. A gene that fewer than 10 fragments covered could reach 10x and read as read. The gene pass now runs without `--fast-mode`; the genome-wide 500 bp pass keeps it. The file name and columns of `<sample>_callability.bed.gz` do not change; its `10X` counts can be lower, so a gene near 90% can move to "not read". Step 16b skips a sample it already ran, so a sample from before this change keeps its old counts: delete `<sample>/mosdepth/<sample>_callability.bed.gz` and run step 16b again, and only the gene pass runs. On public HG002 30x, 4 of 1,553 genes moved (DHX40, SPATA31C2, GOLGA6L10 and UGT2B17), and the pass takes about 37% more CPU time, 3.5-4 min on a 10-core Mac mini. The step doc and `docs/lessons-learned.md` said `--fast-mode` drops the per-base output; `--no-per-base` does that, and they now say so. A new e2e case, `i411-callability-overlap-1-mates`, runs both paths on a small BAM with overlapping pairs and a deletion.
- The output contract check now compares the contract with the last release, so a stable output cannot disappear between releases. Before, `og dev contract --static` compared the code with the contract of the same checkout only: a change that renamed a stable output in the code and in its row, removed a stable param with its row, or made a stable row experimental passed. `og dev contract --baseline FILE` checks every stable row of a release's contract against this one: still listed, still stable, the same format, every leg that wrote the file and every required field. New rows and changes to experimental and internal rows pass. Guard's `rust` job runs it against the contract of the newest `v1.x.y` tag; until 1.0.0 is tagged there is none, and the step says so. The HG002 e2e case now also checks 31 stable files the base run writes whose `e2e` cell was `-` (the clinical summary, the slivar, mitochondrial and pypgx tables, the ClinVar cDNA and deep-intronic tables, the gVCF, the read count, the charts and search index, the PharmCAT JSON and HTML, the coverage beds, the sex check, indexcov, the Stranger VCF of ExpansionHunter, the indexes of the clinical, slivar and mitochondrial VCFs, and the Mutect2 stats of chrM). `run_manifest.tsv`, those three indexes and the Mutect2 stats are now listed on both legs, since the step scripts write them too. No output changes. `.github/CODEOWNERS` now says it routes review requests: main requires its status checks, not an approving review.
- Step 50 marks a known disease insertion (the FKTN, TAF1 and BBS1 SVAs of `assets/mei_sites.tsv`) seen only when MEGAnE's call there passed its own filters (FILTER `PASS`). Before, a call MEGAnE had flagged, such as `LC` for low confidence, also made the site read seen, so a call MEGAnE did not trust could read as carrier status. Now a site whose only calls are flagged reads `call filtered out (LC)` (with the call's flags) in the `status` column of `mei/<sample>_mei_sites.tsv`, and `call` names the flagged call. A `PASS` call in the same window still makes the site seen. That status value is new; the columns and the other values (seen, not seen, not callable) do not change. A window with zero depth and no call reads not callable, never not seen, as before; a new test now holds that. The calls table `mei/<sample>_mei.tsv` does not change: it lists flagged calls with their `filter`, as before. Tests: `rust/crates/og-reports/tests/i411_mei_seen_pass.rs`. Part of #411.
Expand Down
16 changes: 15 additions & 1 deletion bin/write_manifest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@
# run written_utc, written_by, git_commit, og_tree_hash (the rust/ tree
# the og binary in bin/.og was built from, or "not built"),
# pipeline_version (the OwnGenome release: the version in
# nextflow.config's manifest block), declared_sex
# nextflow.config's manifest block), declared_sex, intervals (the
# regions the run called, INTERVALS as run-all.sh passes it to
# --intervals; no row for a whole-genome run, nor for step 29's
# genome and chip)
# version every NAME=VALUE line of versions.env
# image each *_IMAGE reference and the digest Docker resolved it to
# ("not pulled" when the image is not on this machine)
Expand Down Expand Up @@ -77,6 +80,17 @@ if [ ! -r "${PGP_ROOT}/nextflow.config" ]; then
fi
[ -z "$release" ] || row run pipeline_version "$release"
[ -z "$DECLARED_SEX" ] || row run declared_sex "$DECLARED_SEX"
# The regions the run was limited to (INTERVALS, run-all.sh's --intervals), on
# one line with single spaces, so a tab or newline in the value cannot break
# the row. og summary collect reads it and marks the genome-wide sections
# partial. No row when the run called the whole genome, nor for genome or
# chip: step 29 reads the same INTERVALS, and those two are its whole genome
# and its CHIP genes, not regions the run was limited to.
read -r -d '' -a regions <<< "${INTERVALS:-}" || true
case "${regions[*]:-}" in
''|genome|chip) ;;
*) row run intervals "${regions[*]}" ;;
esac

# versions.env as written (comments and blank lines dropped)
while IFS= read -r line || [ -n "$line" ]; do
Expand Down
2 changes: 1 addition & 1 deletion docs/results/quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ So `summary.json` records the run's scope under `scope`. It comes from the chrom

- `autosomes_covered`: how many of chr1 to chr22 have calls, and `autosomes`: which ones.
- `chrx`: whether chrX has calls. chrX alone does not make a run partial, because these results read the autosomes.
- `intervals`: the intervals the run was limited to, from a `run intervals` row of `run_manifest.tsv`, else `null`. No step writes this row yet, so today it is `null`: a run limited with `--intervals` is caught only by the chromosomes its calls are on.
- `intervals`: the intervals the run was limited to, from the `run intervals` row of `run_manifest.tsv`, else `null`. `run-all.sh` writes that row when it was given `INTERVALS` (its `--intervals`), and so does the Nextflow report for `--intervals`; a whole-genome run has no such row. So a run limited to parts of all 22 autosomes is marked too.
- `partial` and `mark`: the run is partial when it covers fewer than 22 autosomes (`partial: covers 2 of 22 autosomes`) or was limited to intervals (`partial: limited to the run's intervals (...)`).

When the run is partial, each of those six sections that has a result of this run gets the same text in its `partial` field. A stale section holds a result from an earlier run, and that run's scope was not recorded: it gets `partial: not known, from an earlier run`, whatever this run covered. Without a sample VCF that can be read to its end, the scope is not known: `summary.json` has no `scope`, and no section is marked. The scope says which chromosomes have calls. It does not say how much of each chromosome was read well; callability does that.
Expand Down
14 changes: 12 additions & 2 deletions modules/local/html_report/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@
vcf/<id>.g.vcf.gz and its .tbi (or .csi), where the bash report finds
step 03's gVCF. A run_manifest.tsv in the sample folder records the
OwnGenome release (nextflow.config's manifest.version), so the report
names it as the bash report does; it is not published, because
names it as the bash report does, and the regions of --intervals when
the run was limited to them, so summary.json marks the run partial as
the bash report does; it is not published, because
run-all.sh writes the full manifest there. Every gene and rsID the
results mention also goes to ${id}_search_index.json beside it
(bin/results_index.py), for the live site's search, and the report's
Expand Down Expand Up @@ -69,6 +71,8 @@ process HTML_REPORT {
def gvcf_index = meta.gvcf ? meta.gvcf[1].toString().replace("'", "'\\''") : ''
def gvcf_suffix = gvcf_index.endsWith('.csi') ? 'csi' : 'tbi'
def release = workflow.manifest.version ?: ''
// --intervals as given, single-quoted below like gvcf; the task script puts it on one line
def intervals = params.intervals ? params.intervals.toString().replace("'", "'\\''") : ''
"""
# Where og summary collect looks for each output (rust/crates/og-summary)
S=layout/${id}
Expand All @@ -82,7 +86,13 @@ process HTML_REPORT {
fi
# The release this run is (bash: bin/write_manifest.sh's run pipeline_version row)
release='${release}'
[ -z "\$release" ] || printf 'run\\tpipeline_version\\t%s\\n' "\$release" > "\$S/run_manifest.tsv"
[ -z "\$release" ] || printf 'run\\tpipeline_version\\t%s\\n' "\$release" >> "\$S/run_manifest.tsv"
# The regions this run was limited to (bash: the run intervals row), so
# og summary collect marks the genome-wide sections partial; one line with
# single spaces, as bin/write_manifest.sh writes INTERVALS
intervals='${intervals}'
read -r -d '' -a regions <<< "\$intervals" || true
[ "\${#regions[@]}" -eq 0 ] || printf 'run\\tintervals\\t%s\\n' "\${regions[*]}" >> "\$S/run_manifest.tsv"
for f in input/*; do
n=\$(basename "\$f")
if [ -n "\$gvcf" ] && { [ "\$n" = "\$gvcf" ] || [ "\$n" = "\$gvcf_index" ]; }; then continue; fi
Expand Down
Loading
Loading