Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/guard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -910,6 +910,14 @@ jobs:
working-directory: .
run: bin/og dev remnants

# EVERYTHING=true and -profile everything run every optional step from
# assets/everything.tsv: a new tool, switch, setup flag, false-by-default
# or opt-in parameter, or optional image with no row there fails here.
# Its planted cases are og-dev's everything::tests, in cargo test.
- name: og dev everything-check
working-directory: .
run: bin/og dev everything-check

# main.nf runs the same check before any task: a stale og stops
# `nextflow run` with the setup command. Without the check, this stub
# run completes and the step fails.
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- One switch now runs every optional step. Before, a run that should run everything had to name each opt-in step in `TOOLS` and set each switch by hand, and nothing said when a new opt-in was missing from that list. `EVERYTHING=true ./scripts/run-all.sh <sample> <sex>` reads the new list `assets/everything.tsv`: it puts every `--tools` name in `TOOLS`, sets `KIR`, `HLA_QC`, `TIDDIT`, `GRIDSS`, `IMPUTATION`, `SOMATIC` and `BENCHMARK` to `true` and `EXTRA_CALLERS` to all four callers. A step whose data is not installed, or a panel set to `none`, stops the run with exit 2 before Nextflow, and the error names the setup command. That includes the classic BWA index that GRIDSS (step 4b) needs, which `setup.sh` does not build. A refused run leaves the last run's `logs/run_status.tsv` as it was. `TOOLS`, or a switch set to another value, also stops it with exit 2. A skip that comes from the sample stays a skip: no BAM, no gVCF, a female sample for the Y steps, a BAM that is not PacBio HiFi for Paraphase. On the Nextflow path, `-profile everything` (`conf/everything.config`) sets the same tools and switches. `og dev everything-check`, in Guard, fails when `main.nf`, `nextflow.config`, `run-all.sh`, `setup.sh` or `versions.env` gains an opt-in with no row in the list. Without `EVERYTHING` a default run does not change, with three exceptions. `TIDDIT=true` is a new switch that runs `04a-tiddit.sh` after the pipeline. Step 37 (Yleaf) on a female sample is now listed as `skipped (female sample: no Y chromosome)`, as step 45 already was; the pipeline skipped it for her before, so no result changes. Step 35b (Paraphase) on a BAM that is not PacBio HiFi, or on a FASTQ row, is now `skipped (not PacBio HiFi: ...)` instead of a failed step: `run-all.sh` runs `og paraphase check-bam` on the BAM first. `logs/run_status.tsv` records `EVERYTHING` and `TIDDIT` with the other switches, and `og recheck --rerun` replays them.
- A third-party outage no longer fails the Freshness check of a pull request. Since 2026-11-18 rest.ensembl.org has not answered, and the pull-request job's dry run of `og dev freshness` failed on every pull request that touched `versions.env` or `og-fresh`. `og dev freshness --unreachable-warns` now lists a lookup whose upstream did not answer (no connection, a timeout, HTTP 5xx or 429 after the retries, or HTTP 403) under "Upstreams that did not answer" as a warning, and its database row reads `NO ANSWER`; it does not change the exit. The 403 counts because rest.ensembl.org came back on 2026-11-19 answering 403 to GitHub's runners and 200 elsewhere, which still failed the pull-request job. A 404, an empty answer or an answer in an odd shape still exits 1. `freshness.yml` passes the flag only in the pull-request dry run. Without the flag the report and the exit are as before, and the monthly report on main still counts an outage as an error in the issue.
- Three statements in the docs were wrong. Troubleshooting said exit code 137 confirms a kill for memory, and checked it with `docker ps -a --latest`. 137 is any SIGKILL, and the step scripts run their containers with `--rm`, so that command shows an older, unrelated container. The page now says how to confirm a kill for memory: `docker events --filter event=oom`, the kernel log, or `.State.OOMKilled` for a container kept without `--rm`. The run-state and troubleshooting pages said `bin/og` does not run on a Mac and told Mac users to build `og` with cargo or run it in a container. `bin/og` runs the native macOS build that `scripts/setup.sh --og` installs, and the pages now say so; on a Mac, `og run headroom` reports the machine's memory as unknown. Troubleshooting, the FAQ and the lessons-learned page said the step scripts run their containers as root, and lessons-learned gave `:latest` images and `chmod 777` as current fixes. The step scripts run their containers as you; those entries are now marked as past fixes and name what the pipeline does today. Documentation only: no option, output or behaviour changes.
- Lint's ShellCheck job now checks every tracked shell script, not only `scripts/`: the e2e cases and their library, the fake-docker and parity tests, `tests/*.sh`, `bin/og` and the fake tools, 482 files in all, at every ShellCheck level. The 265 findings these files hold today are listed in `tests/shellcheck-baseline.txt` and still pass; any other finding fails lint. A listed finding is matched by its file, its code and the text of its line, so it keeps matching when lines above it move, and a new line with the same mistake still fails. Each run plants an unquoted `$1` in a new test script and on a new line of a listed script, and fails if either passes or if the file search misses the new script or `bin/og`. The `scripts/` check at warning level stays as it was. Part of #411 (plat-10).
Expand Down
159 changes: 159 additions & 0 deletions assets/everything.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# assets/everything.tsv: every optional part of a run, and what EVERYTHING=true (scripts/run-all.sh) and
# -profile everything (conf/everything.config) do with it. One row per item; `og dev everything-check` fails when
# main.nf, nextflow.config, run-all.sh, setup.sh or versions.env gains an item with no row here.
# kind: tool (a main.nf --tools name: EVERYTHING names it), switch (a run-all.sh switch: EVERYTHING sets it to
# value), setup (a setup.sh flag: value names the rows whose data it installs), input (a choice or a file only
# the user can give: EVERYTHING leaves it to them), image (an optional image of versions.env: value names the row
# whose step runs it), excluded (left out: reason says why).
# value: tool and input rows: how to install the data, '-' when none is needed; switch rows: the value EVERYTHING
# sets; setup and image rows: the rows they serve, comma separated.
# nextflow: switch rows with a Nextflow parameter: that parameter as conf/everything.config sets it, else '-'.
# Rows named --<param> stand for a Nextflow parameter that has no run-all.sh switch.
kind name value nextflow reason
tool pharmcat - - step 07
tool cpic - - step 27
tool clinvar setup.sh - step 06
tool pypgx the pypgx-bundle, by hand: docs/00-reference-setup.md - step 32
tool vep setup.sh --vep-cache - step 13
tool vcfanno setup.sh --vep-cache, then the score files of docs/00-reference-setup.md - step 30
tool slivar setup.sh --vep-cache - step 31
tool clinical_filter setup.sh --vep-cache - step 23
tool cpsr setup.sh --clinical-data - step 17
tool roh - - step 11
tool prs setup.sh - step 25
tool ancestry setup.sh --ancestry-panel - step 26
tool ancestry_fine setup.sh --ancestry-fine-panel - step 26b
tool mito_haplogroup - - step 12
tool hla_typing setup.sh - step 08
tool expansion_hunter - - step 09
tool stranger - - step 9b
tool telomere_hunter - - step 10
tool mosdepth - - step 16b
tool mito_variants - - step 20
tool cyrius setup.sh --cyrius - step 21; its licence is non-commercial, and installing it accepts that licence
tool manta - - step 04
tool delly - - step 19
tool cnvpytor the CNVpytor resources, by hand: docs/00-reference-setup.md - step 18
tool duphold - - step 15
tool annotsv setup.sh --annotsv-data - step 05
tool survivor_merge - - step 22
tool sample_qc setup.sh --sample-qc-data - step 33
tool parascopy setup.sh --parascopy-data - step 35
tool paraphase - - step 35b; a BAM that is not PacBio HiFi is skipped
tool y_haplogroup setup.sh --yleaf-data - step 37; a female sample is skipped
tool html_report - - step 24; run-all.sh writes the report itself after the pipeline
tool multiqc - - step 28
tool consumer_export setup.sh --consumer-sites - step 40
tool bloodtype - - step 41
tool lpa - - step 46
tool vntyper - - step 52
tool traits - - step 42
tool archaic setup.sh --archaic - step 43
tool ancient setup.sh --ancient-panel - step 44
tool y_str - - step 45; a female sample is skipped
tool viral setup.sh --viral-data - step 47
tool chip setup.sh --chip-data - step 49
tool mei setup.sh --megane-kmers - step 50
tool mocha setup.sh --mocha-data - step 51
tool locityper setup.sh --locityper-data - step 53
input exomiser setup.sh --exomiser-data - step 48 runs when HPO (or --hpo) names the symptoms; naming it without them stops the run
excluded cram_archive - - a CRAM copy of each BAM for the archive: it costs disk and adds no result
switch KIR true --kir true step 08 types the KIR genes too; setup.sh --kir-data
switch HLA_QC true --hla_qc true step 08 runs OptiType beside T1K
switch CPSR_SECONDARY_FINDINGS true --cpsr_secondary_findings true step 17 also screens the ACMG SF genes
switch GRIDSS true - script 04b after the pipeline
switch TIDDIT true - script 04a after the pipeline
switch IMPUTATION true - script 14 after the pipeline
switch SOMATIC true - script 29 after the pipeline
switch EXTRA_CALLERS gatk,freebayes,strelka2,octopus - scripts 03a to 03d after the pipeline
switch BENCHMARK true - benchmark-variants.sh after the extra callers
input HPO setup.sh --exomiser-data - the symptoms as HPO ids; only the user knows them
input FAMILIES - - who is related to whom; steps 38 and 39 run when the family has what they need
input ALIGN_DIR - - the BAM of another aligner (script 02a writes aligned_bwamem2/)
input REF_FASTA - - another GRCh38 build
input EH_CATALOG - - another catalog for steps 09 and 9b
input MANTA_CALL_REGIONS - - the regions Manta calls on
input PARASCOPY_POPULATION - - the Parascopy model; EUR when unset
input PARAPHASE_REGIONS - - the Paraphase regions; all when unset
input ANCESTRY_PANEL - - another panel for step 26; none skips the step, which EVERYTHING=true refuses
input ANCESTRY_FINE_DIR - - another panel for step 26b; none skips the step, which EVERYTHING=true refuses
input ARCHAIC_DIR - - another table for step 43; none skips the step, which EVERYTHING=true refuses
input ARCHAIC_POP - - the 1000 Genomes group of step 43; ALL when unset
input PGSC_CALC_DIR - - another pgsc_calc checkout
input ANCIENT_PANEL_DIR - - another panel for step 44
input VIRAL_REF - - other viral genomes for step 47
input MEGANE_KMERS - - another k-mer set for step 50
input MOCHA_DATA - - another panel for step 51
input LOCITYPER_DATA - - another database for step 53
input LOCITYPER_BG_REGION - - another background region for step 53
input long-read - - scripts 02b, 03e and 04c read ONT or PacBio HiFi reads: run them by hand on long-read data
input --pedigree - - the family file of the Nextflow path (FAMILIES on run-all.sh)
input --medications - - the drugs taken, for step 27's phenoconversion
excluded EVERYTHING - - this switch
excluded TOOLS - - EVERYTHING=true names every tool itself, so it refuses TOOLS
excluded GENOME_DIR - - where the data is, not a step
excluded SKIP_VALIDATION - - skips a check, not a step
excluded THREADS - - a resource cap, not a step
excluded SKIP_TRIM - - skips fastp, so it removes a step
excluded INTERVALS - - calls part of the genome, so it removes work
excluded OWNGENOME_LIVE_URL - - the live site's address, not a step
excluded NXF_VER - - the Nextflow release, not a step
excluded NXF_WORK - - Nextflow's work folder, not a step
excluded NXF_HOME - - Nextflow's home folder, not a step
excluded NXF_OFFLINE - - Nextflow's offline mode, not a step
excluded --allow_unfiltered - - accepts a VCF with no FILTER=PASS record: an input choice, not a step
excluded --skip_trim - - skips fastp, so it removes a step
excluded --prs_keep_ambiguous - - changes which rows of a score count, not which steps run
setup --cyrius cyrius - Cyrius, under its non-commercial licence
setup --parascopy-data parascopy - Parascopy's table and models
setup --kir-data KIR - IPD-KIR
setup --ancestry-panel ancestry - pgsc_calc's ancestry panel
setup --ancestry-fine-panel ancestry_fine - the fine-grained ancestry panel
setup --yleaf-data y_haplogroup - Yleaf's tables
setup --consumer-sites consumer_export - the 23andMe v5 site table
setup --ancient-panel ancient - the AADR panel
setup --viral-data viral - the viral genomes
setup --exomiser-data exomiser,HPO - Exomiser's data
setup --chip-data chip - Ensembl's gene annotation
setup --strchive-data expansion_hunter,stranger - the STRchive catalog; part of a full setup
setup --megane-kmers mei - MEGAnE's k-mer set
setup --mocha-data mocha - MoChA's panel
setup --locityper-data locityper - Locityper's database
setup --vep-cache vep,vcfanno,slivar,clinical_filter - the VEP cache; part of a full setup
setup --annotsv-data annotsv - AnnotSV's data; part of a full setup
setup --archaic archaic - the archaic table
setup --clinical-data cpsr,vep - the VEP caches and the CPSR bundle; part of a full setup
setup --sample-qc-data sample_qc - somalier's sites and VerifyBamID2's panel; part of a full setup
excluded --masked-reference - - another reference: every sample is aligned again on it
excluded --og - - builds og, which a full setup does too
excluded --no-clinical-data - - installs less
excluded --pull-only - - pulls the images only
excluded --refresh - - replaces ClinVar with a newer release
image BWAMEM2_IMAGE ALIGN_DIR - script 02a
image BWA_IMAGE GRIDSS - the classic BWA index of script 04b
image FREEBAYES_IMAGE EXTRA_CALLERS - script 03b
image STRELKA_IMAGE EXTRA_CALLERS - script 03c
image OCTOPUS_IMAGE EXTRA_CALLERS - script 03d
image CLAIR3_IMAGE long-read - script 03e
image TIDDIT_IMAGE TIDDIT - script 04a
image SNIFFLES_IMAGE long-read - script 04c
image HAPPY_IMAGE BENCHMARK - benchmark-variants.sh
image YLEAF_IMAGE y_haplogroup - step 37
image HIPSTR_IMAGE y_str - step 45
image OPTITYPE_IMAGE HLA_QC - step 08 with HLA_QC=true
image PARASCOPY_IMAGE parascopy - step 35
image PARAPHASE_IMAGE paraphase - step 35b
image MEGANE_IMAGE mei - step 50
image VNTYPER_IMAGE vntyper - step 52
image LOCITYPER_IMAGE locityper - step 53
image GLNEXUS_IMAGE FAMILIES - step 38
image HAP_IBD_IMAGE FAMILIES - step 38
image EXOMISER_IMAGE exomiser - step 48
image ADMIXTURE_IMAGE ancestry_fine - step 26b
image BEAGLE_IMAGE ancestry_fine,mocha - steps 26b and 51
image RFMIX_IMAGE ancestry_fine - step 26b
image BCFTOOLS_HTTPS_IMAGE --ancestry-fine-panel - setup.sh --ancestry-fine-panel
image MOCHA_IMAGE mocha - step 51
image IBDMIX_IMAGE archaic - step 43
image EIGENSOFT_IMAGE ancient - step 44
excluded SAMTOOLS_HTTPS_IMAGE - - docs/quick-test.md reads a CRAM over https with it; no step runs it
2 changes: 2 additions & 0 deletions assets/output_contract.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,7 @@ switch VIRAL_REF - - - - experimental - -
switch MEGANE_KMERS - - - - experimental - -
switch MOCHA_DATA - - - - experimental - -
switch GRIDSS - - - - stable - -
switch TIDDIT - - - - stable - -
switch IMPUTATION - - - - stable - -
switch SOMATIC - - - - stable - -
switch EXTRA_CALLERS - - - - stable - -
Expand All @@ -433,6 +434,7 @@ switch FAMILIES - - - - experimental - -
switch OWNGENOME_LIVE_URL - - - - experimental - -
switch LOCITYPER_DATA - - - - experimental - -
switch LOCITYPER_BG_REGION - - - - experimental - -
switch EVERYTHING - - - - stable - -
switch NXF_VER - - - - stable - -
switch NXF_WORK - - - - stable - -
switch NXF_HOME - - - - stable - -
Expand Down
1 change: 1 addition & 0 deletions assets/steps.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ step label tool recorded processes sections upstream hours hours_from hours_low
03c Strelka2 yes 02
03d Octopus yes 02 4 estimate
04 Manta manta yes MANTA manta 02 0.68 measured 0.3666
04a TIDDIT yes 02
04b GRIDSS yes 02 8 estimate
05 AnnotSV annotsv yes ANNOTSV annotsv 15,19,18 0.03 measured 0.0166
06 ClinVar screen clinvar yes CLINVAR_SCREEN,CLINVAR_CDNA_CHECK clinvar,carrier 03 0.02 measured 0.0083 clinvar_file_date
Expand Down
2 changes: 1 addition & 1 deletion bin/collect_summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ def collect(sample, sample_dir, declared_sex=None, out=None):
# equal; the set stays written here because this file also runs where only
# bin/ is mounted). A section whose step is not here (variant calling, step
# 03, which run-all reuses on purpose) is never marked stale.
RUN_ALL_STEPS = {"03a", "03b", "03c", "03d", "04", "04b", "05", "06", "07", "08", "09", "09b", "10", "11",
RUN_ALL_STEPS = {"03a", "03b", "03c", "03d", "04", "04a", "04b", "05", "06", "07", "08", "09", "09b", "10", "11",
"12", "13", "14", "15", "16", "16b", "17", "18", "19", "20", "21", "22", "23", "25", "26",
"26b", "27", "28", "29", "30", "31", "32", "33", "35", "35b", "37", "38", "39", "40", "41",
"42", "43", "44", "45", "46", "47", "48", "49", "50", "51", "52", "53"}
Expand Down
23 changes: 23 additions & 0 deletions conf/everything.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Nextflow config: every optional step
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nextflow run main.nf -profile docker,everything --input ... --reference ... \
--<database> <path> ...

The tool rows and the switch rows of assets/everything.tsv, as
EVERYTHING=true in scripts/run-all.sh sets them: every --tools name but
exomiser (it runs when --hpo names the symptoms) and cram_archive (an
archive copy of each BAM), plus KIR typing and OptiType. Pass every
database the tools need: main.nf stops before any task and names the
option a tool lacks. `og dev everything-check` fails when this file and
the list differ.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*/

params {
tools = 'pharmcat,cpic,clinvar,pypgx,vep,vcfanno,slivar,clinical_filter,cpsr,roh,prs,ancestry,ancestry_fine,mito_haplogroup,hla_typing,expansion_hunter,stranger,telomere_hunter,mosdepth,mito_variants,cyrius,manta,delly,cnvpytor,duphold,annotsv,survivor_merge,sample_qc,parascopy,paraphase,y_haplogroup,html_report,multiqc,consumer_export,bloodtype,lpa,vntyper,traits,archaic,ancient,y_str,viral,chip,mei,mocha,locityper'
kir = true
hla_qc = true
cpsr_secondary_findings = true
}
Loading
Loading