Skip to content

feat(run): add EVERYTHING=true so the owner's runs never miss an optional step - #538

Open
geiresx wants to merge 5 commits into
GeiserX:mainfrom
geiresx:pgp/screen-everything-switch
Open

geiresx wants to merge 5 commits into
GeiserX:mainfrom
geiresx:pgp/screen-everything-switch

Conversation

@geiresx

@geiresx geiresx commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A run that should run every step had to name each opt-in step in TOOLS and set each switch by hand. Such a hand-made list drifts: a new opt-in step is missed, and nothing says so. A step whose data was never installed was skipped with one line in the plan, and the run went on, so a run meant to cover everything could end with steps missing.

Fix

  • assets/everything.tsv is the one list of every optional item: each --tools name of main.nf, each run-all.sh switch, each setup.sh flag, each false-by-default or opt-in Nextflow parameter and each optional image of versions.env. Each row is tool, switch, setup, input, image or excluded, with the reason.
  • EVERYTHING=true ./scripts/run-all.sh <sample> <sex> puts every tool row in TOOLS and sets every switch row: KIR, HLA_QC, TIDDIT, GRIDSS, IMPUTATION, SOMATIC, BENCHMARK to true, EXTRA_CALLERS to all four callers. TOOLS, or a switch set to another value, stops it with exit 2.
  • A step that would be skipped because its data is not installed, or because ANCESTRY_PANEL, ANCESTRY_FINE_DIR or ARCHAIC_DIR is none, stops the run with exit 2 before Nextflow. The error lists every such step with the command that installs it, for example setup.sh --yleaf-data. The same check covers the classic BWA index that GRIDSS (script 04b, after the pipeline) needs beside the reference: setup.sh does not build it, so without it the run would fail at 04b hours in. The error names bwa index (docs/04b-gridss.md).
  • A refused run puts the last run's logs/run_status.tsv back and posts no run-finished event, so og recheck --rerun still replays the last real run. The message says Nextflow did not start.
  • A skip that comes from the sample stays a skip: no BAM, no gVCF, a female sample for steps 37 and 45, a BAM that is not PacBio HiFi for step 35b.
  • TIDDIT=true is a new switch: it runs 04a-tiddit.sh after the pipeline, so every optional step is reachable from one command.
  • -profile everything (conf/everything.config) gives the Nextflow path the same tools and switches.
  • og dev everything-check (Rust, in Guard's rust job) fails when one of those items has no row, when a row names an item that is gone, when conf/everything.config differs from the rows, or when conf/test_all.config misses a tool.

What changes without EVERYTHING:

  • Step 37 (Yleaf) on a female sample is listed as skipped (female sample: no Y chromosome), as step 45 already was. The pipeline skipped it for her before, so no result changes.
  • Step 35b (Paraphase) on a BAM that is not PacBio HiFi, or on a FASTQ row, is skipped (not PacBio HiFi: ...) instead of a failed step. run-all.sh runs og paraphase check-bam, the check PARAPHASE_CHECK runs, on the BAM first.
  • logs/run_status.tsv records EVERYTHING and TIDDIT with the other switches. og recheck --rerun replays them. With TIDDIT=true it gains a step 04a row. No output file is renamed or removed.

Decisions

  • Files outside the package's list, each needed to keep main true. rust/crates/og-recheck/src/rerun.rs gains TIDDIT and EVERYTHING: a unit test keeps its replay list equal to SWITCHES, and without them a replay would drop the switch. assets/output_contract.tsv gains the two switch rows that og dev contract --static asks for. Step 04a joins assets/steps.tsv and RUN_ALL_STEPS, in og-summary and in the remnant bin/collect_summary.py (one token), because the tests keep them equal to the steps run-all.sh records. The regex of tests/test_run_state.py that finds the script steps now accepts 04a as it accepts 04b. No new Python logic. docs/pipeline-overview.md lists TIDDIT=true with the opt-in switches and points to EVERYTHING=true; without it the page would say 4a runs only by hand.
  • assets/output_contract.tsv and bin/collect_summary.py are also changed by other open lanes. On a merge conflict keep both sides' rows.
  • The BWA index check runs only when the sample has reads (a FASTQ or BAM row): a VCF-only sample has no BAM for GRIDSS, which is a skip from the input.
  • The list has a sixth kind, image: an optional image names the row whose step runs it. That ties each image to a step the switch covers.
  • exomiser is an input row: naming it without --hpo stops main.nf, and only the user knows the symptoms. cram_archive is excluded: it writes a CRAM copy of each BAM, which costs disk and adds no result.
  • The stub check is -profile test_all,everything, not test,everything. The test profile has no stub databases, so main.nf stops at the first tool without its option, which is the behaviour we want on a real run.
  • PRS_ACADEMIC is not in the list: the switch does not exist on main yet. When its lane adds it to SWITCHES, the check asks for its row.

How it was checked

  • og dev everything-check on this head: ok assets/everything.tsv: 147 rows cover every tool, switch, setup flag and optional image; conf/everything.config and conf/test_all.config agree. On main plus the new list it fails with exit 1 and names TIDDIT and EVERYTHING.
  • cargo test -p og-dev --lib everything: 10 passed. They are positive controls on copies of the files, and each one must fail and name the item: a new name in known_tools, a new foo = false param, a new // opt-in param, a new setup flag in the case line, a new # optional image, a new switch, a stale row, a drifted conf/everything.config and a test_all without a tool.
  • cargo test -p og-recheck, -p og-summary, -p og-dev --test repo, -p og --test port_scaffold, cargo fmt --check and clippy -D warnings: green.
  • Fake-docker case run-all-everything, run in a Linux container: it passes on this head. With every data folder stubbed, --tools is every tool row but html_report, no step is skipped, --kir true and --hla_qc true are passed, and scripts 04a, 04b, 14, 29, 03a to 03d and benchmark-variants.sh run. With the Yleaf folder removed it exits 2 before Nextflow and names setup.sh --yleaf-data. With female it runs, and steps 37 and 45 are skipped (female sample: no Y chromosome). The same case against main's run-all.sh fails: EVERYTHING is ignored and the opt-in steps are skipped.
  • Stub runs in the pinned Nextflow 26.10.5: -profile test_all,everything -stub completes with 112 processes, the 113 of -profile test_all -stub minus CRAM_ARCHIVE.
  • CI on the earlier head 87fc304, the fast checks on the geiresx pool, all green:
  • Head c3a3c78 (this round): the fake-docker case run-all-everything passes on macOS with GNU coreutils. It now stubs the five BWA index files in the all-data set, and a new no-bwa case removes them and expects exit 2 with 04b GRIDSS (GRIDSS=true): classic BWA index not installed: ...; install: bwa index (docs/04b-gridss.md) and no Nextflow call. The no-yleaf case also checks that logs/run_status.tsv is byte for byte the last run's and that no .prev file is left. Both are shown red: with the BWA check turned off the case fails with no-bwa exited 1, expected 2; with the status restore removed it fails with a refused EVERYTHING=true run changed the last run's status. ShellCheck on both scripts: no new finding.
  • The red runs: each of the 10 unit tests plants the missing item on a copy and asserts the check fails and names it. The fake-docker case was run red against main's run-all.sh in a local Linux container, as said above. E2E for this PR runs on the batch tip.

Beads: D8

… script 04a

EVERYTHING=true reads assets/everything.tsv: every tool row goes into TOOLS
(html_report aside: run-all.sh writes the report itself), every switch row
is set to its value, and TOOLS or a switch set to another value stops the run
with exit 2. A step skipped because its data is not installed, or because a
panel is set to none, stops the run with exit 2 before Nextflow and names the
setup command. Skips that come from the sample stay skips: no BAM, no gVCF,
a female sample for steps 37 and 45, a BAM that is not PacBio HiFi for 35b.

Step 37 on a female sample is now listed as skipped, as step 45 was; the
pipeline skipped it already. Step 35b runs og paraphase check-bam on the BAM
first, so a BAM that is not PL:PACBIO, or a FASTQ row, is a skip instead of a
failed step.

TIDDIT=true runs 04a-tiddit.sh after the pipeline, so every optional step is
reachable from one command. EVERYTHING and TIDDIT join SWITCHES, the replay
list of og recheck, the output contract, and step 04a joins assets/steps.tsv
and RUN_ALL_STEPS, which the tests keep equal to what run-all.sh records.

Bead: D8
assets/everything.tsv lists every optional item: each --tools name of
main.nf, each run-all.sh switch, each setup.sh flag, each false-by-default or
opt-in parameter of nextflow.config and each optional image of versions.env,
as tool, switch, setup, input, image or excluded with the reason.
og dev everything-check fails when one of them has no row, when a row names
an item that is gone, when conf/everything.config sets other tools or
parameters than the rows, or when conf/test_all.config misses a tool. Guard
runs it; its planted cases are og-dev unit tests.

-profile everything (conf/everything.config) gives the Nextflow path the same
tools and switches as EVERYTHING=true.

Bead: D8
Every data folder stubbed: every tool row runs and every script step runs
after the pipeline. Without Yleaf's tables it exits 2 before Nextflow and
names setup.sh --yleaf-data. A female sample still runs with the Y steps
skipped; a BAM that is not PacBio HiFi skips Paraphase. TOOLS, KIR=false or
ARCHAIC_DIR=none with EVERYTHING=true are refused.

Bead: D8
@geiresx
geiresx requested a review from GeiserX as a code owner November 20, 2026 10:21
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 47 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between fd81a5c and c3a3c78.

Files ignored due to path filters: 3

  • assets/everything.tsv is excluded by !**/*.tsv
  • assets/output_contract.tsv is excluded by !**/*.tsv
  • assets/steps.tsv is excluded by !**/*.tsv
Files selected for processing: 14

  • .github/workflows/guard.yml
  • CHANGELOG.md
  • bin/collect_summary.py
  • conf/everything.config
  • docs/getting-started.md
  • docs/parameters.md
  • docs/pipeline-overview.md
  • nextflow.config
  • rust/crates/og-dev/src/everything.rs
  • rust/crates/og-recheck/src/rerun.rs
  • rust/crates/og-summary/src/collect/model.rs
  • scripts/run-all.sh
  • tests/fake-docker/run-all-everything.sh
  • tests/test_run_state.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…index

GRIDSS (script 04b, after the pipeline) needs the classic BWA index, which
setup.sh does not build; an EVERYTHING=true run now refuses up front instead
of failing hours in. A refused run puts the last run's run_status.tsv back
and posts no run-finished event. pipeline-overview.md lists TIDDIT=true.
geiresx pushed a commit to geiresx/OwnGenome that referenced this pull request Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants