Repository navigation
Conversation
… script 04a EVERYTHING=true reads assets/everything.tsv: every tool row goes into TOOLS (html_report aside: run-all.sh writes the report itself), every switch row is set to its value, and TOOLS or a switch set to another value stops the run with exit 2. A step skipped because its data is not installed, or because a panel is set to none, stops the run with exit 2 before Nextflow and names the setup command. Skips that come from the sample stay skips: no BAM, no gVCF, a female sample for steps 37 and 45, a BAM that is not PacBio HiFi for 35b. Step 37 on a female sample is now listed as skipped, as step 45 was; the pipeline skipped it already. Step 35b runs og paraphase check-bam on the BAM first, so a BAM that is not PL:PACBIO, or a FASTQ row, is a skip instead of a failed step. TIDDIT=true runs 04a-tiddit.sh after the pipeline, so every optional step is reachable from one command. EVERYTHING and TIDDIT join SWITCHES, the replay list of og recheck, the output contract, and step 04a joins assets/steps.tsv and RUN_ALL_STEPS, which the tests keep equal to what run-all.sh records. Bead: D8
assets/everything.tsv lists every optional item: each --tools name of main.nf, each run-all.sh switch, each setup.sh flag, each false-by-default or opt-in parameter of nextflow.config and each optional image of versions.env, as tool, switch, setup, input, image or excluded with the reason. og dev everything-check fails when one of them has no row, when a row names an item that is gone, when conf/everything.config sets other tools or parameters than the rows, or when conf/test_all.config misses a tool. Guard runs it; its planted cases are og-dev unit tests. -profile everything (conf/everything.config) gives the Nextflow path the same tools and switches as EVERYTHING=true. Bead: D8
Every data folder stubbed: every tool row runs and every script step runs after the pipeline. Without Yleaf's tables it exits 2 before Nextflow and names setup.sh --yleaf-data. A female sample still runs with the Y steps skipped; a BAM that is not PacBio HiFi skips Paraphase. TOOLS, KIR=false or ARCHAIC_DIR=none with EVERYTHING=true are refused. Bead: D8
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 47 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…index GRIDSS (script 04b, after the pipeline) needs the classic BWA index, which setup.sh does not build; an EVERYTHING=true run now refuses up front instead of failing hours in. A refused run puts the last run's run_status.tsv back and posts no run-finished event. pipeline-overview.md lists TIDDIT=true.
geiresx
pushed a commit
to geiresx/OwnGenome
that referenced
this pull request
Nov 20, 2026
# Conflicts: # CHANGELOG.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A run that should run every step had to name each opt-in step in
TOOLSand set each switch by hand. Such a hand-made list drifts: a new opt-in step is missed, and nothing says so. A step whose data was never installed was skipped with one line in the plan, and the run went on, so a run meant to cover everything could end with steps missing.Fix
assets/everything.tsvis the one list of every optional item: each--toolsname ofmain.nf, eachrun-all.shswitch, eachsetup.shflag, each false-by-default or opt-in Nextflow parameter and each optional image ofversions.env. Each row istool,switch,setup,input,imageorexcluded, with the reason.EVERYTHING=true ./scripts/run-all.sh <sample> <sex>puts every tool row inTOOLSand sets every switch row:KIR,HLA_QC,TIDDIT,GRIDSS,IMPUTATION,SOMATIC,BENCHMARKtotrue,EXTRA_CALLERSto all four callers.TOOLS, or a switch set to another value, stops it with exit 2.ANCESTRY_PANEL,ANCESTRY_FINE_DIRorARCHAIC_DIRisnone, stops the run with exit 2 before Nextflow. The error lists every such step with the command that installs it, for examplesetup.sh --yleaf-data. The same check covers the classic BWA index that GRIDSS (script 04b, after the pipeline) needs beside the reference:setup.shdoes not build it, so without it the run would fail at 04b hours in. The error namesbwa index(docs/04b-gridss.md).logs/run_status.tsvback and posts no run-finished event, soog recheck --rerunstill replays the last real run. The message says Nextflow did not start.TIDDIT=trueis a new switch: it runs04a-tiddit.shafter the pipeline, so every optional step is reachable from one command.-profile everything(conf/everything.config) gives the Nextflow path the same tools and switches.og dev everything-check(Rust, in Guard's rust job) fails when one of those items has no row, when a row names an item that is gone, whenconf/everything.configdiffers from the rows, or whenconf/test_all.configmisses a tool.What changes without
EVERYTHING:skipped (female sample: no Y chromosome), as step 45 already was. The pipeline skipped it for her before, so no result changes.skipped (not PacBio HiFi: ...)instead of a failed step.run-all.shrunsog paraphase check-bam, the checkPARAPHASE_CHECKruns, on the BAM first.logs/run_status.tsvrecordsEVERYTHINGandTIDDITwith the other switches.og recheck --rerunreplays them. WithTIDDIT=trueit gains astep 04arow. No output file is renamed or removed.Decisions
rust/crates/og-recheck/src/rerun.rsgainsTIDDITandEVERYTHING: a unit test keeps its replay list equal toSWITCHES, and without them a replay would drop the switch.assets/output_contract.tsvgains the two switch rows thatog dev contract --staticasks for. Step 04a joinsassets/steps.tsvandRUN_ALL_STEPS, inog-summaryand in the remnantbin/collect_summary.py(one token), because the tests keep them equal to the stepsrun-all.shrecords. The regex oftests/test_run_state.pythat finds the script steps now accepts04aas it accepts04b. No new Python logic.docs/pipeline-overview.mdlistsTIDDIT=truewith the opt-in switches and points toEVERYTHING=true; without it the page would say 4a runs only by hand.assets/output_contract.tsvandbin/collect_summary.pyare also changed by other open lanes. On a merge conflict keep both sides' rows.image: an optional image names the row whose step runs it. That ties each image to a step the switch covers.exomiseris aninputrow: naming it without--hpostopsmain.nf, and only the user knows the symptoms.cram_archiveis excluded: it writes a CRAM copy of each BAM, which costs disk and adds no result.-profile test_all,everything, nottest,everything. Thetestprofile has no stub databases, somain.nfstops at the first tool without its option, which is the behaviour we want on a real run.PRS_ACADEMICis not in the list: the switch does not exist on main yet. When its lane adds it toSWITCHES, the check asks for its row.How it was checked
og dev everything-checkon this head:ok assets/everything.tsv: 147 rows cover every tool, switch, setup flag and optional image; conf/everything.config and conf/test_all.config agree. On main plus the new list it fails with exit 1 and namesTIDDITandEVERYTHING.cargo test -p og-dev --lib everything: 10 passed. They are positive controls on copies of the files, and each one must fail and name the item: a new name inknown_tools, a newfoo = falseparam, a new// opt-inparam, a new setup flag in the case line, a new# optionalimage, a new switch, a stale row, a driftedconf/everything.configand atest_allwithout a tool.cargo test -p og-recheck,-p og-summary,-p og-dev --test repo,-p og --test port_scaffold,cargo fmt --checkandclippy -D warnings: green.run-all-everything, run in a Linux container: it passes on this head. With every data folder stubbed,--toolsis every tool row buthtml_report, no step is skipped,--kir trueand--hla_qc trueare passed, and scripts 04a, 04b, 14, 29, 03a to 03d andbenchmark-variants.shrun. With the Yleaf folder removed it exits 2 before Nextflow and namessetup.sh --yleaf-data. Withfemaleit runs, and steps 37 and 45 areskipped (female sample: no Y chromosome). The same case against main'srun-all.shfails:EVERYTHINGis ignored and the opt-in steps are skipped.-profile test_all,everything -stubcompletes with 112 processes, the 113 of-profile test_all -stubminusCRAM_ARCHIVE.og dev everything-checkprinting theokline above, and the fake-docker caserun-all-everythingwith exit 0): https://github.com/geiresx/OwnGenome/actions/runs/44718662809run-all-everythingpasses on macOS with GNU coreutils. It now stubs the five BWA index files in the all-data set, and a newno-bwacase removes them and expects exit 2 with04b GRIDSS (GRIDSS=true): classic BWA index not installed: ...; install: bwa index (docs/04b-gridss.md)and no Nextflow call. Theno-yleafcase also checks thatlogs/run_status.tsvis byte for byte the last run's and that no.prevfile is left. Both are shown red: with the BWA check turned off the case fails withno-bwa exited 1, expected 2; with the status restore removed it fails witha refused EVERYTHING=true run changed the last run's status. ShellCheck on both scripts: no new finding.run-all.shin a local Linux container, as said above. E2E for this PR runs on the batch tip.Beads: D8