Skip to content

fix(telomere): read the GRCh38 bands we ship instead of falling back to hg19 - #276

Closed
geiresx wants to merge 138 commits into
mainfrom
pgp/i411-telomere-bands
Closed

geiresx wants to merge 138 commits into
mainfrom
pgp/i411-telomere-bands

Conversation

@geiresx

@geiresx geiresx commented Nov 20, 2026

Copy link
Copy Markdown
Owner

…ants only

og prs table takes the genotypes pgsc_calc scored (--target) and counts
their called and reference calls. When under one in five is 0/0, every
missing site was read as the reference, so each score keeps its raw sum,
loses its percentile and effect, and the new Percentile_Note column says
why. The rule reads the genotypes, never --input-kind: chip VCFs list
their reference calls and keep their percentiles.

Bead: lin-42
Step 25 passes target.vcf.gz and PRS_SUMMARY gets PRS_SCORE_SITES'
output. The NOTE on missing hom-ref sites now comes from og prs table and
follows the genotypes instead of the input kind.

Bead: lin-42
og summary collect writes percentile_note for a table that has the
Percentile_Note column. The c-full fixture gains the column with one
withheld row; the goldens of the cases built on it change on purpose.

Bead: lin-42
…en no percentile is left

pgsc_calc drops a score position the genotypes leave out, so it adds
nothing to the sum; it does not read it as the reference. The NOTE,
the doc comments, docs/25-prs.md and the CHANGELOG now say so.

Step 25's closing banner said the percentile compares the score with
the reference samples even when every percentile was withheld. It now
says that only when a row has a numeric Percentile.

docs/25-prs.md now says the reports still print the Raw score only
line when every percentile was withheld, and where the reason is.
…alled

When the panel ran but every percentile was withheld for a variant-only
input, adjusted is false, and both reports said no ancestry reference panel
was installed and told the user to add one. The not-assessed list said no
panel was used. Both were false.

Both reports (bin/render_report.py and og report render) now print a
"Percentile withheld: the input lists variants only" line with the reason
when a score carries a percentile_note, and og summary collect writes a
not-assessed line that says the percentiles were withheld. A run without the
panel keeps its lines. The report-render golden a-prs-raw records the new
line: that case's T2D score carries the note.
The a-prs-raw summary is c-full's with adjusted false, so it took c-full's
withheld note on the T2D score, and og now renders the withheld line there.
The report-render oracle's commit predates that line, so the case needed a
golden: a whole report, whose LPA lines the personal-data scan reads as
genotypes. Clearing the note keeps the case what it was meant to be, a run
without the panel, which og and the oracle render alike. The cargo test
prs_withheld.rs and tests/test_render_report.py cover the withheld line.
… a gene was read

The gene callability pass of step 16b and the MOSDEPTH module ran mosdepth
with --fast-mode. That mode counts every base from a read's start to its
end: the overlap of two mates counts twice, and a CIGAR deletion counts as
read. A gene that fewer than 10 fragments covered could reach 10x and read
as read. The gene pass now runs without --fast-mode; the genome-wide 500 bp
pass keeps it. The step doc and the lessons-learned entry no longer say
that --fast-mode drops the per-base output (--no-per-base does).

The new e2e case builds a small BAM: 5 overlapping pairs over one gene
(0 bases at 10x, 10 on main), 10 pairs over another (10) and a deletion
over a third (0, 10 on main), on both paths, plus a --fast-mode control.

Bead: plat-50
rust/ was outside the e2e scope (but for two og-dev files), so a pull
request that changed only Rust code ran no e2e case, and its merge
started no run on main. A change under rust/ now counts as a change to
bin/og: it picks every case whose paths row has a glob that matches
bin/og. The push paths of e2e.yml gain rust/**. 19 rows of cases that
run og through a step, helper or module now list bin/og.

Bead: plat-09
The e2e job ran under always(), so when a run was cancelled (a newer
push, or by hand) it ran anyway, found the legs cancelled and failed.
It now runs under !cancelled(): in a cancelled run it is cancelled with
the run, as the other legs are. A cancelled check still blocks a merge.
A leg cut by its time limit in a run that was not cancelled still turns
e2e red, and the message says so.

Bead: plat-05
Bead: plat-06
container-ok passes on a pull request that changes no image, with no
smoke row run. The testing guide now says so, and why the full-data
AnnotSV row stays monthly and on dispatch.

Bead: plat-08
…k genome-wide sections partial

The test fails on main: summary.json has no scope, and no genome-wide
section says it covers part of the genome.

Bead: D14
…genome-wide results partial

og summary collect reads the chromosomes of the sample VCF's records and
writes scope {autosomes_covered, autosomes, chrx, intervals, partial,
mark}. When fewer than 22 autosomes have calls, or the run manifest
records intervals, callability, prs, ancestry_fine, telomere, roh and
archaic get a partial field with the mark. Without a readable sample VCF
no scope is written. The whole-file parity goldens all and all-history
change on purpose: their fixture VCF has calls on chr22 only.

Bead: D14
…c HG002 30x

The gene callability pass took 3 min 26 s and 3 min 50 s on HG002 30x
without --fast-mode (1 min 46 s and 2 min 29 s with it), about 37% more
CPU time. Four of 1,553 catalog genes moved from read to not read.

Bead: plat-50
… file type

The withheld line said a gVCF or a chip VCF keeps the percentile. A target
of either kind with too few 0/0 calls still fails the one-in-five rule, so
both reports and the docs now say the reference calls keep it, and that
those files usually have them.
# Conflicts:
#	CHANGELOG.md
#	scripts/ci/e2e-case-paths.tsv
The case runs the pipeline from docs/vcf-first.md and reads PharmCAT's
report with og dev pharmcat diplotypes, so its row lists bin/og. The
real-table check in tests/test_e2e_scope.sh now names it.
…eline)

og dev contract --static compares the code with the contract in the same
checkout only, so a change that renames a stable output and its row
together, removes a stable param with its row, or demotes a stable row
passes. --baseline FILE now checks every stable row of a release's
contract against this one: still listed, still stable, same format,
every leg and every required field kept. New rows and changes to
experimental and internal rows pass.

Bead: plat-03
26 stable file rows had e2e '-' although the base cases write them on
every run (the clinical summary, the slivar tables, the mito tables,
pypgx, the ClinVar cDNA and deep-intronic tables, the gVCF, the read
count, the charts and search index, the PharmCAT JSON and HTML, the
coverage beds, the sex check, indexcov), so the HG002 contract case
never checked them. Their e2e cells now name the legs the base run
writes, from the e2e-base artifact of main run 44444355103 and the
observed lists of three main runs. The annotated VCF index is checked on
both legs now, not only Nextflow. run_manifest.tsv is written by step 24
too, so its legs are both and the bash leg checks it. The VEP output of
step 13 stays '-': the fixture provides it, the step does not run.

Bead: plat-04
Guard's rust job now reads the newest v1.x.y tag of the upstream
repository and runs og dev contract --baseline against that tag's
contract. Before the first 1.x tag it says there is nothing to compare
with; after it, a tag that cannot be listed, fetched or read fails the
step. Its control plants a stable row that is gone since the baseline,
which must fail.

Bead: plat-03
The comment said owner review is required. main requires a pull request
with its 14 status checks and no approving review (the decision is to
keep checks only), so the comment now says the file routes review
requests. The output contract joins the routed paths.

Bead: plat-07
Bead: D6
docs/output-contract.md names the third check, the comparison with the
newest 1.x release, says what it refuses and what it lets through, and
adds the breaks it now catches (a stable row made experimental or
internal, another format, a dropped leg). The e2e paragraph says which
rows stay '-' and why, and that the bash leg checks run_manifest.tsv.
The contract's header and the changelog say the same.

Bead: plat-03
Bead: plat-04
Red first: an LC-flagged SVA call in the FKTN window reads seen today.
Zero depth without a call must read not callable, never not seen.

Bead: sv-33
Bead: sv-34
…E's filters

A catalog site is seen only when a call of its element in the window has
FILTER PASS. When only flagged calls are there, the site reads
"call filtered out (FLAGS)" and names the first one. A PASS call wins
over a flagged one in the same window.

Bead: sv-33
GeiserX#522 (i411-sample-qc-honest) now calls below() directly in
tests/e2e/qc-1-sample-qc.sh, so ShellCheck no longer reports SC2329 there.
The baseline line came with main's ShellCheck scope change, which GeiserX#522's
branch predates, so the merged tree held a baseline line with no finding.
Lint prints a notice for it; a stale line could also hide a new finding
with the same text.
A homozygous hypomorphic allele inside a deletion is one copy, so the two-hit
verdict for RBM8A is now 'possibly biallelic'. The og-render clinical test
still asserted the old 'biallelic' and was the one red test of Guard rust on
this branch.

Bead: clin-32
…ll-floor

# Conflicts:
#	CHANGELOG.md
#	tests/e2e/cn-engine-1-parity.sh
Step 10 warned and ran TelomereHunter without -b when the band file was
missing, and the Nextflow module did so whenever --cytoband was unset
(the default), so reads were sorted by hg19 band ends on GRCh38
positions. The script now installs assets/reference/cytoBand.hg38.txt
first; the workflow passes that copy when --cytoband is unset.

Bead: trait-46
Bead: trait-47
TelomereHunter 1.1.0 is Python 2.7 code. TelomereHunter2 1.0.12 gives
the same tel_content on the planted test BAM (5347.255643 with 1,100
reads, 5337.585253 with 1,098, both versions). Its image has no
samtools, so the image test builds telo.bam with the samtools image
(new input telobam). It needs more than 1,024 open files on a BAM with
many contigs; the script and module raise the limit.

Bead: trait-49
The step script and module called the value a biological age
baseline. Reworded, the step doc covers TelomereHunter2 and the bands,
and tests/test_telomere_wording.sh fails when a file that describes
step 10 uses the phrase (with a planted control).

Bead: trait-48
Bead: trait-49
…off an unread ABO row

Also correct the gVCF block wording (DP, else MIN_DP) and the platelet
control label.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…dings card

When step 23 is not ok in this run and its clinical VCF predates the run,
sections.acmg_sf still reads that VCF. The card now says so in
high_impact_note, with the VCF's date and step 23's result, as the clinical
section goes stale.
… FCGR

Since GeiserX#519, og locityper report writes filter "uncalibrated" outside the
HLA and KIR loci, but the demo sample still wrote "pass" for every locus,
so the docs pictures and the live demo showed CYP2D6 and FCGR passing a
filter the real pipeline no longer applies there.

The demo now writes a made-up res.json.gz per locus (the same seeded
values as before) and runs og locityper report on them, as its module
comment promises, so the table cannot drift from the verb again. The
Python remnant writer gets the same filter and reason. A new test in
port_demo_sample.rs holds the demo table to og-reports' calibrated() and
UNCALIBRATED; it failed on the FCGR row before the fix.
…ll requests

The drug-interactions row reads the FDA table from the Wayback Machine,
because fda.gov refuses GitHub's runners. It turned every error of that second
read into a plain failure, so a day when archive.org does not answer would
have failed the Freshness check of every pull request that touches the
pinned versions. The row now keeps 'did not answer' as it is, and the dry
run warns. A new test asks with a fetcher that never answers; it failed
before this change.

docs/53-locityper.md named Locityper 1.7.14 in one paragraph and 1.7.15 in
another after two changes met. The step runs 1.7.15.
…nger's VCF

The base leg runs no step 09b, so HG002_eh_stranger.vcf exists only when
bash-step-09-str ran earlier in the same shard. The contract case passed on
main by shard placement; adding one case moved it to another shard and it
failed. The needs column now keeps both in one shard, in order.
# Conflicts:
#	CHANGELOG.md
#	tests/parity/golden/summary-collect/all-history/all-history.json
#	tests/parity/golden/summary-collect/all/all.json
…d agreement test

GeiserX#531 (i411-sf-haplotype-count) gives each reported row of a recessive gene
the gene's own verdict, `pair` and `phase`. GeiserX#533 (screen-acmg-card-section)
checks that every ClinVar row of the card without VEP is in the card with
VEP, unchanged. Together, cb-full fails: with step 23's clinical VCF a
HIGH-impact MUTYH row joins, and the gene's phase sentence names that pair
instead of the deletion. The row's own fields are the same. Rule (a) now
compares the row without `pair` and `phase` only when a HIGH-impact row of
its gene joined it; any other change still fails.
…iserX#531 adds

GeiserX#533 (screen-acmg-card-section) recorded the new sections.acmg_sf card in
the all, all-nf and all-history goldens before GeiserX#531 (i411-sf-haplotype-count)
gave each row of the card its gene's `pair` and `phase` and kept the
`combined_allele` note. The merged og writes the card with those fields,
the same card as sections.clinical.data.acmg_sf, so oracle-diff reported
3 items that differ. The goldens are og's output for the merged tree;
only those card rows and the masked generated_utc change.
… too

c9961b9 let a HIGH-impact row of a gene change that gene's pair and phase
in the og-summary agreement test. The e2e case acmg-sf-rules-1-reports
checks the same rule on a real run and still demanded strict equality, so
the two copies of the rule disagreed. The jq check now ignores pair and
phase only when a HIGH-impact row of that gene is in the full run's card;
any other change still fails. Checked with jq on hand-made cards: a pair
change with a HIGH-impact row passes, and a pair change without one, a
changed field and a lost row all fail. The old check failed the first.
@geiresx geiresx closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants