Skip to content

fix(viewer): serve the local genome viewer from og and refuse a reversed locus - #281

Open
geiresx wants to merge 62 commits into
mainfrom
pgp/port-viewer
Open

geiresx wants to merge 62 commits into
mainfrom
pgp/port-viewer

Conversation

@geiresx

@geiresx geiresx commented Nov 20, 2026

Copy link
Copy Markdown
Owner

…es of its records

Two variants phased on one copy, a combined allele counted with its own
parts, or a record with no alt copy called could complete the two hits a
recessive ACMG SF gene needs. The tier now counts haplotypes: a phased
heterozygote needs a "|" GT, a non-empty PS and every copy called; a half
call or a "|" GT without PS is phase unknown. Each finding of a two-hit
gene gets pair (biallelic or possibly biallelic) and phase fields, and a
ClinVar finding that is a combined allele keeps its combined_allele note.

Bead: clin-01
Bead: clin-02
Bead: clin-03
Bead: clin-04
Bead: clin-06
Bead: clin-29
…bly biallelic

Nothing says whether the deletion is real. A full loss of function called
homozygous hits both copies either way, so it stays biallelic with a
reason that says so; a hypomorph is biallelic only with a real deletion.

Bead: clin-32
… record the goldens

The summary-collect goldens change only by the new pair, phase and
combined_allele fields and the two-hit reason wording.

Bead: clin-01
Bead: D14
… phase

A present genotype was used whatever its quality. The reader now marks a
call with GQ under 20 or DP (else MIN_DP) under 8 as weak, keeping its
genotype, and keeps the haplotype of the marker's ALT and the phase set
of a phased call, for the rules to use.

Bead: trait-31
Bead: D10
…ssing chr9 as type O

A weak call gives its system confidence low and is named in the note.
Without a BAM, a system with no VCF record at any of its sites reads not
assessed instead of the reference phenotype at confidence medium. Duffy
places a GATA box phased with one Fyb on the chromosome the VCF gives. An
ABO A allele reads low, since no variant shows it. The ABO, RhD and
null-row notes say what the DNA cannot check. The bloodtype parity
goldens record these changes on purpose.

Bead: trait-31
Bead: trait-28
Bead: trait-32
Bead: trait-33
Bead: trait-35
Bead: trait-36
Bead: trait-37
Bead: D10
… and fix the position count

The step page, the script and module headers, the VCF-only page and the
e2e case now match the new rules. The crate doc said the VCF is read at
318 positions; the marker table has 271.

Bead: trait-39
Bead: trait-33
…s a reference call

A reference block that starts before a marker and covers it through
INFO/END was ignored, so without a BAM a gVCF could read a system as not
assessed although it covers it. Such a block is now a reference call at
the marker, with its GQ and MIN_DP. ABO's c.796C>A call now counts in
the ABO confidence and notes, and the VCF-only page says the blood
groups are opt-in.

Bead: trait-28
Bead: D10
Red first: summary.json has no ACMG card when clinical/ is missing, and
the agreement test compares every group b parity folder with and without
its clinical VCF. summary-clinical.sh gets NO_ORACLE=1 so the test can
write the folders without the oracle's helper modules.

Bead: D11
Bead: D14
… step 06

sections.acmg_sf follows step 06: ok when the ClinVar hits VCF is there,
not run or failed with step 06's reason otherwise. The verdicts come from
the same acmg::acmg_sf the clinical section calls. Without step 23's
clinical VCF the card says high_impact_checked false and high_impact_note
names why and what it can miss; an unreadable clinical VCF costs only the
HIGH-impact source. The clinical section keeps its nested card for the
readers that still use it. The full-summary parity goldens gain the
section.

Bead: D11
Bead: D14
… in e2e

acmg-sf-rules-1-reports gains a cell without clinical/: the card is
there with the 84 rules, says the HIGH-impact source was not screened and
keeps every ClinVar row of the full run's card. nf-wiring-1-report-tables
compares the card of the bash and the Nextflow summaries.

Bead: D11
Bead: D14
…ssed systems and new RhD notes

Without a BAM, the platelet case's control VCF had no record at the
HPA-1, HNA-3 and HNA-4 sites, which now read not assessed; it now plants
0/0 there. The copy-number case compared step 41's whole RhD row with
the tree before the engine, notes included; it now compares the
phenotype, genotype and confidence, the engine's result.

Bead: trait-28
Bead: trait-35
A homozygous hypomorphic allele inside a deletion is one copy, so the two-hit
verdict for RBM8A is now 'possibly biallelic'. The og-render clinical test
still asserted the old 'biallelic' and was the one red test of Guard rust on
this branch.

Bead: clin-32
…ll-floor

# Conflicts:
#	CHANGELOG.md
#	tests/e2e/cn-engine-1-parity.sh
Step 10 warned and ran TelomereHunter without -b when the band file was
missing, and the Nextflow module did so whenever --cytoband was unset
(the default), so reads were sorted by hg19 band ends on GRCh38
positions. The script now installs assets/reference/cytoBand.hg38.txt
first; the workflow passes that copy when --cytoband is unset.

Bead: trait-46
Bead: trait-47
TelomereHunter 1.1.0 is Python 2.7 code. TelomereHunter2 1.0.12 gives
the same tel_content on the planted test BAM (5347.255643 with 1,100
reads, 5337.585253 with 1,098, both versions). Its image has no
samtools, so the image test builds telo.bam with the samtools image
(new input telobam). It needs more than 1,024 open files on a BAM with
many contigs; the script and module raise the limit.

Bead: trait-49
The step script and module called the value a biological age
baseline. Reworded, the step doc covers TelomereHunter2 and the bands,
and tests/test_telomere_wording.sh fails when a file that describes
step 10 uses the phrase (with a planted control).

Bead: trait-48
Bead: trait-49
…off an unread ABO row

Also correct the gVCF block wording (DP, else MIN_DP) and the platelet
control label.
…dings card

When step 23 is not ok in this run and its clinical VCF predates the run,
sections.acmg_sf still reads that VCF. The card now says so in
high_impact_note, with the VCF's date and step 23's result, as the clinical
section goes stale.
# Conflicts:
#	CHANGELOG.md
#	tests/parity/golden/summary-collect/all-history/all-history.json
#	tests/parity/golden/summary-collect/all/all.json
…d agreement test

GeiserX#531 (i411-sf-haplotype-count) gives each reported row of a recessive gene
the gene's own verdict, `pair` and `phase`. GeiserX#533 (screen-acmg-card-section)
checks that every ClinVar row of the card without VEP is in the card with
VEP, unchanged. Together, cb-full fails: with step 23's clinical VCF a
HIGH-impact MUTYH row joins, and the gene's phase sentence names that pair
instead of the deletion. The row's own fields are the same. Rule (a) now
compares the row without `pair` and `phase` only when a HIGH-impact row of
its gene joined it; any other change still fails.
…st show in the annotation

The m.315 C>CC call is the m.310 T>TC insertion ClinVar and gnomAD list,
and m.308 CC>C is m.302 AC>A; today neither meets its record. A run whose
NuMT filter had no autosomal depth must say so in the table. Both tests
fail on main.

Bead: lin-35
Bead: lin-32
… a NuMT filter without depth

og mito annotate matched an indel only on its exact trimmed spelling, and
looked for ClinVar's record at most one base away, so m.315 C>CC never met
the m.310 T>TC record of the same poly-C insertion. When the exact spelling
finds nothing, both sides are now moved to the left end of their repeat on
the rCRS (embedded, GRCh38's chrM, MD5 c68f5267...) and matched there. An
exact match still wins, so every table that matched before is unchanged;
an allele whose REF is not the rCRS is not moved. ClinVar's chrM records
are now read whole, since a record can sit anywhere in its repeat.

--numt-default adds a '# numt' header line: the NuMT filter used a default
depth (0) and marked nothing. Without it the table is as before.

Bead: lin-35
Bead: lin-32
…mal depth

Step 20 already warned; it now passes --numt-default to og mito annotate
when neither mosdepth output nor AUTOSOMAL_COVERAGE gives a depth. The
Nextflow MITO_VARIANTS prints its message as a WARNING on stderr, and
MITO_VARIANTS_ANNOTATE passes --numt-default when mosdepth is not in
--tools, the one case where MITO_VARIANTS has no depth files. The
fake-docker case checks both paths and fails on main.

Bead: lin-32
og report charts --summary IN --charts-json OUT writes what
bin/render_report.py --charts-json writes, byte for byte, and prints the
same Sections line, so both run with the same arguments. The specs are
built once (charts::build) for the HTML report (charts::figure) and the
live page (charts::json).

Bead: D1
Every non-browser check of tests/test_report_charts.py now runs on og
report charts in rust/crates/og-render/tests/charts.rs, each with its red
control, plus frozen hashes of the oracle's charts JSON on the fixture and
five changed copies. The Python file keeps only the live page's browser
check, which needs the Python live server.

Bead: D1
… script 04a

EVERYTHING=true reads assets/everything.tsv: every tool row goes into TOOLS
(html_report aside: run-all.sh writes the report itself), every switch row
is set to its value, and TOOLS or a switch set to another value stops the run
with exit 2. A step skipped because its data is not installed, or because a
panel is set to none, stops the run with exit 2 before Nextflow and names the
setup command. Skips that come from the sample stay skips: no BAM, no gVCF,
a female sample for steps 37 and 45, a BAM that is not PacBio HiFi for 35b.

Step 37 on a female sample is now listed as skipped, as step 45 was; the
pipeline skipped it already. Step 35b runs og paraphase check-bam on the BAM
first, so a BAM that is not PL:PACBIO, or a FASTQ row, is a skip instead of a
failed step.

TIDDIT=true runs 04a-tiddit.sh after the pipeline, so every optional step is
reachable from one command. EVERYTHING and TIDDIT join SWITCHES, the replay
list of og recheck, the output contract, and step 04a joins assets/steps.tsv
and RUN_ALL_STEPS, which the tests keep equal to what run-all.sh records.

Bead: D8
assets/everything.tsv lists every optional item: each --tools name of
main.nf, each run-all.sh switch, each setup.sh flag, each false-by-default or
opt-in parameter of nextflow.config and each optional image of versions.env,
as tool, switch, setup, input, image or excluded with the reason.
og dev everything-check fails when one of them has no row, when a row names
an item that is gone, when conf/everything.config sets other tools or
parameters than the rows, or when conf/test_all.config misses a tool. Guard
runs it; its planted cases are og-dev unit tests.

-profile everything (conf/everything.config) gives the Nextflow path the same
tools and switches as EVERYTHING=true.

Bead: D8
Every data folder stubbed: every tool row runs and every script step runs
after the pipeline. Without Yleaf's tables it exits 2 before Nextflow and
names setup.sh --yleaf-data. A female sample still runs with the Y steps
skipped; a BAM that is not PacBio HiFi skips Paraphase. TOOLS, KIR=false or
ARCHAIC_DIR=none with EVERYTHING=true are refused.

Bead: D8
og view serve does what assets/viewer/serve.py did: it builds the igv.js
config from the sample's files, checks the locus against the reference,
and serves the page, igv.js, the config and byte ranges of the files on
127.0.0.1 only, with the viewer's Content-Security-Policy.

Two refusals are new. A locus whose end comes before its start is refused
at start (serve.py accepted chr1:8-2). A request with a Forwarded,
X-Forwarded-* or X-Real-IP header gets 403: the viewer has no password,
so a reverse proxy in front of it would hand the sample to anyone who
reaches the proxy. An ssh port forward adds no such header.

sample_files picks Manta's diploid SVs before step 22's consensus, for
the live server to reuse. The cargo tests carry the server checks of
tests/test_viewer.py and the page check in headless Chrome.

Bead: plat-11
Bead: D12
Bead: D1
scripts/owngenome-view.sh now runs og view serve (bin/og, or OG) and no
longer needs python3 on the host. Its default SV track is Manta's diploid
SVs, else step 22's consensus: the consensus keeps only calls two callers
agree on, and the reads are where a single-caller call gets checked.

tests/test_owngenome_view.sh checks the launcher with a python3 that fails
first on PATH: the pinned igv.js and its sha256, the printed address,
Manta's SV VCF first, 403 for a foreign Host and a forwarded request, and
exit 2 for a reversed locus. Against the old launcher it fails (python3
was called).

Bead: plat-11
Bead: D12
Bead: D1
…nd needs no python3

docs/viewer.md now names og view serve, drops the python3 requirement,
says every account on the machine can reach 127.0.0.1, documents the
reversed-locus and forwarded-request refusals and Manta's SVs first, and
lists the tests that check it. docs/live-site.md says the live viewer
still takes step 22's consensus first until its server moves to og.

assets/viewer/serve.py keeps only what bin/owngenome_live_viewer.py
imports (sample_files, build_config, send_file, LIVE_CSP and what they
call); its server is gone. tests/test_viewer.py keeps its config and
report-link checks and the helpers tests/test_live_viewer.py imports; the
server, launcher and browser checks moved to Rust and shell. Both are
deletions plus their docstrings: no Python line was added.

Bead: plat-11
Bead: D12
Bead: D1
…iserX#531 adds

GeiserX#533 (screen-acmg-card-section) recorded the new sections.acmg_sf card in
the all, all-nf and all-history goldens before GeiserX#531 (i411-sf-haplotype-count)
gave each row of the card its gene's `pair` and `phase` and kept the
`combined_allele` note. The merged og writes the card with those fields,
the same card as sections.clinical.data.acmg_sf, so oracle-diff reported
3 items that differ. The goldens are og's output for the merged tree;
only those card rows and the masked generated_utc change.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

… too

c9961b9 let a HIGH-impact row of a gene change that gene's pair and phase
in the og-summary agreement test. The e2e case acmg-sf-rules-1-reports
checks the same rule on a real run and still demanded strict equality, so
the two copies of the rule disagreed. The jq check now ignores pair and
phase only when a HIGH-impact row of that gene is in the full run's card;
any other change still fails. Checked with jq on hand-made cards: a pair
change with a HIGH-impact row passes, and a pair change without one, a
changed field and a lost row all fail. The old check failed the first.
…epth

A successful task's stderr stays in its work directory, so the
MITO_VARIANTS WARNING never reached a Nextflow user. The workflow now
says it with log.warn when mito_variants runs without mosdepth.
…and open Manta's SVs first on the live site too

og's main holds stderr's lock while og view serve runs, so a handler thread
that logged with eprintln! blocked forever before it closed its connection:
no request was logged, every socket stayed in CLOSE_WAIT, and the viewer
stopped answering once it ran out of file descriptors. The handlers now send
their lines over a channel and the thread that owns stderr writes them.
tests/test_owngenome_view.sh checks that a request is logged and that the
server closes the connection; both fail on the previous build.

The live site's viewer picks its SV track with serve.sample_files, so the
order swap there makes both viewers open Manta's diploid SVs first.
… figures

Old checks 7 and 8 test what bin/render_report.py still ships: each chart once in its section below the caveat, every anchor resolving, and two renders giving the same charts file. og report render does not embed the figures yet, so they stay in Python until html.rs does. The charts.rs header and the changelog now say which checks moved.
left_allele trimmed first, so m.314 CG>CCG passed as C>CC on the rCRS
and met m.310's record. The docs now say the table writes the trimmed
spelling, not the moved one.
…index

GRIDSS (script 04b, after the pipeline) needs the classic BWA index, which
setup.sh does not build; an EVERYTHING=true run now refuses up front instead
of failing hours in. A refused run puts the last run's run_status.tsv back
and posts no run-finished event. pipeline-overview.md lists TIDDIT=true.
…r attributes were set in

The live page sets the <html> data-pgp-* attributes as its files arrive, so
two renders can carry the same attributes in another order. CI failed once
on T7's and once on GeiserX#536's Guard on 'child with no act, a search typed' and
passed on a rerun. Each element of the clone now gets its attributes back
sorted by name before outerHTML: an order swap passes, while a changed value,
an added or a missing attribute still differ (checked in Chromium on a planted
order swap, value change, added and missing attribute).
…rpus summaries

GeiserX#537 ported the charts but left its corpus rows out of the parity list, so CI
held the charts only on the six frozen summaries. One report-charts case per
report-render summary, compared as JSON, and the oracle row at 65095e9.
scripts/ci/oracle-diff.sh report-charts: all 616 items in 154 cases agree;
a planted key rename in og's charts file makes both cases it touches differ.
It writes the chart specs as render_report.py --charts-json does (GeiserX#537); no
step calls it yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants