Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Step 22's SV consensus no longer counts a record a caller did not call as a second caller. Before, every PASS record went to SURVIVOR, so a PASS Delly record genotyped `0/0`, a Delly call whose sample filter (`FT`) says `LowQual`, or a CNVpytor segment no quality rule had read sat beside a Manta-only call and made it read "2 callers". A new verb, `og sv survivor-prep --caller NAME IN OUT`, now writes each caller's input for both the script and the Nextflow `SURVIVOR_PREP`, in place of the bcftools and awk they each carried. A record votes when its FILTER and its sample `FT` are `PASS` or `.` and its genotype is not reference only (`0/0`, `0|0`, haploid `0`, `0/.`); a no-call (`./.`) and a sites-only record still vote. A CNVpytor call votes when its e-value is below 0.01 and its share of MAPQ 0 reads below 0.5, the rule step 05 already applies when it adds CNVpytor's calls; the VCF step 22 makes from CNVpytor's table now carries the table's e-value, `q0` and `pN` as `pytorP1`, `pytorQ0` and `pytorPN`, so the rule reads both inputs. A header without `GT` or without these tags no longer matters. Step 18's closing hint and `docs/18-cnvpytor.md` gave e-value below 0.05 and size over 1 kb; they now give the same rule. What changes in the output: `sv_merged/inputs/<caller>.vcf` holds only the records that vote, and the consensus can hold fewer records. No file, column or option is renamed or removed. Part of #411 (sv-05, sv-06, sv-09, sv-10, sv-11, sv-40).
- Step 10 (telomere content) always reads the GRCh38 chromosome bands now. When `reference/cytoBand.hg38.txt` was missing, `scripts/10-telomere-hunter.sh` warned and ran TelomereHunter without `-b`, so it sorted reads by its own hg19 band ends on GRCh38 positions; the Nextflow module did the same whenever `--cytoband` was not set, which is the default. The script now installs the copy the repository ships (`assets/reference/cytoBand.hg38.txt`) before it runs, and the pipeline passes that copy when `--cytoband` is not set. The step also moves from TelomereHunter 1.1.0, Python 2.7 code, to its maintained successor TelomereHunter2 1.0.12 (`TELOMEREHUNTER_IMAGE`, `quay.io/biocontainers/telomerehunter2`). On the image test's input both give the same `tel_content` (5347.255643 with 1,100 planted telomeric reads, 5337.585253 with 1,098), so earlier values stay comparable. THREADS now sets its worker count (`-c`), and the container gets a higher open-file limit, which it needs on a BAM with many contigs. Output change: `telomere/<sample>/<sample>/<sample>_summary.tsv` keeps its name, but its columns follow TelomereHunter2 (`tel_content` third, `intratelomeric_reads`, `tel_read_count`, `read_lengths`, plus three read-class counts). `summary.json` and the reports read both layouts by name and do not change. The docs no longer describe the value as an age measure: `tests/test_telomere_wording.sh` fails when a file that describes step 10 does. Part of #411 (trait-46, trait-47, trait-48, trait-49).
- Step 25 and the Nextflow `PRS_SUMMARY` withhold a score's percentile when the genotypes pgsc_calc scored list variants only. Every site such a VCF leaves out adds nothing to the sum, so the sum leans away from the panel's sums, which come from complete genotypes, and the percentile was not comparable. Each score keeps its raw sum. Its `Percentile` is `NA`, its `Effect` is empty, and a new column before `Input`, `Percentile_Note`, says `percentile withheld: the input lists variants only`. The decision reads the genotypes, not the kind of input: at least one genotyped position in five must be a reference call (0/0), the rule step 26b applies to the panel's sites. Genotypes from a gVCF and chip VCFs from `scripts/chip-to-vcf.sh`, which list their reference calls, keep their percentiles. `og prs table` takes the genotypes with a new option, `--target VCF`, and prints the NOTE that step 25 printed before; the NOTE now follows the genotypes too, so a gVCF without reference blocks at the score positions gets it. `og summary collect` passes the note on as `percentile_note`. When every percentile was withheld, both reports say "Percentile withheld: the input lists variants only" and why, and the summary's `not_assessed` list says the percentiles were withheld, in place of the lines for a run without the panel, which told the user to install a panel the run had used. The output gains one column; no column, option or file name changes.
- Step 41 (blood groups) no longer reads a VCF that does not cover a gene as the reference allele. Without a BAM, a system with no VCF record at any of its sites now reads `not assessed` (phenotype and confidence) instead of the reference phenotype at confidence medium, so a VCF with no chr9 record no longer reads ABO as O. A call with a genotype quality (`GQ`) under 20 or a read depth (`DP`, else `MIN_DP`) under 8 keeps its genotype but gives its system confidence low, and the note and the evidence file name the values. An ABO A allele now reads at confidence low, because no variant shows it: it is what is left of a functional allele with neither c.802G>A nor c.803G>C. Duffy reads the VCF's phase: a GATA box phased with one Fyb in one phase set goes on the chromosome the VCF gives (on Fya that is Fy(a-b+)). When H reads H- and ABO is `not assessed` or `not determined`, the ABO row no longer carries the Bombay note about its alleles. The notes now say what the DNA cannot check: the ABO alleles outside the five variants, RHD null and DEL alleles behind a D+, and null alleles the table does not hold. The table keeps its columns; `not assessed` is a new value in the phenotype and confidence columns.
Expand Down
6 changes: 4 additions & 2 deletions docs/18-cnvpytor.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,11 @@ The `1000` parameter is the bin size in base pairs (must be divisible by 100). U
- `${SAMPLE}_cnvs.vcf.gz` (+ `.tbi`) — normalized VCF with `SVTYPE=DEL/DUP`, `END`, `SVLEN`, `GT`, and `CN`

## Filtering
One quality rule reads CNVpytor's calls everywhere in the pipeline: e-value (column 5, `pytorP1` in the VCF) below 0.01 and `q0` (column 9, `pytorQ0`) below 0.5. Step 5 adds only those calls to AnnotSV's input (`og sv add-callers`), and in step 22 only those vote in the SV consensus (`og sv survivor-prep`).

```bash
# Keep confident CNVs (e-value in col 5 < 0.05, size in col 3 > 1 kb)
awk -F'\t' '$5 < 0.05 && $3 > 1000' ${SAMPLE}_cnvs.txt
# The calls steps 5 and 22 use (e-value in col 5 < 0.01, q0 in col 9 < 0.5)
awk -F'\t' '$5 < 0.01 && $9 < 0.5' ${SAMPLE}_cnvs.txt
```
The first call is often a low-`level` artifact spanning the chromosome-start N gap (`pN` near 1.0); the `pN`/`q0` columns let you drop such regions.

Expand Down
32 changes: 25 additions & 7 deletions docs/22-survivor-merge.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

Keeps the structural variants (SVs) that two or more callers agree on. SURVIVOR merge pairs two calls when both of their breakpoints lie within 1,000 bp of each other, their SV type and strands agree, and the event is at least 50 bp long. Each consensus record says how many callers support it and which. That is caller agreement, not validation (see Why).

A caller votes only with the calls it makes (see [Which records vote](#which-records-vote)): a record the caller genotyped as reference, or filtered, is not a second caller.

## Why

Individual SV callers each have distinct biases and false-positive profiles:
Expand All @@ -14,18 +16,19 @@ Individual SV callers each have distinct biases and false-positive profiles:

An SV that two callers report is less likely to be one caller's artefact. Agreement is not independent proof, though. Manta and Delly both read split and paired reads from the same alignment, so a misaligned read cluster both of them see can reach the consensus. Only CNVpytor reads a different signal, depth, and its breakpoints are coarse (see Limitations).

The step used to bin calls by chromosome, `int(POS/1000)` and SV type. That split one deletion called at positions 999 and 1001 over two bins, and counted two deletions of very different size that start in one bin as agreement. SURVIVOR compares both breakpoints, so neither happens: the e2e case `tests/e2e/sv-mito-telomere-steps-1-sv-merge.sh` runs both situations on the synthetic three-caller set in `tests/fixtures/sv/`.
The step used to bin calls by chromosome, `int(POS/1000)` and SV type. That split one deletion called at positions 999 and 1001 over two bins, and counted two deletions of very different size that start in one bin as agreement. SURVIVOR compares both breakpoints, so neither happens: the e2e case `tests/e2e/sv-mito-telomere-steps-1-sv-merge.sh` runs both situations on the synthetic three-caller set in `tests/fixtures/sv/`. The e2e case `tests/e2e/sv-merge-votes.sh` runs the records that must not vote on the same set.

## Tool

- **SURVIVOR** 1.0.7 (Jeffares et al., Nat Commun 2017), `SURVIVOR merge`
- **bcftools** for the PASS filter before the merge and the sorted, indexed output
- **og** (`og sv survivor-prep`) for the records each caller votes with
- **bcftools** for the sorted, indexed output

Not chosen: `truvari collapse`, which also compares sequence similarity. It would need a comparison run against SURVIVOR on real calls first.

## Docker Image

- `SURVIVOR_IMAGE` (the merge), `BCFTOOLS_IMAGE` (filter, sort, index)
- `SURVIVOR_IMAGE` (the merge), `BCFTOOLS_IMAGE` (runs `og` for the records that vote; sort, index)

Pinned in `versions.env`; [Image versions](versions.md) lists the current tags.

Expand Down Expand Up @@ -53,19 +56,34 @@ The Nextflow pipeline (`survivor_merge` in `--tools`, with at least two of `mant

## What the Script Does Internally

1. Turns CNVpytor's table into a VCF when step 18 left no VCF
2. Writes each caller's PASS (or unfiltered) records as plain VCF under `sv_merged/inputs/`, with one sample column named after the caller. SURVIVOR names its output columns after the input samples, so three inputs that all name the sample would give one name three times
1. Turns CNVpytor's table into a VCF when step 18 left no VCF, or when the VCF there has no `pytorP1` (an earlier version of this step made it from the table without the e-value). Each record carries the table's e-value, `q0` and `pN` under the INFO names CNVpytor's own VCF uses (`pytorP1`, `pytorQ0`, `pytorPN`)
2. Writes the records each caller votes with (below) as plain VCF under `sv_merged/inputs/`, with one sample column named after the caller (`og sv survivor-prep --caller NAME IN OUT`, which prints how many records it left out and why). SURVIVOR names its output columns after the input samples, so three inputs that all name the sample would give one name three times
3. Runs `SURVIVOR merge inputs/sv_files.txt 1000 2 1 1 0 50`: maximum breakpoint distance 1,000 bp, support from 2 callers, same type, same strands, no size-scaled distance, minimum size 50 bp
4. Checks that SURVIVOR wrote a VCF (it exits 0 when it cannot open an input), then sorts, compresses and indexes it
5. Prints the count, and the count per type and caller combination

### Which records vote

SURVIVOR counts a caller as support for every record of its input. A record goes in only when it is a call the caller makes:

| Record | Votes | Why |
|---|---|---|
| FILTER `PASS` or `.`, genotype with an ALT allele (`0/1`, `1/1`, `1`) | yes | a call |
| genotype `./.`, or no `GT`, or a sites-only VCF | yes | nothing says the call is absent; the VCF made from CNVpytor's table writes `./.` for every call |
| FILTER other than `PASS` or `.` (Delly's `LowQual`) | no | the caller filtered it |
| sample filter `FT` other than `PASS` or `.` (Delly's per-sample `LowQual`) | no | the caller filtered it for this sample |
| genotype with a called allele and no ALT (`0/0` phased or not, haploid `0`, `0/.`) | no | the caller tested the site and found the reference |
| a CNVpytor call with e-value (`pytorP1`) 0.01 or more, `q0` (`pytorQ0`) 0.5 or more, or either value missing | no | CNVpytor's quality rule, the one step 5 applies when it adds CNVpytor's calls ([step 18](18-cnvpytor.md#filtering)) |

Delly genotypes every site it reports, so a PASS Delly record with `0/0` is common. Before this rule, such a record next to a Manta call read as two callers.

## Output

| File | Contents |
|---|---|
| `${SAMPLE}_sv_consensus.vcf.gz` | Consensus SVs supported by 2+ callers, with `SUPP` and `SUPP_VEC` in INFO |
| `${SAMPLE}_sv_consensus.vcf.gz.tbi` | Tabix index |
| `inputs/sv_files.txt`, `inputs/<caller>.vcf` | The PASS records SURVIVOR read, in `SUPP_VEC` order |
| `inputs/sv_files.txt`, `inputs/<caller>.vcf` | The records each caller voted with, which SURVIVOR read, in `SUPP_VEC` order |

All output is written to `${GENOME_DIR}/${SAMPLE}/sv_merged/`.

Expand Down Expand Up @@ -101,7 +119,7 @@ docker run --rm -v "${GENOME_DIR}:/genome" "${BCFTOOLS_IMAGE}" \

- Single-caller SVs are discarded even if they are real. If you suspect a specific SV, check the individual caller outputs directly.
- CNVpytor's calls are depth-only and their breakpoints are coarse (its bin size), so a CNVpytor call often lies more than 1 kb from the Manta or Delly breakpoint of the same event and does not pair with it.
- The CNVpytor VCF made from its table carries no genotype and no quality.
- The CNVpytor VCF made from its table carries no genotype (`./.`); its e-value and `q0` come from the table.
- A breakend (BND) record pairs with a breakend, an inversion (INV) or a translocation (TRA) of another caller: SURVIVOR 1.0.7 lets a BND match those two types. That includes a lone Manta inversion junction of 1 Mb or more, which step 4 keeps as its two breakends instead of a whole-span inversion.

## Notes
Expand Down
18 changes: 8 additions & 10 deletions modules/local/survivor_merge/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SURVIVOR_MERGE — SV consensus: the calls two or more callers agree on
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SURVIVOR_PREP each caller's PASS (or unfiltered) records as plain VCF,
one sample column named after the caller (bcftools)
SURVIVOR_PREP the calls each caller votes with, as plain VCF, one sample
column named after the caller (og sv survivor-prep): no
record whose FILTER or sample FT is not PASS, none whose
genotype is reference only (0/0, 0|0, 0, 0/.), and for
CNVpytor none that fails step 05's e-value and q0 rule
SURVIVOR_MERGE `SURVIVOR merge LIST 1000 2 1 1 0 50`: both breakpoints
within 1,000 bp, same type and strands, >= 50 bp, kept
when 2+ callers support it; SUPP and SUPP_VEC say which
Expand All @@ -30,15 +33,10 @@ process SURVIVOR_PREP {
script:
// SURVIVOR names its output columns after the input samples: three
// columns all named after the sample would be one name three times.
// og (bin/) reads the gzip input itself; scripts/22-survivor-merge.sh
// runs the same verb.
"""
bcftools view -f PASS,. ${vcf} | awk -v s=${caller} 'BEGIN { FS = OFS = "\\t" }
/^##/ { print; next }
/^#CHROM/ {
if (NF < 10) { print "##FORMAT=<ID=GT,Number=1,Type=String,Description=\\"Genotype\\">"; sites = 1; print \$1, \$2, \$3, \$4, \$5, \$6, \$7, \$8, "FORMAT", s }
else print \$1, \$2, \$3, \$4, \$5, \$6, \$7, \$8, \$9, s
next
}
{ if (sites) print \$1, \$2, \$3, \$4, \$5, \$6, \$7, \$8, "GT", "./."; else print \$1, \$2, \$3, \$4, \$5, \$6, \$7, \$8, \$9, \$10 }' > ${caller}.vcf
og sv survivor-prep --caller ${caller} ${vcf} ${caller}.vcf

cat <<-END_VERSIONS > versions.yml
"${task.process}":
Expand Down
12 changes: 10 additions & 2 deletions rust/crates/og-steps/src/sv/add_callers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,14 @@ const STANDARD_INFO: [(&str, &str); 7] = [
),
];

/// CNVpytor's quality rule, the defaults of `--max-eval` and `--max-q0`: a
/// call counts when its e-value (`pytorP1`) is below [`MAX_EVAL`] and its
/// share of MAPQ 0 reads (`pytorQ0`) below [`MAX_Q0`]. Step 22's
/// `og sv survivor-prep --caller cnvpytor` applies the same two numbers.
pub const MAX_EVAL: f64 = 0.01;
/// See [`MAX_EVAL`].
pub const MAX_Q0: f64 = 0.5;

/// The INFO keys an added record keeps, in this order.
const KEPT: [&str; 4] = ["END", "SVTYPE", "CIPOS", "CIEND"];

Expand Down Expand Up @@ -530,8 +538,8 @@ fn opts(argv: &[String]) -> Result<Opts, u8> {
sex,
par_bed: given("--par-bed"),
cnvpytor: given("--cnvpytor"),
max_eval: float("--max-eval", 0.01)?,
max_q0: float("--max-q0", 0.5)?,
max_eval: float("--max-eval", MAX_EVAL)?,
max_q0: float("--max-q0", MAX_Q0)?,
min_overlap: float("--min-overlap", 0.5)?,
})
}
Expand Down
Loading
Loading