Prediction of genomic measurements from DNA sequence in observed and unobserved cell types and assays
DNACipher is a DNA sequence deep learning model that also includes cell type and assay information on the model input.
DNACipher, a deep learning framework that integrates long-range sequence modeling with biological context imputation, enabling variant effect prediction across 38,000+ experimental contexts, a 6-fold improvement over previous state-of-the-art models such as AlphaGenome.
Deep Variant Impact Mapping (DVIM) is an analysis framework built on DNACipher that calls significant variant effects at GWAS loci, to call common and rare 'impact' variants. These impact variants have the following properties:
* They occur within the GWAS hit locus.
* Have significant predicted molecular effects in >=1 biological contexts above non-trait associated common variants at a similar genome location.
* They can be common or rare variants - enabling the study of rare variants previously inaccessible to classical statistical methods.
DNACipher is made available here as both a Python interface and a command-line-interface, with tutorials below reproducing key results from the DNACipher manuscript from an inference perspective.
Because training the model requires additional data, software dependencies, and complexity, we only support training with the commandline interface.
The following tutorials are written in google collab, showing how to use the DNACipher Python interface.
- Tutorial 1: DNACipher inference of genetic variant effects https://colab.research.google.com/github/BradBalderson/DNACipher/blob/main/tutorials/DNACipher_inference_collab.ipynb
- Tutorial 2: Deep Variant Impact Mapping (DVIM) with DNACipher to infer common and rare genetic variants with significant effects at GWAS loci: https://colab.research.google.com/github/BradBalderson/DNACipher/blob/main/tutorials/DeepVariantImpactMapping.ipynb
DNACipher and DVIM analysis using a command-line interface, so that R users and non-Python programmers can utilize the model and analysis.
Because of the complexity of this step, the tutorial is kept separately. The following steps do not require training; the model weights from pre-trained DNACipher (DNACipher-Enformer by default, DNACipher-Borzoi also available) can be used for variant effect prediction and DVIM analysis.
See https://github.com/BradBalderson/DNACipher/blob/main/TRAIN.md
Please replace 'mamba' with 'conda' if not installed, mamba much faster however (recommend installing mamba!).
Expected install time is approximately 3-minutes.
Installing from source
The current version has been tested with python 3.10 using the conda environment setup specified below, with the following systems:
* Linux Ubuntu 22.04.3 LTS with a Nvidia A40 GPU (40Gb GPU RAM, 40Gb of CPU RAM) running Cuda 12.4 driver version 550.90.07.
* Rocky Linux 8.10 using just CPU (tested by Dr Qiongyi Zhao, UQ IMB).
* Nvidia L40 GPU (Will Rieger, UQ SCMB).
Basic commands to install.
mamba create -n dnac_env python=3.13.15
mamba activate dnac_env
mamba install matplotlib seaborn pandas scipy pytorch-lightning zlib ipykernel bioconda::bedtools bioconda:pybigwig
# ensure zlib is found
export LDFLAGS="-L$CONDA_PREFIX/lib"
export CPPFLAGS="-I$CONDA_PREFIX/include"
export LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"
pip install numba typer pyfaidx kipoiseq enformer-pytorch borzoi-pytorch==0.5.1 pybedtools
pip install transformers==4.50.0
git clone https://github.com/BradBalderson/DNACipher.git
cd DNACipher
pip install .Verify install with:
dnacipher versionDNACipher 1.5.0: ๐ฆพMetalGrip
๐ฃDepending on your system, the above may not install the correct Pytorch to detect your GPU. To check this, run the following:
dnacipher deviceWill use cuda GPU
๐งTrouble shooting no GPU detected:
If you have a GPU but it was not detected, please check you have installed the correct Pytorch by:nvidia-smiThen cross-reference with this Pytorch resource to ensure you are installing the correct Pytorch:
https://pytorch.org/get-started/locally/
Once you have done this, re-check if the GPU is detected with the dnacipher device command.
dnacipher --help Usage: dnacipher [OPTIONS] COMMAND [ARGS]...
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ --install-completion Install completion for the current shell. โ
โ --show-completion Show completion for the current shell, to copy it or customize the installation. โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Commands โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ version Prints the DNACipher version. โ
โ device Prints the device that is detected and will be used by DNACipher. โ
โ precompute-embeds Precomputes the sequence embeddings at desired genome sequence locations, as mean embeddings around the input coordinates. โ
โ precompute-signals Precomputes the functional signals to be predicted with given input sequence, as mean signal across the input locations. โ
โ split-h5 Converts a .h5 file containing a single matrix, to a split.h5 which has an index, allows for faster random indexing. โ
โ fit-norm Fits a normalization to the data - so that the mean signal values across assays are scaled to the median observed value. โ
โ Because the data is very sparse, this mean signal is relative to the non-zero entries in the dataset, and also the โ
โ normalization is only applied to the non-zero entries. โ
โ RNA-seq is dealt with in a specific way, because of the different representation of this assay. โ
โ apply-norm Applies a pre-fitted normalization to the inputted data. โ
โ norm Performs both the fitting and applying of the normalization. โ
โ train Trains the DNACipher model. โ
โ eval Runs evaluation on a trained the DNACipher model. โ
โ celltypes Prints the available celltypes to infer effects for. โ
โ assays Prints the available assays to infer effects for. โ
โ infer-signals Infer signals across genomic regions for given celltype/assays. โ
โ infer-effects Infers a single varaints effects across celltypes and assays, and optionally across the sequence. โ
โ infer-multivariant-effects Takes as input a vcf file, in format, CHR, POS, REF, ALT as columns. Outputs a dataframe with rows per โ
โ variant, and predicted effect sizes across the columns for celltype/assay combinations. โ
โ stratify-variants Performs stratification of variants at GWAS loci to categories: โ
โ * 'candidate' variants (common significant variants), โ
โ * 'rare' variants (non-significant rare variants in the same region as the candidate variants), โ
โ * 'background' variants (common non-significant variants), and 'other' variants (rare variants outside of the hit locus). โ
โ impact-map Calls 'impact' variants - variants with significant predicted effects in particular cell types / assays compared with background variants. โ
โ plot-signals Plots DNACipher signal tracks and optional gene/cCRE annotations. โ
โ plot-variant-stats Manhattan-like plot for variant statistics. โ
โ plot-volcano Volcano plot for Deep Variant Impact Mapping predicted molecular effects. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Step 0: Required data download: reference genome in order to load the sequences for variant effect inference
Most steps below need the reference genome in order to load the sequences for variant effect inference
wget -O - http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz | gunzip -c > hg38.fa
mamba install bioconda::samtools
samtools faidx hg38.fa2.2.1 Inferring the effects for a single variant
The minimal inputs here are just the variant CHR, POS, REF, ALT of the variant, the celltypes and assays to infer the effects for, the path to the fasta file and the prefix for the output files.
dnacipher infer-effects --help Usage: dnacipher infer-effects [OPTIONS] {out_prefix} {chr_} {pos} {ref} {alt}
{celltypes} {assays} {fasta_file_path}
Infers a single varaints effects across celltypes and assays, and optionally across the sequence.
โญโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * out_prefix <str> Prefix for all outputs files. [required] โ
โ * chr_ <str> Chromosome of variant. [required] โ
โ * pos <int> Position of variant on chromosome. [required] โ
โ * ref <str> Reference allele of variant. [required] โ
โ * alt <str> Alternate allele of variant. [required] โ
โ * celltypes <str> Celltypes to infer effects for. File in format: ct1,,ct2,,ct3 [required] โ
โ * assays <str> Assays to infer effects for. File in format: assay1,,assay2,,assay3 [required] โ
โ * fasta_file_path <str> FASTA file path for the reference genome. Must have .fai index. [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ -d,-device <str> Device to run model on. โ
โ -sf,-sample_file <str> Sample metadata used to train the DNACipher model to be used for inference. If not provided uses default DNACipher. โ
โ -m,-model_config <str> .yaml file specifying the model parameters, such as layer sizes, number of layers, etc. If not provided, defaults to an internal config specifying the โ
โ original model. โ
โ -w,-weights <str> .pth specifying the pretrained DNACipher model weights. if None will use default pretrained DNACipher model. โ
โ -mn,-model_name <str> Name of the model to use for the sequence. If None defaults to the enformer model. โ
โ -i,-index_base <int> Whether the variant position is 0-based or 1-based indexing. [default: 0] โ
โ -correct_ref -no-correct_ref Correct the reference genome sequence if disagrees with the inputted ref allele. [default: no-correct_ref] โ
โ -s,-seq_pos <int> Where to centre the query sequence. โ
โ -ers,-effect_region_start <int> Where to start measuring the effect in the genome. โ
โ -ere,-effect_region_end <int> Where to end measuring the effect in the genome. โ
โ -b,-batch_size <int> How many effects to infer at a time. โ
โ -by,-batch_by <str> Indicates how to batch the data when fed into the model, either by 'experiment', 'sequence', or None. If None, will automatically choose whichever is the โ
โ larger axis. โ
โ -all_combinations -no-all_combinations Generate predicetions for all combinations of inputted cell types and assays. [default: all_combinations] โ
โ -return_all -no-return_all Return the signals across the ref and alt sequences [default: no-return_all] โ
โ -sm,-scoring_method <str> Refers to how to compare the ref and alt prediction to score variant effects. Supported methods are 'sign_sum_abs' which will take the SUM(|ALT-REF|)*{-1 if โ
โ SUM(ALT-REF) < 0}. 'sum_logfc' which will score as log2( (SUM(ALT) / SUM(REF)) + 1). โ
โ [default: signed_sum_abs] โ
โ -verbose -quiet Enable or disable verbose output [default: verbose] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Running example, showing the variant effect inference for a causal eQTL at the WRN gene locus:
To see the available cell types and assays available for inference, can run dnacipher celltypes or dnacipher assays.
By specifying '-mn borzoi' in the command below, the DNACipher model trained on Borzoi embedding will be used for effect inference. Otherwise defaults to enformer, i.e. '-mn enformer'.
out_prefix="WRN_eQTL_"
fasta_path="hg38.fa"
chr_="chr8"
pos="31119876"
index_base="1"
ref="T"
alt="C"
seq_pos="31076838"
effect_start="31119776"
effect_end="31119976"
echo "prostate gland" > celltypes.txt
echo "plus strand polyA plus RNA-seq,,minus strand polyA plus RNA-seq" > assays.txt
# Scores the effects for the variant in the inputted contexts, with scoring referring to the SUM(ALT-REF) at the indicated positions
dnacipher infer-effects ${out_prefix} ${chr_} ${pos} ${ref} ${alt} celltypes.txt assays.txt ${fasta_path} -mn borzoi -i ${index_base} -s ${seq_pos} -ers ${effect_start} -ere ${effect_end}# Inferring effects for a single variant with signals along sequence outputted, by the -return_all flag.
dnacipher infer-effects ${out_prefix} ${chr_} ${pos} ${ref} ${alt} celltypes.txt assays.txt ${fasta_path} -mn borzoi -i ${index_base} -s ${seq_pos} -ers ${effect_start} -ere ${effect_end} -return_all2.2.2 Plotting results for a single variant
dnacipher plot-signals --help Usage: dnacipher plot-signals [OPTIONS] SIGNALS_PATH OUT_PREFIX
Plots DNACipher signal tracks and optional gene/cCRE annotations.
โญโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * signals_path TEXT Path to the signal predictions across contexts for a given variant. [default: None] [required] โ
โ * out_prefix TEXT Prefix for all outputs files. [default: None] [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ -gtf,-gtf_file_path TEXT Optional: GTF file path for gene annotation overlap. [default: None] โ
โ -cres,-encode_cres_path TEXT Optional: BED file path for cCREs annotation overlap. [default: None] โ
โ -chr,-variant_chr TEXT Chromosome of the variant (for plotting vertical line). [default: None] โ
โ -pos,-variant_pos INTEGER Position of the variant (for plotting vertical line). [default: None] โ
โ -ref,-variant_ref TEXT Reference allele of the variant. [default: None] โ
โ -alt,-variant_alt TEXT Alternate allele of the variant. [default: None] โ
โ -plot_delta -no-plot_delta Whether to plot the difference between ref and alt signals. [default: no-plot_delta] โ
โ -xtick_freq INTEGER Spacing between x-ticks in base pairs. [default: 200] โ
โ -title TEXT Plot title. [default: DNACipher track predictions] โ
โ -verbose -quiet Enable or disable verbose output. [default: verbose] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
The signals_path points to one of the signal files outputted from the infer-effects command above with the -return-all option, which could be the ref signals, the alt signals, or the diff signals which refers to the difference between ALT-REF signals.
These outputted signal files have this format:
prostate gland---plus strand polyA plus RNA-seq prostate gland---minus strand polyA plus RNA-seq
chr8_30978534_30978566 -1.9446015e-06 -0.00014594942
chr8_30978566_30978598 -2.4095178e-05 -0.00021222234
chr8_30978598_30978630 6.7520887e-06 -8.9131296e-05
These are tab-separated files, with columns in format CELLTYPE---ASSAY, and rows are in format CHR_START_END referring to positions in the genome where the inferred difference occurs. When using the Enformer model as the based model, the resolution is 128bp resolution bins, for 896 bins (total ~114kb, in the middle of the inputted 196kb sequence). The values in this case are ALT-REF signals as each position. The above uses Borzoi as the base model for higher resolution and longer input sequence and longer output signal prediction. Can also use a some custom model, but need to specify additional inputs (see Train.md).
The below code produces signal plots for the diff-signals shown above, which is also produced in Tutorial 1 of the Python API.
# OPTIONAL annotation downloads, to make visualization more compelling
wget https://downloads.wenglab.org/V3/GRCh38-cCREs.bed
wget -O - https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_26/gencode.v26.annotation.gtf.gz | gunzip -c > gencode.v26.annotation.gtf
dnacipher plot-signals ${out_prefix}diff_signals.txt ${out_prefix} -gtf gencode.v26.annotation.gtf -cres GRCh38-cCREs.bed -chr ${chr_} -pos ${pos} -ref ${ref} -alt ${alt}
2.2.3 Inferring effects for multiple variants outputting the summed effect across the locus
This shows how to infer effects for multiple variants, just outputing the SIGN*SUM(ABS(ALT-REF)), where SIGN refers to if most positions were negative or positive along the sequence (SIGN = {-1, +1}).
The input file to this command is a VCF-like tab-separated file, that looks like this:
CHR POS REF ALT
chr8 31119876 T C
chr13 50909867 G C
๐Each line is a 'query' to the DNACipher model. Can include additional columns that specify where to centre the query sequence (otherwise defaults to centring ON the variant location), and also where to determine ALT-REF predictions. For example, can include an addition two columns specifying where to take the difference (SUM(ALT[start_effect:end_effect]-REF[start_effect:end_effect])). These coordinates are given relative to the reference genome, with the chromosome assumed to be the same as the specified chromosome for the variant. For example, could have an input file like this:
CHR POS REF ALT SEQ_POS START_EFFECT END_EFFECT
chr8 31119876 T C 31119866 31119856 31119886
In this above example, would infer the effect for the variant, with the query sequence centred at SEQ_POS, and measuring the effect only at positions START_EFFECT:END_EFFECT. So with this input, for the same variant, we could alter SEQ_POS and START_EFFECT and END_EFFECT in order to measure different predicted effects along the genome for the same variant.
Inferring effects for multiple variants like this is achieved with the command below:
dnacipher infer-multivariant-effects --help Usage: dnacipher infer-multivariant-effects [OPTIONS] {out_prefix} {vcf_path}
{celltypes} {assays}
{fasta_file_path}
Takes as input a vcf file, in format, CHR, POS, REF, ALT as columns. Outputs a dataframe with rows per variant, and predicted effect sizes across the columns for celltype/assay combinations.
โญโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * out_prefix <str> Prefix for all outputs files. [required] โ
โ * vcf_path <str> Rows represent particular genetic variants, columns are CHR, POS, REF, ALT [required] โ
โ * celltypes <str> Celltypes to infer effects for. File in format: ct1,,ct2,,ct3 [required] โ
โ * assays <str> Assays to infer effects for. File in format: assay1,,assay2,,assay3 [required] โ
โ * fasta_file_path <str> FASTA file path for the reference genome. Must have .fai index. [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ -sf,-sample_file <str> Sample metadata used to train the DNACipher model to be used for inference. If not provided uses default DNACipher. โ
โ -m,-model_config <str> .yaml file specifying the model parameters, such as layer sizes, number of layers, etc. If not provided, defaults to an internal config specifying the โ
โ original model. โ
โ -w,-weights <str> .pth specifying the pretrained DNACipher model weights. if None will use default pretrained DNACipher model. โ
โ -mn,-model_name <str> Name of the model to use for the sequence. If None defaults to the enformer model. โ
โ -d,-device <str> Device to run model on. โ
โ -i,-index_base <int> Whether the variant position is 0-based or 1-based indexing. [default: 0] โ
โ -correct_ref -no-correct_ref Correct the reference genome sequence if disagrees with the inputted ref allele. [default: no-correct_ref] โ
โ -sc,-seq_pos_col <str> Column in vcf that specifies the position to centre the query sequence on, must be within dnacipher.seqlen_max in order to predict effect of the genetic โ
โ variant. If None then will centre the query sequence on the inputted variant. โ
โ -ersc,-effect_region_start_col <str> Specifies column in the inputted data frame specifying the start position (in genome coords) to measure the effect โ
โ -erec,-effect_region_end_col <str> Specifies column in the inputted data frame specifying the end position (in genome coords) to measure the effect โ
โ -b,-batch_size <int> How many effects to infer at a time. โ
โ -by,-batch_by <str> Indicates how to batch the data when fed into the model, either by 'experiment', 'sequence', or None. If None, will automatically choose whichever is โ
โ the larger axis. โ
โ -all_combinations -no-all_combinations Generate predictions for all combinations of inputted cell types and assays. [default: all_combinations] โ
โ -sm,-scoring_method <str> Refers to how to compare the ref and alt prediction to score variant effects. Supported methods are 'signed_sum_abs' which will take the โ
โ SUM(|ALT-REF|)*{-1 if SUM(ALT-REF) < 0}. 'sum_logfc' which will score as log2( (SUM(ALT) / SUM(REF)) + 1). โ
โ [default: signed_sum_abs] โ
โ -rrf,-res_reduce_factor <int> Decrease the resolution of the imputed signals by this factor. E.g. rrf=4, 32bp/bin goes to 128bp/bin. If None maintains same resolution as embedding โ
โ model. โ
โ -check -unchecked Whether to check if the input variants are valid or not, by loading all of the sequences first. Helps prevent errors during a long inference run. โ
โ [default: check] โ
โ -verbose -quiet Enable or disable verbose output [default: verbose] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Working example using provided tutorial data, this will take ~0.4 mins and ~4.2Gb of GPU RAM (A40 GPU), or ~41 min for CPU.
out_prefix="eQTLs"
vcf_path="tutorials/data/gtex_variants_SMALL.vcf"
dnacipher infer-multivariant-effects ${out_prefix} ${vcf_path} celltypes.txt assays.txt ${fasta_path} -mn borzoi -i ${index_base}The above does a pre-flight check for any model queries that are invalid, such as INDELs, and filters them out, writing out a new file ${vcf_path}.fixed, as exemplified by the above where an INDEL was filtered out.
The output file is for the valid variants outputted as ${vcf_path}.fixed, or ${vcf_path} if all variants were valid, and is formatted as below:
prostate gland---plus strand polyA plus RNA-seq prostate gland---minus strand polyA plus RNA-seq
3.4083834 -0.5303463
38.672764 -57.34453
2.648003 -3.7050495
13.718397 -16.052166
Each row refers to a row in the vcf_path input or the ${vcf_path}.fixed, and each column is the CELLTYPE---ASSAY, with SUM(ALT-REF) for the given query outputted.
DVIM analysis is performed one locus at a time, and assumes that the inputted VCF-like file has been subsetted to the variants surrounding a given GWAS hit locus.
The below will reproduce the DVIM at the RUNX3 locus of T1D.
The first input file looks like this, but it is flexible of the order and naming of the columns. Required information is the chromosome, position, ref, alt, allele_frequency, p_value. This is necessary to perform the variant stratification to decide what are the different variant types present at the locus for DVIM.
variant_id p_value chromosome base_pair_location effect_allele other_allele effect_allele_frequency beta standard_error odds_ratio ci_lower ci_upper -log10_pval
rs6657823 0.774 1 24890003 G A 0.138 0.00593 0.020636 0.11125903931710739
rs6663476 0.539 1 24890041 C T 0.831 -0.012048 0.019613
2.3.1 Stratifying the variants at the RUNX3 T1D GWAS loci into the common, rare, background variants and other variants
dnacipher stratify-variants --help Usage: dnacipher stratify-variants [OPTIONS] {out_prefix}
{signal_gwas_stats_path} {var_ref_col}
{var_alt_col} {var_loc_col} {p_col}
{allele_freq_col}
Performs stratification of variants at GWAS loci to categories: * 'candidate' variants (common significant variants), * 'rare' variants (non-significant rare variants in the same region as the candidate variants), *
'background' variants (common non-significant variants), and 'other' variants (rare variants outside of the hit locus).
โญโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * out_prefix <str> Prefix for all outputs files. [required] โ
โ * signal_gwas_stats_path <str> Path to GWAS summary statistics for each of the variants at a GWAS locus, within range model predictions. [required] โ
โ * var_ref_col <str> Column in the input dataframe that specifies the variant reference sequence as a string. [required] โ
โ * var_alt_col <str> Column in the input dataframe that specifies the variant alternate sequence as a string. [required] โ
โ * var_loc_col <str> Column in the input dataframe that specifies the variant position as an integer. [required] โ
โ * p_col <str> Name of column in the input dataframe that contains the p-values of the variant-trait associations. [required] โ
โ * allele_freq_col <str> Column in the input dataframe that specifies the variant allele frequency as a float. [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ -pc,-p_cut <float> P-value cutoff to consider a variant significantly associated with the trait. [default: 5e-07] โ
โ -lc,-lowsig_cut <float> Cutoff to consider variants confidently not-significant. [default: 0.001] โ
โ -nt,-n_top <int> If no significant variants, will take this many as the top candidates. [default: 10] โ
โ -afc,-allele_freq_cut <float> If a variant is above this minor allele frequency AND is considered confidently non-significant, then is considered a 'background' variant. If is below this allele โ
โ frequency, then considered a rare variant. โ
โ [default: 0.05] โ
โ -mbv,-min_bg_variants <int> If have less than this number of background variants, will rank-order potential background variants by scoring allele frequency and significance, and take this many variants โ
โ as significant. โ
โ [default: 100] โ
โ -verbose -quiet Enable or disable verbose output [default: verbose] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Now running the example at the RUNX3 T1D locus, note that the columns with the required information are specified on input to enable flexible input formats:
out_prefix="dvim_runx3"
runx3_gwas_stats_path="tutorials/data/Chiou-2021-T1D-GWAS_RUNX3-signal_variant_stats.txt.gz"
dnacipher stratify-variants ${out_prefix} ${runx3_gwas_stats_path} other_allele effect_allele base_pair_location p_value effect_allele_frequency As output, this gives the same input file, but with an extra column 'variant_label' specifying the stratification of the variant:
variant_id p_value chromosome base_pair_location effect_allele other_allele effect_allele_frequency beta standard_error odds_ratio ci_lower ci_upper log10_pval var_label
rs72657048 1.11e-07 1 24963243 G C 0.509 0.074632 0.01406 6.954677021213342 candidate
rs6672420 1.74e-07 1 24964519 T A 0.483 0.07369 0.014103 6.7594507517174005 candidate
The labels are explained in the dnacipher stratify-variants --help documentation above.
2.3.2 Plotting the variant stratification results
Easy to check this result with the following command:
dnacipher plot-variant-stats --help Usage: dnacipher plot-variant-stats [OPTIONS] {out_prefix}
{stratified_gwas_stats_path} {y_axis_col}
{color_by} {var_loc_col}
Manhattan-like plot for variant statistics.
โญโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * out_prefix <str> Prefix for output files. [required] โ
โ * stratified_gwas_stats_path <str> Path to stratified GWAS statistics file. [required] โ
โ * y_axis_col <str> Column name to use for y-axis values (e.g., -log10_pval). [required] โ
โ * color_by <str> Column name to color points by (e.g., var_label). [required] โ
โ * var_loc_col <str> Column in the input dataframe that specifies the variant position as an integer. [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ -li,-locus <int> Which locus to plot, as an integer specifying the locus ordering, if multiple loci represented by a seq_pos column. If None will plot first locus are all variants, โ
โ for latter case assuming file represents single locus. โ
โ -chr,-chrom_col <str> Name of column that contains the chromosome. โ
โ -gtf,-gtf_file_path <str> Optional GTF file path for plotting gene annotations. โ
โ -cmap,-color_map <str> Colormap name for continuous coloring. [default: magma] โ
โ -alpha <float> Opacity of the scatter plot points. [default: 0.5] โ
โ -order_points -no-order_points Whether to plot points ordered by statistic. [default: order_points] โ
โ -reverse_order -no-reverse_order Whether to reverse point order. [default: no-reverse_order] โ
โ -show_legend -no-show_legend Whether to display the plot legend. [default: show_legend] โ
โ -verbose -quiet Enable or disable verbose output. [default: verbose] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
We can now plot to see if the variant stratification make sense. Need to specify the chromosome column to plot the gene annotations.
dnacipher plot-variant-stats ${out_prefix} ${out_prefix}stratified_gwas_stats.txt log10_pval var_label base_pair_location -gtf gencode.v26.annotation.gtf -chrom_col chromosome
2.3.3 Performing the variant effect inference in the relevant cell types
Creating the input files required for the DNACipher variant effect inference for cell types and assays.
echo "natural killer cell,,B cell,,T-cell,,pancreas" > t1d_runx3_celltypes.txt
echo 'minus strand polyA plus RNA-seq,,plus strand polyA plus RNA-seq,,plus strand total RNA-seq,,minus strand total RNA-seq,,ATAC-seq,,H3K27ac_ChIP-seq,,H2BK20ac_ChIP-seq,,H3K27me3_ChIP-seq,,CTCF_ChIP-seq' > t1d_runx3_assays.txtNeed to re-arrange the frame to make sure is in format CHR, POS, REF, ALT, required by dnacipher infer-multivariant-effects
awk -F'\t' 'NR==1 || $NF != "other"' ${out_prefix}stratified_gwas_stats.txt > ${out_prefix}selected_gwas_stats.txt
awk 'BEGIN {OFS="\t"} {print $3, $4, $6, $5, $NF}' ${out_prefix}selected_gwas_stats.txt > ${out_prefix}selected_gwas_stats.reformatted.txt
awk 'BEGIN {OFS=FS="\t"} NR==1 || $1 ~ /^chr/ {print $0; next} { $1 = "chr" $1; print }' ${out_prefix}selected_gwas_stats.reformatted.txt > ${out_prefix}selected_gwas_stats.reformatted.chr_named.txt๐IMPORTANT need to set the seq_pos column, so that each variant is being scored consistently for DVIM and comparing variant effects in a apples-to-apples fashion at each locus.
If there are variants from multiple GWAS loci in this input file, they need to be labelled with a different variant-centre position, which is what DNACipher will use to determine which variants below to which loci for analysis. I recommend setting this to be the centre of the GWAS signal.
In the future, I will implement another function which automatically formats such a query file appropriately from different GWAS summary statistics outputs.
runx3_stats=${out_prefix}selected_gwas_stats.reformatted.chr_named.txt
runx3_stats_dvim=${out_prefix}selected_gwas_stats.reformatted.chr_named.seq_pos.txt
awk 'BEGIN {OFS=FS="\t"} NR==1 {print $0, "seq_pos"; next} {print $0, "24970252"}' ${runx3_stats} > ${runx3_stats_dvim}This is the resulting file format:
chromosome base_pair_location other_allele effect_allele var_label seq_pos
chr1 24963243 C G candidate 24970252
chr1 24964519 A T candidate 24970252
chr1 24966177 C T candidate 24970252
chr1 24966593 T C candidate 24970252
Now running the dnacipher effect inference for these variants:
dnacipher infer-multivariant-effects ${out_prefix} ${runx3_stats_dvim} t1d_runx3_celltypes.txt t1d_runx3_assays.txt ${fasta_path} -i 1 -seq_pos_col seq_pos2.3.4 Calculating variant-effect p-values and calling significant variant effects
dnacipher impact-map --help Usage: dnacipher impact-map [OPTIONS] {out_prefix} {selected_gwas_stats_path}
{pred_effects_path}
Calls 'impact' variants - variants with significant predicted effects in particular cell types / assays compared with background variants.
โญโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * out_prefix <str> Prefix for all outputs files. [required] โ
โ * selected_gwas_stats_path <str> Path to GWAS summary statistics for each of the variants at a GWAS locus, with a column 'var_labels' indicating candidate, rare, and background variants. Each row is a variant. โ
โ [required] โ
โ * pred_effects_path <str> Path to predicted effects for each variant. Each row is a variant, and each column is a predicted molecular effect for that variant. [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ -c,-cpu <int> No. of CPUs to use. Parallelizes across the number of loci, so no effect if only one locus. [default: 3] โ
โ -nb,-n_boots <int> No. of boot-straps of re-selecting the background variants. [default: 10000] โ
โ -pc,-p_cutoff <float> P-value below which a non-background variant predicted molecular effect is considered significantly different to the background vars. [default: 0.05] โ
โ -fc,-fc_cutoff <float> Fold-change cutoff to be considered significant. [default: 0] โ
โ -std,-min_std <float> Minimum standard deviation for the background variant effects. Set to avoid 0 std for 0 effects of background variants causing infinite z-scores. [default: 0.01] โ
โ -p,-pseudocount <int> Value added to boot-strap counts to prevent 0 p-values, defines lower-bound for minimum p-values, should be set to 1. [default: 1] โ
โ -v,-verbosity <int> Verbosity levels. 0 errors only, 1 prints processing progress, 2 prints debugging information. [default: 1] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
This command will perform boot-strapping of the background variant effects, and perform a one-sample z-test for each predicted effect of the candidate common and rare variants to determine p-values with the measured mean and std of the background variant effects as the null distribution. Very fast due to Numba just-in-time (JIT) C-compilation (~0.1min for 10,000 boot-straps).
runx3_effects=${out_prefix}_var_context_effects.txt.gz
dnacipher impact-map ${out_prefix} ${runx3_stats_dvim} ${runx3_effects} -nb 1000000 -c 1The output files include {out_prefix}_boot_pvals.txt and {out_prefix}_boot_counts.txt. Former is the p-values of the predicted molecular effect significance. Output looks like this:
effector memory CD4-positive, alpha-beta T cell---ATAC-seq effector memory CD4-positive, alpha-beta T cell---DNase-seq
1.0 1.0
0.9998000199980002 0.9998000199980002
Rows correspond to a variant, and the columns correspond to the context in format CELLTYPE---ASSAY, and the values are the p-values for the inferred effect being significantly different from the background variants.
Most important output is {runx3_stats}.impact_calls.txt, which specifies which of the variants are 'impact' variants and the number of significant effects they have:
chromosome base_pair_location other_allele effect_allele var_label n_sig_effects impact_variant
chr1 24963243 C G candidate 0 False
chr1 24964519 A T candidate 2 True
chr1 24966177 C T candidate 18 True
2.3.5 Plotting the impact variant results
Can now-replot the variant stats, this time with the n_sig_effects to show the impact variants! We need to specify the column that has the chromosome information to enable plotting the gene annotations.
By default, the below will plot the results for the first GWAS locus. To plot the results for others, specify the '-locus_index'. NOTE however that the p-values shown are corrected for MHT within the locus, but not between loci, to tell if a locus has any common or rare imVars, checked the
dnacipher plot-variant-stats ${out_prefix} ${out_prefix}selected_gwas_stats.reformatted.chr_named.seq_pos.impact_calls.txt n_sig_effects var_label base_pair_location -gtf gencode.v26.annotation.gtf -chrom_col chromosome
Also volcano plots to assess the cutoffs used to call the impact variants:
dnacipher plot-volcano ${out_prefix} candidate ${runx3_stats} ${out_prefix}_sig_effects.txt ${out_prefix}_fold_changes.txt ${out_prefix}_boot_pvals.txt
dnacipher plot-volcano ${out_prefix} rare ${runx3_stats} ${out_prefix}_sig_effects.txt ${out_prefix}_fold_changes.txt ${out_prefix}_boot_pvals.txt
NOTE the below preprint to be updated soon
Comprehensive molecular impact mapping of common and rare variants at GWAS loci. Brad Balderson, Sanjana Tule, Mei-Lin Okino, William JF Rieger, Sierra Corban, Jeff Jaureguy, Nathan Palpant, Kyle J. Gaulton, Mikael Boden, Graham McVicker bioRxiv 2025.06.05.658079; doi: https://doi.org/10.1101/2025.06.05.658079
Authors: Brad Balderson
Contact: uqbbalde@uq.edu.au

