StableForm-TTS - Pytorch Implementation

PyTorch unofficial implementation of "Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation".

Getting Started

Dependencies

pip install -r requirements.txt

Vocoder

• HiFi-GAN
• BigVGAN

Dataset

Modify the config file to fit your environment.

# configs/datasets/lj/base_text2mel.yaml
raw_data_dir: ./dataset/LJSpeech-1.1
preprocess_cls: configs.datasets.lj.preprocess.LJPreprocess
processed_data_dir: ./dataset/processed/ljspeech
binary_data_dir: ./dataset/binary/ljspeech
...

The public dataset can be downloaded from the following link. LJSpeech, VCTK, LibriTTS

Preprocess

# preprocess.sh
export PYTHONPATH=.
CONFIG="configs/models/tts/stableform_tts.yaml";
python data_gen/tts/runs/preprocess.py --config $CONFIG
python data_gen/tts/runs/binarize.py --config $CONFIG

Training

# run.sh 
export PYTHONPATH=.
DEVICE=0;
CONFIG="configs/models/tts/stableform_tts.yaml";
MODEL_NAME="StableFormTTS_libri16k";

CUDA_VISIBLE_DEVICES=$DEVICE python tasks/run.py \
    --config $CONFIG \
    --exp_name $MODEL_NAME \
    --reset \
    --hparams=$HPARAMS

Inference

CUDA_VISIBLE_DEVICES=$DEVICE python tasks/run.py \
    --config $CONFIG \
    --exp_name $MODEL_NAME \
    --infer \
    --hparams=$HPARAMS \
    --reset

Tensorboard

Loss

Mel-spectrogram

Alignment

References

• NATSpeech
• RAD-TTS
• Meta-StyleSpeech
• BigVGAN