Skip to content
sh-lee-prmlPublic

About

[NeurIPS 2026] WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

Resources

Stars

63 stars

Watchers

0 watching

Forks

Repository files navigation

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

NeurIPS 2026

WaveGen Paper Python 3.10+ Hugging Face Models WaveGen Live Demo WaveGen Demo Page WaveGen Benchmark Demo Spectral Forcing Demo WaveGen Studio for Windows

Sang-Hoon Lee1,* and Ha-Yeong Choi2,*

1Department of Artificial Intelligence, Ajou University, Suwon, Korea
2Frontier AI Lab, KT Corp., Seoul, Korea

WaveGen is the first end-to-end waveform text-to-speech model achieving state-of-the-art performance on the Emilia training dataset

  • The base model, training, inference, and evaluation code is in src/wavegen.
  • WaveGen V-Spec is the recommended model for inference. Its 1400k EMA Safetensors bundle is in ckpts/WaveGen_VSpec/; training, inference, and evaluation are packaged in src/wavegen_v_spec.
  • WaveGen-Flash generates speech in four steps. Its inference code is in src/wavegen_flash, with a self-contained EMA Safetensors bundle in ckpts/WaveGen_Flash/.

GPU Sponsorship

We welcome GPU resources to support the continued development of WaveGen. If you are interested in supporting this work, please contact Sang-Hoon Lee (sanghoonlee@ajou.ac.kr).

Update

26.11.12

  • Released WaveGen Studio, a native Windows app built with C++ and ONNX Runtime CUDA, using BF16 inference, with CUDA Graph support in the native command-line runner. No Python installation is required.

26.11.10

  • We released the WaveGen source code and checkpoint (WaveGen, WaveGen-VSpec) trained on Emilia (24 kHz audio, 50 Hz waveform patches)

  • WaveGen-Flash (4 Steps) is a four-step model distilled from the WaveGen (Vspec). We will share the distillation details soon

26.09.24

  • WaveGen was accepted to NeurIPS 2026.

26.04.25

  • WaveGen was submitted to NeurIPS 2026.

Installation

Use Python 3.10 or newer. Install PyTorch and torchaudio for your system, then install this repository:

cd WaveGen
pip install -e .

FFmpeg is required for audio preprocessing. Evaluation dependencies are optional:

pip install -e ".[eval]"

Models

Model Checkpoint Use
WaveGen 1300k EMA Pre-trained model
WaveGen (Fine-tuned with v-spec losses) 1400k EMA Recommended; default for CLI, Gradio, and V-Spec batch evaluation
WaveGen-Flash (4 Steps) 48k EMA Fixed four-step inference

The public checkpoints use Safetensors and contain FP32 EMA weights only. Download the selected model's checkpoint, YAML, and vocabulary from Hugging Face, then place its model folder under ckpts/ as shown below.

Seed-TTS evaluation

Results on the complete Seed-en (1,088 utterances) and Seed-zh (2,020 utterances) test sets. WaveGen and V-Spec use 50 Euler steps; WaveGen-Flash uses four Euler steps. All three models use Mel temperature 0.

Model EN ZH
WER (%) ↓ SIM ↑ UTMOS ↑ CER (%) ↓ SIM ↑ UTMOS ↑
WaveGen
Pre-trained · 1300k EMA
1.47 0.755 3.68 1.75 0.805 2.87
WaveGen V-Spec
1400k EMA · Recommended
1.39 0.743 3.59 1.65 0.783 2.79
WaveGen-Flash
48k EMA · 4 steps
1.49 0.727 3.53 1.59 0.768 2.74

WaveGen achieves state-of-the-art performance among TTS models trained on the Emilia dataset.

See the Hugging Face model card for the evaluation settings.

Files

WaveGen/
├── ckpts/
│   ├── WaveGen/         # base 1300k checkpoint and vocabulary
│   ├── WaveGen_VSpec/   # 1400k EMA checkpoint, configuration, and vocabulary
│   └── WaveGen_Flash/   # four-step EMA checkpoint, configuration, and vocabulary
├── data/               # dataset metadata and evaluation lists
├── docs/static/images/ # model overview (Figure 1)
└── src/
    ├── wavegen/        # base model
    │   ├── configs/
    │   ├── eval/
    │   ├── infer/
    │   ├── model/
    │   ├── scripts/
    │   └── train/
    ├── wavegen_v_spec/ # V-Spec training, inference, and evaluation
    └── wavegen_flash/  # four-step inference only

The recommended default is WaveGen V-Spec, using the 1,400,000-update EMA checkpoint:

ckpts/WaveGen_VSpec/model_1400000.safetensors
ckpts/WaveGen_VSpec/vocab.txt
ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml

Weights stay in the local ckpts/ directory and are excluded from Git. The matching vocabulary and model configuration are included with the source.

Inference

Run from the repository root:

wavegen_infer-cli \
  --ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \
  --model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \
  --vocab_file ckpts/WaveGen_VSpec/vocab.txt \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference audio." \
  --gen_text "The text to synthesize."

The example configuration uses the same checkpoint:

wavegen_infer-cli -c src/wavegen_v_spec/infer/examples/basic.toml

The defaults are EMA 1400k, Prompt Anchor (Prompt_xt_Anchor), Scheduled K3 (base CFG 1.25), 50 steps, Mel temperature 0, and seed 0. Inference dropout is read from the model YAML (0.2). Prompt Anchor interpolates the prompt waveform and Mel states from noise to the reference at each ODE step and masks prompt states in the joint-null CFG branch. To use baseline prompt conditioning:

wavegen_infer-cli -c src/wavegen_v_spec/infer/examples/basic.toml --inference_method baseline

The inference method and CFG schedule are separate options. Select constant guidance with --cfg_schedule constant (CFG 3 by default).

wavegen_infer-cli and wavegen_v_spec_infer-cli both use the V-Spec package. Without reinstalling an existing environment, use python -m wavegen_v_spec.infer.infer_cli with src on PYTHONPATH. For the pre-trained 1300k Safetensors release, select its assets explicitly:

python -m wavegen.infer.infer_cli \
  --ckpt_file ckpts/WaveGen/model_1300000.safetensors \
  --model_cfg ckpts/WaveGen/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml \
  --vocab_file ckpts/WaveGen/vocab.txt \
  --cfg_schedule scheduled_k3 \
  --cfg_strength 1.25 \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference audio." \
  --gen_text "The text to synthesize."

The base module retains its original local defaults; the paths above select the public Safetensors bundle. Existing .pt checkpoints remain supported through an explicit --ckpt_file override with matching model assets.

See inference for the available options, including Hugging Face loading from the shared repository's WaveGen_VSpec/ folder.

WaveGen-Flash (4 Steps)

Place model.safetensors, WaveGen_Flash.yaml, and vocab.txt in ckpts/WaveGen_Flash/, then run:

python -m wavegen_flash.infer.infer_cli \
  --ref-audio reference.wav \
  --ref-text "The transcript of the reference audio." \
  --gen-text "The text to synthesize." \
  --output generated_flash.wav

Flash uses four Euler steps with the time grid [0, 0.1, 0.25, 0.5, 1.0], CFG 0, and inference bottleneck dropout 0. The matching YAML stores the complete sampling settings, with waveform temperature 1 and initial Mel temperature 0. Mel temperature scales only the initial Mel noise; reference conditioning and later Mel predictions remain active. Use --mel-temperature 1 to reproduce the previous initialization. The final Mel output head is skipped by default. The Safetensors file contains the complete merged EMA model; no separate base checkpoint is needed. This release includes inference code and weights; the Flash training recipe is not included.

See the Flash guide for Python, Hugging Face, and batch inference examples.

Gradio demo

Run the local demo with the Python environment that already runs WaveGen:

python app.py

Open http://127.0.0.1:7860, upload reference speech, enter its transcript and the text to generate. Advanced settings expose Prompt Anchor/baseline inference and constant/Scheduled K3 guidance. The demo defaults to Prompt Anchor with Scheduled K3 (base CFG 1.25), the recommended V-Spec 1400k EMA Safetensors checkpoint, and 50 steps. Inference dropout uses model.cfm.inference_bottleneck_dropout from the loaded model YAML (0.2 in the supplied config).

Select WaveGen-Flash in the model selector for the fixed four-step model. Its sampling schedule, CFG, and dropout follow the Flash configuration.

The app was tested with the existing Gradio 6.17.3 environment. No package installation or version change is needed in that environment. It loads the local checkpoint on the first generation request and does not download models. Use python app.py --help for checkpoint, device, and server options.

Training

Prepare dataset metadata with the scripts in src/wavegen/train/datasets, and set the dataset and output paths in src/wavegen/configs/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml.

accelerate config
accelerate launch src/wavegen/train/train.py --config-name WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml

See training for data preparation and training options.

Evaluation

The default V-Spec 1400k Safetensors batch inference and evaluation settings are in src/wavegen_v_spec/eval/eval_infer_batch.toml. See the evaluation guide for dataset setup and metrics.

python -m wavegen_v_spec.eval.eval_infer_batch \
  --eval-config src/wavegen_v_spec/eval/eval_infer_batch.toml --full-eval

Set the dataset and local metric-model paths in the TOML before running evaluation.

For Flash, generate the complete metadata list with python -m wavegen_flash.eval.eval_infer_batch --meta path/to/meta.lst --output-dir results/flash, then use the existing metric tools described in the Flash evaluation notes.

Safeguards

WaveGen can reproduce voice characteristics from a reference recording. To support responsible use of the source code and checkpoints, please follow these guidelines:

  • Consent: Use reference speech only with the speaker's informed consent and permission to use the recording. Obtain permission before sharing generated speech that reproduces their voice.
  • Misuse: Do not use WaveGen for deceptive impersonation, fraud, harassment, or fabricated statements presented as authentic speech.
  • Transparency: Clearly identify shared or published outputs as AI-generated audio.
  • Privacy: Protect reference recordings and personal information; do not redistribute them without authorization.

These are usage guidelines, not automatic safety filters. Public-facing services should add appropriate access controls, abuse reporting, and review of suspected misuse.

Reference

Our Previous Work

  • PeriodWave [ICLR 2025]: The first CFM-based waveform generation model
  • FlowTokenizer [ACL Findings 2026]: The first CFM-based neural audio codec
  • ReGen [ICML 2026]: The first CFM-based audio VAE

Baseline Models

  • F5-TTS [ACL 2025]: A zero-shot TTS model
  • WavTTS: Concurrent work from ByteDance. We used the official WavTTS implementation as a baseline during the rebuttal period.

License

WaveGen source-code contributions are Copyright (c) 2026 Sang-Hoon Lee and released under the MIT License. Reused components retain their original copyright notices and licenses; see third-party notices.

The published model checkpoints are licensed separately under CC BY-NC 4.0.

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2025-02283048, Developing the Next-Generation General AI with Reliability, Ethics, and Adaptability, IITP-2026-RS-2023-00255968, the Artificial Intelligence Convergence Innovation Human Resources Development, RS-2026-25617480, AI Star Fellowship Support Program(Gyeongsang National University)), National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2026-25518808), the “Advanced GPU Utilization Support Program” funded by the Government of the Republic of Korea (Ministry of Science and ICT), and Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea)&Gwangju Metropolitan City.

  • Advanced GPU Utilization Support Program (WaveGen-S, WaveGen-B)
  • Artificial intelligence industrial convergence cluster development project (WaveGen, WaveGen (V-Spec), WaveGen-Flash)

About

[NeurIPS 2026] WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

Resources

Stars

63 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages