SLIP

SLIP evaluates the local smoothness of continuous speech-VAE latent spaces. It perturbs latent frames toward nearby frames from other utterances, decodes the perturbed latents, and compares the result with real speech. The final score is a Fréchet distance; lower is better.

Leaderboard

All models are evaluated with the same SLIP protocol. Lower is better.

VAE	Frame rate (Hz)	Latent dim	SeedTTS-Eval	LibriSpeech-PC test-clean
Semantic-VAE (dim 16)	40	16	0.2025	0.1778
KVAE-Audio	50	64	0.2066	0.2428
Semantic-VAE (1000k)	40	64	0.2231	0.1911
Semantic-VAE (600k)	40	64	0.2247	0.1926
MMAudio-16k	31.25	20	0.4800	0.5842
LTX-2 Audio	25	128	0.4835	0.4644
Ming-Omni 12.5Hz	12.5	64	0.4912	0.8372
MingTok-Audio	50	64	0.5758	0.5641
Semantic-VAE (dim 64)	40	64	0.6803	0.7222
VoxCPM AudioVAE	25	64	0.7023	0.9020
AudioLDM2	25	128	0.7103	0.8175
Stable Audio Open VAE	21	64	0.7796	0.9137
OmniVAE	50	128	0.7902	0.7891
LongCat Wav-VAE	12	64	0.8613	1.0146
HunyuanVideo-Foley	50	128	0.9315	0.9386
DAC	25	128	0.9569	1.0049
SAME-L	10.77	256	1.0894	1.2404
SAME-S	10.77	256	1.5220	1.7352
AudioLDM	25	128	1.8676	2.2150

SeedTTS-Eval contains 3,108 evaluated utterances. LibriSpeech-PC test-clean contains the 1,127 target utterances from the F5-TTS cross-sentence evaluation list. Exact values and model links are available in leaderboard.csv.

Installation

conda create -n slip python=3.10 -y
conda activate slip
python -m pip install -e ".[datasets]"

Install a CUDA-enabled PyTorch build before running a full evaluation.

Data

Prepare an extracted SeedTTS-Eval dataset:

slip datasets prepare-seedtts \
  --source data/seedtts_testset \
  --output data/seedtts-eval-v1

Or prepare LibriSpeech-PC test-clean. The target set follows the F5-TTS cross-sentence list, with audio obtained from AudioLLMs/librispeech_test_clean:

slip datasets download-librispeech-pc \
  --output data/librispeech-pc-v1

Evaluation

Connect a VAE through an adapter, point a YAML config to the adapter and dataset manifest, then run:

slip check-adapter configs/my_vae.yaml
slip evaluate configs/my_vae.yaml
slip report runs/my_vae

See examples/README.md for the adapter template, config fields, and a complete Semantic-VAE example. Interrupted evaluations can be resumed by running the same command again.

Metric

SLIP uses exact cross-utterance cosine top-10 retrieval, deterministic neighbor sampling, trajectory-level Slerp at alpha=0.5, and HuBERT Base layer 9 features. The fixed settings are listed in docs/slip.md.

Development

python -m pip install -e ".[datasets,dev]"
python -m pytest -q
ruff check src tests examples