model: add IndexTTS-2.5 family - #210
Conversation
Add support for IndexTTS-2.5 (GPT + DiT CFM + BigVGAN, zh/en/ja/es/ar) as a new model family alongside index_tts2: - tiktoken BPE tokenizer (multilingual_zh_ja_yue_char_del, 60509 vocab) on top of the vendored llama bpe-core, with <|lang|> prefixes, pronunciation annotations (<text|reading>), and language-aware casing - GPT speaker conditioning via CAMPPlus linear projection (3-token prefix) replacing the conformer+perceiver path, plus lang_embedding - semantic codec: full EnhancedCodec decode chain (decoder backbone + 2x nearest upsample + up conv); prompt_condition now consumes raw normalized w2v-bert semantics as upstream 2.5 does - s2mel/bucket flow drops the gpt-latent second pass (use_gpt_latent defaults to false upstream) - model spec, GGUF converter inputs, warm bench tests, CLI longform cases, WebUI catalog entries, and docs/tts.md section Verified against the official Python reference on CUDA: greedy zh output matches upstream token-for-prefix (99/143 codes, f16 drift after) with identical 5.70s duration; ASR transcripts match for zh/en/ja; q8_0 and native safetensors layouts both pass.
The zero skip-connections accumulator in cfm_wavenet was built with sub(ctx, input_bct, input_bct) where input_bct is a permuted (transposed) view. The ggml CPU binary-op kernels only require contiguous rows for src1 and silently misread a permuted src0 (the CUDA/HIP kernels handle it), so on CPU the accumulator started as garbage and the mel output collapsed into noise. Materialize the tensor before the sub. Also adds env-gated debug taps used for the investigation: INDEXTTS25_DUMP_DIR (NPY dumps of GPT/codec/LR/CFM stages) and INDEXTTS25_CFM_NOISE_NPY (diffusion noise injection); zero cost when unset. Verified: CPU/CUDA/HIP backends all pass ASR transcription checks against the official Python reference.
|
@0xShug0 This is a real big project. I'd appreciate it if you could do a quick check to see which parts are reusable. Thanks! |
tools/convert_index_tts2_5.py stages an official IndexTeam/IndexTTS-2.5 snapshot (.pth checkpoints) into the Safetensors layout the engine expects — unwrapping the s2mel/codec container keys, prefixing CAMPPlus tensors with speaker_encoder., stripping BigVGAN's generator. prefix, wrapping the feat matrices — assembles the sidecar root (config, tiktoken vocabulary, auxiliary configs), and prints or runs the audiocpp_gguf command. Verified end-to-end on CUDA: fresh staging -> f16 GGUF -> inference -> ASR transcript matches the official Python reference; staged tensor name sets match the hand-verified staging for all 10 namespaces.
--native-dir assembles a directly loadable model directory (the spec's safetensors source layout: feat1/feat2.safetensors, semantic_codec_model.safetensors, bigvgan/ and w2v-bert-2.0/ subdirectories, sidecar configs) hardlinked from the staging output. Verified on CUDA: script-produced native directory loads and its ASR transcript matches the official reference.
|
@IIIIIllllIIIIIlllll Thanks for the work! Based on your code, I think IndexTTS2.5 is close enough to IndexTTS2 that we should extend the existing IndexTTS2 implementation additively instead of copying the whole model into a new family. The main 2.5 differences seem to be variant-level pieces: multilingual tiktoken/BPE tokenizer + language embedding, CAMPPlus-projected GPT speaker conditioning, changed GPT prefill condition layout, and semantic-codec 2x upsample + up conv. Most other files are identical or near-identical to IndexTTS2 after renaming. Most request/session options seem to be shared with existing IndexTTS2. Could you refactor this as an IndexTTS2 variant rather than adding copied index_tts2_5 files? Suggested shape:
This should keep the implementation smaller, reduce maintenance cost, and avoid two copies drifting independently. |
Replace the copied index_tts2_5 model family with an additive variant of
the existing index_tts2 implementation, selected from the model config
version field ("2.5") instead of a separate family or tensor-name probing:
- one IndexTTS2TextTokenizer interface with per-variant backends: v2 keeps
the SentencePiece behavior, v2.5 uses the multilingual tiktoken/BPE
vocabulary with language special tokens and language-id handling
- GPT speaker conditioning split into explicit variant modes: v2 keeps the
conditioning_encoder + perceiver_encoder path with speed embeddings, v2.5
loads spk_emb_proj + lang_embedding and builds the campplus conditioning
prefix inside the prefill graph; decode/cache logic stays shared
- semantic codec gains a variant decode path: v2.5 runs the full EnhancedCodec
decoder with the 2x nearest upsample and up convolution
- v2.5 regulates the raw w2v-bert semantic directly (no codec quantize, no
GPT latent projection) while v2 keeps its existing flow
- 2.5 packages and the tiktoken tokenizer file join the index_tts2 model
spec (tokenizer files are optional per variant); the conversion tool
normalizes the staged config version to "2.5" and targets index_tts2
- the lang request option is parsed for the v2.5 tokenizer
- drop the debug-only dump paths, noise injection and environment parity
hooks; the s2mel CFM wavenet keeps the contiguous zero accumulator so the
CPU backend computes permuted operands correctly
Greedy (do_sample=false, seed-pinned) outputs are bit-identical to the
previous split implementation for both variants on CUDA: v2.5 zh/ja GGUF
and v2 zh native Safetensors.
Just for test now.
Add support for IndexTTS-2.5 (GPT + DiT CFM + BigVGAN, zh/en/ja/es/ar) as a new model family alongside index_tts2:
The architecture of IndexTTS-2.5 has many similarities to IndexTTS-2, but I created a new architecture to avoid interfering with the existing code.
GGUF is also supported (GGUF weights are not uploaded).
Test on my 2080TI gfx1151 CPU
The generated content was consistent with the official deployment method.
About convert:
AI usage: Kimi k3 developed this implementation based on the original project.
Side note (not fixed in this PR): the existing index_tts2 family produces noise on the CPU backend, same root cause as the 2.5 fix here — src/models/index_tts2/s2mel.cpp builds the wavenet zero accumulator via sub(ctx, input_bct, input_bct) where input_bct is a permuted view; ggml's CPU binary-op kernels only check src1 row-contiguity and silently misread a permuted src0 (asserts are compiled out under NDEBUG; CUDA/HIP kernels handle it, so only CPU is affected). The same one-line materialize-before-sub fix applies. A more thorough fix belongs in ggml: check row contiguity for both operands in the binary-op path (add/sub/mul/div family).