Skip to content

model: add IndexTTS-2.5 family - #210

Draft
IIIIIllllIIIIIlllll wants to merge 5 commits into
0xShug0:mainfrom
IIIIIllllIIIIIlllll:feature/index-tts2_5-pr
Draft

model: add IndexTTS-2.5 family#210
IIIIIllllIIIIIlllll wants to merge 5 commits into
0xShug0:mainfrom
IIIIIllllIIIIIlllll:feature/index-tts2_5-pr

Conversation

@IIIIIllllIIIIIlllll

@IIIIIllllIIIIIlllll IIIIIllllIIIIIlllll commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Just for test now.

Add support for IndexTTS-2.5 (GPT + DiT CFM + BigVGAN, zh/en/ja/es/ar) as a new model family alongside index_tts2:

The architecture of IndexTTS-2.5 has many similarities to IndexTTS-2, but I created a new architecture to avoid interfering with the existing code.

GGUF is also supported (GGUF weights are not uploaded).

Test on my 2080TI gfx1151 CPU

The generated content was consistent with the official deployment method.

About convert:

python tools/convert_index_tts2_5.py \
    --model-dir /path/to/IndexTTS-2.5 \
    --output-dir /path/to/staging \
    --native-dir /path/to/IndexTTS-2.5-native \
    --run-converter /path/to/gguf_file --type q8_0

AI usage: Kimi k3 developed this implementation based on the original project.

Side note (not fixed in this PR): the existing index_tts2 family produces noise on the CPU backend, same root cause as the 2.5 fix here — src/models/index_tts2/s2mel.cpp builds the wavenet zero accumulator via sub(ctx, input_bct, input_bct) where input_bct is a permuted view; ggml's CPU binary-op kernels only check src1 row-contiguity and silently misread a permuted src0 (asserts are compiled out under NDEBUG; CUDA/HIP kernels handle it, so only CPU is affected). The same one-line materialize-before-sub fix applies. A more thorough fix belongs in ggml: check row contiguity for both operands in the binary-op path (add/sub/mul/div family).

Add support for IndexTTS-2.5 (GPT + DiT CFM + BigVGAN, zh/en/ja/es/ar)
as a new model family alongside index_tts2:

- tiktoken BPE tokenizer (multilingual_zh_ja_yue_char_del, 60509 vocab)
  on top of the vendored llama bpe-core, with <|lang|> prefixes,
  pronunciation annotations (<text|reading>), and language-aware casing
- GPT speaker conditioning via CAMPPlus linear projection (3-token
  prefix) replacing the conformer+perceiver path, plus lang_embedding
- semantic codec: full EnhancedCodec decode chain (decoder backbone +
  2x nearest upsample + up conv); prompt_condition now consumes raw
  normalized w2v-bert semantics as upstream 2.5 does
- s2mel/bucket flow drops the gpt-latent second pass (use_gpt_latent
  defaults to false upstream)
- model spec, GGUF converter inputs, warm bench tests, CLI longform
  cases, WebUI catalog entries, and docs/tts.md section

Verified against the official Python reference on CUDA: greedy zh output
matches upstream token-for-prefix (99/143 codes, f16 drift after) with
identical 5.70s duration; ASR transcripts match for zh/en/ja; q8_0 and
native safetensors layouts both pass.
The zero skip-connections accumulator in cfm_wavenet was built with
sub(ctx, input_bct, input_bct) where input_bct is a permuted (transposed)
view. The ggml CPU binary-op kernels only require contiguous rows for
src1 and silently misread a permuted src0 (the CUDA/HIP kernels handle
it), so on CPU the accumulator started as garbage and the mel output
collapsed into noise. Materialize the tensor before the sub.

Also adds env-gated debug taps used for the investigation:
INDEXTTS25_DUMP_DIR (NPY dumps of GPT/codec/LR/CFM stages) and
INDEXTTS25_CFM_NOISE_NPY (diffusion noise injection); zero cost when
unset.

Verified: CPU/CUDA/HIP backends all pass ASR transcription checks
against the official Python reference.
@IIIIIllllIIIIIlllll

Copy link
Copy Markdown
Contributor Author

@0xShug0 This is a real big project. I'd appreciate it if you could do a quick check to see which parts are reusable. Thanks!

tools/convert_index_tts2_5.py stages an official IndexTeam/IndexTTS-2.5
snapshot (.pth checkpoints) into the Safetensors layout the engine
expects — unwrapping the s2mel/codec container keys, prefixing CAMPPlus
tensors with speaker_encoder., stripping BigVGAN's generator. prefix,
wrapping the feat matrices — assembles the sidecar root (config,
tiktoken vocabulary, auxiliary configs), and prints or runs the
audiocpp_gguf command.

Verified end-to-end on CUDA: fresh staging -> f16 GGUF -> inference ->
ASR transcript matches the official Python reference; staged tensor
name sets match the hand-verified staging for all 10 namespaces.
--native-dir assembles a directly loadable model directory (the spec's
safetensors source layout: feat1/feat2.safetensors,
semantic_codec_model.safetensors, bigvgan/ and w2v-bert-2.0/
subdirectories, sidecar configs) hardlinked from the staging output.
Verified on CUDA: script-produced native directory loads and its ASR
transcript matches the official reference.
@0xShug0

0xShug0 commented Aug 11, 2026

Copy link
Copy Markdown
Owner

@IIIIIllllIIIIIlllll Thanks for the work! Based on your code, I think IndexTTS2.5 is close enough to IndexTTS2 that we should extend the existing IndexTTS2 implementation additively instead of copying the whole model into a new family.

The main 2.5 differences seem to be variant-level pieces: multilingual tiktoken/BPE tokenizer + language embedding, CAMPPlus-projected GPT speaker conditioning, changed GPT prefill condition layout, and semantic-codec 2x upsample + up conv. Most other files are identical or near-identical to IndexTTS2 after renaming. Most request/session options seem to be shared with existing IndexTTS2.

Could you refactor this as an IndexTTS2 variant rather than adding copied index_tts2_5 files? Suggested shape:

  1. Keep shared audio features, wav2vec2bert, qwen emotion, style encoder, BigVGAN vocoder, and most S2Mel code in the current IndexTTS2 implementation.

  2. Add a variant-specific text tokenizer behind one common IndexTTS2 tokenizer interface:

    • v2 keeps the current IndexTTS2 tokenizer behavior.
    • v2.5 uses the multilingual tiktoken/BPE tokenizer, language special tokens, and language-id handling.
  3. Split GPT speaker-conditioning into explicit variant modes:

    • v2 mode should preserve the current IndexTTS2 GPT speaker-conditioning path: use speaker semantic features with the existing conditioning_encoder + perceiver_encoder.
    • v2.5 mode loads and uses spk_emb_proj with the CAMPPlus speaker/style embedding, plus lang_embedding during text prefill.
    • Keep the shared GPT decode/cache logic in one place.
  4. Extend semantic codec with an explicit variant option:

    • v2 keeps the current code embedding path.
    • v2.5 applies the extra 2x nearest upsample and up convolution after the decoder projection.
    • The branch should be selected from model config/variant, not by probing tensor names in hot paths.
  5. Add 2.5 packages/spec entries under the existing IndexTTS2 family/variant surface.

  6. Remove debug-only production code such as dump paths, injected noise files, and environment-variable parity hooks before PR is ready.

This should keep the implementation smaller, reduce maintenance cost, and avoid two copies drifting independently.

Replace the copied index_tts2_5 model family with an additive variant of
the existing index_tts2 implementation, selected from the model config
version field ("2.5") instead of a separate family or tensor-name probing:

- one IndexTTS2TextTokenizer interface with per-variant backends: v2 keeps
  the SentencePiece behavior, v2.5 uses the multilingual tiktoken/BPE
  vocabulary with language special tokens and language-id handling
- GPT speaker conditioning split into explicit variant modes: v2 keeps the
  conditioning_encoder + perceiver_encoder path with speed embeddings, v2.5
  loads spk_emb_proj + lang_embedding and builds the campplus conditioning
  prefix inside the prefill graph; decode/cache logic stays shared
- semantic codec gains a variant decode path: v2.5 runs the full EnhancedCodec
  decoder with the 2x nearest upsample and up convolution
- v2.5 regulates the raw w2v-bert semantic directly (no codec quantize, no
  GPT latent projection) while v2 keeps its existing flow
- 2.5 packages and the tiktoken tokenizer file join the index_tts2 model
  spec (tokenizer files are optional per variant); the conversion tool
  normalizes the staged config version to "2.5" and targets index_tts2
- the lang request option is parsed for the v2.5 tokenizer
- drop the debug-only dump paths, noise injection and environment parity
  hooks; the s2mel CFM wavenet keeps the contiguous zero accumulator so the
  CPU backend computes permuted operands correctly

Greedy (do_sample=false, seed-pinned) outputs are bit-identical to the
previous split implementation for both variants on CUDA: v2.5 zh/ja GGUF
and v2 zh native Safetensors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants