diff --git a/README.md b/README.md index 6fb0615d..457e7a38 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,7 @@ Runtime tags: safetensors is the default model loading path. `GGUF 16/Q8/Q4` mea | **vibevoice_asr** | ASR | auto | VibeVoice ASR | GGUF 16/Q8 | | **voxtral_realtime** | ASR | auto | Voxtral-Mini-4B-Realtime-2602 | GGUF 16/Q8/Q4, Stream | | **voxcpm2** | TTS, Clone, Design, Ctrl | ar, da, de, el, en, es, fi, fr, he, hi, id, it, ja, km, ko, lo, ms, my, nl, no, pl, pt, ru, sv, sw, th, tl, tr, vi, zh | VoxCPM2-2B, 48 kHz | GGUF 16/Q8, Stream | -| **index_tts2** | TTS, Clone, Ctrl | zh, en | IndexTTS-2 | GGUF 16/Q8 | +| **index_tts2** | TTS, Clone, Ctrl | zh, en, ja, es, ar | IndexTTS-2, IndexTTS-2.5 (variant) | GGUF 16/Q8 | | **irodori_tts** | TTS, Clone, Design, Ctrl | ja | Irodori-TTS-v4-Small, Irodori-TTS-500M-v3, Irodori-TTS-600M-v3-VoiceDesign | GGUF 16/Q8 | | **moss_tts_nano** | TTS, Clone | auto | MOSS-TTS-Nano-100M | GGUF 16/Q8 | | **moss_tts_local** | TTS, Clone, Ctrl | auto, optional language hint | MOSS-TTS-Local-Transformer-v1.5 | GGUF 16/Q8 | diff --git a/docs/gguf.md b/docs/gguf.md index 86873359..37f8a9ee 100644 --- a/docs/gguf.md +++ b/docs/gguf.md @@ -69,7 +69,7 @@ Status labels: | `htdemucs` | Done | Pass | --- | Pass | Pass (drift) | | `hviske_asr` | Done | Pass | --- | --- | Pass | | `inflect_v2` | Done | Pass | Pass | --- | --- | -| `index_tts2` | Done | Pass | Pass | Pass (drift) | Pass (ASR match, drift) | +| `index_tts2` | Done (v2 + v2.5 variant) | Pass | Pass | Pass (drift) | Pass (ASR match, drift) | | `irodori_tts` | Done | Pass | --- | Pass | Pass (ASR match, drift) | | `kroko_asr` | Done | Pass | --- | --- | Pass | | `marblenet_vad` | Bundled (tiny model) | Pass | --- | --- | --- | diff --git a/docs/tts.md b/docs/tts.md index a7f2ee59..d0d68999 100644 --- a/docs/tts.md +++ b/docs/tts.md @@ -15,6 +15,7 @@ | Higgs Audio v3 TTS | `higgs_audio_tts` | `tts` | [Higgs Audio v3 TTS](#higgs-audio-v3-tts) | | Fish Audio S2 Pro | `fish_audio` | `tts` | [Fish Audio S2 Pro](#fish-audio-s2-pro) | | IndexTTS2 | `index_tts2` | `tts` | [IndexTTS2](#indextts2) | +| IndexTTS2.5 | `index_tts2` (variant `2.5`) | `tts` | [IndexTTS2.5](#indextts25) | | Irodori-TTS | `irodori_tts` | `tts`, `vdes` | [Irodori-TTS](#irodori-tts) | | GLM-TTS | `glm_tts` | `tts`, `clon` | [GLM-TTS](#glm-tts) | | Inflect Micro v2 | `inflect_v2` | `tts` | [Inflect v2](#inflect-v2) | @@ -546,6 +547,89 @@ audiocpp_cli --task tts --family index_tts2 --model /path/to/IndexTTS-2 --backen | `--session-option index_tts2.emotion_text_max_new_tokens=` | tokens | `256` | Maximum generated tokens for emotion-text classification. | | `--session-option index_tts2.weight_context_mb=` | MB | `32` | Shared ggml weight metadata context size. | +## IndexTTS2.5 + +IndexTTS2.5 is IndexTeam/bilibili's multilingual zero-shot TTS model (released 2026-07): a 0.8B GPT (autoregressive) + DiT CFM + BigVGAN stack that keeps IndexTTS2's timbre-emotion decoupling and adds Japanese, Spanish, and Arabic on top of Chinese and English. It requires a speaker reference through the framework `--voice-ref` path. Inline `<文字|发音>` pronunciation overrides (pinyin, CMU phonemes, or kana) are supported. Upstream weights live at [IndexTeam/IndexTTS-2.5](https://huggingface.co/IndexTeam/IndexTTS-2.5); the reference implementation is [index-tts/index-tts](https://github.com/index-tts/index-tts) branch `indextts-2.5`. + +IndexTTS2.5 is implemented as a variant of the `index_tts2` family rather than a separate family: both variants share the audio features, wav2vec2bert, Qwen emotion, style encoder, BigVGAN vocoder, S2Mel, and the GPT decode/cache code, while the tokenizer (SentencePiece vs multilingual tiktoken), GPT speaker conditioning (conditioning encoder + perceiver vs CAMPPlus `spk_emb_proj` + `lang_embedding`), and the semantic-codec decode path (v2.5 adds a 2x nearest upsample + `up` conv) are selected per variant from the model config `version` field (`"2.5"`). All IndexTTS2 session options (`index_tts2.*`) apply to both variants. + +| Field | Value | +|---|---| +| Family | `index_tts2` (the `2.5` variant is selected from the model config `version` field; no separate family) | +| Model directory | `models/IndexTTS2.5-GGUF` (default GGUF package `index_tts2_5_q8_0`; `index_tts2_5_f16` and `index_tts2_5_orig` also available) | +| Task | `tts`, `clon` | +| Modes | `offline` | +| Languages | `zh`, `en`, `ja`, `es`, `ar` | +| Voice input | Required reference WAV through `--voice-ref` | +| Built-in voices | Not exposed | + +Voice clone: + +```bash +audiocpp_cli --task clon --family index_tts2 --model /path/to/IndexTTS2.5-GGUF --backend cuda --text "Hello from IndexTTS2.5." --voice-ref /path/to/reference.wav --out out.wav +``` + +Emotion text: + +```bash +audiocpp_cli --task tts --family index_tts2 --model /path/to/IndexTTS2.5-GGUF --backend cuda --text "今天的演示会更有情绪。" --voice-ref /path/to/reference.wav --emotion "你吓死我了！你是鬼吗？" --request-option emotion_alpha=0.6 --out out.wav +``` + +The `lang` request option selects the text language (`auto`, `zh`, `en`, `ja`, `es`, `ar`, or any tokenizer language code). The default `auto` picks `zh` when the text contains Han characters and `en` otherwise, so mixed Japanese/Spanish/Arabic text should set `--request-option lang=ja|es|ar` explicitly. + +Emotion conditioning supports all three IndexTTS2 paths: an emotion reference WAV through `--audio`, an explicit `emotion_vector`, and Qwen-based emotion-text classification through `--emotion` / `use_emotion_text`. Known limitation: the NeMo text normalizers for Japanese and Spanish are not ported, so ja/es input text is passed through without upstream-style normalization. + +License: IndexTTS-2.5 weights are distributed under the bilibili Model Use License, which is not OSI-approved. It requires separate commercial authorization when monthly active users exceed 100 million or annual revenue exceeds 1 billion RMB, and it forbids using model outputs to improve other AI models. Check the upstream repository for the full terms before redistribution or commercial use. + +| Option | Values | Default | Meaning | +|---|---|---:|---| +| `--voice-ref` | WAV path | required | Reference speaker audio. | +| `--request-option lang=` | `auto`, `zh`, `en`, `ja`, `es`, `ar`, ... | `auto` | Text language hint; `auto` infers `zh` when the text contains Han characters, otherwise `en`. | +| `--emotion` | text | not set | Emotion-text conditioning through the framework style field. | +| `--request-option emotion_alpha=` | float in `[0, 1]` | `1.0` | Blend strength for explicit emotion conditioning. | +| `--request-option emotion_vector=` | 8 floats | not set | Explicit emotion vector. | +| `--request-option use_emotion_text=true|false` | bool | `false` | Infer emotion from text. | +| `--request-option use_random_emotion=true|false` | bool | `false` | Use random emotion weights in the emotion mixer. | +| `--request-option interval_silence_ms=` | milliseconds | `200` | Silence inserted between generated text chunks. | +| `--text-chunk-size` | characters | not set | Optional framework outer text chunk size. When omitted, IndexTTS2.5 keeps its internal tokenizer segmentation. | +| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework chunking mode used only when `--text-chunk-size` is set. | +| `--max-tokens` | integer | `1500` | Maximum generated GPT mel tokens. | +| `--temperature` | float | `0.8` | GPT sampling temperature. | +| `--top-p` | float | `0.8` | GPT nucleus sampling limit. | +| `--top-k` | integer | `30` | GPT top-k sampling limit. | +| `--repetition-penalty` | float | `10.0` | GPT repetition penalty. | +| `--do-sample` | `true`, `false` | `true` | Enable stochastic GPT sampling. | +| `--request-option length_penalty=` | float | `0.0` | GPT beam-search length penalty. | +| `--request-option num_beams=` | integer | `3` | GPT beam count. | +| `--session-option index_tts2.mem_saver=true|false` | bool | `false` | Release staged reference and conditioning graphs after request phases. | +| `--session-option index_tts2.weight_type=native|f32|f16|bf16|q8_0` | enum | `native` | Matmul weight storage type. | +| `--session-option index_tts2.conv_weight_type=native|f32|f16` | enum | `native` | Convolution weight storage type. | +| `--session-option index_tts2.speaker_cache_slots=` | integer slots | `1` | Prepared speaker-reference cache slots; set `0` to disable reuse. | +| `--session-option index_tts2.emotion_cache_slots=` | integer slots | `1` | Prepared emotion-reference cache slots; set `0` to disable reuse. | +| `--session-option index_tts2.emotion_text_cache_slots=` | integer slots | `1` | Emotion-text weight cache slots; set `0` to disable reuse. | +| `--session-option index_tts2.gpt_graph_arena_mb=` | MB | model default | GPT graph arena size. | +| `--session-option index_tts2.s2mel_graph_arena_mb=` | MB | model default | S2Mel graph arena size. | +| `--session-option index_tts2.reference_graph_arena_mb=` | MB | model default | Reference encoder and codec graph arena size. | +| `--session-option index_tts2.emotion_text_prefill_graph_arena_mb=` | MB | model default | Emotion-text prefill graph arena size. | +| `--session-option index_tts2.emotion_text_decode_graph_arena_mb=` | MB | model default | Emotion-text cached-step graph arena size. | +| `--session-option index_tts2.emotion_text_max_new_tokens=` | tokens | `256` | Maximum generated tokens for emotion-text classification. | +| `--session-option index_tts2.weight_context_mb=` | MB | `32` | Shared ggml weight metadata context size. | + +### Converting From Upstream Weights + +`tools/convert_index_tts2_5.py` turns an official `IndexTeam/IndexTTS-2.5` snapshot (the `.pth` checkpoints) into the Safetensors staging layout the engine expects, and prints (or runs) the matching `audiocpp_gguf` command. The w2v-bert-2.0, CAMPPlus, and BigVGAN checkpoints are auto-detected under `/hf_cache/` (run the official inference once to populate it) and each has an explicit override flag: + +```bash +python tools/convert_index_tts2_5.py \ + --model-dir /path/to/IndexTTS-2.5 \ + --output-dir /path/to/staging \ + --run-converter /path/to/audiocpp_gguf --type q8_0 +``` + +Pass `--native-dir /path/to/IndexTTS-2.5-native` to also emit a directly loadable native Safetensors model directory (hardlinked from the staging files), no GGUF conversion required. + +The script repackages the checkpoints the loader needs (unwraps the `s2mel.pth`/`codec.pth` container keys, prefixes CAMPPlus tensors with `speaker_encoder.`, strips BigVGAN's `generator.` prefix, wraps the `feat1/feat2.pt` matrices as a single `tensor`) and assembles the sidecar `root/` (config, tiktoken vocabulary, auxiliary model configs) that gets embedded into the GGUF. The staged `config.yaml` has its `version` field normalized to `"2.5"` (the official snapshot ships `version: 2.0`); the engine uses that field to select the IndexTTS2 family variant. + ## Irodori-TTS Irodori-TTS is Japanese TTS under `--family irodori_tts`. v4 Small is the preferred GGUF-first package and supports no-reference speech, reference-conditioned speech, and caption-based voice design in one checkpoint. The older 500M v3 and 600M v3 VoiceDesign packages remain supported for existing users. See [Irodori-TTS](models/irodori_tts.md) for v3/v4 differences, GGUF variants, options, and compatibility aliases. diff --git a/include/engine/framework/text/text_normalization.h b/include/engine/framework/text/text_normalization.h index ec026cde..2b10c051 100644 --- a/include/engine/framework/text/text_normalization.h +++ b/include/engine/framework/text/text_normalization.h @@ -10,6 +10,9 @@ struct EnglishTextNormalizationOptions { bool spell_numbers = true; bool index_tts_punctuation = false; bool uppercase_ascii = false; + // Verbalize standalone ASCII symbols like the official wetext English + // normalizer ("a_b" -> "a underscore b", "C++" -> "C plus plus"). + bool verbalize_symbols = false; }; std::string replace_all(std::string text, std::string_view from, std::string_view to); diff --git a/include/engine/models/index_tts2/gpt.h b/include/engine/models/index_tts2/gpt.h index 7df7b34c..93472fe0 100644 --- a/include/engine/models/index_tts2/gpt.h +++ b/include/engine/models/index_tts2/gpt.h @@ -88,6 +88,12 @@ struct IndexTTS2GptWeights { engine::modules::LinearWeights emotion_vec_projection; engine::modules::LinearWeights emotion_layer; std::vector speed_embedding_values; + // v2.5 campplus speaker conditioning (spk_cond_mode="campplus" in the + // official model_v2.py): projects the 192-dim CAMPPlus speaker/style + // embedding into a GPT speaker token. + engine::modules::LinearWeights spk_emb_proj; + // v2.5: row of this table is added to every text embedding during prefill. + engine::core::TensorValue lang_embedding; std::vector gpt_layers; engine::modules::NormWeights gpt_final_norm; engine::modules::NormWeights final_norm; @@ -109,8 +115,14 @@ struct IndexTTS2GptGeneration { struct IndexTTS2GptGenerationRequest { std::vector text_tokens; + // v2 speaker conditioning: wav2vec2bert semantic features of the reference. std::vector speaker_semantic; int64_t speaker_frames = 0; + // v2.5 speaker conditioning: 192-dim CAMPPlus speaker embedding, projected + // by spk_emb_proj inside the prefill graph. + std::vector speaker_style; + // v2.5: row of the GPT lang_embedding table added to every text embedding. + int32_t lang_id = 0; std::vector emotion_semantic; int64_t emotion_frames = 0; std::vector emotion_vector; @@ -125,6 +137,12 @@ struct IndexTTS2GptGenerationRequest { uint32_t seed = 0; }; +// Mirrors the valid_mask filtering in the official v2.5 prepare_gpt_inputs: +// any start/stop text tokens in the segment (including the trailing pad +// appended by the tokenizer) are dropped before the start/stop pair is +// re-added around it. +std::vector align_index_tts2_gpt_text_tokens(const std::vector & text_tokens); + std::shared_ptr load_index_tts2_gpt_weights( const IndexTTS2Assets & assets, ggml_backend_t backend, diff --git a/include/engine/models/index_tts2/request.h b/include/engine/models/index_tts2/request.h index a63d8ca7..8f28737c 100644 --- a/include/engine/models/index_tts2/request.h +++ b/include/engine/models/index_tts2/request.h @@ -5,6 +5,10 @@ namespace engine::models::index_tts2 { +// Normalizes the "lang" request option (v2.5): trims, lowercases, and maps +// "auto" to an empty string (tokenizer-side language inference). +std::string normalize_index_tts2_lang(const std::string & value); + IndexTTS2Request parse_index_tts2_request(const runtime::TaskRequest & request); } // namespace engine::models::index_tts2 diff --git a/include/engine/models/index_tts2/semantic_codec.h b/include/engine/models/index_tts2/semantic_codec.h index 0c3e4c84..dcece494 100644 --- a/include/engine/models/index_tts2/semantic_codec.h +++ b/include/engine/models/index_tts2/semantic_codec.h @@ -41,6 +41,8 @@ struct IndexTTS2SemanticCodecWeights { engine::modules::Conv1dWeights quantizer_out; IndexTTS2VocosBackboneWeights decoder_backbone; engine::modules::LinearWeights decoder_projection; + // v2.5 only: conv applied after the 2x nearest upsample in the decode path. + engine::modules::Conv1dWeights up; }; struct IndexTTS2SemanticCodecOutput { diff --git a/include/engine/models/index_tts2/session.h b/include/engine/models/index_tts2/session.h index 8a2a8db6..5f99a22b 100644 --- a/include/engine/models/index_tts2/session.h +++ b/include/engine/models/index_tts2/session.h @@ -73,6 +73,7 @@ class IndexTTS2Session final const EmotionState & emotion); runtime::AudioBuffer synthesize_segment( const std::vector & text_tokens, + int32_t lang_id, const SpeakerState & speaker, const EmotionState & emotion, const std::vector & emotion_vector, diff --git a/include/engine/models/index_tts2/tokenizer_text.h b/include/engine/models/index_tts2/tokenizer_text.h index a20d144e..4999fcfd 100644 --- a/include/engine/models/index_tts2/tokenizer_text.h +++ b/include/engine/models/index_tts2/tokenizer_text.h @@ -9,39 +9,92 @@ #include #include +namespace llama_tokenizer_vendor { +struct BpeVocabulary; +} // namespace llama_tokenizer_vendor + namespace engine::models::index_tts2 { struct IndexTTS2TextEncoding { + // v2.5: resolved language code; empty for v2. + std::string lang; std::string normalized_text; + // v2 SentencePiece pieces; empty for v2.5. std::vector pieces; std::vector token_ids; + // v2: piece strings per segment; v2.5: processed text per segment. std::vector> segments; std::vector> segment_token_ids; }; +// Variant-aware IndexTTS2 text tokenizer. v2 keeps the SentencePiece bpe.model +// behavior; v2.5 uses the multilingual tiktoken/BPE vocabulary with language +// special tokens and language-id handling. The variant is selected from the +// model config version, not from probing tokenizer files. class IndexTTS2TextTokenizer { public: explicit IndexTTS2TextTokenizer(std::shared_ptr assets); + IndexTTS2Variant variant() const noexcept { + return variant_; + } + std::string normalize_english(const std::string & text) const; std::string normalize_chinese(const std::string & text) const; - std::string normalize_text(const std::string & text) const; + + // v2: SentencePiece encode of the normalized text. + // v2.5: raw tiktoken encode with allowed_special="all"; does not apply any + // text normalization. Special tokens present in the text are recognized + // directly. std::vector encode(const std::string & text) const; + + // v2 only: normalize then tokenize helpers kept for parity/debug. + std::string normalize_text(const std::string & text) const; std::vector tokenize_to_pieces(const std::string & text) const; + + // v2.5 only: returns the id of an exact token text (e.g. "<|zh|>"), or -1 + // when unknown. + int32_t special_token_id(const std::string & token_text) const; + + // v2.5 only: maps a language code to the GPT lang_embedding row, following + // the LANGUAGES order of indextts/utils/tokenizer.py (en=0, zh=1, ...). + // Unknown codes map to "common". + static int32_t lang_to_id(const std::string & lang); + + // v2: normalize -> SentencePiece encode -> segment pieces by token budget. + // v2.5: normalize -> case rules -> pronunciation annotations -> + // special-token name uppercasing -> segment by token budget. Each segment + // is encoded as encode("<|{lang}|> " + segment) plus a trailing pad token + // id 1. When lang is empty, it is inferred (Han -> zh, else en). IndexTTS2TextEncoding encode_for_inference( const std::string & text, - int max_text_tokens_per_segment) const; + int max_text_tokens_per_segment, + const std::string & lang = "") const; private: + // v2 SentencePiece path. + IndexTTS2TextEncoding encode_for_inference_v2( + const std::string & text, + int max_text_tokens_per_segment) const; int32_t piece_to_id(const std::string & piece) const; std::string id_to_piece(int32_t id) const; std::vector> split_segments( const std::vector & pieces, int max_text_tokens_per_segment) const; + // v2.5 tiktoken path. + IndexTTS2TextEncoding encode_for_inference_v2_5( + const std::string & text, + int max_text_tokens_per_segment, + const std::string & lang) const; + std::shared_ptr assets_; + IndexTTS2Variant variant_ = IndexTTS2Variant::kV2; + // v2 SentencePiece model. std::vector pieces_; std::unordered_map piece_to_id_; + // v2.5 tiktoken vocabulary. + std::shared_ptr vocab_; }; } // namespace engine::models::index_tts2 diff --git a/include/engine/models/index_tts2/types.h b/include/engine/models/index_tts2/types.h index a89a1327..0171e776 100644 --- a/include/engine/models/index_tts2/types.h +++ b/include/engine/models/index_tts2/types.h @@ -5,10 +5,28 @@ #include #include #include +#include #include namespace engine::models::index_tts2 { +// Model variant, selected from the model config "version" field ("2.5" -> +// kV2_5, anything else -> kV2). Variant branches must be driven by this value, +// never by probing weight tensor names in hot paths. +enum class IndexTTS2Variant { + kV2, + kV2_5, +}; + +inline IndexTTS2Variant index_tts2_variant_from_version(std::string_view version) { + return version == "2.5" ? IndexTTS2Variant::kV2_5 : IndexTTS2Variant::kV2; +} + +// Rows in the v2.5 GPT lang_embedding table. indextts/utils/tokenizer.py +// defines 106 language codes (including "common"); the checkpoint table has +// one extra unused row. +constexpr int64_t kIndexTTS2LangEmbeddingRows = 107; + struct IndexTTS2GptConfig { int64_t model_dim = 1280; int64_t max_mel_tokens = 1815; @@ -132,6 +150,9 @@ struct IndexTTS2Request { std::string text; std::optional speaker_audio = std::nullopt; std::optional emotion_audio = std::nullopt; + // Text language hint (v2.5 only); empty means auto (zh when the text + // contains Han characters, otherwise en). + std::string lang; float emotion_alpha = 1.0F; std::optional> emotion_vector = std::nullopt; bool use_emotion_text = false; diff --git a/model_specs/index_tts2.json b/model_specs/index_tts2.json index 3f6c737c..72c680eb 100644 --- a/model_specs/index_tts2.json +++ b/model_specs/index_tts2.json @@ -1,7 +1,7 @@ { "family": "index_tts2", "display_name": "IndexTTS2", - "description": "Zero-shot TTS system for Chinese and English speech synthesis with voice cloning, emotion-speaker decoupling, text or audio emotion control, and explicit duration control.", + "description": "Zero-shot TTS system with voice cloning, emotion-speaker decoupling, text or audio emotion control, and explicit duration control. Variant v2 covers Chinese and English; variant v2.5 (config version 2.5) adds multilingual tiktoken tokenization with Japanese, Spanish and Arabic support.", "category": "tts", "status": "supported", "tasks": [ @@ -13,7 +13,10 @@ ], "languages": [ "zh", - "en" + "en", + "ja", + "es", + "ar" ], "capabilities": { "tts": [ @@ -99,6 +102,39 @@ "kind": "huggingface_snapshot", "repo": "mlx-community/index-tts2-mlx" } + }, + { + "id": "index_tts2_5_q8_0", + "display_name": "IndexTTS2.5 Q8_0 GGUF", + "format": "gguf", + "precision": "q8_0", + "target_directory": "IndexTTS2.5-GGUF", + "files": [ + "IndexTTS2.5-GGUF/index-tts2_5-q8_0.gguf" + ], + "strip_prefix": "IndexTTS2.5-GGUF" + }, + { + "id": "index_tts2_5_f16", + "display_name": "IndexTTS2.5 F16 GGUF", + "format": "gguf", + "precision": "f16", + "target_directory": "IndexTTS2.5-GGUF", + "files": [ + "IndexTTS2.5-GGUF/index-tts2_5-f16.gguf" + ], + "strip_prefix": "IndexTTS2.5-GGUF" + }, + { + "id": "index_tts2_5_orig", + "display_name": "IndexTTS2.5 Original-Dtype GGUF", + "format": "gguf", + "precision": "orig", + "target_directory": "IndexTTS2.5-GGUF", + "files": [ + "IndexTTS2.5-GGUF/index-tts2_5-orig.gguf" + ], + "strip_prefix": "IndexTTS2.5-GGUF" } ], "sources": [ @@ -110,7 +146,6 @@ }, "files": { "config": "model:config.yaml", - "bpe": "model:bpe.model", "wav2vec2bert_config": "model:w2v-bert-2.0/config.json", "wav2vec2bert_preprocessor_config": "model:w2v-bert-2.0/preprocessor_config.json", "bigvgan_config": "model:bigvgan/config.json", @@ -121,6 +156,10 @@ "qwen_emotion_vocab": "model:qwen0.6bemo4-merge/vocab.json", "qwen_emotion_merges": "model:qwen0.6bemo4-merge/merges.txt" }, + "optional_files": { + "bpe": "model:bpe.model", + "tiktoken": "model:multilingual_zh_ja_yue_char_del.tiktoken" + }, "tensors": { "gpt": { "source": "weights:", @@ -171,7 +210,6 @@ }, "files": { "config": "model:config.yaml", - "bpe": "model:bpe.model", "wav2vec2bert_config": "model:w2v-bert-2.0/config.json", "wav2vec2bert_preprocessor_config": "model:w2v-bert-2.0/preprocessor_config.json", "bigvgan_config": "model:bigvgan/config.json", @@ -182,6 +220,10 @@ "qwen_emotion_vocab": "model:qwen0.6bemo4-merge/vocab.json", "qwen_emotion_merges": "model:qwen0.6bemo4-merge/merges.txt" }, + "optional_files": { + "bpe": "model:bpe.model", + "tiktoken": "model:multilingual_zh_ja_yue_char_del.tiktoken" + }, "tensors": { "gpt": "model:gpt.safetensors", "s2mel": "model:s2mel.safetensors", diff --git a/src/framework/sampling/torch_random.cpp b/src/framework/sampling/torch_random.cpp index 70b1bb7d..82367d9a 100644 --- a/src/framework/sampling/torch_random.cpp +++ b/src/framework/sampling/torch_random.cpp @@ -8,7 +8,9 @@ #include #include +#include #include +#include #include #include @@ -299,6 +301,48 @@ void log_default_policy(std::string_view category, std::string_view reason) { + "(multiprocessor_count=1, max_threads_per_multiprocessor=256): " + std::string(reason)); } +// ENGINE_TORCH_SAMPLING_POLICY pins the TensorIterator RNG layout instead of +// probing the CUDA device, making the noise realization identical across +// backends (CUDA/HIP/CPU) and machines. Accepted values: "default" (1x256) +// or "x" (e.g. +// "68x1024"). Unset keeps the legacy behavior (device probe on CUDA, default +// layout elsewhere). The pinned layout never uses the CUDA fast path so every +// backend computes the same Philox element mapping on the host. +std::optional pinned_policy_from_env(std::string_view log_category) { + const char * value = std::getenv("ENGINE_TORCH_SAMPLING_POLICY"); + if (value == nullptr || *value == '\0') { + return std::nullopt; + } + TorchCudaSamplingPolicy policy; + std::string text(value); + if (text != "default") { + const auto cross = text.find('x'); + if (cross == std::string::npos) { + throw std::runtime_error( + "ENGINE_TORCH_SAMPLING_POLICY must be \"default\" or \"x\", got: " + text); + } + try { + policy.multiprocessor_count = std::stoll(text.substr(0, cross)); + policy.max_threads_per_multiprocessor = std::stoll(text.substr(cross + 1)); + } catch (const std::exception &) { + throw std::runtime_error( + "ENGINE_TORCH_SAMPLING_POLICY must be \"default\" or \"x\", got: " + text); + } + if (policy.multiprocessor_count <= 0 || policy.max_threads_per_multiprocessor <= 0) { + throw std::runtime_error("ENGINE_TORCH_SAMPLING_POLICY values must be positive: " + text); + } + } + policy.cuda_fast_path = false; + engine::debug::log_message( + engine::debug::LogLevel::Warning, + log_category, + "using pinned Torch RNG layout policy from ENGINE_TORCH_SAMPLING_POLICY " + "(multiprocessor_count=" + std::to_string(policy.multiprocessor_count) + + ", max_threads_per_multiprocessor=" + std::to_string(policy.max_threads_per_multiprocessor) + + ")"); + return policy; +} + } // namespace TorchCudaSamplingPolicy resolve_torch_cuda_sampling_policy( @@ -307,6 +351,9 @@ TorchCudaSamplingPolicy resolve_torch_cuda_sampling_policy( std::string_view log_category, std::string_view model_name, TorchCudaSamplingPolicyFailureMode failure_mode) { + if (const auto pinned = pinned_policy_from_env(log_category)) { + return *pinned; + } TorchCudaSamplingPolicy policy; if (backend_type != engine::core::BackendType::Cuda) { log_default_policy(log_category, "backend is not CUDA"); diff --git a/src/framework/text/text_normalization.cpp b/src/framework/text/text_normalization.cpp index 8c5c494e..be6b7512 100644 --- a/src/framework/text/text_normalization.cpp +++ b/src/framework/text/text_normalization.cpp @@ -462,6 +462,13 @@ std::string normalize_english_numbers(std::string text) { [](const std::smatch & match) { return english_ordinal_from_digits(match[1].str()); }); + // Split letter<->digit boundaries so digits attached to letters verbalize + // like the official wetext English normalizer ("DS4" -> "DS four", + // "R2D2" -> "R two D two", "4K" -> "four K"). Runs after dates/decimals/ + // ordinals so "1st", "2.5" and friends have already been expanded; the + // standalone cardinal rule below then spells out each digit group. + text = std::regex_replace(std::move(text), std::regex(R"(([A-Za-z])(\d))"), "$1 $2"); + text = std::regex_replace(std::move(text), std::regex(R"((\d)([A-Za-z]))"), "$1 $2"); text = normalize_english_regex( std::move(text), std::regex(R"(\b(\d+)\b)"), @@ -482,6 +489,27 @@ std::string normalize_english_text(std::string_view text, const EnglishTextNorma if (options.spell_numbers) { out = normalize_english_numbers(std::move(out)); } + if (options.verbalize_symbols) { + // Match the official wetext English normalizer: standalone ASCII + // symbols are verbalized ("a_b" -> "a underscore b", + // "C++" -> "C plus plus", "a=b" -> "a equal sign b"). Runs after + // number spelling so "50%" has already become "fifty percent". + const std::pair symbol_words[] = { + {"_", " underscore "}, + {"+", " plus "}, + {"=", " equal sign "}, + {"*", " asterisk "}, + {"&", " and "}, + {"#", " hash "}, + {"%", " percent "}, + {"|", " vertical bar "}, + {"~", " tilde "}, + }; + for (const auto & [from, to] : symbol_words) { + out = replace_all(std::move(out), from, to); + } + out = collapse_ascii_whitespace(out); + } if (options.index_tts_punctuation) { out = apply_index_tts_punctuation_map(std::move(out)); } diff --git a/src/models/index_tts2/assets.cpp b/src/models/index_tts2/assets.cpp index 35ad21ea..9a5b4ac5 100644 --- a/src/models/index_tts2/assets.cpp +++ b/src/models/index_tts2/assets.cpp @@ -17,13 +17,27 @@ IndexTTS2Config parse_config(const assets::ResourceBundle & resources) { const auto document = resources.parse_flattened_yaml("config"); IndexTTS2Config config; config.version = yaml::optional_string(document, "version", config.version); - config.dataset_sample_rate = static_cast(yaml::require_i64(document, "dataset.sample_rate")); + // The official IndexTTS-2.5 config has no dataset section; these values are + // parsed for compatibility but not used at inference time. + if (const auto value = yaml::optional_int(document, "dataset.sample_rate")) { + config.dataset_sample_rate = *value; + } config.dataset_squeeze = yaml::optional_bool(document, "dataset.squeeze", config.dataset_squeeze); - config.dataset_mel_sample_rate = static_cast(yaml::require_i64(document, "dataset.mel.sample_rate")); - config.dataset_mel_n_fft = yaml::require_i64(document, "dataset.mel.n_fft"); - config.dataset_mel_hop_length = yaml::require_i64(document, "dataset.mel.hop_length"); - config.dataset_mel_win_length = yaml::require_i64(document, "dataset.mel.win_length"); - config.dataset_mel_n_mels = yaml::require_i64(document, "dataset.mel.n_mels"); + if (const auto value = yaml::optional_int(document, "dataset.mel.sample_rate")) { + config.dataset_mel_sample_rate = *value; + } + if (const auto value = yaml::optional_int(document, "dataset.mel.n_fft")) { + config.dataset_mel_n_fft = *value; + } + if (const auto value = yaml::optional_int(document, "dataset.mel.hop_length")) { + config.dataset_mel_hop_length = *value; + } + if (const auto value = yaml::optional_int(document, "dataset.mel.win_length")) { + config.dataset_mel_win_length = *value; + } + if (const auto value = yaml::optional_int(document, "dataset.mel.n_mels")) { + config.dataset_mel_n_mels = *value; + } config.dataset_mel_fmin = yaml::optional_f32(document, "dataset.mel.mel_fmin", config.dataset_mel_fmin); config.dataset_mel_normalize = yaml::optional_bool(document, "dataset.mel.normalize", config.dataset_mel_normalize); @@ -154,7 +168,14 @@ void validate_gpt_weights(const IndexTTS2Config & config, const assets::TensorSo assets::require_tensor_shape(source, "gpt.h.0.attn.c_proj.weight", {config.gpt.model_dim, config.gpt.model_dim}); assets::require_tensor_shape(source, "gpt.h.0.mlp.c_fc.weight", {config.gpt.model_dim, config.gpt.model_dim * 4}); assets::require_tensor_shape(source, "gpt.h.0.mlp.c_proj.weight", {config.gpt.model_dim * 4, config.gpt.model_dim}); - assets::require_tensor_shape(source, "conditioning_encoder.after_norm.weight", {config.gpt.condition_output_size}); + if (index_tts2_variant_from_version(config.version) == IndexTTS2Variant::kV2_5) { + // v2.5 campplus speaker conditioning: projected CAMPPlus embedding plus + // the language embedding table; no conditioning_encoder/speed_emb. + assets::require_tensor_shape(source, "spk_emb_proj.weight", {config.gpt.model_dim, config.s2mel.style_dim}); + assets::require_tensor_shape(source, "lang_embedding.weight", {kIndexTTS2LangEmbeddingRows, config.gpt.model_dim}); + } else { + assets::require_tensor_shape(source, "conditioning_encoder.after_norm.weight", {config.gpt.condition_output_size}); + } assets::require_tensor_shape(source, "emo_conditioning_encoder.after_norm.weight", {config.gpt.emo_condition_output_size}); } @@ -195,6 +216,11 @@ void validate_semantic_codec_weights(const IndexTTS2Config & config, const asset assets::require_tensor_shape(source, "quantizer.quantizers.0.codebook.weight", {config.semantic_codec.codebook_size, config.semantic_codec.codebook_dim}); assets::require_tensor_shape(source, "encoder.1.weight", {config.semantic_codec.hidden_size, config.semantic_codec.vocos_dim}); assets::require_tensor_shape(source, "decoder.1.weight", {config.semantic_codec.hidden_size, config.semantic_codec.vocos_dim}); + if (index_tts2_variant_from_version(config.version) == IndexTTS2Variant::kV2_5) { + // v2.5 decodes codes through the full EnhancedCodec path ending in the + // 2x nearest upsample and the `up` conv. + assets::require_tensor_shape(source, "up.weight", {config.semantic_codec.hidden_size, config.semantic_codec.hidden_size, 3}); + } } void validate_qwen_weights(const assets::TensorSource & source) { diff --git a/src/models/index_tts2/gpt.cpp b/src/models/index_tts2/gpt.cpp index 45e885d9..2647efac 100644 --- a/src/models/index_tts2/gpt.cpp +++ b/src/models/index_tts2/gpt.cpp @@ -39,7 +39,6 @@ constexpr int64_t kSpeakerConditionLayers = 6; constexpr int64_t kEmotionConditionLayers = 4; constexpr int64_t kGptLayers = 24; constexpr int64_t kGptMlpDim = 5120; -constexpr int64_t kTextTokens = 12001; constexpr int64_t kMelCodes = 8194; constexpr int64_t kMelPositions = 1818; constexpr int64_t kTextPositions = 602; @@ -47,12 +46,22 @@ constexpr int64_t kConditionPosFrames = 5000; constexpr int64_t kConditionConvKernel = 15; constexpr int64_t kGptHeads = 20; constexpr int64_t kGptHeadDim = kModelDim / kGptHeads; -constexpr int64_t kConditionTokens = 34; +// v2: 32 perceiver speaker tokens + two speed embedding tokens. +constexpr int64_t kV2ConditionTokens = 34; +constexpr int64_t kV2TextTokens = 12001; +// v2.5 spk_cond_mode="campplus": the projected 192-dim CAMPPlus embedding +// forms a single speaker token, followed by two all-zero tokens. +constexpr int64_t kCampplusStyleDim = 192; +constexpr int64_t kV2_5ConditionTokens = 3; constexpr int32_t kStartTextToken = 0; constexpr int32_t kStopTextToken = 1; constexpr int32_t kStartMelToken = 8192; constexpr int32_t kStopMelToken = 8193; +int64_t gpt_text_vocab_size(const IndexTTS2Config & config) { + return config.gpt.number_text_tokens + 1; +} + struct GgmlContextDeleter { void operator()(ggml_context * ctx) const noexcept { if (ctx != nullptr) { @@ -920,15 +929,20 @@ std::shared_ptr load_index_tts2_gpt_weights( weight_context_bytes); const auto & source = *assets.gpt_weights; - weights->speaker_conditioner = load_condition_encoder( - *weights->store, - source, - "conditioning_encoder", - kSpeakerConditionLayers, - 2048, - 8, - matmul_storage_type, - conv_storage_type); + const bool campplus_conditioning = + index_tts2_variant_from_version(assets.config.version) == IndexTTS2Variant::kV2_5; + const int64_t text_vocab = gpt_text_vocab_size(assets.config); + if (!campplus_conditioning) { + weights->speaker_conditioner = load_condition_encoder( + *weights->store, + source, + "conditioning_encoder", + kSpeakerConditionLayers, + 2048, + 8, + matmul_storage_type, + conv_storage_type); + } weights->emotion_conditioner = load_condition_encoder( *weights->store, source, @@ -938,16 +952,18 @@ std::shared_ptr load_index_tts2_gpt_weights( 4, matmul_storage_type, conv_storage_type); - weights->speaker_perceiver = load_perceiver( - *weights->store, - source, - "perceiver_encoder", - 32, - kModelDim, - kConditionDim, - 512, - 3412, - matmul_storage_type); + if (!campplus_conditioning) { + weights->speaker_perceiver = load_perceiver( + *weights->store, + source, + "perceiver_encoder", + 32, + kModelDim, + kConditionDim, + 512, + 3412, + matmul_storage_type); + } weights->emotion_perceiver = load_perceiver( *weights->store, source, @@ -962,7 +978,7 @@ std::shared_ptr load_index_tts2_gpt_weights( source, "text_embedding.weight", matmul_storage_type, - {kTextTokens, kModelDim}); + {text_vocab, kModelDim}); weights->mel_embedding = weights->store->load_tensor( source, "mel_embedding.weight", @@ -992,7 +1008,23 @@ std::shared_ptr load_index_tts2_gpt_weights( kModelDim, kModelDim, true); - weights->speed_embedding_values = source.require_f32("speed_emb.weight", {2, kModelDim}); + if (campplus_conditioning) { + weights->spk_emb_proj = binding::linear_from_source( + *weights->store, + source, + "spk_emb_proj", + matmul_storage_type, + kModelDim, + kCampplusStyleDim, + true); + weights->lang_embedding = weights->store->load_tensor( + source, + "lang_embedding.weight", + matmul_storage_type, + {kIndexTTS2LangEmbeddingRows, kModelDim}); + } else { + weights->speed_embedding_values = source.require_f32("speed_emb.weight", {2, kModelDim}); + } weights->gpt_layers.reserve(static_cast(kGptLayers)); for (int64_t i = 0; i < kGptLayers; ++i) { weights->gpt_layers.push_back(load_gpt2_layer(*weights->store, source, i, matmul_storage_type)); @@ -1012,7 +1044,7 @@ std::shared_ptr load_index_tts2_gpt_weights( source, "text_head", matmul_storage_type, - kTextTokens, + text_vocab, kModelDim, true); @@ -1298,13 +1330,18 @@ class IndexTTS2GptRuntime::PrefillGraph { core::ExecutionContext & execution, std::shared_ptr weights, int64_t text_tokens, + bool campplus_conditioning, + int64_t text_vocab, size_t graph_arena_bytes) : execution_(execution), weights_(std::move(weights)), + campplus_conditioning_(campplus_conditioning), + condition_tokens_(campplus_conditioning ? kV2_5ConditionTokens : kV2ConditionTokens), + text_vocab_(text_vocab), text_tokens_(text_tokens), text_steps_(text_tokens + 2), - prompt_steps_(kConditionTokens + text_tokens + 3) { - if (weights_ == nullptr || text_tokens_ <= 0) { + prompt_steps_(condition_tokens_ + text_tokens + 3) { + if (weights_ == nullptr || text_tokens_ < 0 || (!campplus_conditioning_ && text_tokens_ == 0)) { throw std::runtime_error("IndexTTS2 GPT prefill graph requires weights and text tokens"); } const auto build_start = Clock::now(); @@ -1332,17 +1369,43 @@ class IndexTTS2GptRuntime::PrefillGraph { output_ctx_.get(), "index_tts2.gpt.prefill.outputs", execution_.backend_type()}; - conds_ = core::make_tensor(input_ctx, GGML_TYPE_F32, core::TensorShape::from_dims({1, kConditionTokens, kModelDim})).tensor; + core::TensorValue conds; + if (campplus_conditioning_) { + style_ = core::make_tensor(input_ctx, GGML_TYPE_F32, core::TensorShape::from_dims({1, kCampplusStyleDim})).tensor; + emo_vec_ = core::make_tensor(input_ctx, GGML_TYPE_F32, core::TensorShape::from_dims({1, kModelDim})).tensor; + lang_id_ = ggml_new_tensor_1d(input_ctx_.get(), GGML_TYPE_I32, 1); + ggml_set_input(style_); + ggml_set_input(emo_vec_); + ggml_set_input(lang_id_); + } else { + conds_ = core::make_tensor(input_ctx, GGML_TYPE_F32, core::TensorShape::from_dims({1, condition_tokens_, kModelDim})).tensor; + ggml_set_input(conds_); + conds = core::wrap_tensor(conds_, core::TensorShape::from_dims({1, condition_tokens_, kModelDim}), GGML_TYPE_F32); + } text_ids_ = ggml_new_tensor_1d(input_ctx_.get(), GGML_TYPE_I32, text_steps_); start_mel_id_ = ggml_new_tensor_1d(input_ctx_.get(), GGML_TYPE_I32, 1); - ggml_set_input(conds_); ggml_set_input(text_ids_); ggml_set_input(start_mel_id_); - auto conds = core::wrap_tensor(conds_, core::TensorShape::from_dims({1, kConditionTokens, kModelDim}), GGML_TYPE_F32); auto text_ids = core::wrap_tensor(text_ids_, core::TensorShape::from_dims({text_steps_}), GGML_TYPE_I32); - auto text = modules::EmbeddingModule({kTextTokens, kModelDim}).build(ctx, text_ids, weights_->text_embedding); + auto text = modules::EmbeddingModule({text_vocab_, kModelDim}).build(ctx, text_ids, weights_->text_embedding); auto text_pos = modules::SliceModule({0, 0, text_steps_}).build(ctx, weights_->text_pos_embedding); text = modules::AddModule{}.build(ctx, text, text_pos); + if (campplus_conditioning_) { + // v2.5 campplus conditioning prefix (model_v2.py inference_speech): + // conds = [spk_emb_proj(style) + emo_vec, zeros, zeros]. + auto style = core::wrap_tensor(style_, core::TensorShape::from_dims({1, kCampplusStyleDim}), GGML_TYPE_F32); + auto speaker_token = build_biased_gpt_projection(ctx, style, kCampplusStyleDim, kModelDim, weights_->spk_emb_proj); + speaker_token = core::reshape_tensor(ctx, speaker_token, core::TensorShape::from_dims({1, 1, kModelDim})); + auto emo_vec = core::wrap_tensor(emo_vec_, core::TensorShape::from_dims({1, kModelDim}), GGML_TYPE_F32); + emo_vec = core::reshape_tensor(ctx, emo_vec, core::TensorShape::from_dims({1, 1, kModelDim})); + conds = modules::AddModule{}.build(ctx, speaker_token, emo_vec); + auto zero_token = modules::RepeatModule({core::TensorShape::from_dims({1, condition_tokens_ - 1, kModelDim})}) + .build(ctx, scale(ctx, conds, 0.0F)); + conds = modules::ConcatModule({1}).build(ctx, conds, zero_token); + auto lang_id = core::wrap_tensor(lang_id_, core::TensorShape::from_dims({1}), GGML_TYPE_I32); + auto lang = modules::EmbeddingModule({kIndexTTS2LangEmbeddingRows, kModelDim}).build(ctx, lang_id, weights_->lang_embedding); + text = modules::AddModule{}.build(ctx, text, modules::RepeatModule({text.shape}).build(ctx, lang)); + } text = core::reshape_tensor(ctx, core::ensure_backend_addressable_layout(ctx, text), core::TensorShape::from_dims({1, text_steps_, kModelDim})); auto mel_id = core::wrap_tensor(start_mel_id_, core::TensorShape::from_dims({1}), GGML_TYPE_I32); auto mel = modules::EmbeddingModule({kMelCodes, kModelDim}).build(ctx, mel_id, weights_->mel_embedding); @@ -1412,17 +1475,48 @@ class IndexTTS2GptRuntime::PrefillGraph { } GptPrefillOutput run(const std::vector & conds, const std::vector & text_tokens) { - if (static_cast(conds.size()) != kConditionTokens * kModelDim || + if (campplus_conditioning_) { + throw std::runtime_error("IndexTTS2 GPT prefill conds input requires the v2 speaker-conditioning mode"); + } + if (static_cast(conds.size()) != condition_tokens_ * kModelDim || + static_cast(text_tokens.size()) != text_tokens_) { + throw std::runtime_error("IndexTTS2 GPT prefill input shape mismatch"); + } + auto timing_start = Clock::now(); + ggml_backend_tensor_set(conds_, conds.data(), 0, conds.size() * sizeof(float)); + return run_with_text_tokens(text_tokens, timing_start); + } + + GptPrefillOutput run( + const std::vector & speaker_style, + const std::vector & emotion_vector, + int32_t lang_id, + const std::vector & text_tokens) { + if (!campplus_conditioning_) { + throw std::runtime_error("IndexTTS2 GPT prefill style/lang inputs require the v2.5 campplus speaker-conditioning mode"); + } + if (static_cast(speaker_style.size()) != kCampplusStyleDim || + static_cast(emotion_vector.size()) != kModelDim || static_cast(text_tokens.size()) != text_tokens_) { throw std::runtime_error("IndexTTS2 GPT prefill input shape mismatch"); } + if (lang_id < 0 || lang_id >= kIndexTTS2LangEmbeddingRows) { + throw std::runtime_error("IndexTTS2 GPT prefill lang id is out of range"); + } + auto timing_start = Clock::now(); + ggml_backend_tensor_set(style_, speaker_style.data(), 0, speaker_style.size() * sizeof(float)); + ggml_backend_tensor_set(emo_vec_, emotion_vector.data(), 0, emotion_vector.size() * sizeof(float)); + ggml_backend_tensor_set(lang_id_, &lang_id, 0, sizeof(int32_t)); + return run_with_text_tokens(text_tokens, timing_start); + } + +private: + GptPrefillOutput run_with_text_tokens(const std::vector & text_tokens, Clock::time_point timing_start) { std::vector ids; ids.reserve(static_cast(text_steps_)); ids.push_back(kStartTextToken); ids.insert(ids.end(), text_tokens.begin(), text_tokens.end()); ids.push_back(kStopTextToken); - auto timing_start = Clock::now(); - ggml_backend_tensor_set(conds_, conds.data(), 0, conds.size() * sizeof(float)); ggml_backend_tensor_set(text_ids_, ids.data(), 0, ids.size() * sizeof(int32_t)); debug::timing_log_scalar("index_tts2.gpt.prefill.input_upload_ms", engine::debug::elapsed_ms(timing_start, Clock::now())); core::set_backend_threads(execution_.backend(), execution_.config().threads); @@ -1452,7 +1546,6 @@ class IndexTTS2GptRuntime::PrefillGraph { return out; } -private: void clear_graph() { if (graph_ != nullptr) { core::release_backend_graph_resources(execution_.backend(), graph_); @@ -1474,6 +1567,9 @@ class IndexTTS2GptRuntime::PrefillGraph { core::ExecutionContext & execution_; std::shared_ptr weights_; + bool campplus_conditioning_ = false; + int64_t condition_tokens_ = kV2ConditionTokens; + int64_t text_vocab_ = 0; int64_t text_tokens_ = 0; int64_t text_steps_ = 0; int64_t prompt_steps_ = 0; @@ -1481,6 +1577,9 @@ class IndexTTS2GptRuntime::PrefillGraph { std::unique_ptr output_ctx_; std::unique_ptr ctx_; ggml_tensor * conds_ = nullptr; + ggml_tensor * style_ = nullptr; + ggml_tensor * emo_vec_ = nullptr; + ggml_tensor * lang_id_ = nullptr; ggml_tensor * text_ids_ = nullptr; ggml_tensor * start_mel_id_ = nullptr; ggml_tensor * latent_ = nullptr; @@ -1526,16 +1625,16 @@ class IndexTTS2GptRuntime::ForwardGraph { input_ctx_.get(), "index_tts2.gpt.forward.inputs", execution_.backend_type()}; - conds_ = core::make_tensor(input_ctx, GGML_TYPE_F32, core::TensorShape::from_dims({1, kConditionTokens, kModelDim})).tensor; + conds_ = core::make_tensor(input_ctx, GGML_TYPE_F32, core::TensorShape::from_dims({1, kV2ConditionTokens, kModelDim})).tensor; text_ids_ = ggml_new_tensor_1d(input_ctx_.get(), GGML_TYPE_I32, text_steps_); mel_ids_ = ggml_new_tensor_1d(input_ctx_.get(), GGML_TYPE_I32, mel_steps_); ggml_set_input(conds_); ggml_set_input(text_ids_); ggml_set_input(mel_ids_); - auto conds = core::wrap_tensor(conds_, core::TensorShape::from_dims({1, kConditionTokens, kModelDim}), GGML_TYPE_F32); + auto conds = core::wrap_tensor(conds_, core::TensorShape::from_dims({1, kV2ConditionTokens, kModelDim}), GGML_TYPE_F32); auto text_ids = core::wrap_tensor(text_ids_, core::TensorShape::from_dims({text_steps_}), GGML_TYPE_I32); - auto text = modules::EmbeddingModule({kTextTokens, kModelDim}).build(ctx, text_ids, weights_->text_embedding); + auto text = modules::EmbeddingModule({kV2TextTokens, kModelDim}).build(ctx, text_ids, weights_->text_embedding); auto text_pos = modules::SliceModule({0, 0, text_steps_}).build(ctx, weights_->text_pos_embedding); text = modules::AddModule{}.build(ctx, text, text_pos); text = core::reshape_tensor(ctx, core::ensure_backend_addressable_layout(ctx, text), core::TensorShape::from_dims({1, text_steps_, kModelDim})); @@ -1553,14 +1652,14 @@ class IndexTTS2GptRuntime::ForwardGraph { x = out.output; } x = modules::LayerNormModule({kModelDim, 1.0e-5F, true, true}).build(ctx, x, weights_->gpt_final_norm); - x = modules::SliceModule({1, kConditionTokens + text_steps_, code_count_}).build(ctx, x); + x = modules::SliceModule({1, kV2ConditionTokens + text_steps_, code_count_}).build(ctx, x); x = modules::LayerNormModule({kModelDim, 1.0e-5F, true, true}).build(ctx, x, weights_->final_norm); output_ = core::ensure_backend_addressable_layout(ctx, x).tensor; ggml_set_output(output_); graph_ = ggml_new_graph_custom( ctx_.get(), - static_cast(std::max(65536, (kConditionTokens + text_steps_ + mel_steps_) * 8192)), + static_cast(std::max(65536, (kV2ConditionTokens + text_steps_ + mel_steps_) * 8192)), false); ggml_build_forward_expand(graph_, output_); input_buffer_ = ggml_backend_alloc_ctx_tensors(input_ctx_.get(), execution_.backend()); @@ -1591,7 +1690,7 @@ class IndexTTS2GptRuntime::ForwardGraph { const std::vector & conds, const std::vector & text_tokens, const std::vector & codes) { - if (static_cast(conds.size()) != kConditionTokens * kModelDim || + if (static_cast(conds.size()) != kV2ConditionTokens * kModelDim || static_cast(text_tokens.size()) != text_tokens_ || static_cast(codes.size()) != code_count_) { throw std::runtime_error("IndexTTS2 GPT forward graph input shape mismatch"); @@ -2022,6 +2121,9 @@ void IndexTTS2GptRuntime::prepare_speaker_conditioning(int64_t frames) { if (execution_ == nullptr) { throw std::runtime_error("IndexTTS2 GPT runtime execution context is missing"); } + if (index_tts2_variant_from_version(assets_->config.version) != IndexTTS2Variant::kV2) { + throw std::runtime_error("IndexTTS2 GPT speaker conditioning is only used by the v2 variant; v2.5 uses campplus conditioning inside the prefill graph"); + } if (frames <= 0) { throw std::runtime_error("IndexTTS2 GPT speaker conditioning prepare requires positive frames"); } @@ -2064,7 +2166,9 @@ void IndexTTS2GptRuntime::prepare_generation(int64_t text_tokens, int64_t max_me if (execution_ == nullptr) { throw std::runtime_error("IndexTTS2 GPT runtime execution context is missing"); } - if (text_tokens <= 0 || max_mel_tokens <= 0) { + const bool campplus_conditioning = + index_tts2_variant_from_version(assets_->config.version) == IndexTTS2Variant::kV2_5; + if (text_tokens < 0 || (!campplus_conditioning && text_tokens == 0) || max_mel_tokens <= 0) { throw std::runtime_error("IndexTTS2 GPT generation prepare requires positive lengths"); } if (num_beams != 1) { @@ -2072,7 +2176,13 @@ void IndexTTS2GptRuntime::prepare_generation(int64_t text_tokens, int64_t max_me } if (prefill_graph_ == nullptr || !prefill_graph_->matches(text_tokens)) { prefill_graph_.reset(); - prefill_graph_ = std::make_unique(*execution_, weights_, text_tokens, graph_arena_bytes_); + prefill_graph_ = std::make_unique( + *execution_, + weights_, + text_tokens, + campplus_conditioning, + gpt_text_vocab_size(assets_->config), + graph_arena_bytes_); } const int64_t required_cache_steps = prefill_graph_->prompt_steps() + max_mel_tokens + 1; const int64_t required_beam_slots = 2 * std::max(1, num_beams); @@ -2113,11 +2223,28 @@ std::vector IndexTTS2GptRuntime::merge_emotion_vector( } IndexTTS2GptGeneration IndexTTS2GptRuntime::generate_speech(const IndexTTS2GptGenerationRequest & request) { - if (request.text_tokens.empty()) { - throw std::runtime_error("IndexTTS2 GPT generation requires text tokens"); + const bool campplus_conditioning = + index_tts2_variant_from_version(assets_->config.version) == IndexTTS2Variant::kV2_5; + std::vector text_tokens = request.text_tokens; + IndexTTS2GptLatent speech_conditioning; + if (campplus_conditioning) { + // v2.5: drop embedded start/stop text tokens (mirrors the valid_mask + // filtering in the official prepare_gpt_inputs); the speaker token is + // built inside the prefill graph from the CAMPPlus style embedding. + text_tokens = align_index_tts2_gpt_text_tokens(request.text_tokens); + if (static_cast(request.speaker_style.size()) != kCampplusStyleDim) { + throw std::runtime_error("IndexTTS2 GPT generation speaker style shape mismatch"); + } + if (request.lang_id < 0 || request.lang_id >= kIndexTTS2LangEmbeddingRows) { + throw std::runtime_error("IndexTTS2 GPT generation lang id is out of range"); + } + } else { + if (request.text_tokens.empty()) { + throw std::runtime_error("IndexTTS2 GPT generation requires text tokens"); + } + prepare_speaker_conditioning(request.speaker_frames); + speech_conditioning = speaker_conditioning(request.speaker_semantic, request.speaker_frames); } - prepare_speaker_conditioning(request.speaker_frames); - const auto speech_conditioning = speaker_conditioning(request.speaker_semantic, request.speaker_frames); std::vector emotion_vector = request.emotion_vector; if (emotion_vector.empty()) { prepare_emotion_conditioning(request.emotion_frames); @@ -2127,21 +2254,26 @@ IndexTTS2GptGeneration IndexTTS2GptRuntime::generate_speech(const IndexTTS2GptGe throw std::runtime_error("IndexTTS2 GPT generation emotion vector shape mismatch"); } prepare_generation( - static_cast(request.text_tokens.size()), + static_cast(text_tokens.size()), request.max_mel_tokens, request.num_beams); - std::vector conds(static_cast(kConditionTokens * kModelDim), 0.0F); - for (int64_t token = 0; token < 32; ++token) { - for (int64_t dim = 0; dim < kModelDim; ++dim) { - conds[static_cast(token * kModelDim + dim)] = - speech_conditioning.values[static_cast(token * kModelDim + dim)] + - emotion_vector[static_cast(dim)]; + GptPrefillOutput prefill; + if (campplus_conditioning) { + prefill = prefill_graph_->run(request.speaker_style, emotion_vector, request.lang_id, text_tokens); + } else { + std::vector conds(static_cast(kV2ConditionTokens * kModelDim), 0.0F); + for (int64_t token = 0; token < 32; ++token) { + for (int64_t dim = 0; dim < kModelDim; ++dim) { + conds[static_cast(token * kModelDim + dim)] = + speech_conditioning.values[static_cast(token * kModelDim + dim)] + + emotion_vector[static_cast(dim)]; + } } + const auto & speed = weights_->speed_embedding_values; + std::copy_n(speed.data() + static_cast(kModelDim), static_cast(kModelDim), conds.data() + static_cast(32 * kModelDim)); + std::copy_n(speed.data(), static_cast(kModelDim), conds.data() + static_cast(33 * kModelDim)); + prefill = prefill_graph_->run(conds, request.text_tokens); } - const auto & speed = weights_->speed_embedding_values; - std::copy_n(speed.data() + static_cast(kModelDim), static_cast(kModelDim), conds.data() + static_cast(32 * kModelDim)); - std::copy_n(speed.data(), static_cast(kModelDim), conds.data() + static_cast(33 * kModelDim)); - auto prefill = prefill_graph_->run(conds, request.text_tokens); const auto sampling_policy = engine::sampling::resolve_torch_cuda_sampling_policy( execution_->backend_type(), execution_->config().device, @@ -2400,6 +2532,9 @@ IndexTTS2GptLatent IndexTTS2GptRuntime::forward_latent( const std::vector & emotion_semantic, int64_t emotion_frames, const std::vector & emotion_vector) { + if (index_tts2_variant_from_version(assets_->config.version) != IndexTTS2Variant::kV2) { + throw std::runtime_error("IndexTTS2 GPT latent forward is only used by the v2 variant; v2.5 decodes codes through the semantic codec"); + } if (speech_conditioning_latent.frames != 32 || speech_conditioning_latent.dims != kModelDim || static_cast(speech_conditioning_latent.values.size()) != 32 * kModelDim) { @@ -2416,7 +2551,7 @@ IndexTTS2GptLatent IndexTTS2GptRuntime::forward_latent( if (static_cast(emo.size()) != kModelDim) { throw std::runtime_error("IndexTTS2 GPT latent forward emotion vector shape mismatch"); } - std::vector conds(static_cast(kConditionTokens * kModelDim), 0.0F); + std::vector conds(static_cast(kV2ConditionTokens * kModelDim), 0.0F); for (int64_t token = 0; token < 32; ++token) { for (int64_t dim = 0; dim < kModelDim; ++dim) { conds[static_cast(token * kModelDim + dim)] = @@ -2449,4 +2584,16 @@ void IndexTTS2GptRuntime::release_generation_graphs() { forward_graph_.reset(); } +std::vector align_index_tts2_gpt_text_tokens(const std::vector & text_tokens) { + std::vector out; + out.reserve(text_tokens.size()); + for (const int32_t token : text_tokens) { + if (token == kStartTextToken || token == kStopTextToken) { + continue; + } + out.push_back(token); + } + return out; +} + } // namespace engine::models::index_tts2 diff --git a/src/models/index_tts2/loader.cpp b/src/models/index_tts2/loader.cpp index f99f3915..cf45822b 100644 --- a/src/models/index_tts2/loader.cpp +++ b/src/models/index_tts2/loader.cpp @@ -13,11 +13,13 @@ runtime::ModelMetadata metadata(const IndexTTS2Assets & assets) { runtime::ModelMetadata out; out.family = "index_tts2"; out.variant = assets.config.version; - out.description = "IndexTTS2 loaded from local extracted assets."; + out.description = index_tts2_variant_from_version(assets.config.version) == IndexTTS2Variant::kV2_5 + ? "IndexTTS2.5 (index_tts2 family variant) loaded from local extracted assets." + : "IndexTTS2 loaded from local extracted assets."; return out; } -runtime::CapabilitySet capabilities(const IndexTTS2Assets &) { +runtime::CapabilitySet capabilities(const IndexTTS2Assets & assets) { runtime::CapabilitySet out; out.supported_tasks = { {runtime::VoiceTaskKind::Tts, {runtime::RunMode::Offline}}, @@ -25,11 +27,15 @@ runtime::CapabilitySet capabilities(const IndexTTS2Assets &) { }; out.supports_speaker_reference = true; out.supports_style_condition = true; - out.languages = {"English", "Chinese"}; + if (index_tts2_variant_from_version(assets.config.version) == IndexTTS2Variant::kV2_5) { + out.languages = {"Chinese", "English", "Japanese", "Spanish", "Arabic"}; + } else { + out.languages = {"English", "Chinese"}; + } return out; } -runtime::ModelCliInterface cli(const IndexTTS2Assets &) { +runtime::ModelCliInterface cli(const IndexTTS2Assets & assets) { runtime::ModelCliInterface out; out.request_options = { {"emotion_alpha", "float", "Blend strength for explicit emotion conditioning."}, @@ -42,6 +48,10 @@ runtime::ModelCliInterface cli(const IndexTTS2Assets &) { {"length_penalty", "float", "GPT beam-search length penalty."}, {"num_beams", "n", "GPT beam count."}, }; + if (index_tts2_variant_from_version(assets.config.version) == IndexTTS2Variant::kV2_5) { + out.request_options.push_back( + {"lang", "auto|zh|en|ja|es|ar|...", "Text language hint; auto infers zh when the text contains Han characters, otherwise en."}); + } out.session_options = { {"index_tts2.weight_type", "native|f32|f16|bf16|q8_0", "Matmul weight storage type."}, {"index_tts2.conv_weight_type", "native|f32|f16", "Convolution weight storage type."}, diff --git a/src/models/index_tts2/request.cpp b/src/models/index_tts2/request.cpp index 25dfda23..abf9af22 100644 --- a/src/models/index_tts2/request.cpp +++ b/src/models/index_tts2/request.cpp @@ -3,6 +3,8 @@ #include "engine/framework/io/text.h" #include "engine/framework/runtime/options.h" +#include +#include #include #include #include @@ -57,6 +59,17 @@ void require_valid_audio(const runtime::AudioBuffer & audio, const char * label) } // namespace +std::string normalize_index_tts2_lang(const std::string & value) { + std::string lang = engine::io::trim_ascii_whitespace(value); + std::transform(lang.begin(), lang.end(), lang.begin(), [](unsigned char ch) { + return static_cast(std::tolower(ch)); + }); + if (lang == "auto") { + lang.clear(); + } + return lang; +} + IndexTTS2Request parse_index_tts2_request(const runtime::TaskRequest & request) { IndexTTS2Request out; if (request.text_input.has_value()) { @@ -74,6 +87,9 @@ IndexTTS2Request parse_index_tts2_request(const runtime::TaskRequest & request) } else { throw std::runtime_error("IndexTTS2 request requires --voice-ref or voice.speaker.audio"); } + if (const auto value = runtime::find_option(request.options, {"lang"})) { + out.lang = normalize_index_tts2_lang(*value); + } if (const auto value = runtime::parse_finite_float_option(request.options, {"emotion_alpha"})) { if (*value < 0.0F || *value > 1.0F) { diff --git a/src/models/index_tts2/s2mel.cpp b/src/models/index_tts2/s2mel.cpp index ab6a80e6..9209d503 100644 --- a/src/models/index_tts2/s2mel.cpp +++ b/src/models/index_tts2/s2mel.cpp @@ -237,7 +237,11 @@ core::TensorValue cfm_wavenet( const IndexTTS2S2MelCfmWeights & weights) { auto g = core::reshape_tensor(ctx, core::ensure_backend_addressable_layout(ctx, timestep_b), core::TensorShape::from_dims({timestep_b.shape.dims[0], kHidden, 1})); g = modules::Conv1dModule({kHidden, 2 * kHidden * kWavenetLayers, 1, 1, 0, 1, true}).build(ctx, g, weights.wavenet_cond); - auto output = sub(ctx, input_bct, input_bct); + // The zero accumulator must come from a contiguous tensor: input_bct is a + // permuted (transposed) view, and the ggml CPU binary-op kernels miscompute + // permuted src operands (the CUDA kernels handle them). + const auto zeros_base = core::ensure_backend_addressable_layout(ctx, input_bct); + auto output = sub(ctx, zeros_base, zeros_base); auto x = input_bct; for (int64_t i = 0; i < kWavenetLayers; ++i) { const int64_t dilation = 1; diff --git a/src/models/index_tts2/semantic_codec.cpp b/src/models/index_tts2/semantic_codec.cpp index c1aff416..f026e601 100644 --- a/src/models/index_tts2/semantic_codec.cpp +++ b/src/models/index_tts2/semantic_codec.cpp @@ -408,10 +408,12 @@ class IndexTTS2SemanticCodecRuntime::CodesGraph { core::ExecutionContext & execution, std::shared_ptr weights, int64_t frames, + bool upsample_decode, size_t graph_arena_bytes) : execution_(execution), weights_(std::move(weights)), - frames_(frames) { + frames_(frames), + upsample_decode_(upsample_decode) { if (frames_ <= 0) { throw std::runtime_error("IndexTTS2 semantic codec code graph requires positive frame count"); } @@ -436,11 +438,28 @@ class IndexTTS2SemanticCodecRuntime::CodesGraph { "index_tts2.semantic_codec.codes.inputs", execution_.backend_type()}; codes_ = core::make_tensor(input_ctx, GGML_TYPE_I32, core::TensorShape::from_dims({1, frames_})).tensor; + if (upsample_decode_) { + upsample_ids_ = ggml_new_tensor_1d(input_ctx_.get(), GGML_TYPE_I32, 2 * frames_); + } ggml_set_input(codes_); auto embedding = embed_codes_bct( ctx, core::wrap_tensor(codes_, core::TensorShape::from_dims({1, frames_}), GGML_TYPE_I32), *weights_); + if (upsample_decode_) { + // v2.5 EnhancedCodec.decode (codec/models.py): decoder backbone + + // projection, then 2x nearest upsample along time and the `up` conv. + auto x = vocos_backbone(ctx, embedding, weights_->decoder_backbone); + x = modules::LinearModule({kVocosDim, kHidden, true}).build(ctx, x, weights_->decoder_projection); + x = core::ensure_backend_addressable_layout(ctx, x); + x = core::wrap_tensor( + ggml_get_rows(ctx.ggml, x.tensor, upsample_ids_), + core::TensorShape::from_dims({1, 2 * frames_, kHidden}), + GGML_TYPE_F32); + x = modules::TransposeModule({{0, 2, 1, 3}, 3}).build(ctx, x); + x = modules::Conv1dModule({kHidden, kHidden, 3, 1, 1, 1, true}).build(ctx, x, weights_->up); + embedding = x; + } embedding_ = core::ensure_backend_addressable_layout(ctx, embedding).tensor; ggml_set_output(embedding_); @@ -450,6 +469,14 @@ class IndexTTS2SemanticCodecRuntime::CodesGraph { if (input_buffer_ == nullptr) { throw std::runtime_error("failed to allocate IndexTTS2 semantic codec code input buffer"); } + if (upsample_decode_) { + std::vector upsample_ids(static_cast(2 * frames_)); + for (int64_t frame = 0; frame < frames_; ++frame) { + upsample_ids[static_cast(2 * frame)] = static_cast(frame); + upsample_ids[static_cast(2 * frame + 1)] = static_cast(frame); + } + ggml_backend_tensor_set(upsample_ids_, upsample_ids.data(), 0, upsample_ids.size() * sizeof(int32_t)); + } gallocr_ = ggml_gallocr_new(ggml_backend_get_default_buffer_type(execution_.backend())); if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || @@ -493,10 +520,10 @@ class IndexTTS2SemanticCodecRuntime::CodesGraph { } IndexTTS2SemanticCodecOutput output; - output.frames = frames_; + output.frames = upsample_decode_ ? 2 * frames_ : frames_; output.dims = kHidden; output.codes = codes; - output.embedding_channel_first.resize(static_cast(kHidden * frames_)); + output.embedding_channel_first.resize(static_cast(kHidden * output.frames)); timing_start = Clock::now(); ggml_backend_tensor_get( embedding_, @@ -528,9 +555,11 @@ class IndexTTS2SemanticCodecRuntime::CodesGraph { core::ExecutionContext & execution_; std::shared_ptr weights_; int64_t frames_ = 0; + bool upsample_decode_ = false; std::unique_ptr input_ctx_; std::unique_ptr ctx_; ggml_tensor * codes_ = nullptr; + ggml_tensor * upsample_ids_ = nullptr; ggml_tensor * embedding_ = nullptr; ggml_cgraph * graph_ = nullptr; ggml_gallocr_t gallocr_ = nullptr; @@ -610,6 +639,17 @@ std::shared_ptr load_index_tts2_semantic_co kHidden, kVocosDim, true); + if (index_tts2_variant_from_version(assets.config.version) == IndexTTS2Variant::kV2_5) { + weights->up = binding::conv1d_from_source( + *weights->store, + source, + "up", + conv_storage_type, + kHidden, + kHidden, + 3, + true); + } weights->store->upload(); assets.semantic_codec_weights->release_storage(); @@ -668,7 +708,12 @@ void IndexTTS2SemanticCodecRuntime::prepare_codes(int64_t frames) { return; } codes_graph_.reset(); - codes_graph_ = std::make_unique(*execution_, weights_, frames, graph_arena_bytes_); + codes_graph_ = std::make_unique( + *execution_, + weights_, + frames, + index_tts2_variant_from_version(assets_->config.version) == IndexTTS2Variant::kV2_5, + graph_arena_bytes_); } IndexTTS2SemanticCodecOutput IndexTTS2SemanticCodecRuntime::quantize(const IndexTTS2SemanticEmbedding & semantic) { diff --git a/src/models/index_tts2/session.cpp b/src/models/index_tts2/session.cpp index 98b025db..8e51de63 100644 --- a/src/models/index_tts2/session.cpp +++ b/src/models/index_tts2/session.cpp @@ -36,6 +36,10 @@ std::shared_ptr require_assets(std::shared_ptr(align_index_tts2_gpt_text_tokens(segment).size()) + : static_cast(segment.size()); gpt_->prepare_generation( - static_cast(segment.size()), + gpt_text_tokens, generation.max_mel_tokens, generation.num_beams); } @@ -384,18 +398,30 @@ const IndexTTS2Session::SpeakerState & IndexTTS2Session::resolve_speaker_state(c true); semantic_encoder_->prepare(prepared.semantic_features.frames); auto semantic = semantic_encoder_->encode(prepared.semantic_features); - semantic_codec_->prepare_quantize(semantic.frames); - auto reference_codes = semantic_codec_->quantize(semantic); - const auto reference_content = channel_first_to_time_major( - reference_codes.embedding_channel_first, - reference_codes.dims, - reference_codes.frames); - debug::trace_log_scalar("index_tts2.s2mel.reference_mel_frames", static_cast(prepared.mel.frames)); - s2mel_->prepare_length_regulator(reference_codes.frames, prepared.mel.frames); - auto prompt_condition = s2mel_->regulate_length( - reference_content, - reference_codes.frames, - prepared.mel.frames); + IndexTTS2S2MelSequence prompt_condition; + if (is_v2_5_variant(*assets_)) { + // Official v2.5 regulates the raw (normalized) w2v-bert semantic + // directly; the semantic codec is only used to decode generated codes. + debug::trace_log_scalar("index_tts2.s2mel.reference_mel_frames", static_cast(prepared.mel.frames)); + s2mel_->prepare_length_regulator(semantic.frames, prepared.mel.frames); + prompt_condition = s2mel_->regulate_length( + semantic.values, + semantic.frames, + prepared.mel.frames); + } else { + semantic_codec_->prepare_quantize(semantic.frames); + auto reference_codes = semantic_codec_->quantize(semantic); + const auto reference_content = channel_first_to_time_major( + reference_codes.embedding_channel_first, + reference_codes.dims, + reference_codes.frames); + debug::trace_log_scalar("index_tts2.s2mel.reference_mel_frames", static_cast(prepared.mel.frames)); + s2mel_->prepare_length_regulator(reference_codes.frames, prepared.mel.frames); + prompt_condition = s2mel_->regulate_length( + reference_content, + reference_codes.frames, + prepared.mel.frames); + } SpeakerState state; state.identity = identity; @@ -592,15 +618,22 @@ std::vector IndexTTS2Session::resolve_emotion_vector( runtime::AudioBuffer IndexTTS2Session::synthesize_segment( const std::vector & text_tokens, + int32_t lang_id, const SpeakerState & speaker, const EmotionState & emotion, const std::vector & emotion_vector, const IndexTTS2GenerationOptions & options, uint32_t segment_seed) { + const bool v2_5 = is_v2_5_variant(*assets_); IndexTTS2GptGenerationRequest generation; generation.text_tokens = text_tokens; - generation.speaker_semantic = speaker.semantic.values; - generation.speaker_frames = speaker.semantic.frames; + if (v2_5) { + generation.speaker_style = speaker.style.values; + generation.lang_id = lang_id; + } else { + generation.speaker_semantic = speaker.semantic.values; + generation.speaker_frames = speaker.semantic.frames; + } generation.emotion_semantic = emotion.semantic.values; generation.emotion_frames = emotion.semantic.frames; generation.emotion_vector = emotion_vector; @@ -624,29 +657,42 @@ runtime::AudioBuffer IndexTTS2Session::synthesize_segment( throw std::runtime_error("IndexTTS2 GPT generated no acoustic codes"); } const int64_t code_frames = static_cast(generated.codes.size()); - const int64_t target_frames = static_cast(static_cast(code_frames) * 1.72F); - const int64_t total_frames = speaker.prompt_condition.frames + target_frames; - const auto forward_start = Clock::now(); - auto latent = gpt_->forward_latent( - generated.speech_conditioning_latent, - text_tokens, - generated.codes, - emotion.semantic.values, - emotion.semantic.frames, - emotion_vector); - debug::timing_log_scalar("index_tts2.gpt.forward_ms", engine::debug::elapsed_ms(forward_start)); const auto s2mel_start = Clock::now(); - s2mel_->prepare_gpt_layer(latent.frames); - auto projected = s2mel_->project_gpt_latent(latent.values, latent.frames); semantic_codec_->prepare_codes(code_frames); auto semantic = semantic_codec_->codes_to_embedding(generated.codes, code_frames); if (mem_saver_) { semantic_codec_->release_graphs(); } - auto content = add_latent_to_semantic(semantic, projected); - s2mel_->prepare_length_regulator(code_frames, target_frames); - auto generated_condition = s2mel_->regulate_length(content, code_frames, target_frames); + std::vector content; + int64_t content_frames = 0; + if (v2_5) { + // v2.5: the codec decode (with its 2x upsample) yields the S2Mel + // content directly; there is no GPT latent projection in this variant. + content = channel_first_to_time_major( + semantic.embedding_channel_first, + semantic.dims, + semantic.frames); + content_frames = semantic.frames; + } else { + const auto forward_start = Clock::now(); + auto latent = gpt_->forward_latent( + generated.speech_conditioning_latent, + text_tokens, + generated.codes, + emotion.semantic.values, + emotion.semantic.frames, + emotion_vector); + debug::timing_log_scalar("index_tts2.gpt.forward_ms", engine::debug::elapsed_ms(forward_start)); + s2mel_->prepare_gpt_layer(latent.frames); + auto projected = s2mel_->project_gpt_latent(latent.values, latent.frames); + content = add_latent_to_semantic(semantic, projected); + content_frames = code_frames; + } + const int64_t target_frames = static_cast(static_cast(content_frames) * 1.72F); + const int64_t total_frames = speaker.prompt_condition.frames + target_frames; + s2mel_->prepare_length_regulator(content_frames, target_frames); + auto generated_condition = s2mel_->regulate_length(content, content_frames, target_frames); auto condition = concat_conditions(speaker.prompt_condition, generated_condition); if (mem_saver_) { gpt_->release_generation_graphs(); @@ -713,15 +759,19 @@ runtime::TaskResult IndexTTS2Session::run(const runtime::TaskRequest & request) throw std::runtime_error("IndexTTS2 text chunking produced no chunks"); } + const bool v2_5 = is_v2_5_variant(*assets_); std::vector> segment_token_ids; + std::vector segment_lang_ids; for (const auto & text_chunk : text_chunks) { const auto text_encoding = tokenizer_.encode_for_inference( text_chunk, - parsed.max_text_tokens_per_segment); - segment_token_ids.insert( - segment_token_ids.end(), - text_encoding.segment_token_ids.begin(), - text_encoding.segment_token_ids.end()); + parsed.max_text_tokens_per_segment, + parsed.lang); + const int32_t lang_id = v2_5 ? IndexTTS2TextTokenizer::lang_to_id(text_encoding.lang) : 0; + for (const auto & ids : text_encoding.segment_token_ids) { + segment_token_ids.push_back(ids); + segment_lang_ids.push_back(lang_id); + } } runtime::AudioBuffer merged; @@ -731,6 +781,7 @@ runtime::TaskResult IndexTTS2Session::run(const runtime::TaskRequest & request) } auto segment_audio = synthesize_segment( segment_token_ids[i], + segment_lang_ids[i], speaker, emotion, emotion_vector, diff --git a/src/models/index_tts2/tokenizer_text.cpp b/src/models/index_tts2/tokenizer_text.cpp index b7b18d80..8b87547c 100644 --- a/src/models/index_tts2/tokenizer_text.cpp +++ b/src/models/index_tts2/tokenizer_text.cpp @@ -3,26 +3,30 @@ #include "engine/framework/text/chinese_normalization.h" #include "engine/framework/text/text_normalization.h" +#include "bpe-core.h" +#include "unicode.h" + #include +#include #include +#include +#include +#include +#include #include #include #include #include +#include namespace engine::models::index_tts2 { namespace { -bool contains_token( - const std::vector & values, - const std::vector & needles) { - for (const auto & value : values) { - if (std::find(needles.begin(), needles.end(), value) != needles.end()) { - return true; - } - } - return false; -} +namespace vendor = llama_tokenizer_vendor; + +// --------------------------------------------------------------------------- +// Shared UTF-8 / text helpers +// --------------------------------------------------------------------------- size_t utf8_codepoint_size(unsigned char byte) { if ((byte & 0x80U) == 0U) { @@ -57,27 +61,22 @@ uint32_t decode_utf8_codepoint(const std::string & text, size_t offset, size_t s return byte(0); } -bool is_han_codepoint(uint32_t cp) { - return (cp >= 0x4E00U && cp <= 0x9FFFU); +uint32_t next_utf8_codepoint(const std::string & text, size_t & offset) { + const size_t size = std::min(utf8_codepoint_size(static_cast(text[offset])), text.size() - offset); + const uint32_t cp = decode_utf8_codepoint(text, offset, size); + offset += size; + return cp; } -bool is_cjk_codepoint(uint32_t cp) { - return (cp >= 0x1100U && cp <= 0x11FFU) - || (cp >= 0x2E80U && cp <= 0xA4CFU) - || (cp >= 0xA840U && cp <= 0xD7AFU) - || (cp >= 0xF900U && cp <= 0xFAFFU) - || (cp >= 0xFE30U && cp <= 0xFE4FU) - || (cp >= 0xFF65U && cp <= 0xFFDCU) - || (cp >= 0x20000U && cp <= 0x2FFFFU); +bool is_han_codepoint(uint32_t cp) { + return cp >= 0x4E00U && cp <= 0x9FFFU; } bool contains_han(const std::string & text) { for (size_t i = 0; i < text.size();) { - const size_t size = std::min(utf8_codepoint_size(static_cast(text[i])), text.size() - i); - if (is_han_codepoint(decode_utf8_codepoint(text, i, size))) { + if (is_han_codepoint(next_utf8_codepoint(text, i))) { return true; } - i += size; } return false; } @@ -101,6 +100,38 @@ std::string uppercase_ascii(std::string text) { return text; } +std::string lowercase_ascii(std::string text) { + for (char & ch : text) { + ch = static_cast(std::tolower(static_cast(ch))); + } + return text; +} + +// --------------------------------------------------------------------------- +// v2 SentencePiece helpers +// --------------------------------------------------------------------------- + +bool contains_token( + const std::vector & values, + const std::vector & needles) { + for (const auto & value : values) { + if (std::find(needles.begin(), needles.end(), value) != needles.end()) { + return true; + } + } + return false; +} + +bool is_cjk_codepoint(uint32_t cp) { + return (cp >= 0x1100U && cp <= 0x11FFU) + || (cp >= 0x2E80U && cp <= 0xA4CFU) + || (cp >= 0xA840U && cp <= 0xD7AFU) + || (cp >= 0xF900U && cp <= 0xFAFFU) + || (cp >= 0xFE30U && cp <= 0xFE4FU) + || (cp >= 0xFF65U && cp <= 0xFFDCU) + || (cp >= 0x20000U && cp <= 0x2FFFFU); +} + std::string tokenize_by_cjk_char(const std::string & text) { std::vector tokens; std::string pending; @@ -207,6 +238,457 @@ std::vector> split_segments_by_token( return merged_segments; } +// --------------------------------------------------------------------------- +// v2.5 tiktoken helpers +// --------------------------------------------------------------------------- + +// IndexTTS-2.5 pads every text segment with a trailing token id 1. +constexpr int32_t kSegmentPadTokenId = 1; + +std::string decode_base64(const std::string & input) { + static const std::array table = [] { + std::array values{}; + values.fill(-1); + for (int i = 0; i < 26; ++i) { + values[static_cast('A' + i)] = static_cast(i); + values[static_cast('a' + i)] = static_cast(26 + i); + } + for (int i = 0; i < 10; ++i) { + values[static_cast('0' + i)] = static_cast(52 + i); + } + values[static_cast('+')] = 62; + values[static_cast('/')] = 63; + return values; + }(); + + std::string out; + int bits = 0; + int value = 0; + for (const unsigned char ch : input) { + if (ch == '=') { + break; + } + const int8_t digit = table[ch]; + if (digit < 0) { + throw std::runtime_error("IndexTTS2 tiktoken vocabulary contains invalid base64 token bytes"); + } + value = (value << 6) | digit; + bits += 6; + if (bits >= 8) { + bits -= 8; + out.push_back(static_cast((value >> bits) & 0xff)); + } + } + return out; +} + +// The vendored llama BPE runtime works in the GPT-2 byte-to-unicode domain +// (e.g. space becomes U+0120) so that byte-level merges, including tokens that +// split a UTF-8 codepoint, are reproduced exactly. tiktoken ranks are keyed by +// raw bytes, so every token is mapped once at load time. +std::string map_token_bytes(const std::string & bytes) { + std::string mapped; + for (const unsigned char byte : bytes) { + mapped += unicode_byte_to_utf8(byte); + } + return mapped; +} + +std::string pair_key(const std::string & left, const std::string & right) { + std::string key = left; + key.push_back('\0'); + key += right; + return key; +} + +// Language codes in the LANGUAGES order of indextts/utils/tokenizer.py. The +// first 99 entries double as the <|lang|> special tokens below; the remaining +// codes (plus the fallback "common") only index the GPT lang_embedding table. +const std::array kLanguages = { + "en", "zh", "de", "es", "ru", "ko", "fr", "ja", "pt", "tr", + "pl", "ca", "nl", "ar", "sv", "it", "id", "hi", "fi", "vi", + "he", "uk", "el", "ms", "cs", "ro", "da", "hu", "ta", "no", + "th", "ur", "hr", "bg", "lt", "la", "mi", "ml", "cy", "sk", + "te", "fa", "lv", "bn", "sr", "az", "sl", "kn", "et", "mk", + "br", "eu", "is", "hy", "ne", "mn", "bs", "kk", "sq", "sw", + "gl", "mr", "pa", "si", "km", "sn", "yo", "so", "af", "oc", + "ka", "be", "tg", "sd", "gu", "am", "yi", "lo", "uz", "fo", + "ht", "ps", "tk", "nn", "mt", "sa", "lb", "my", "bo", "tl", + "mg", "as", "tt", "haw", "ln", "ha", "ba", "jw", "su", +}; +const std::array kEmbeddingOnlyLanguages = { + "yue", "minnan", "wuyu", "dialect", "zh/en", "en/zh", "common", +}; +constexpr int32_t kCommonLangId = 105; + +void add_special_token(vendor::BpeVocabulary & vocab, const std::string & text, int32_t id) { + vocab.token_to_id.emplace(text, id); + vocab.id_to_token.emplace(id, vendor::TokenData{text, vendor::TOKEN_ATTR_CONTROL}); +} + +// Special token order must match indextts/utils/tokenizer.py exactly: +// ids are assigned sequentially starting right after the mergeable ranks. +void register_special_tokens(vendor::BpeVocabulary & vocab, int32_t base_id) { + static const std::array kAudioEvents = { + "ASR", "AED", "SER", "Speech", "/Speech", "BGM", "/BGM", + "Laughter", "/Laughter", "Applause", "/Applause", + }; + static const std::array kEmotions = { + "HAPPY", "SAD", "ANGRY", "NEUTRAL", + }; + static const std::array kTasks = { + "translate", "transcribe", "startoflm", "startofprev", "nospeech", "notimestamps", + }; + static const std::array kTtsVocal = { + "TTS/B", "TTS/O", "TTS/Q", "TTS/A", "TTS/CO", "TTS/CL", "TTS/H", + }; + + int32_t id = base_id; + add_special_token(vocab, "<|endoftext|>", id++); + add_special_token(vocab, "<|startoftranscript|>", id++); + for (const char * lang : kLanguages) { + add_special_token(vocab, "<|" + std::string(lang) + "|>", id++); + } + for (const char * event : kAudioEvents) { + add_special_token(vocab, "<|" + std::string(event) + "|>", id++); + } + for (const char * emotion : kEmotions) { + add_special_token(vocab, "<|" + std::string(emotion) + "|>", id++); + } + for (const char * task : kTasks) { + add_special_token(vocab, "<|" + std::string(task) + "|>", id++); + } + for (int i = 1; i <= 30; ++i) { + add_special_token(vocab, "<|SPECIAL_TOKEN_" + std::to_string(i) + "|>", id++); + } + for (const char * vocal : kTtsVocal) { + add_special_token(vocab, "<|" + std::string(vocal) + "|>", id++); + } + for (int i = 1; i <= 13; ++i) { + char name[32]; + std::snprintf(name, sizeof(name), "<|TTS/SP%02d|>", i); + add_special_token(vocab, name, id++); + } + // Timestamps <|0.00|> .. <|30.00|> in 0.02 steps; i * 0.02 == i / 50. + for (int i = 0; i <= 1500; ++i) { + char name[32]; + std::snprintf(name, sizeof(name), "<|%d.%02d|>", i / 50, (i * 2) % 100); + add_special_token(vocab, name, id++); + } +} + +std::shared_ptr load_tiktoken_vocabulary(const std::filesystem::path & vocab_path) { + std::ifstream input(vocab_path, std::ios::binary); + if (!input) { + throw std::runtime_error("IndexTTS2 failed to open tiktoken vocabulary: " + vocab_path.string()); + } + + auto vocab = std::make_shared(); + vocab->pre_type = vendor::PreTokenizerType::Gpt2; + + std::string line; + int64_t mergeable_count = 0; + while (std::getline(input, line)) { + if (!line.empty() && line.back() == '\r') { + line.pop_back(); + } + if (line.empty()) { + continue; + } + std::istringstream parts(line); + std::string token_base64; + int64_t rank = -1; + if (!(parts >> token_base64 >> rank) || rank < 0 || rank > INT32_MAX) { + throw std::runtime_error("IndexTTS2 tiktoken vocabulary has an invalid line: " + line); + } + const std::string bytes = decode_base64(token_base64); + const auto token_id = static_cast(rank); + const std::string mapped = map_token_bytes(bytes); + vocab->token_to_id.emplace(mapped, token_id); + vocab->id_to_token.emplace(token_id, vendor::TokenData{mapped, 0}); + // tiktoken ranks double as merge priorities: an adjacent pair merges + // iff its concatenation is a token, with that token's rank. Register + // every split so find_bpe_rank(left, right) == rank(left + right). + for (size_t split = 1; split < bytes.size(); ++split) { + vocab->bpe_ranks.emplace( + pair_key(map_token_bytes(bytes.substr(0, split)), map_token_bytes(bytes.substr(split))), + token_id); + } + ++mergeable_count; + } + if (mergeable_count == 0) { + throw std::runtime_error("IndexTTS2 tiktoken vocabulary is empty: " + vocab_path.string()); + } + + register_special_tokens(*vocab, static_cast(mergeable_count)); + vendor::rebuild_special_tokens_cache(*vocab); + return vocab; +} + +bool is_kana(const std::string & text) { + if (text.empty()) { + return false; + } + bool all_hiragana = true; + bool all_katakana = true; + for (size_t i = 0; i < text.size();) { + const uint32_t cp = next_utf8_codepoint(text, i); + if (cp < 0x3040U || cp > 0x309FU) { + all_hiragana = false; + } + if (cp < 0x30A0U || cp > 0x30FFU) { + all_katakana = false; + } + } + return all_hiragana || all_katakana; +} + +struct AnnotationMatch { + size_t end = 0; // one past the match; 0 when there is no match at pos + size_t word_begin = 0; + size_t word_end = 0; + size_t pron_begin = 0; + size_t pron_end = 0; +}; + +// Matches <([^|>\n]+)\|([^>\n]+)> anchored at pos. +AnnotationMatch match_pronunciation_annotation(const std::string & text, size_t pos) { + AnnotationMatch match; + if (text[pos] != '<') { + return match; + } + size_t cursor = pos + 1; + const size_t word_begin = cursor; + while (cursor < text.size() && text[cursor] != '|' && text[cursor] != '>' && text[cursor] != '\n') { + ++cursor; + } + if (cursor == word_begin || cursor >= text.size() || text[cursor] != '|') { + return match; + } + match.word_begin = word_begin; + match.word_end = cursor; + const size_t pron_begin = ++cursor; + while (cursor < text.size() && text[cursor] != '>' && text[cursor] != '\n') { + ++cursor; + } + if (cursor == pron_begin || cursor >= text.size()) { + return AnnotationMatch{}; + } + match.pron_begin = pron_begin; + match.pron_end = cursor; + match.end = cursor + 1; + return match; +} + +// Base-26 spreadsheet-style index ("a".."z", "aa"..), mirroring the official +// TextNormalizer._protect_pronunciation_annotations placeholder naming. +std::string alpha_placeholder_index(size_t n) { + std::string s; + while (true) { + s.insert(s.begin(), static_cast('a' + (n % 26))); + const size_t q = n / 26; + if (q == 0) { + break; + } + n = q - 1; + } + return s; +} + +using PronunciationPlaceholders = std::vector>; + +// Replaces annotations with letter-only placeholders so +// text normalization cannot rewrite their digits/symbols (e.g. XING2). +std::pair protect_pronunciation_annotations(const std::string & text) { + std::string out; + out.reserve(text.size()); + PronunciationPlaceholders placeholders; + size_t pos = 0; + while (pos < text.size()) { + const auto match = match_pronunciation_annotation(text, pos); + if (match.end == 0) { + out.push_back(text[pos++]); + continue; + } + std::string key = "PRONPLACEHOLDER" + alpha_placeholder_index(placeholders.size()) + "PRONPLACEHOLDER"; + placeholders.emplace_back(key, text.substr(pos, match.end - pos)); + out += key; + pos = match.end; + } + return {out, placeholders}; +} + +std::string restore_pronunciation_annotations(std::string text, const PronunciationPlaceholders & placeholders) { + for (const auto & [key, original] : placeholders) { + size_t at = 0; + while ((at = text.find(key, at)) != std::string::npos) { + text.replace(at, key.size(), original); + at += original.size(); + } + } + return text; +} + +// Expands annotations (see infer_v2_5.py +// apply_pronunciation_annotations): +// Chinese word -> <|SPECIAL_TOKEN_2|>PRON<|SPECIAL_TOKEN_2|> +// other word -> <|SPECIAL_TOKEN_1|>PRON<|SPECIAL_TOKEN_1|> +// kana pron -> inlined as " PRON " +std::string apply_pronunciation_annotations(const std::string & text) { + std::string out; + out.reserve(text.size()); + size_t pos = 0; + while (pos < text.size()) { + const auto match = match_pronunciation_annotation(text, pos); + if (match.end == 0) { + out.push_back(text[pos++]); + continue; + } + const std::string word = text.substr(match.word_begin, match.word_end - match.word_begin); + const std::string pron = uppercase_ascii(text.substr(match.pron_begin, match.pron_end - match.pron_begin)); + if (is_kana(pron)) { + out.push_back(' '); + out += pron; + out.push_back(' '); + } else { + const char * wrapper = contains_han(word) ? "<|SPECIAL_TOKEN_2|>" : "<|SPECIAL_TOKEN_1|>"; + out += wrapper; + out += pron; + out += wrapper; + } + pos = match.end; + } + return out; +} + +// Uppercases the name inside <|...|> markers: re.sub(r'<\|([^|]+)\|>', upper). +std::string uppercase_special_token_names(const std::string & text) { + std::string out; + out.reserve(text.size()); + size_t pos = 0; + while (pos < text.size()) { + if (text[pos] != '<' || pos + 1 >= text.size() || text[pos + 1] != '|') { + out.push_back(text[pos++]); + continue; + } + size_t cursor = pos + 2; + while (cursor < text.size() && text[cursor] != '|') { + ++cursor; + } + if (cursor == pos + 2 || cursor + 1 >= text.size() || text[cursor + 1] != '>') { + out.push_back(text[pos++]); + continue; + } + out += "<|"; + out += uppercase_ascii(text.substr(pos + 2, cursor - (pos + 2))); + out += "|>"; + pos = cursor + 2; + } + return out; +} + +bool is_segment_delimiter(uint32_t cp) { + switch (cp) { + case U',': + case U'.': + case U'!': + case U'?': + case U';': + case U':': + case U'\n': + case 0xFF0CU: // ， + case 0x3002U: // 。 + case 0xFF01U: // ！ + case 0xFF1FU: // ？ + case 0x3001U: // 、 + case 0xFF1BU: // ； + case 0xFF1AU: // ： + return true; + default: + return false; + } +} + +// re.split(r'(?<=[，。！？、；：,\.!\?;:\n])', piece): split after each delimiter. +std::vector split_after_delimiters(const std::string & piece) { + std::vector parts; + std::string current; + for (size_t i = 0; i < piece.size();) { + const size_t begin = i; + const uint32_t cp = next_utf8_codepoint(piece, i); + current.append(piece, begin, i - begin); + if (is_segment_delimiter(cp)) { + parts.push_back(std::move(current)); + current.clear(); + } + } + if (!current.empty()) { + parts.push_back(std::move(current)); + } + return parts; +} + +// Matches "<|SPECIAL_TOKEN_|>" at pos; returns the match length or 0. +size_t match_special_token_marker(const std::string & text, size_t pos) { + static const std::string kPrefix = "<|SPECIAL_TOKEN_"; + if (text.compare(pos, kPrefix.size(), kPrefix) != 0) { + return 0; + } + size_t cursor = pos + kPrefix.size(); + const size_t digits_begin = cursor; + while (cursor < text.size() && std::isdigit(static_cast(text[cursor])) != 0) { + ++cursor; + } + if (cursor == digits_begin || cursor + 1 >= text.size() || text[cursor] != '|' || text[cursor + 1] != '>') { + return 0; + } + return cursor + 2 - pos; +} + +// SPLIT_PROTECTED_PATTERN spans (<|SPECIAL_TOKEN_n|>...<|SPECIAL_TOKEN_n|>) +// are kept atomic during segmentation. +std::vector> split_atomic_pieces(const std::string & text) { + std::vector> pieces; + size_t pos = 0; + while (pos < text.size()) { + size_t opener = std::string::npos; + size_t opener_len = 0; + for (size_t i = pos; i < text.size(); ++i) { + const size_t len = match_special_token_marker(text, i); + if (len > 0) { + opener = i; + opener_len = len; + break; + } + } + if (opener == std::string::npos) { + break; + } + size_t closer = std::string::npos; + size_t closer_len = 0; + for (size_t i = opener + opener_len; i < text.size(); ++i) { + const size_t len = match_special_token_marker(text, i); + if (len > 0) { + closer = i; + closer_len = len; + break; + } + } + if (closer == std::string::npos) { + break; + } + if (opener > pos) { + pieces.emplace_back(text.substr(pos, opener - pos), false); + } + pieces.emplace_back(text.substr(opener, closer + closer_len - opener), true); + pos = closer + closer_len; + } + if (pos < text.size()) { + pieces.emplace_back(text.substr(pos), false); + } + return pieces; +} + } // namespace IndexTTS2TextTokenizer::IndexTTS2TextTokenizer(std::shared_ptr assets) @@ -214,10 +696,15 @@ IndexTTS2TextTokenizer::IndexTTS2TextTokenizer(std::shared_ptrresources.require_file("bpe")); - piece_to_id_.reserve(pieces_.size()); - for (const auto & piece : pieces_) { - piece_to_id_.emplace(piece.text, static_cast(piece.id)); + variant_ = index_tts2_variant_from_version(assets_->config.version); + if (variant_ == IndexTTS2Variant::kV2_5) { + vocab_ = load_tiktoken_vocabulary(assets_->resources.require_file("tiktoken")); + } else { + pieces_ = engine::tokenizers::load_sentencepiece_model(assets_->resources.require_file("bpe")); + piece_to_id_.reserve(pieces_.size()); + for (const auto & piece : pieces_) { + piece_to_id_.emplace(piece.text, static_cast(piece.id)); + } } } @@ -225,7 +712,8 @@ std::string IndexTTS2TextTokenizer::normalize_english(const std::string & text) engine::text::EnglishTextNormalizationOptions options; options.expand_common_contractions = true; options.index_tts_punctuation = true; - options.uppercase_ascii = true; + options.uppercase_ascii = variant_ == IndexTTS2Variant::kV2; + options.verbalize_symbols = true; return engine::text::normalize_english_text(text, options); } @@ -235,15 +723,24 @@ std::string IndexTTS2TextTokenizer::normalize_chinese(const std::string & text) engine::text::ChineseTextNormalizationTarget::IndexTTS); } -std::string IndexTTS2TextTokenizer::normalize_text(const std::string & text) const { - return contains_han(text) ? tokenize_by_cjk_char(normalize_chinese(text)) : normalize_english(text); -} - std::vector IndexTTS2TextTokenizer::encode(const std::string & text) const { + if (variant_ == IndexTTS2Variant::kV2_5) { + return vendor::tokenize_bpe(*vocab_, text, true); + } return engine::tokenizers::tokenize_sentencepiece(pieces_, normalize_text(text)); } +std::string IndexTTS2TextTokenizer::normalize_text(const std::string & text) const { + if (variant_ != IndexTTS2Variant::kV2) { + throw std::runtime_error("IndexTTS2 normalize_text is only available for the v2 SentencePiece tokenizer"); + } + return contains_han(text) ? tokenize_by_cjk_char(normalize_chinese(text)) : normalize_english(text); +} + std::vector IndexTTS2TextTokenizer::tokenize_to_pieces(const std::string & text) const { + if (variant_ != IndexTTS2Variant::kV2) { + throw std::runtime_error("IndexTTS2 tokenize_to_pieces is only available for the v2 SentencePiece tokenizer"); + } const auto ids = encode(text); std::vector out; out.reserve(ids.size()); @@ -253,12 +750,45 @@ std::vector IndexTTS2TextTokenizer::tokenize_to_pieces(const std::s return out; } +int32_t IndexTTS2TextTokenizer::special_token_id(const std::string & token_text) const { + if (variant_ != IndexTTS2Variant::kV2_5) { + throw std::runtime_error("IndexTTS2 special_token_id is only available for the v2.5 tiktoken tokenizer"); + } + const auto it = vocab_->token_to_id.find(token_text); + return it == vocab_->token_to_id.end() ? -1 : it->second; +} + +int32_t IndexTTS2TextTokenizer::lang_to_id(const std::string & lang) { + const std::string normalized = lowercase_ascii(lang); + for (size_t i = 0; i < kLanguages.size(); ++i) { + if (normalized == kLanguages[i]) { + return static_cast(i); + } + } + for (size_t i = 0; i < kEmbeddingOnlyLanguages.size(); ++i) { + if (normalized == kEmbeddingOnlyLanguages[i]) { + return static_cast(kLanguages.size() + i); + } + } + return kCommonLangId; +} + IndexTTS2TextEncoding IndexTTS2TextTokenizer::encode_for_inference( const std::string & text, - int max_text_tokens_per_segment) const { + int max_text_tokens_per_segment, + const std::string & lang) const { if (max_text_tokens_per_segment <= 0) { throw std::runtime_error("IndexTTS2 max_text_tokens_per_segment must be positive"); } + if (variant_ == IndexTTS2Variant::kV2_5) { + return encode_for_inference_v2_5(text, max_text_tokens_per_segment, lang); + } + return encode_for_inference_v2(text, max_text_tokens_per_segment); +} + +IndexTTS2TextEncoding IndexTTS2TextTokenizer::encode_for_inference_v2( + const std::string & text, + int max_text_tokens_per_segment) const { IndexTTS2TextEncoding encoding; encoding.normalized_text = normalize_text(text); encoding.token_ids = engine::tokenizers::tokenize_sentencepiece(pieces_, encoding.normalized_text); @@ -279,6 +809,108 @@ IndexTTS2TextEncoding IndexTTS2TextTokenizer::encode_for_inference( return encoding; } +IndexTTS2TextEncoding IndexTTS2TextTokenizer::encode_for_inference_v2_5( + const std::string & text, + int max_text_tokens_per_segment, + const std::string & lang) const { + std::string resolved_lang = lowercase_ascii(lang); + if (resolved_lang.empty()) { + resolved_lang = contains_han(text) ? "zh" : "en"; + } + + std::string processed = text; + if (resolved_lang == "zh" || resolved_lang == "en") { + // Protect annotations from the normalizer, as the + // official TextNormalizer does inside normalize(). + auto protected_text = protect_pronunciation_annotations(processed); + protected_text.first = resolved_lang == "zh" + ? normalize_chinese(protected_text.first) + : normalize_english(protected_text.first); + processed = restore_pronunciation_annotations(std::move(protected_text.first), protected_text.second); + } + // ja/es/ar and other languages currently pass through without TN. + if (resolved_lang == "zh" || resolved_lang == "ja" || resolved_lang == "en") { + processed = lowercase_ascii(std::move(processed)); + } else if (resolved_lang == "es") { + processed = uppercase_ascii(std::move(processed)); + } + processed = apply_pronunciation_annotations(processed); + processed = uppercase_special_token_names(processed); + + const std::string lang_prefix = "<|" + resolved_lang + "|> "; + const auto prefix_tokens = static_cast(encode(lang_prefix).size()); + const int64_t capacity = assets_->config.gpt.max_text_tokens; + int64_t budget = std::min(max_text_tokens_per_segment, capacity - 2) - prefix_tokens; + budget = std::max(budget, 1); + + std::vector segments; + const auto token_len = [this](const std::string & value) { + return static_cast(encode(value).size()); + }; + if (token_len(processed) <= budget) { + segments.push_back(processed); + } else { + std::vector chunks; + for (const auto & [piece, atomic] : split_atomic_pieces(processed)) { + if (atomic) { + chunks.push_back(piece); + continue; + } + for (const auto & part : split_after_delimiters(piece)) { + if (token_len(part) <= budget) { + chunks.push_back(part); + continue; + } + std::string current; + for (size_t i = 0; i < part.size();) { + const size_t begin = i; + next_utf8_codepoint(part, i); + const std::string ch = part.substr(begin, i - begin); + if (!current.empty() && token_len(current + ch) > budget) { + chunks.push_back(std::move(current)); + current = ch; + } else { + current += ch; + } + } + if (!current.empty()) { + chunks.push_back(std::move(current)); + } + } + } + std::string current; + for (const auto & chunk : chunks) { + if (!current.empty() && token_len(current + chunk) > budget) { + segments.push_back(std::move(current)); + current = chunk; + } else { + current += chunk; + } + } + if (!current.empty()) { + segments.push_back(std::move(current)); + } + if (segments.empty()) { + segments.push_back(processed); + } + } + + IndexTTS2TextEncoding encoding; + encoding.lang = resolved_lang; + encoding.normalized_text = processed; + encoding.segments.reserve(segments.size()); + for (const auto & segment : segments) { + encoding.segments.push_back({segment}); + } + encoding.segment_token_ids.reserve(segments.size()); + for (const auto & segment : segments) { + std::vector ids = encode(lang_prefix + segment); + ids.push_back(kSegmentPadTokenId); + encoding.segment_token_ids.push_back(std::move(ids)); + } + return encoding; +} + int32_t IndexTTS2TextTokenizer::piece_to_id(const std::string & piece) const { const auto it = piece_to_id_.find(piece); if (it != piece_to_id_.end()) { diff --git a/tests/index_tts2/index_tts2_5_warm_bench_cases.json b/tests/index_tts2/index_tts2_5_warm_bench_cases.json new file mode 100644 index 00000000..f2992421 --- /dev/null +++ b/tests/index_tts2/index_tts2_5_warm_bench_cases.json @@ -0,0 +1,131 @@ +{ + "voice_clone": { + "requests": [ + { + "text": "The palace is strict, no false rumors, Lady Qi!", + "voice_ref": "resources/index_tts2_5/official_examples/voice_02.wav", + "seed": 1234 + } + ] + }, + "chinese_voice_clone": { + "requests": [ + { + "text": "这个呀，就是我们精心制作准备的纪念品，大家可以看到这个色泽和这个材质啊，哎呀多么的光彩照人。", + "language": "zh", + "lang": "zh", + "voice_ref": "resources/index_tts2_5/official_examples/voice_03.wav", + "seed": 1245 + } + ] + }, + "chinese_emotion_text": { + "requests": [ + { + "text": "快躲起来！是他要来了！他要来抓我们了！", + "language": "zh", + "lang": "zh", + "voice_ref": "resources/index_tts2_5/official_examples/voice_12.wav", + "use_emotion_text": true, + "emotion_text": "你吓死我了！你是鬼吗？", + "emotion_alpha": 0.6, + "use_random_emotion": false, + "seed": 1246 + } + ] + }, + "emotion_reference": { + "requests": [ + { + "text": "The old theater was empty, but every chair still seemed to remember the audience.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_07.wav", + "audio": "resources/index_tts2_5/official_examples/emo_sad.wav", + "seed": 1235 + } + ] + }, + "emotion_reference_alpha": { + "requests": [ + { + "text": "I tried to sound calm, but the storm outside made every word feel heavier.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_07.wav", + "audio": "resources/index_tts2_5/official_examples/emo_sad.wav", + "emotion_alpha": 0.9, + "seed": 1236 + } + ] + }, + "emotion_vector": { + "requests": [ + { + "text": "I'm sorry, I really did forget, but I promise I will remember the important things.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_09.wav", + "emotion_vector": [0.0, 0.0, 0.8, 0.0, 0.0, 0.0, 0.0, 0.0], + "use_random_emotion": false, + "seed": 1237 + } + ] + }, + "emotion_text": { + "requests": [ + { + "text": "Hide quickly. Someone is coming, and I do not think they are here to help us.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_12.wav", + "use_emotion_text": true, + "emotion_alpha": 0.6, + "use_random_emotion": false, + "seed": 1238 + } + ] + }, + "emotion_text_description": { + "requests": [ + { + "text": "Hide quickly. Someone is coming, and I do not think they are here to help us.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_12.wav", + "use_emotion_text": true, + "emotion_text": "You scared me. Are you a ghost?", + "emotion_alpha": 0.6, + "use_random_emotion": false, + "seed": 1239 + } + ] + }, + "long_session_changed_requests": { + "use_all_requests_by_default": true, + "requests": [ + { + "text": "The palace is strict, no false rumors, Lady Qi!", + "voice_ref": "resources/index_tts2_5/official_examples/voice_02.wav", + "seed": 1240 + }, + { + "text": "Please lower your voice. The guards are already listening outside the door.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_02.wav", + "seed": 1241 + }, + { + "text": "I am trying to stay brave, but every shadow in this room feels alive.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_02.wav", + "use_emotion_text": true, + "emotion_text": "You scared me. Are you a ghost?", + "emotion_alpha": 0.6, + "use_random_emotion": false, + "seed": 1242 + }, + { + "text": "The old theater was empty, but every chair still seemed to remember the audience.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_07.wav", + "audio": "resources/index_tts2_5/official_examples/emo_sad.wav", + "seed": 1243 + }, + { + "text": "I tried to sound calm, but the storm outside made every word feel heavier.", + "voice_ref": "resources/index_tts2_5/official_examples/voice_07.wav", + "audio": "resources/index_tts2_5/official_examples/emo_sad.wav", + "emotion_alpha": 0.9, + "seed": 1244 + } + ] + } +} diff --git a/tests/index_tts2/index_tts2_warm_bench.cpp b/tests/index_tts2/index_tts2_warm_bench.cpp index 8c32172b..01c49e53 100644 --- a/tests/index_tts2/index_tts2_warm_bench.cpp +++ b/tests/index_tts2/index_tts2_warm_bench.cpp @@ -127,6 +127,7 @@ engine::runtime::TaskRequest make_request(const engine::io::json::Value & object set_optional_option(request, object, "emotion_vector"); set_optional_option(request, object, "use_emotion_text"); set_optional_option(request, object, "emotion_text"); + set_optional_option(request, object, "lang"); set_optional_option(request, object, "use_random_emotion"); set_optional_option(request, object, "interval_silence_ms"); set_optional_option(request, object, "text_chunk_size"); @@ -217,7 +218,10 @@ int main(int argc, char ** argv) { load_request.model_path = model_path; load_request.family_hint = "index_tts2"; auto registry = engine::runtime::make_default_registry(); + const auto load_start = Clock::now(); auto model = registry.load(load_request); + const auto load_end = Clock::now(); + const double load_ms = std::chrono::duration(load_end - load_start).count(); engine::runtime::TaskSpec task; task.task = engine::runtime::VoiceTaskKind::Tts; @@ -230,6 +234,7 @@ int main(int argc, char ** argv) { session_options.options[key] = value; } auto requests = parse_requests(request_sequence_json); + const auto session_start = Clock::now(); auto session_base = model->create_task_session(task, session_options); auto * session = dynamic_cast(session_base.get()); if (session == nullptr) { @@ -247,11 +252,17 @@ int main(int argc, char ** argv) { : std::nullopt; preparation.options = requests.front().options; session->prepare(preparation); + const auto session_end = Clock::now(); + const double session_ms = std::chrono::duration(session_end - session_start).count(); std::vector steps; std::vector timing_lines; timing_lines.push_back("index_tts2.backend " + backend_name); timing_lines.push_back("index_tts2.model_root " + model_path.string()); + timing_lines.push_back("index_tts2.load_ms " + engine::io::json::stringify_number(load_ms)); + timing_lines.push_back("index_tts2.session_prepare_ms " + engine::io::json::stringify_number(session_ms)); + std::cout << "index_tts2.load_ms=" << load_ms << "\n"; + std::cout << "index_tts2.session_prepare_ms=" << session_ms << "\n"; for (int i = 0; i < warmup; ++i) { (void) session->run(requests.front()); } diff --git a/tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json b/tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json index 6f214309..4ad10c62 100644 --- a/tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json +++ b/tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json @@ -288,6 +288,26 @@ } ] }, + { + "id": "index_tts2_5_voice_clone_longform", + "coverage": "IndexTTS2.5 voice clone with shared long-form text for chunking and RTF measurement", + "family": "index_tts2", + "model": "models/IndexTTS2.5-GGUF", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "clone_longform", + "text": "At dawn the harbor station opens its tall windows and the first clerk begins a careful report for the day. She notes the weather above the river, the slow cargo boats beyond the bridge, and the market voices arriving from the eastern road. A brass clock marks each quarter hour while porters stack wooden crates, bakers carry warm bread across the square, and a violinist practices the same bright phrase under the stone archway. By midmorning the keeper of the lighthouse sends a message about shifting currents, the museum guide unlocks a cabinet of maps, and a teacher leads a quiet line of students toward the ferry. In the afternoon a painter describes the silver color of the water, a mechanic jokes with the tram driver, and the station master reads an announcement that asks every traveler to keep close watch over letters, tickets, and parcels. After sunset the same clerk continues the report because new visitors keep arriving from the inland road. She explains that a florist carries pale roses past the fountain, two carpenters compare measurements beside the warehouse door, and the watchman checks each lock before the tide reaches its highest mark. A child laughs when the tram bell rings, a cook lowers a basket of fruit to the cellar, and three sailors unfold a chart that shows old channels, sandbars, and safe turning points for the morning crossing. Near midnight the lamps still glow on wet stone, the last cart rattles toward the market gate, and the report ends by saying that the harbor remains orderly, the wind has softened, the ferries are secure, and the town can rest until the next sunrise returns over the water. On the following morning the clerk resumes the record with even greater care because a week of inspections is about to begin. She writes that a ferry captain checks the mooring ropes one by one, a bookseller arranges travel guides beside the station cafe, and a pair of gardeners lift wet soil into bright clay pots near the west entrance. The bakery sends out trays of seed bread, the telegraph operator copies three official notices, and a tailor unfolds navy cloth across a polished wooden counter while customers wait in a line that bends toward the fountain. Before noon a surveyor compares bridge numbers against an old ledger, two cousins argue cheerfully about the best route to the fish market, and a choir director rehearses a patient scale that echoes against the warehouse wall. The lighthouse keeper reports that the northern channel is calmer than expected, the harbor pilot recommends a slower turn near the sandbar, and the customs officer stamps a packet of forms before waving a cart through the side gate. Later the schoolteacher returns with another group of students, asking them to observe the colors of rope, paint, stone, and water so they can write more exact descriptions in the classroom. A photographer kneels beside a rain barrel to capture the reflection of the clock tower, a mechanic tightens a brass hinge on the tram door, and an elderly traveler asks the clerk whether the evening ferry still stops at the orchard village beyond the marsh. As dusk arrives, lamps are trimmed again, shutters are tested against the wind, and the station kitchen sends bowls of soup to workers who remain on the late shift. The report continues with notes about a carpenter measuring floorboards in the east hall, a florist tying silver ribbon around the last stems of the day, and a violin case resting open on a bench beside the ticket window while its owner copies melody marks into a notebook. Long after the market gate closes, the clerk still writes that the harbor road stays busy, the river glints beneath scattered lamps, and the town maintains its patient rhythm of signals, footsteps, voices, bells, and distant engines. On the third day the clerk decides the record should be more precise, so she marks each event by the quarter hour and notes which sounds carry farthest through the station concourse. At first light she hears broom bristles on the stone steps, kettle lids in the cafe kitchen, and the slow scrape of crates being nudged across a loading cart beside the river wall. A messenger in a green coat delivers two canvas pouches, the ticket agent counts rolled coins into a brass tray, and a mother reads directions aloud while her son traces the painted ferry schedule with one curious finger. Midmorning brings a burst of sunlight across the waiting hall, making every brass handle shine while the museum guide escorts visitors toward the gallery of maps and navigational instruments. A porter pauses to describe the oldest compass in the display, a student sketches the harbor outline in graphite, and an apprentice clockmaker compares the station bell to a pocket watch that once belonged to his grandfather. By noon the fish market sends salt and seaweed scents through the open doors, tram wheels hiss at the curb, and the baker from the square exchanges a laugh with the florist who is carrying fresh lilies to the hotel veranda. The clerk writes that a cooper rolls three narrow barrels toward the cellar ramp, a translator copies weather bulletins for inland travelers, and a painter in a blue scarf studies the changing color of the tide as if each small wave might explain a different part of the sky. In the late afternoon the station master reviews freight tags, the customs officer checks a parcel of glassware, and a choir of children crosses the square singing a phrase so soft that the watchman removes his cap to listen. Evening settles slowly; lamps brighten in sequence, a cook inventories apples and onions in the pantry, and two sailors spread a faded chart on a crate so they can debate whether the shoals have shifted since the previous autumn. Before sleep the clerk closes the day with a final note that every vessel is accounted for, every platform has been swept, every lock has been tested twice, and the harbor seems ready to welcome another tide, another market, and another patient stream of voices at sunrise.", + "language": "en", + "voice_ref": "resources/index_tts2/official_examples/voice_12.wav", + "seed": 1234 + } + ] + }, { "id": "supertonic_tts_longform", "coverage": "Supertonic preset voice TTS with shared long-form text for chunking and RTF measurement", @@ -382,6 +402,36 @@ "seed": 1234 } ] + }, + { + "id": "index_tts2_5_longform_voice_clone_6000_emotion_text", + "coverage": "IndexTTS2.5 longform voice clone with 6000-character text plus long emotion-text conditioning", + "family": "index_tts2", + "model": "models/IndexTTS2.5-GGUF", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "longform_voice_clone_emotion_text", + "text_repeat": { + "text": "The archivist kept her voice steady while reading the witness report aloud, pausing after each sentence so the council could follow the chain of events without losing the emotional thread. ", + "chars": 6000 + }, + "voice_ref": "resources/index_tts2/official_examples/voice_12.wav", + "emotion_repeat": { + "text": "Speak with restrained fear, controlled urgency, and a careful softness, as if protecting a secret while trying not to alarm the listener. ", + "chars": 2400 + }, + "options": { + "emotion_alpha": 0.6, + "use_random_emotion": false + }, + "seed": 1234 + } + ] } ] } diff --git a/tools/convert_index_tts2_5.py b/tools/convert_index_tts2_5.py new file mode 100644 index 00000000..59cbe6e9 --- /dev/null +++ b/tools/convert_index_tts2_5.py @@ -0,0 +1,344 @@ +#!/usr/bin/env python3 +"""Prepare an official IndexTTS-2.5 checkpoint for audio.cpp's GGUF converter. + +Point --model-dir at a complete IndexTeam/IndexTTS-2.5 snapshot. The script writes +a staging directory whose Safetensors layout matches the tensor namespaces the +index_tts2_5 engine expects, plus a root/ directory with the sidecar files +(config, tokenizer, auxiliary model configs) that audiocpp_gguf embeds into the +final GGUF. + +The w2v-bert-2.0, CAMPPlus and BigVGAN checkpoints are not part of the official +snapshot; they are auto-detected under /hf_cache/ (where the official +downloader places them) and each can be overridden explicitly. + +This tool does not download anything and never writes into --model-dir. + +Example: + python tools/convert_index_tts2_5.py \ + --model-dir /path/to/IndexTTS-2.5 \ + --output-dir /path/to/staging + +Then run the printed audiocpp_gguf command, or let the script run it: + + python tools/convert_index_tts2_5.py \ + --model-dir /path/to/IndexTTS-2.5 \ + --output-dir /path/to/staging \ + --run-converter /path/to/audiocpp_gguf --type f16 + +Add --native-dir /path/to/native-model to also emit a directly loadable +native Safetensors model directory (no GGUF conversion needed). +""" + +from __future__ import annotations + +import argparse +import os +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Dict + +import torch +from safetensors.torch import save_file + +# GGUF tensor namespaces (must match the index_tts2 model spec) and the +# staging file each one is produced from. +TENSOR_OUTPUTS = [ + ("gpt", "gpt.safetensors"), + ("s2mel", "s2mel.safetensors"), + ("speaker_matrix", "speaker_matrix.safetensors"), + ("emotion_matrix", "emotion_matrix.safetensors"), + ("wav2vec2bert_stats", "wav2vec2bert_stats.safetensors"), + ("wav2vec2bert", "wav2vec2bert.safetensors"), + ("semantic_codec", "semantic_codec.safetensors"), + ("campplus", "campplus.safetensors"), + ("bigvgan", "bigvgan.safetensors"), + ("qwen_emotion", "qwen_emotion.safetensors"), +] + +QWEN_SIDECARS = ( + "config.json", + "generation_config.json", + "tokenizer.json", + "tokenizer_config.json", + "vocab.json", + "merges.txt", +) + + +def _require_file(path: Path, label: str) -> Path: + if not path.is_file(): + raise FileNotFoundError(f"missing {label}: {path}") + return path + + +def _load_checkpoint(path: Path): + return torch.load(path, map_location="cpu", weights_only=False) + + +def _flatten(obj, prefix: str = "", out: Dict[str, torch.Tensor] | None = None) -> Dict[str, torch.Tensor]: + if out is None: + out = {} + if isinstance(obj, dict): + for key, value in obj.items(): + _flatten(value, f"{prefix}{key}.", out) + elif hasattr(obj, "shape"): + out[prefix.rstrip(".")] = obj.contiguous() + else: + raise TypeError(f"unexpected non-tensor leaf at {prefix!r}: {type(obj)}") + return out + + +def _save_safetensors(tensors: Dict[str, torch.Tensor], path: Path) -> None: + save_file(tensors, str(path)) + print(f"wrote {path} ({len(tensors)} tensors)") + + +def _convert_gpt(model_dir: Path, output_dir: Path) -> None: + # gpt.pth is already a flat tensor dict (includes spk_emb_proj and + # lang_embedding; the 2.5 checkpoint has no conditioning_encoder/speed_emb). + obj = _load_checkpoint(_require_file(model_dir / "gpt.pth", "gpt.pth")) + _save_safetensors(_flatten(obj), output_dir / "gpt.safetensors") + + +def _convert_s2mel(model_dir: Path, output_dir: Path) -> None: + # s2mel.pth wraps the state dict under "net" (cfm.*/length_regulator.*/gpt_layer.*). + obj = _load_checkpoint(_require_file(model_dir / "s2mel.pth", "s2mel.pth")) + if isinstance(obj, dict) and isinstance(obj.get("net"), dict): + obj = obj["net"] + _save_safetensors(_flatten(obj), output_dir / "s2mel.safetensors") + + +def _convert_semantic_codec(model_dir: Path, output_dir: Path) -> None: + # codec.pth wraps the state dict under "model" (encoder.*/decoder.*/quantizer.*/down/up). + obj = _load_checkpoint(_require_file(model_dir / "codec.pth", "codec.pth")) + if isinstance(obj, dict) and isinstance(obj.get("model"), dict): + obj = obj["model"] + _save_safetensors(_flatten(obj), output_dir / "semantic_codec.safetensors") + + +def _convert_emotion_matrices(model_dir: Path, output_dir: Path) -> None: + # feat1.pt/feat2.pt hold a single root-level tensor each: (73, 192) speaker + # matrix and (73, 1280) emotion matrix. + speaker = _load_checkpoint(_require_file(model_dir / "feat1.pt", "feat1.pt")) + emotion = _load_checkpoint(_require_file(model_dir / "feat2.pt", "feat2.pt")) + _save_safetensors({"tensor": speaker.float().contiguous()}, output_dir / "speaker_matrix.safetensors") + _save_safetensors({"tensor": emotion.float().contiguous()}, output_dir / "emotion_matrix.safetensors") + + +def _convert_wav2vec2bert_stats(model_dir: Path, output_dir: Path) -> None: + obj = _load_checkpoint(_require_file(model_dir / "wav2vec2bert_stats.pt", "wav2vec2bert_stats.pt")) + flat = {key: value.float().contiguous() for key, value in _flatten(obj).items()} + _save_safetensors(flat, output_dir / "wav2vec2bert_stats.safetensors") + + +def _convert_campplus(campplus_checkpoint: Path, output_dir: Path) -> None: + # The engine binds CAMPPlus weights under the "speaker_encoder." prefix. + obj = _load_checkpoint(_require_file(campplus_checkpoint, "campplus checkpoint")) + flat = _flatten(obj) + flat = {key if key.startswith("speaker_encoder.") else f"speaker_encoder.{key}": value for key, value in flat.items()} + _save_safetensors(flat, output_dir / "campplus.safetensors") + + +def _convert_bigvgan(bigvgan_dir: Path, output_dir: Path) -> None: + # bigvgan_generator.pt stores keys with a "generator." prefix; the engine + # expects bare names (conv_pre/ups.N/...). + obj = _load_checkpoint(_require_file(bigvgan_dir / "bigvgan_generator.pt", "bigvgan generator")) + flat = _flatten(obj) + flat = {key[len("generator."):] if key.startswith("generator.") else key: value for key, value in flat.items()} + _save_safetensors(flat, output_dir / "bigvgan.safetensors") + + +def _copy(src: Path, dst: Path, label: str) -> None: + _require_file(src, label) + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(src, dst) + print(f"copied {src} -> {dst}") + + +def _stage_config_v2_5(src: Path, dst: Path) -> None: + """Stage config.yaml with the version field normalized to "2.5". + + The official IndexTTS-2.5 snapshot ships config.yaml with `version: 2.0` + (inherited from IndexTTS-2). audio.cpp selects the IndexTTS2 family variant + from this field, so the staged copy must declare 2.5 explicitly. + """ + _require_file(src, "config.yaml") + dst.parent.mkdir(parents=True, exist_ok=True) + lines = src.read_text(encoding="utf-8").splitlines() + replaced = False + for i, line in enumerate(lines): + if line.strip().startswith("version:"): + lines[i] = 'version: "2.5"' + replaced = True + break + if not replaced: + lines.append('version: "2.5"') + dst.write_text("\n".join(lines) + "\n", encoding="utf-8") + print(f"staged {src} -> {dst} (version normalized to \"2.5\")") + + +def build_converter_command(output_dir: Path, converter: str, quant_type: str) -> list[str]: + command = [converter] + for namespace, filename in TENSOR_OUTPUTS: + command += ["--input", f"{namespace}={output_dir / filename}"] + command += [ + "--root", str(output_dir / "root"), + "--family", "index_tts2", + "--type", quant_type, + "--output", str(output_dir / f"index-tts2_5-{quant_type}.gguf"), + ] + return command + + +# Native Safetensors model directory layout: the spec's safetensors source maps +# logical tensor groups to these paths under the model root. +NATIVE_TENSOR_LAYOUT = [ + # (staging filename, relative path in the native model directory) + ("gpt.safetensors", "gpt.safetensors"), + ("s2mel.safetensors", "s2mel.safetensors"), + ("speaker_matrix.safetensors", "feat1.safetensors"), + ("emotion_matrix.safetensors", "feat2.safetensors"), + ("wav2vec2bert_stats.safetensors", "wav2vec2bert_stats.safetensors"), + ("wav2vec2bert.safetensors", "w2v-bert-2.0/model.safetensors"), + ("semantic_codec.safetensors", "semantic_codec_model.safetensors"), + ("campplus.safetensors", "campplus.safetensors"), + ("bigvgan.safetensors", "bigvgan/model.safetensors"), + ("qwen_emotion.safetensors", "qwen0.6bemo4-merge/model.safetensors"), +] + + +def _link_or_copy(src: Path, dst: Path) -> None: + dst.parent.mkdir(parents=True, exist_ok=True) + if dst.exists(): + dst.unlink() + try: + os.link(src, dst) + except OSError: + shutil.copyfile(src, dst) + + +def write_native_layout(output_dir: Path, native_dir: Path) -> None: + """Assemble the directly loadable native Safetensors model directory.""" + for staging_name, relative in NATIVE_TENSOR_LAYOUT: + _link_or_copy(_require_file(output_dir / staging_name, staging_name), native_dir / relative) + root_dir = output_dir / "root" + for name in ("config.yaml", "multilingual_zh_ja_yue_char_del.tiktoken", + "w2v-bert-2.0/config.json", "w2v-bert-2.0/preprocessor_config.json", + "bigvgan/config.json"): + _link_or_copy(_require_file(root_dir / name, name), native_dir / name) + for name in QWEN_SIDECARS: + _link_or_copy(_require_file(root_dir / "qwen0.6bemo4-merge" / name, name), + native_dir / "qwen0.6bemo4-merge" / name) + print(f"native model directory written to {native_dir}") + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Stage an official IndexTTS-2.5 snapshot for audio.cpp's GGUF converter.", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--model-dir", type=Path, required=True, + help="Path to the official IndexTTS-2.5 snapshot (gpt.pth, s2mel.pth, ...).") + parser.add_argument("--output-dir", type=Path, required=True, + help="Staging directory to write; created if missing. Never inside --model-dir.") + parser.add_argument("--w2v-bert-dir", type=Path, default=None, + help="Directory with w2v-bert-2.0 model.safetensors/config.json/preprocessor_config.json. " + "Default: /hf_cache/w2v-bert-2.0") + parser.add_argument("--campplus-checkpoint", type=Path, default=None, + help="Path to campplus_cn_common.bin. Default: /hf_cache/campplus/campplus_cn_common.bin") + parser.add_argument("--bigvgan-dir", type=Path, default=None, + help="Directory with bigvgan_generator.pt and config.json. Default: /hf_cache/bigvgan") + parser.add_argument("--run-converter", type=str, default=None, + help="Path to the audiocpp_gguf executable; when given, run the GGUF conversion right away.") + parser.add_argument("--native-dir", type=Path, default=None, + help="Also write a directly loadable native Safetensors model directory (the layout of " + "the spec's safetensors source: feat1/feat2.safetensors, semantic_codec_model.safetensors, " + "bigvgan/model.safetensors, w2v-bert-2.0/, qwen0.6bemo4-merge/). Files are hardlinked " + "from the staging directory when possible, copied otherwise.") + parser.add_argument("--type", dest="quant_type", default="f16", + choices=("orig", "f16", "bf16", "q8_0", "q2_k", "q3_k", "q4_k", "q5_k", "q6_k"), + help="GGUF weight type used with --run-converter and in the printed command (default: f16).") + args = parser.parse_args() + + model_dir = args.model_dir.resolve() + output_dir = args.output_dir.resolve() + if not model_dir.is_dir(): + print(f"error: --model-dir does not exist: {model_dir}", file=sys.stderr) + return 1 + if output_dir == model_dir or model_dir in output_dir.parents: + print("error: --output-dir must not be inside --model-dir", file=sys.stderr) + return 1 + + hf_cache = model_dir / "hf_cache" + w2v_bert_dir = (args.w2v_bert_dir or hf_cache / "w2v-bert-2.0").resolve() + campplus_checkpoint = (args.campplus_checkpoint or hf_cache / "campplus" / "campplus_cn_common.bin").resolve() + bigvgan_dir = (args.bigvgan_dir or hf_cache / "bigvgan").resolve() + + output_dir.mkdir(parents=True, exist_ok=True) + root_dir = output_dir / "root" + root_dir.mkdir(parents=True, exist_ok=True) + + _convert_gpt(model_dir, output_dir) + _convert_s2mel(model_dir, output_dir) + _convert_semantic_codec(model_dir, output_dir) + _convert_emotion_matrices(model_dir, output_dir) + _convert_wav2vec2bert_stats(model_dir, output_dir) + _convert_campplus(campplus_checkpoint, output_dir) + _convert_bigvgan(bigvgan_dir, output_dir) + + # Auxiliary checkpoints ship as Safetensors already; copy them through. + _copy(w2v_bert_dir / "model.safetensors", output_dir / "wav2vec2bert.safetensors", "w2v-bert-2.0 weights") + _copy(model_dir / "qwen0.6bemo4-merge" / "model.safetensors", output_dir / "qwen_emotion.safetensors", + "qwen emotion weights") + + # Sidecar files embedded into the GGUF via --root. + _stage_config_v2_5(model_dir / "config.yaml", root_dir / "config.yaml") + _copy(model_dir / "multilingual_zh_ja_yue_char_del.tiktoken", + root_dir / "multilingual_zh_ja_yue_char_del.tiktoken", "tiktoken vocabulary") + _copy(w2v_bert_dir / "config.json", root_dir / "w2v-bert-2.0" / "config.json", "w2v-bert-2.0 config") + _copy(w2v_bert_dir / "preprocessor_config.json", root_dir / "w2v-bert-2.0" / "preprocessor_config.json", + "w2v-bert-2.0 preprocessor config") + _copy(bigvgan_dir / "config.json", root_dir / "bigvgan" / "config.json", "bigvgan config") + for name in QWEN_SIDECARS: + _copy(model_dir / "qwen0.6bemo4-merge" / name, root_dir / "qwen0.6bemo4-merge" / name, f"qwen sidecar {name}") + + if args.native_dir is not None: + native_dir = args.native_dir.resolve() + if native_dir == model_dir or model_dir in native_dir.parents: + print("error: --native-dir must not be inside --model-dir", file=sys.stderr) + return 1 + native_dir.mkdir(parents=True, exist_ok=True) + write_native_layout(output_dir, native_dir) + + converter = args.run_converter or "audiocpp_gguf" + command = build_converter_command(output_dir, converter, args.quant_type) + lines = [command[0]] + index = 1 + while index < len(command): + flag = command[index] + if flag.startswith("--") and index + 1 < len(command) and not command[index + 1].startswith("--"): + lines.append(f" {flag} {command[index + 1]} \\") + index += 2 + else: + lines.append(f" {flag} \\") + index += 1 + lines[-1] = lines[-1].rstrip(" \\") + print() + print("staging complete. Convert to GGUF with:") + print("\n".join(lines)) + + if args.run_converter is not None: + print() + print("running converter...") + result = subprocess.run(command) + if result.returncode != 0: + print(f"error: converter exited with {result.returncode}", file=sys.stderr) + return result.returncode + print(f"GGUF written to {output_dir / f'index-tts2_5-{args.quant_type}.gguf'}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/webui/configs/model_params.json b/webui/configs/model_params.json index 407bc115..e414f7e7 100644 --- a/webui/configs/model_params.json +++ b/webui/configs/model_params.json @@ -164,6 +164,7 @@ ], "index_tts2": [ + {"name": "lang", "type": "choice", "label": "lang（语种提示, 仅 IndexTTS2.5 模型）", "label_en": "lang (language hint, IndexTTS2.5 models only)", "default": "auto", "choices": ["auto", "zh", "en", "ja", "es", "ar"], "info": "仅对 IndexTTS2.5（多语种）模型生效：auto 含汉字按中文，否则按英文；日/西/阿建议显式选择", "info_en": "Only applies to IndexTTS2.5 (multilingual) models: auto picks zh when the text contains Han characters, otherwise en; set ja/es/ar explicitly"}, {"name": "emotion_text", "type": "text", "label": "emotion_text（情绪参考文本）", "label_en": "emotion_text (emotion reference text)", "default": "", "placeholder": "例：你吓死我了！你是鬼吗？", "placeholder_en": "e.g. You scared me to death!", "info": "填写后自动开启情感条件（use_emotion_text）", "info_en": "Setting this enables emotion conditioning."}, {"name": "emotion_alpha", "type": "slider", "label": "emotion_alpha（情感强度）", "label_en": "emotion_alpha", "default": 1.0, "minimum": 0.0, "maximum": 1.0, "step": 0.05}, {"name": "use_emotion_text", "type": "bool", "label": "use_emotion_text（从朗读文本推断情感）", "label_en": "use_emotion_text (infer from text)", "default": false}, diff --git a/webui/configs/models_catalog.json b/webui/configs/models_catalog.json index 671c8b7b..6e19b9bd 100644 --- a/webui/configs/models_catalog.json +++ b/webui/configs/models_catalog.json @@ -15,6 +15,9 @@ { "id": "voxcpm2", "display_name": "VoxCPM2 (tts)", "family": "voxcpm2", "path": "models/VoxCPM2", "task": "tts", "mode": "offline", "download_id": "voxcpm2", "session_options": { "voxcpm2.weight_type": "q8_0" }, "min_vram_gb": 6 }, { "id": "vibevoice", "display_name": "VibeVoice 1.5B (tts, long-form/multi-speaker)", "family": "vibevoice", "path": "models/VibeVoice-1.5B", "task": "tts", "mode": "offline", "download_id": "vibevoice_1_5b", "min_vram_gb": 7 }, { "id": "index-tts2", "display_name": "IndexTTS2 (tts 中英克隆+情感)", "display_name_en": "IndexTTS2 (tts, zh/en clone + emotion)", "family": "index_tts2", "path": "models/IndexTTS-2", "task": "tts", "mode": "offline", "download_id": "index_tts2", "min_vram_gb": 8 }, + { "id": "index-tts2.5", "display_name": "IndexTTS2.5 (tts 多语种克隆+情感, GGUF Q8)", "display_name_en": "IndexTTS2.5 (tts, zh/en/ja/es/ar clone + emotion, GGUF Q8)", "family": "index_tts2", "path": "models/IndexTTS2.5-GGUF", "task": "tts", "mode": "offline", "download_id": "index_tts2_5_q8_0", "min_vram_gb": 8, + "input_hint": "**IndexTTS2.5**：中/英/日/西/阿零样本克隆；上传参考音色即克隆；可在『其它参数(JSON)』里传 `lang`（默认 auto：含汉字按中文，否则按英文）与情感选项。许可证为 bilibili Model Use License（非 OSI），商用前请确认条款。", + "input_hint_en": "**IndexTTS2.5**: zero-shot cloning in zh/en/ja/es/ar. Upload a reference voice to clone; pass `lang` (default auto: zh when the text contains Han characters, otherwise en) and emotion options through the JSON box. Weights are under the bilibili Model Use License (not OSI-approved) — check terms before commercial use." }, { "id": "irodori-tts", "display_name": "Irodori-TTS v4 Small (tts 日语, GGUF Q8)", "display_name_en": "Irodori-TTS v4 Small (ja tts, GGUF Q8)", "family": "irodori_tts", "path": "models/Irodori-TTS-v4-Small-GGUF", "task": "tts", "mode": "offline", "download_id": "irodori_tts_v4_small_q8_0", "min_vram_gb": 4, "input_hint": "**Irodori-TTS v4 Small**：日语 TTS；可不上传参考音色直接生成，也可上传参考音色进行克隆；可在声音设计页用日语 caption 描述音色。", "input_hint_en": "**Irodori-TTS v4 Small**: Japanese TTS. Generate without a reference voice, clone from an uploaded reference, or use the voice-design page with a Japanese voice caption." }, diff --git a/webui/configs/required_files.json b/webui/configs/required_files.json index dbc337e4..fc556abc 100644 --- a/webui/configs/required_files.json +++ b/webui/configs/required_files.json @@ -366,6 +366,15 @@ "qwen0.6bemo4-merge/merges.txt", "qwen0.6bemo4-merge/model.safetensors" ], + "index_tts2_5_q8_0": [ + "index-tts2_5-q8_0.gguf" + ], + "index_tts2_5_f16": [ + "index-tts2_5-f16.gguf" + ], + "index_tts2_5_orig": [ + "index-tts2_5-orig.gguf" + ], "mel_band_roformer": [ "config.json", "model.safetensors"