Feat/stable models

PROJECT.md

@@ -178,21 +178,37 @@ VoiceStudio to restore it — don't fork the whole processor.
 
 ## Status tracking
 
-Update this table as each model's migration lands.
-
-| Model | Status | Notes |
-|---|---|---|
-| Qwen3-TTS | In progress | Model/config are import relays to transformers-tts. Processor subclass adds `encode`/`encode_voice_design`/`encode_custom_voice` task dispatch with `RuntimeError` on task mismatch. `encode_voice_clone` raises `NotImplementedError`: transformers-tts's `Qwen3TTSProcessor` has no reference-audio input path yet. |
-| Parler-TTS | Not started | Second in migration order. Target: use HF-registered `dac`, closest transformers lineage TBD |
-| Higgs-Audio v2 | Done | Import relay to transformers-tts. Reference processor pattern for other audio-tokenizer models |
-| Higgs-Audio v3 | In progress | `bosonai/higgs-tts-3-4b`. Weights-only checkpoint (no upstream v3 code); backbone is a plain `Qwen3Model` (no dual-FFN). Reuses transformers-tts's `Qwen3Model`, `HiggsAudioV2Embeddings`, `HiggsAudioV2PreTrainedModel`, and `HiggsAudioV2TokenizerModel`; `HiggsAudioV3Model`/`ForConditionalGeneration` are new since v2's dual-FFN decoder layer does not fit. No checkpoint conversion or real-weights test run yet; the reference-audio prompt layout in `HiggsAudioV3Processor` is unverified against the real checkpoint. |
-| Chroma | In progress | Backbone/decoder/generation loop reimplemented against transformers-tts's Llama, Qwen2.5-Omni thinker, and Mimi codec classes (no full Chroma architecture exists upstream in transformers-tts to relay to). Processor subclasses `Qwen2_5OmniProcessor` to add the reference-audio voice-cloning prompt. |
-| Spark-TTS | Not started | Already vendored in `dep/Spark-TTS` |
-| Dia | Not started | Already marked "fully tested (by HF)" in old README |
-| CosyVoice (v1/v2/v3) | In progress | Three separate model folders (`cosyvoice_v1`, `cosyvoice_v2`, `cosyvoice_v3`), matching the upstream repo's own class inheritance: v2's Qwen2-backbone LLM and causal flow decoder subclass v1's flow-matching classes, v3's LLM and DiT flow decoder subclass v2's. `cosyvoice_v1` reuses `Wav2Vec2ConformerEncoder` for the text encoder/LLM backbone; `cosyvoice_v2`/`v3` reuse `Qwen2Model`. Flow-matching U-Net/DiT estimators and the HiFTNet vocoder are new code (no transformers-tts lineage). `dep/CosyVoice` deleted. See each folder's README for known gaps (JIT/TRT/vLLM/streaming/bistream not ported; speech tokenizer and speaker encoder are external ONNX artifacts). |
-| F5-TTS | In progress | Full reimplementation (DiT flow-matching, no existing transformers-tts lineage). `RMSNorm` reuses `LlamaRMSNorm`. Model predicts mel spectrograms only; `F5TTSProcessor.decode` requires an external vocoder (e.g. `vocos`) to render audio. `dep/F5-TTS` deleted. |
-| PromptTTS++ (`prompt_tts_pp`) | In progress | Acoustic model and vocoder are `FastSpeech2Conformer`/`FastSpeech2ConformerHifiGan` from transformers-tts, conditioned via `speaker_embedding`. The BERT-based prompt encoder is newly implemented since it has no equivalent in transformers-tts. No pretrained checkpoint conversion done yet (upstream ships no public checkpoint). |
-| audiotools dependency removal | Not started | |
-| vocos dependency removal | Not started | |
-| speechbrain fork removal | Not started | Check if upstream now supports the required torch version |
-| UTMOSv2 decoupling | Not started | Route through `evaluate`, may need `voicestudio/metrics/` |
+Update this table as each model's migration lands. "Code landed" means the folder exists,
+imports, and passed a static review pass (architecture, CE-training requirement, CLAUDE.md
+conventions) with known issues fixed. It does NOT mean the model has been run against a real
+pretrained checkpoint yet; see "Runtime-verified" for that.
+
+| Model | Code landed | Runtime-verified (real checkpoint) | Upstream git history preserved | Notes |
+|---|---|---|---|---|
+| Qwen3-TTS | Yes | No | No | Import relays to transformers-tts. Processor subclass adds `encode`/`encode_voice_design`/`encode_custom_voice` task dispatch with `RuntimeError` on task mismatch. `encode_voice_clone` raises `NotImplementedError`. |
+| Parler-TTS | Yes | No | No | Import bug (`isin_mps_friendly`) fixed. Still a ~3300-line near-verbatim port with scattered `# Copied from` comments rather than a `modular_parler_tts.py`; revisit for a proper modular conversion. |
+| Higgs-Audio v2 | Yes | No | No | Import relay to transformers-tts. Reviewed clean, no issues found. |
+| Higgs-Audio v3 | Yes | No | No | `bosonai/higgs-tts-3-4b`, weights-only checkpoint (no upstream v3 code). Reuses transformers-tts's `Qwen3Model`/`HiggsAudioV2*` classes; `HiggsAudioV3Model`/`ForConditionalGeneration` are new. `audio_labels`-only crash and default-off text CE loss fixed. |
+| Chroma | Yes | No | No | Backbone/decoder reimplemented against transformers-tts's Llama, Qwen2.5-Omni thinker, and Mimi codec classes. Processor kwargs-merging and a labels-dereference guard fixed. |
+| Spark-TTS | Yes | No | No | Config previously had no LLM sub-config, so plain construction silently produced an untrained model; fixed. License header consolidated onto `modeling_spark_tts.py` only; dead code and processor/model duplication cleaned up. |
+| Dia | Yes | No | No | Import relay to transformers-tts's native Dia. Reviewed clean. |
+| CosyVoice v1 | Yes | No | No | `Wav2Vec2ConformerEncoder`-based text encoder/LLM backbone; own flow-matching/vocoder code. |
+| CosyVoice v2 | Yes | No | No | Subclasses v1; `Qwen2Model` LLM backbone. |
+| CosyVoice v3 | Yes | No | No | Subclasses v2. DiT attention-mask shape bug (crashed on padded batches) fixed. |
+| F5-TTS | Yes | No | No | Full reimplementation, DiT flow-matching. Predicts mel spectrograms only; `F5TTSProcessor.decode` needs an external vocoder. `forward()` previously had no `labels`/loss path at all; added. |
+| PromptTTS++ (`prompt_tts_pp`) | Yes | No | No | `FastSpeech2Conformer`/`FastSpeech2ConformerHifiGan` from transformers-tts, conditioned via a newly-implemented BERT-based prompt encoder. `return_dict=False` tuple-indexing bug fixed. No public upstream checkpoint exists. |
+| OmniVoice | Yes | No | No | No transformers-tts lineage (closest in spirit to CSM/Moshi); modeling code is new, audio tokenizer reused from transformers-tts's `HiggsAudioV2TokenizerModel`. Training-time sample masking and reference-audio auto-transcription (ASR) not ported. |
+| audiotools dependency removal | Not started | | | |
+| vocos dependency removal | Not started | | | |
+| speechbrain fork removal | Not started | | | Check if upstream now supports the required torch version |
+| UTMOSv2 decoupling | Not started | | | Route through `evaluate`, may need `voicestudio/metrics/` |
+
+### Known gap: upstream git history
+
+The per-model migration procedure above calls for preserving each upstream repo's git
+history via `git subtree`/`filter-repo`. In practice none of the models above got this: every
+folder's earliest commit in this repo is a fresh "Add X" commit with no upstream authorship in
+its ancestry. Retrofitting this means, per model: clone upstream fresh, `git filter-repo
+--to-subdirectory-filter` to rewrite its history under that model's folder path, then graft it
+onto `develop`'s history (e.g. `git merge --allow-unrelated-histories -X ours`, since the
+grafted paths land in a non-colliding subfolder) and force-push. In progress.

voicestudio/models/__init__.py

@@ -0,0 +1,32 @@
+from . import (
+    chroma,
+    cosyvoice_v1,
+    cosyvoice_v2,
+    cosyvoice_v3,
+    dia,
+    f5_tts,
+    higgs_audio_v2,
+    higgs_audio_v3,
+    omnivoice,
+    parler_tts,
+    prompt_tts_pp,
+    qwen3_tts,
+    spark_tts,
+)
+
+
+__all__ = [
+    "chroma",
+    "cosyvoice_v1",
+    "cosyvoice_v2",
+    "cosyvoice_v3",
+    "dia",
+    "f5_tts",
+    "higgs_audio_v2",
+    "higgs_audio_v3",
+    "omnivoice",
+    "parler_tts",
+    "prompt_tts_pp",
+    "qwen3_tts",
+    "spark_tts",
+]

voicestudio/models/chroma/_history/.gitignore

@@ -0,0 +1,207 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+.vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/