Feat/stable models

.gitignore

@@ -230,7 +230,3 @@ wandb/
 # AI Agents
 .claude
 .agents
-
-# VoiceStudio migration workspace (vendored source + downloaded checkpoints)
-/dep
-/ckpts

PROJECT.md

@@ -100,12 +100,14 @@ and the vendored copy is removed from `dep/`.
 
 - **https://github.com/latentforge/audiotools** — analyze what depends on it, remove the
   dependency, then delete the repo.
-- **https://github.com/latentforge/vocos** — analyze what depends on it, remove the
-  dependency, then delete the repo.
+- **https://github.com/latentforge/vocos** — dependency was unused anywhere in this repo
+  (`F5TTSProcessor.decode` already took a generic vocoder callable), dropped. Repo not
+  yet deleted.
 - **https://github.com/latentforge/speechbrain** — this fork exists only to support a
-  newer torch version. If upstream speechbrain has since caught up, drop the fork and
-  depend on upstream (or drop the dependency entirely if unused after migration); then
-  delete the fork.
+  newer torch version. Upstream speechbrain has caught up (v1.1.0 supports
+  `torch>=2.1.0` with no upper bound), and the dependency turned out to be unused
+  anywhere in this repo, so it was dropped entirely rather than switched to upstream.
+  Delete the fork.
 - **https://github.com/sarulab-speech/UTMOSv2** — decouple via the `evaluate` library
   using https://huggingface.co/spaces/sarulab-speech/UTMOSv2/ as the reference
   implementation. A new `voicestudio/metrics/` folder may be created if needed for this.
@@ -198,10 +200,10 @@ pretrained checkpoint yet; see "Runtime-verified" for that.
 | F5-TTS | Yes | No | No | Full reimplementation, DiT flow-matching. Predicts mel spectrograms only; `F5TTSProcessor.decode` needs an external vocoder. `forward()` previously had no `labels`/loss path at all; added. |
 | PromptTTS++ (`prompt_tts_pp`) | Yes | No | No | `FastSpeech2Conformer`/`FastSpeech2ConformerHifiGan` from transformers-tts, conditioned via a newly-implemented BERT-based prompt encoder. `return_dict=False` tuple-indexing bug fixed. No public upstream checkpoint exists. |
 | OmniVoice | Yes | No | No | No transformers-tts lineage (closest in spirit to CSM/Moshi); modeling code is new, audio tokenizer reused from transformers-tts's `HiggsAudioV2TokenizerModel`. Training-time sample masking and reference-audio auto-transcription (ASR) not ported. |
-| audiotools dependency removal | Not started | | | |
-| vocos dependency removal | Not started | | | |
-| speechbrain fork removal | Not started | | | Check if upstream now supports the required torch version |
-| UTMOSv2 decoupling | Not started | | | Route through `evaluate`, may need `voicestudio/metrics/` |
+| audiotools dependency removal | Done | | | No reference to `audiotools` remains in `pyproject.toml` or `voicestudio/`; already dead after the model migrations, nothing to change. |
+| vocos dependency removal | Done | | | Dependency was declared in `pyproject.toml` but never imported anywhere in the codebase; `F5TTSProcessor.decode` already took a generic `vocoder` callable rather than importing `vocos` directly. Removed the unused `pyproject.toml` entry and reworded docstrings/README that named `vocos` as if required. No transformers-tts-native vocoder matches F5-TTS's mel config (24kHz, 100 mel channels), so a caller-supplied vocoder is still required at `decode()` time; repo not deleted per task instructions. |
+| speechbrain fork removal | Done | | | Unused in repo; dependency dropped entirely (not switched to upstream). |
+| UTMOSv2 decoupling | Done | | | `voicestudio/metrics/utmos.py` adapts UTMOSv2 to the `evaluate.Metric` interface; `utmosv2` git dependency dropped from `pyproject.toml`, now an optional runtime import. Upstream model code (five-fold SSL + image-classifier ensemble, hydra config system) not vendored, see module docstring. |
 
 ### Known gap: upstream git history
 

pyproject.toml

@@ -61,8 +61,7 @@ eval = [
     "pyworld>=0.3.5",
     "pysptk>=1.0.1",
     "jiwer>=4.0.0",
-    "speechbrain",
-    "utmosv2",
+    "evaluate>=0.4.0",
 ]
 f5_tts = [
     "pypinyin>=0.51.0",
@@ -93,9 +92,6 @@ torchaudio = [
     { index = "pytorch-windows", marker = "platform_system == 'Windows'" },
 ]
 transformers = { git = "https://github.com/latentforge/transformers-tts", branch = "research" }
-vocos = { git = "https://github.com/latentforge/vocos" }
-speechbrain = { git = "https://github.com/latentforge/speechbrain", rev = "fix/support_torch2.9" }
-utmosv2 = { git = "https://github.com/sarulab-speech/UTMOSv2" }
 
 [[tool.uv.index]]
 name = "pytorch-windows"

voicestudio/metrics/__init__.py

@@ -0,0 +1,4 @@
+from .utmos import UTMOSv2Metric
+
+
+__all__ = ["UTMOSv2Metric"]

voicestudio/metrics/utmos.py

@@ -0,0 +1,96 @@
+"""UTMOSv2 naturalness MOS metric."""
+
+from typing import List, Optional, Union
+
+import numpy as np
+
+import evaluate
+
+
+_CITATION = """\
+@inproceedings{baba2024utmosv2,
+    title = {The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech},
+    author = {Baba, Kaito and Nakata, Wataru and Saito, Yuki and Saruwatari, Hiroshi},
+    booktitle = {IEEE Spoken Language Technology Workshop (SLT)},
+    year = {2024},
+}
+"""
+
+_DESCRIPTION = """\
+UTMOSv2 predicts the naturalness Mean Opinion Score (MOS) of synthesized speech without a
+reference recording. This metric wraps the pretrained ensemble released by the UTMOSv2 authors
+(https://github.com/sarulab-speech/UTMOSv2) behind the standard `evaluate.Metric` interface, so
+callers depend on `evaluate` rather than importing the upstream training/eval repository directly.
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions: list of speech file paths, or a list of waveform arrays paired with `sampling_rate`.
+    sampling_rate (`int`, *optional*): sampling rate of the waveform arrays in `predictions`. Required
+        when `predictions` are arrays rather than file paths.
+    device (`str`, *optional*): device to run inference on, e.g. `"cuda"` or `"cpu"`. Defaults to the
+        device UTMOSv2 selects automatically.
+
+Returns:
+    mos (`List[float]`): predicted naturalness MOS, one value per prediction, on the 1-5 scale.
+"""
+
+
+class UTMOSv2Metric(evaluate.Metric):
+    """
+    Reference-free naturalness MOS metric backed by the pretrained UTMOSv2 ensemble.
+
+    Loading the pretrained weights requires the `utmosv2` package (`pip install
+    git+https://github.com/sarulab-speech/UTMOSv2`). It is not a VoiceStudio dependency: UTMOSv2's
+    architecture is a five-fold ensemble that fuses an SSL branch with an image-classifier branch
+    over mel-spectrogram crops, driven by a hydra config system for architecture/fold selection.
+    That is real model code, not a thin scoring call, so it is not vendored here; this class only
+    adapts the upstream package's public `create_model`/`predict` API to the `evaluate.Metric`
+    shape and lazily imports it so `voicestudio` itself never requires it at import time.
+    """
+
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=evaluate.Features(
+                {
+                    "predictions": evaluate.Value("string"),
+                }
+            ),
+        )
+
+    def _download_and_prepare(self, dl_manager):
+        try:
+            import utmosv2
+        except ImportError as e:
+            raise ImportError(
+                "UTMOSv2Metric requires the `utmosv2` package: "
+                "pip install git+https://github.com/sarulab-speech/UTMOSv2"
+            ) from e
+
+        self._model = utmosv2.create_model(pretrained=True)
+
+    def _compute(
+        self,
+        predictions: List[Union[str, np.ndarray]],
+        sampling_rate: Optional[int] = None,
+        device: Optional[str] = None,
+    ):
+        if device is not None:
+            self._model = self._model.to(device)
+
+        mos = []
+        for prediction in predictions:
+            if isinstance(prediction, str):
+                score = self._model.predict(input_path=prediction)
+            else:
+                if sampling_rate is None:
+                    raise ValueError(
+                        "`sampling_rate` is required when `predictions` are waveform arrays."
+                    )
+                score = self._model.predict(data=prediction, sr=sampling_rate)
+            mos.append(float(score))
+
+        return {"mos": mos}