Feat/stable models - #5
Closed
b-re-w wants to merge 2857 commits into
Closed
Conversation
support vLLM >=0.11.0 (V1 engine) for better performance
support vLLM >=0.11.0 (V1 engine) for better performance
support vLLM >=0.11.0 (V1 engine) for better performance
…ialization [Security] Fix CRITICAL vulnerability: V-005
…ialization [Security] Fix CRITICAL vulnerability: V-005
…ialization [Security] Fix CRITICAL vulnerability: V-005
Added citation and license sections to README.
…#212) Under accelerate mixed_precision=bf16 (fp32 master weights), Qwen3's q_norm/k_norm multiply fp32 weights with bf16 hidden states, promoting q/k (and v via fp32 RoPE constants) to fp32. SDPA is rescued by autocast's op list, but flex_attention is not covered by autocast, so the default flex training path silently runs all attention math in fp32. At head_dim=128 the fp32 flex backward is ~12x slower than bf16 (61.8ms vs 5.1ms per layer-call, H100, identical mask/shape/layout), costing ~4-5x end-to-end training throughput; it also makes the sdpa and flex attn_implementation paths numerically inconsistent with each other. Register an 'omnivoice_flex_attention' implementation via the public AttentionInterface API that casts q/k/v to the active autocast dtype at the kernel boundary -- exactly what autocast already does for every other matmul (softmax accumulation inside the kernel stays fp32 either way) -- and select it for the backbone whenever flex_attention is configured. No-op when autocast is off or inputs are already low precision; no global attention registry entries are modified. Measured on our production fork (0.6B backbone, 8xH100 and 3xH100): step time 3.240 -> 0.755 s/it from this cast alone, loss parity within 0.43% worst-rel over 300 steps.
Previously, the asr_model_name passed to OmniVoice.from_pretrained() was silently discarded unless load_asr=True was also given: the lazy ASR load in generate() always downloaded the hardcoded openai/whisper-large-v3-turbo. Store the configured name on the instance and make load_asr_model() default to it, so local or alternative Whisper models are honored on every path.
estimate_sample_total_duration() only needs the reference duration in
seconds, but it called load_audio(), which decodes the entire file and
resamples it to SAMPLING_RATE. The resample is pointless here: duration
in seconds is invariant under resampling.
This runs once per sample inside _sort_samples_by_duration(), which is a
plain serial loop, so the cost scales linearly with the size of the test
list and cannot be hidden by --nj_per_gpu. With a single shared reference
audio the same file is decoded once per line.
Read the duration from the file header via soundfile.info() instead, and
fall back to the previous full decode for formats soundfile cannot
inspect (e.g. MP3/M4A on older libsndfile builds).
Measured on a 7.4s FLAC reference (identical duration before/after,
delta 0.0):
per call: 36.1 ms -> 2.9 ms (12x)
batch preparation, before any GPU work starts:
500 samples: 0.3 min -> 0.02 min
10,000 samples: 6.0 min -> 0.49 min
76,274 samples: 45.9 min -> 3.71 min
…(normalize_text=True) (#227)
* Add opt-in text normalization via generate(normalize_text=True)
Numbers, dates, currency, etc. are converted to their spoken form
(e.g. "2345" -> "twenty three forty five", "199" -> "一百九十九") so the
model reads them correctly. Opt-in, default False, so existing behavior
and paper reproducibility are unaffected. Chinese/English use
WeTextProcessing; other languages fall back to num2words for integers.
Inline control syntax is preserved: bracketed non-verbal tags
("[laughter]") and CMU pronunciation ("[B EY1 S]"), and Chinese pinyin
tone markers, are held out so their embedded digits are not read as
numbers. Normalization is applied to the target text only, leaving
ref_text aligned with the reference audio.
WeTextProcessing is an optional extra (omnivoice[tn]); a clear
ImportError with install guidance is raised when it is missing.
Closes #208.
* Address review: use 一-鿿 escape for CJK range; match lowercase code against LANG_IDS
- _CJK_RE: use the standard 一-鿿 unicode-range escape (readability;
byte-for-byte equivalent to the literal CJK range).
- _resolve_lang_code: check and return the lowercased `code` against LANG_IDS
instead of the raw `language`, so the canonical lowercase id is returned.
When gen_text is split into multiple chunks by chunk_text(), each chunk was generated with the full-sentence fix_duration, producing ~N×target seconds. Allocate fix_duration across chunks by UTF-8 byte weight (compensating for cross-fade overlap) so each chunk only generates its proportional share. fix_duration=None and single-chunk paths unchanged.
fix: allocate fix_duration across chunks to avoid N× duration blow-up
* feat: FlashInfer-accelerated inference (2-2.9x lossless speedup) Add omnivoice/models/omnivoice_flashinfer.py: apply_flashinfer() patches a loaded OmniVoice instance in place (no weight changes): - cond/uncond CFG sequences packed into one row with flashinfer ragged attention (no padding compute, no (2B,1,S,S) masks), KV cache disabled - fused kernels: flashinfer rmsnorm (7-kernel chain -> 1), rope apply_rope_pos_ids_inplace in NHD layout (no transposes/cat), qkv and MLP gate+up GEMM fusion with silu_and_mul, audio_heads computed on target positions only, batched unmask scoring - optional CUDA graphs: per-shape or duration-bucketed (recommended for batch=1, where kernel-launch overhead dominates) Wire --enable_flashinfer into omnivoice-infer-batch and document installation, usage and seed-tts zh benchmarks (2.1x at batch=1 eager, 2.4x with CUDA graphs, 2.6x at batch=8; outputs ASR-verified lossless) in the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> * docs: drop optimize.md reference from README (file not in repo) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds parameter-efficient finetuning as an opt-in alternative to full finetuning (use_lora: false by default keeps existing behavior). - TrainingConfig: use_lora, lora_r/alpha/dropout/bias, lora_target_modules, lora_modules_to_save - builder.py wraps the model with a PEFT LoraConfig when enabled - New omnivoice-merge-lora CLI to merge an adapter into a deployable checkpoint; omnivoice-infer gains --lora_adapter for adapter inference - Adds docs/lora_finetuning.md, example config/script, and tests/test_lora.py New optional dependency: peft (via the "lora" extra).
Member
Author
|
@copilot resolve the merge conflicts in this pull request |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.