Feature/multilingual tts - #154
Open
dezihh wants to merge 40 commits into
Open
Conversation
|
Someone is attempting to deploy a commit to the CJ Dyas' projects Team on Vercel. A member of the Team first needs to authorize it. |
Member
|
Hey Derrick, went through this one properly today, ran everything rather than just reading it. The OmniVoice side is in good shape: the compiler and chat-text modules are well covered, ELD really does catch the mis-tags (a German sentence with one "ñ" gets pulled back to de), and the TTS request bodies for OpenAI TTS, Kokoro and plain single-language OmniVoice come out identical to main apart from the quote stripping you documented. Nice work on that. A few things in the shared paths bit users who never touch OmniVoice though, and since they're mostly stuff you couldn't have seen from the local/direct path, I pushed fixes straight to your branch rather than bouncing it back and forth. Four commits, all with tests where there's pure logic:
Two things I'm leaving with you since they're your parser and your call:
Vercel still needs me to authorise the fork deploy, I'll click that. Once you've had a look at the pushed commits and the buffer bit, this should be good to go. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Add a second TTS voice for foreign-language segments spoken per word in their own language, with persistent voice profiles, cloning and the OmniVoice settings UI. The prompt now prescribes inline speak() text notation and injects generic alternative-language switching rules only when the second voice is enabled. Streaming segments get buffered, split at sentence boundaries and synthesised while the model still writes.
…mplification parseActionsEnvelope, parseXmlAttributes, parseXmlSpeakTags and findLastTagOpener are still imported by chat-text and the streaming buffer, but the compiler reduction removed them, breaking SSR at startup. Restore them from the pre-simplification state.
parseXmlSpeakTags stripped <pause ms="..."/> without producing a pause segment, so pauses between XML speech tags were lost. hasIncompleteTrailingMarkup treated a closing paren inside the speak() text as the call terminator, letting raw incomplete syntax through to the chat bubble; it now balances parens/braces while skipping quoted strings. The helper moved to chat-text so it is unit-testable.
A language-teacher reply quotes foreign words inside the text field
({"text":"Das Wort "ir" bedeutet gehen."}), which broke JSON.parse
and silently dropped the whole call. A quote is now only treated as the
string boundary when a structural delimiter follows (, } ) ] : or a new
key); nested quotes are escaped during the JS-literal repair, and the
brace scanner uses the same rule so call boundaries stay aligned.
Measured against the live proxy: inputs wrapped in quote marks always
return empty audio, and one-word inputs randomly return empty audio in
about one out of five runs. Quote marks are now stripped (apostrophes
between letters are kept) and single words are repeated ("ir" ->
"ir, ir."), which synthesised reliably across all tested languages.
Segments without any speakable character (a bare ".") no longer send a
request at all. OpenAI and other providers are untouched.
…guage Models forget or mistype the speak() lang tag; a Spanish question was then synthesised as accented German. Segments tagged with the primary language are now retagged when the text clearly shows the alternative language - never the reverse, since missing diacritics prove nothing. The heuristic now also covers non-Latin scripts (Japanese, Korean, Chinese, Cyrillic, Arabic, Hebrew, Thai, Devanagari, Greek), and the Romance diacritics lists no longer contain Germanic letters (a German "fur" must never flip to Spanish).
A model emitting a regional tag ("es-ES") no longer misses the
configured alternative language ("es"); the switch compares primary
subtags. The segment's own language value stays untouched so requests
keep their full tag.
Single foreign words being taught now get their own alternative-voice call instead of riding inside the primary call (the previous rule forbade exactly that), quote marks around words are banned because they break the synthesis, and the seven prose rules condense into four plus one concrete call pattern the model can copy.
Models tend to pack gender variants ("el gato" / "la gata") into the
primary-language sentence; each variant and each gloss now gets its own
call so the alternative voice speaks every one of them.
…up detection The XML check only matched <speak/<gesture, so a partial <pause tag slipped into the chat bubble; and an escaped \" inside the text toggled the string state, misreading complete calls as still open.
The value was stored and passed along but never read by any decision; removing it keeps the alt-voice resolution state honest.
"go!" now repeats as "go, go!" while trailing commas or colons are dropped, so the enriched input never ends mid-phrase.
…h timer Alt segments must inherit primary synthesis parameters when no alt override is set; altLanguage equal to the primary language must never switch voices; and the 1500 ms flush timer must emit trailing fragments.
Two-word foreign phrases are the worst case for the diffusion model:
"por favor" returned empty audio in 5/5 measured runs, and neither
num_step, temperatures nor speed help. Short inputs in a non-primary
language are now wrapped in a carrier of their own language ("Se dice:
por favor - por favor."), which synthesised reliably in every run;
primary-language fragments stay untouched.
A model that wraps its state block in a markdown fence can split the
fence from its label at a chunk boundary (" ```" then "json"). The
fence was stripped immediately, orphaning the label, which was then
synthesised as the word "json". Trailing naked backticks are now held
back until the label arrives.
Enriching a lone marker destroyed its syntax and read the marker word
aloud ("laughter, laughter."). Marker-only inputs now pass through
unchanged so OmniVoice renders them as audio.
…nd TTS reliability tweaks Core architecture – 4 layers for reliable language switching: 1. LLM function calling: speak_segment tool with required language enum 2. ELD validation: nitotm/eld with setLanguageSubset for mis-tag detection 3. TTS reliability: cap.+period, guidance_scale=6, postprocess_output=false 4. Fallback: unset/invalid language → primary voice Key changes: - Proxy: postprocess_output, duration, guidance_scale API parameters - Prompt: buildOmniVoiceLayer() shortened to 15 lines (better LLM compliance) - Tool calling: native speak_segment in both client-direct and server-route - Language detection: looksLikeAltLanguage extended with function-word comparison (DE/ES/FR) - Inline speak() fallback: unchanged and fully functional for models without tool support - Configurable toggle in settings (default: on, only visible with alt voice) - ELD browser-compatible via eld/medium static import - All review findings (H1-H3, M1-M4, M-1, order inversion) addressed
- Remove [reminder:Nmin]...[/reminder] in stripSpeechArtifacts
- Server-side state block filter: no longer requires leading '{'
(catches blocks inside code fences)
The server-side state block filter dropped every text delta from the first one containing a state key until a closing brace. With providers that send multi-token chunks the whole block vanished before the client could parse it, so mood, affection and new_memory updates were silently lost for hosted users. The client already strips the block for display and speech, so the proxy forwards text as-is again. Sampling params stay scoped to openai-compatible endpoints as before; the tools payload gets its own spread for every non-Anthropic provider.
- sentence split only breaks on whitespace after Latin punctuation again, so 3.14, example.com and v0.13.1 stay in one sentence; CJK marks still split without a following space - the brace balancer remembers which quote opened a string, so an apostrophe inside a double-quoted value no longer ends it early and leaves the raw JSON to be spoken - complete json fences are removed whole again (JSON is never speech), dangling fence markers are still stripped afterwards
eld/medium is ~2 MB of ngram data and was statically imported on the chat page for every user. It now loads the first time a session with the alternative voice enabled starts, and the orchestrator kicks the load off at beginSession so it is usually ready before the first segment.
With speech switched off but OmniVoice still selected, every LLM request carried the speech_output_control layer and the speak_segment tool. Both now check the speech module state. Language fallbacks use en to match the prompt layer and the module default.
dezihh
force-pushed
the
feature/multilingual-tts
branch
from
August 29, 2026 17:19
60409fb to
aefb7e8
Compare
added 13 commits
August 31, 2026 14:48
Split-at-every-position harness (byte-wise + every single boundary) over an
8-entry corpus of taught speech syntax. Fixes surfaced by it:
- hold paren-less calls so the trailing ")" is consumed with the call
- hold partial fence labels (label fragments no longer leak as es-tagged word)
- hold truncated call-opener prefixes (spea/paus/gest...) in markup mode
- drop bare marker names at flush, case-sensitively (German "Pause" survives)
- never speak punctuation-only or incomplete-JSON flush fragments
- drop orphan "<" of mixed "<speak({...})" syntax before a call
- gate XML/envelope after-prose at incomplete tags, opener prefixes and calls
- defer calls preceded by a complete XML speak tag to the XML pass
Language heuristic: expand the primary function-word list (pronouns incl.
"es") so an ambiguous "es" can no longer flip an explicitly German-tagged
segment; the maintainer's "el amigo" class still flips.
Regression tests for all three maintainer observations included.
When OmniVoice speech is active with the alternative voice and tool calling enabled, the speech layer now mandates tool calls instead of teaching the inline syntax: models mixed both syntaxes before, which fed the fragile text path. The inline speak()/pause()/gesture() teaching stays as the documented fallback (tool calling off, providers without tool support). - PromptContext.ttsToolCalling mirrors the ttsTools gate exactly - pause_segment and gesture_segment join speak_segment as native tools - tool calls are converted atomically via pseudoCallFromTool (send-when-set validation: invalid pause/gesture args are dropped, never rendered) - alternative-language rule is language-neutral (articles depend on the target language, no hardcoded Spanish forms)
The OmniVoice proxy accepts a duration parameter but nothing sent it. Now: - new speech setting "Target Duration (s)" (0 = automatic, default) - TTSOptions.duration flows into the OpenAI-compatible request body with send-when-set semantics: unset/0/NaN/negative omit the field entirely, so older proxies and providers behave exactly as before - settings UI gets a number input with "auto" placeholder Duration is session-level (not per-voice), so it lives in TTSOptions only; StreamOptions stays untouched to avoid a dead configuration field.
The proxied /api/chat route only converted speak_segment tool calls back to pseudo-call text, so native pause_segment/gesture_segment calls were silently dropped on the Web/cloud path while the direct client path converted them. Both paths now share pseudoCallFromTool, keeping their behavior identical (AGENTS: duplicate surfaces must stay identical). Unknown tools are still dropped; malformed tool-call args no longer risk killing the stream (JSON parse is guarded).
Review follow-up on pseudoCallFromTool: - pause_segment: finite ms values are clamped to the taught range (100-5000) so a pause intent survives; wrong-typed values still drop - gesture_segment: type is validated against the taught gesture set (GESTURE_TYPES) instead of accepting any non-empty string - unknown tool names already returned null (tested with set_event)
Review finding: the tool-calling gate is '!== false' (default on), so existing OmniVoice + alternative-voice users get the tool-mandate layer without explicit opt-in. State the default explicitly instead of changing the default, which would be a breaking behavior change for existing users.
Teacher-style language switching broke on two wiring flaws: - pushSegment only validated segments whose declared language differs from the primary, so German-tagged segments quoting Spanish were never checked by the detector at all - validateLanguageTag resolved every disagreement to the primary voice, so even checked segments could only ever end up German Now every segment is validated and a reliable ELD detection wins: the detected alt language flips the segment to its own voice (H2), while a detection of the primary language itself stays conservative (avoids a wrong accent, keeps the M2 purpose). An isReliable gate prevents short-phrase misdetections from overriding correct tags. Existing 2-argument call sites keep their behavior (detection wins, regional tags stay raw).
Teacher-style mixes quote foreign phrases inside a primary-language
sentence. Whole-segment detection cannot help there: ELD returns one
language per text and the dominant one wins ('El coche es rojo. - Das
Auto ist rot.' detects as German). The foreign quote stayed on the
primary voice.
splitByDetectedLanguage fragments segment text at sentence and dash
boundaries (separators stay attached to the preceding fragment, so
joining reproduces the input exactly), resolves each fragment with ELD
and merges adjacent same-language fragments. Unreliable or undetected
fragments inherit the declared language. pushSegment now validates,
splits and enqueues one pipeline item per language part; separator-only
fragments are filtered so no empty TTS segments are produced.
Mid-sentence quotes without a sentence boundary ("el coche es rojo, das
Auto ist rot.") were resolved wholesale by ELD, so one of the two
languages was spoken with the wrong voice. Signal-anchored token runs
(romance articles, pronouns, ¿/¡, alt-only diacritics, bare -ar/-ir
infinitives) now hand the affected phrase to the alt voice in both
directions, with an article guard against German look-alikes and
reliable-detection demotion against false positives. Whitespace-only
pieces are folded into their neighbour so joining still reproduces the
input exactly.
…edges English quotes inside German sentences now anchor on tokens German text practically never contains (the, and, of, to, you, I, it, …, n't contractions); a German "es" closes English runs without breaking the Spanish verb "es". Romance-only signals (diacritics, -ar/-ir infinitives) are gated to romance alt languages, possessives close Spanish runs, closing ¿…? questions are detected at sentence end, and a reliable primary detection demotes false positives such as proper names starting with "El".
Legacy inline language markup ("[lang:es]…[/lang:es]") was only cleaned
on the envelope/xml paths of the speech buffer — the plaintext path
spoke the raw tags. cleanSpeechMarkers now also runs in emit(), closing
tags tolerate a repeated code ("[/lang:es]", "</speak:es>"), orphan
tags without a matching partner are dropped, and unclosed "[lang:"
openers hold the display chunk back until the closer arrives.
Manual testing of the tea vocabulary dialogue surfaced four issues, all
spoken with the wrong voice:
- An es-declared short sentence kept its German gloss in the alt voice
("El té – der Tee."): the run-less path short-circuited before the
German stop-token check. Alt-declared pieces now split at the first
German stop token.
- German nouns collide with lowercase romance articles ("Los geht's" vs
Spanish "los"): articles only anchor in lowercase or at piece start.
- ELD misreads such colloquialisms as Spanish outright: apostrophe-s
contractions (impossible in romance languages) now close runs and
demote the piece to the declared language.
- The buffer pre-tag fired on é ("Café") — removed from the es
diacritics like in the detector.
Surrounding parts of a detected run are now resolved recursively, so a
quote before a German clause that itself contains a Spanish quote keeps
all three voices.
"la noche bedeutet die Nacht." swallowed "bedeutet" into the Spanish run; the over-extended run then made ELD demote the whole run to German, so "la noche" was spoken with the German voice. heißt/bedeutet/sagt/ meint/nennt now close alt runs like the other German stop words.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds full multi-language TTS support for OmniVoice across three iterative improvements (branches v1–v3), consolidated into one feature branch. The feature is built for language-teaching use cases where the LLM switches between languages mid-conversation.
Alternative Voice & Language (base)
A new Alternative Language section in the OmniVoice settings lets users configure a second voice for foreign-language text:
es,fr)speak({"text":"...","lang":"es"})calls that trigger the voice switchLLM Prompt & Speech Pipeline (v2)
buildOmniVoiceLayer()– instructs the LLM to producespeak({text, lang})pseudo-calls for every spoken segmentstreaming-speech-buffer.ts– real-time parsing of pseudo-calls, language-aware retagging via diacritics and scriptsspeech-compiler.ts– segment validation, splitting, merging, and language resolutionvoice-orchestrator.ts– resolves segment language to the configured primary or alternative voiceopenai-tts.ts– sanitizes OmniVoice input (quote removal, text enrichment)Reliability Improvements (v3)
1. LLM Function Calling –
speak_segmenttool withlanguageas a required enum field. Every speech segment must explicitly declare its language, preventing silent mis-tagging. Backed by a toggle in settings (Force language per segment).2. ELD Validation (Efficient Language Detector) – We chose nitotm/eld after benchmarking several approaches:
langdetect: 41 % accuracy on short words (rejected)tinyld: 68 % accuracy on short words (rejected)eldwithsetLanguageSubset: 100 % accuracy on our test set (DE/ES classification) when restricted to the two configured languageseldis a pure-JS library (< 2 MB), browser-compatible via the staticeld/mediumimport, and requires no external API calls. The detector validates every segment's declared language against the actual text – a mis-tagged segment (e.g. the LLM tagged Spanish text as German) falls back to the primary voice.3. TTS Tweaks – Short foreign words get punctuation boundaries (
"ir"→"Ir."), higher guidance scale (guidance_scale=6), and disabled silence removal (postprocess_output=false). The proxy received new optional API parameters.4. Graceful Fallback – Models without function-calling support ignore the tools parameter and continue using
speak({...})syntax. The toggle can be disabled.5. Language Detection – Extended
looksLikeAltLanguage()with function-word comparison for DE/ES/EN (articles, prepositions, conjunctions), primary-subtag normalization for regional tags.Result
The combined approach achieves high accuracy even for single-word foreign segments – previously the main pain point. With the correct voice clone configured, a word like "ir" (Spanish, 2 letters) is reliably spoken with the Spanish voice.
Files Changed (36 files, +5354/−249)
OmniVoiceSettings.svelteomnivoice-proxy.pyprompt-builder.tsclient-chat.ts,companion-chat.ts,+server.tsopenai-tts.ts,speech-compiler.ts,streaming-speech-buffer.ts,voice-orchestrator.tschat-text.ts,language-detector.ts(ELD),eld.d.tslanguage-detector.tssentences.tsstreaming-speech-buffer.tsomnivoice.mdNon-OmniVoice changes
sentences.ts: +4 lines (reminder tag stripping, utility shared with all TTS providers)prompt-builder.ts: ThebuildOmniVoiceLayer()function is only called for OmniVoiceAll other changes are in OmniVoice-specific files or new files.
Testing
638 tests pass, 0 svelte-check errors. Coverage includes:
Beta Note
Multi-language is actively tested for DE, ES, EN. Other languages may work but are less mature.