Add DiffSinger BRAPA Phonemizer and BRAPA G2P model#1841
Conversation
- Add comprehensive G2P dictionary support with replacement types (single, merge, split, many-to-many) - Implement sophisticated phoneme validation and language prefix handling - Add timing-based phoneme processing with duration-dependent modifications - Include word-level phoneme editing capabilities for cross-phoneme modifications - Integrate ONNX machine learning models (linguistic and duration prediction) - Support multi-language models with language ID mapping - Add comprehensive speaker embedding management - Include tensor caching for performance optimization - Implement robust error handling and validation throughout - Add extensive XML documentation for all public APIs This phonemizer provides a robust foundation for DiffSinger voice synthesis with support for complex phoneme transformations and cross-note processing.
- Add BRAPA (Brazilian Portuguese Phonetic Alphabet) G2P
- Update brapa-g2p.zip - Added support on phonetic assistant
Introduces DiffSingerBrapaPhonemizer, a phonemizer for Brazilian Portuguese (BRAPA) in the DiffSinger engine. Implements phonetic rules, word boundary processing, and duration-based phoneme replacements tailored for the language.
Refactored EditTimedPhonemes to accept next note and its first phoneme duration. Updated ProcessPart to collect phoneme timing data in two passes, allowing timing edits with access to neighboring note information
Updated the class name to match the phonemizer name
|
Close as replied in #1840. |
|
Thank you for your response. Here is my reply on why these codes should be implemented. On Phoneme Replacement Sorry, but I don't think you fully understood the Pull Request. This is not about "blurring responsibility." The For context, Brazilian Portuguese has 16 recognized regional accents, and BRAPA for DiffSinger supports the most common phonemes found across them. Here is the full list of replacements currently implemented in the PR: - {from: h9, to: pt/h} # R-sound before a vowel. Can be changed to [h], [hr], or [x]
- {from: r9, to: pt/h} # Rhotic. Can be changed to [h], [hr], [r], [rw], or [x]
- {from: s9, to: pt/s} # S-sound before a consonant or at the end of a word. Can be changed to [s] or [sh]
- {from: z9, to: pt/z} # Z-sound before a voiced consonant. Can be changed to [z] or [j]The voice model should have these replacements added in their dict.yaml file to properly work On Sandhi Phenomena You mentioned that context should handle these changes, but it's not that simple. These hardcoded rules are actually essential for capturing the connection and fluidity that are natural in Portuguese. Here are some examples: If you look into the Refined Phonemizer API, you should notice that those changes occur before applying the duration, so the model get those timmings right. As you can see, some phonemes merge while others change. The ones that merge can be handled by writing words together or separating them by syllables, but the ones that change cannot be addressed that easily. As you know, when labeling, the same sound should be mapped to the same phoneme. We can't label an /s/ as a /z/ and expect DiffSinger to infer from context that it's at a word boundary and should be pronounced as /z/. That's why we need a hardcoded rule, because this change always happens, and the G2P simply does not have the capability to handle it. For reference, SynthV uses a similar approach. synth_phonemizer.mp4On Stressed vs. Non-Stressed Syllables Based on Timing OpenUTAU G2P is an RNN-T model from TorchAudio 2.8.0, and with PyTorch dropping support, this G2P has become outdated as it never recieved updates within OpenUTAU. This creates limitations, since RNN-T is not good for complex languages like Portuguese. If you need confirmation, you can ask @Cadlaxa, who faced these challenges while working on the ARPA+ dictionary. (Apologies for the mention.) To address this, we created a new custom dictionary by hand, designed to be predictable for users and easy to G2P to understand. If an out-of-vocabulary (OOV) word appears or the output is incorrect, users can simply type the word as it is spoken or divide it into syllables to get a predictable result. One of the challenges we tackled is stressed vs. non-stressed vowels. Non-stressed vowels occur in words with phonetic repetition, such as "papada," "papaya," and "tititi," as well as at the end of words. Since non-stressed vowels naturally have shorter durations in our language and, in singing, what should be non-stressed often becomes stressed, we established a rule: if the duration is below a certain millisecond threshold, it should be changed to the non-stressed variant. This approach was tested and approved by our community. And remember: all of these hardcoded rules can be turned off by adding a On the CL and VF Situation As you know, all modern voicebanks support the These symbols are simply a more convenient way to call those phonemes without needing phonetic hints. We observed this approach in SynthV, found it effective, and implemented it here. We can even find a similar system in the ARPA+ Phonemizer, where adding We are not here to debate linguistic or phonetic theory, we are here to discuss whether this works as a system. In the end, users don't care where it came from; they care about results. However, if you still believe these symbols should be removed, we are not strictly opposed. Alternatively, we would welcome a better implementation suggestion that we can adopt immediately. ARPA+ SynthV We Are Not Beginners BRAPA and Team BRAPA exists since 2022. We research, test, gather feedback from our users, and update our models and voicebanks across platforms, from UTAU and NNSVS to now DiffSinger. We can confidently say that we are very knowledgeable about our language and understand what it works with and what doesn't in various synthesis systems. You are welcome to use our models, as well as those from other teams that support BRAPA: Many other individuals also have access to our private corpus and have built their own voicebanks. On the Review Philosophy Even though we do believe DiffSinger is fully capable of making context-based decisions, we also believe the review process behind PRs should take a different approach for different languages. Regardless of philosophy, a "one-size-fits-all" solution is hardly ever viable in any project, especially considering how vastly different the structure of each language can be. It's also worth pointing out that we do not wish to interfere with DiffSinger itself, our objective aims toward OpenUTAU, which is one of the reasons we're requesting this reassessment. On the topic of context: yes, DiffSinger can learn from context, but sometimes it takes a lot of data to get it right. Many developers work with limited resources when building voicebanks, and in such cases, it's better not to rely on DiffSinger to infer context, but rather to "force" recognition by adding another phoneme. In other words, it's often more efficient to add more rules than to spend time and resources training the system to understand context. In our reality, it's better to include the phonemes you consider "unnecessary" and produce a high-quality voicebank with just one or two hours of data than to reduce the phoneme set and require significantly more data to achieve the same result. A good case study is ACE Studio. The developers of that software followed a similar philosophy to what is being suggested here, and it ultimately failed to provide intelligible results to native speakers. By relying on context alone, their voicebanks failed to reproduce any of the rules previously mentioned in this response, resulting in complete abandonment by the community. In contrast, Synthesizer V's structured approach follows a similar method to ours. Although it still lacks native voicebanks, it has gathered user interest because it produces results comparable to ours. I hope my explanation helps you see our perspective. Modern singing voice synthesis evolves every day with every new paper published, and DiffSinger changes a lot too. But in the end, both approaches work and both produce results. It ultimately depends on what the user wants and how the developer chooses to approach it. |
Thank you for the detailed response. I do not question Team BRAPA's knowledge of Brazilian Portuguese or its experience building voicebanks. However, experience and community adoption cannot replace technical justification for adding language-specific behavior to OpenUtau core. I will respond to the main points separately. 1. Phoneme replacement and regional accentsI now understand that My concern is more fundamental than whether a particular combination corresponds to a named regional accent. An accent is not simply a rule that every occurrence of The proposed If these replacements are only compatibility aliases from BRAPA symbols to the tokens on which a particular model was trained, then they can be reasonable, but the general mechanism in #1840 is sufficient. If they are presented as accent modeling or accent switching, changing the same global constants through a future UI does not address the underlying limitation; it merely selects a different static categorization. At most, this defines preferred default phones, not context-sensitive accent behavior. This also does not justify the separate word-boundary and duration rules in this PR. Accepting configurable intermediate symbols is not the same as accepting every transformation currently implemented by 2. Sandhi and contextual labelingThe fact that OpenUtau's G2P queries one word at a time only proves that the G2P itself does not see the next word. DiffSinger still receives a phrase-level phoneme sequence, so the neural model can use surrounding phonemes when producing the acoustic realization. The statement that "the same sound should be mapped to the same phoneme" exposes the central disagreement. It conflates surface phones with phonemes. A surface If every surface realization is instead assigned a separate input token, the label has already encoded the contextual decision. The model is never given the task of learning The listed processes are also not unusually complex compared with allophony, assimilation, liaison, reduction, and sandhi in many other languages. More importantly, external sandhi is not determined by phoneme adjacency alone. Related research on Brazilian Portuguese external vowel sandhi describes its application as dependent on phonological-phrase domains and stress or intonational prominence, and as optional outside certain prosodic domains (Santos, 2021, The Acquisition of External Vowel Sandhi in Brazilian Portuguese). A 2025 production study of the sibilant at issue here likewise found word-final sibilants to be more voiced before a vowel than before a pause, but also found substantial devoicing conditioned by the preceding consonant and lexical considerations; both voiced and voiceless realizations occurred (Mendes, 2025). Therefore the claim that these changes "always happen" is too strong, and inspecting only the first phoneme of the next word is not a sufficient model of the phenomenon. If existing BRAPA models were trained with separate 3. Duration-based vowel replacementThe proposed More importantly, assigning different input phoneme symbols to the long and short or full and reduced realizations is itself what prevents the model from learning vowel reduction. If both are realizations of the same underlying vowel, consistently labeling them with the same vowel token would let the model observe how its acoustic realization changes with duration and surrounding context. By labeling the reduced realization as This repeats the same problem as the separate There are therefore only two possibilities: either vowel reduction is genuinely continuous and context-dependent, in which case a fixed threshold and separate categorical labels are an inadequate representation; or the 45 ms threshold is an adequate description, in which case there is no convincing reason that a duration-aware neural model cannot learn such a simple relationship. Community approval of a convenient heuristic does not resolve this contradiction or make it an appropriate built-in default. 4.
|
|
Referenced Rule:
I don't really know much about the languages of the Iberian Peninsula... but I am very curious about this rule. How was it derived? Is it based on some linguistic study? Are there any relevant papers on this? I am quite skeptical about this rule. What is so magical about 45ms? Is it statistically derived? Since songs have different BPMs and performance techniques vary in speed—some fast, some slow—how did this very specific number come about? On the other hand, does this rule even exist during the training process? Does the model actually learn such a linguistic context in practice? |
There is a Paper regarding that https://kuscholarworks.ku.edu/entities/publication/1963fa35-0591-44ed-b865-f0815ff35c8c 
As shown in this picture, the minimum size that every /a/ /i/ and /i/ became unstressed is 44ms. So we added a rule that if it's less then 45, it turns into non-stressed |
|
@yqzhishen BRAPA for DiffSinger has existed since DiffSinger was first introduced to the community.From the outset, our goal was not to create such extensive phoneme inventory. On the contrary, we initially aimed to keep the phonetic system as compact as possible, relying on contextual information and deterministic rules to resolve pronunciation whenever feasible. If you remeber, have some time that DiffSinger couldn't reliably distinguish between /s/ and /z/ sounds in English contexts (and somewat still doesn't), and plosives had to be manually drawn and carefully timed to produce the correct pronunciation. This led to additions such as /i0/, /u0/, and /ax/, which were introduced to improve phonetic consistency where contextual inference alone proved insufficient. More recent dictionary revisions also introduced phonemes such as /yn/ and /wn/ as our research identified further distinctions that benefited model stability and pronunciation quality. We initially tried to derive these pronunciations automatically from context, but DiffSinger still does not consistently infer the nasal variants of /y/ and /w/; in practice, they are generally produced only when followed by an explicitly nasal vowel. Because of this, introducing dedicated phonemes was not a matter of preference but a necessary design decision. Although DiffSinger has evolved considerably since those early days, our phonetic system was never intended to simply compensate for temporary implementation details. It gradually became an independent phonetic standard, shaped by experimentation, linguistic analysis, and practical experience with Brazilian Portuguese singing synthesis. We found that preserving these language-specific distinctions is more effective than repeatedly retraining models or relabeling datasets while expecting inference to resolve ambiguities automatically. This approach provides greater model stability with less training data while giving users more control over pronunciation, including support for deterministic accent patterns and additional phonetic distinctions when desired. As a result, users spend less time manually correcting phonemes and more time creating. This philosophy is what ultimately shaped our acoustic models, phonetic standard, G2P, and phonemizers. We contributed this work to OpenUTAU because it has become the primary platform for users coming from UTAU. It is a community-driven project where tools, research, and model development are built by and for the community. We believed that our vision, study and effort would also be welcomed and implemented here. However, it now appears that we are expected to conform to the design decisions and conventions of another system integrated into OpenUTAU. As I understand it from your own comments back in the day, that system was only intended to be temporary. And until recently it was discovered that its language tag system was not functioning correctly when training. What concerns us most is the underlying assumption that phonetic decisions made for one language should automatically take precedence over solutions developed specifically for another. From our perspective, this overlooks the linguistic realities of different language communities instead of recognizing them as equally valid design choices. If adhering to those conventions is a requirement for our phonemizer to be accepted, then we would rather continue distributing it as a DLL for our own users. They seem to appreciate it. Despite our disagreement, we genuinely hope DiffSinger continues to grow and eventually surpasses commercial singing synthesis software while supporting many languages under your philosophy. If that day comes and those principles prove equally effective for Brazilian Portuguese, we'll gladly adopt them for our own models. And if that happens, perhaps ChatGPT can let you know. |
|
@overdramatic I only let ChatGPT to explain and summarize your reply, and search for evidence so I can confirm it's weakness of your simple static rules instead of my lack of linguistic knowledge. The final reply to you is just a re-statement of my previous opinion. And let me just be brief - your 45ms hard threshold and the special symbols representing specific phoneme patterns are just nonsense and rediculous. Not a single experienced deep learning researcher will prefer these things instead of letting model to learn. Continuing to argue on these rules will only make your proposal questionable. From your previous reply, I found you have gone through the improper way from the very beginning when you labeled your data, and you are too confident of what you have done - you said Worth to mention: I don't think your language is that much special among all languages in the world. When talking about dialects and sandhi things, many languages would have even more varieties than yours, many are simply not commonly documented. As a common phenomenon, we just accept the nature of a voicebank, instead of forcing it to sing in another dialect or accent (which cannot produce good results, and you end up fighting with it). Naturalness, instead of controllability, is what a generative model primarily seeks for. Even when it comes to linguistics itself, linguists doesn't have agreement on many things. The more you learn linguistics, the more you will realize its shortage and weakness to describe real world, which is much more complex than you may think, and many linguists also admit this.
Actually I saw the long-standing difference in data processing toolkits and labeling styles between Chinese community and Discord community since years ago. It's not because Chinese has fewer variants; it's because we accepted some uncertainty and randomness of AI models, and agreed with the imperfection of human voice. This is also partially caused by historical reasons where we lack powerful yet universal infrastructure - many of you utilize NNSVS-stuff, which only has one phoneme labeling layer, and may mislead you to label too honest phonemes instead of preserving the full original semantic context. Infrastructure will hopefully improve, but it's more important that many community members should change their mind on what a proper labeling style is, and what information we should bring to the model. This is DiffSinger, not UTAU. Don't bring too much linguistics in, and be ready to live with semantics, context and data.
|
Lack of controllability is slop. While a default generation should ideally be as natural as possible, without the ability to force compliance, you have a mindless generator instead of an instrument to be played. To borrow the example used for explaining sandhi: In your scenario, you would create a model that is completely incapable of producing the sounds |
|
We want the Diffsinger vbs singing in a UNDERSTANDABLE and GOOD portuguese, not what you are thinking you can do. |
|
and also, sometimes your models kinda suck ngl principally the chinese |
I don’t see what relevance a 1990 paper has in 2026—especially when even in 1990 it was already outdated. As early as the 1970s, researchers were questioning whether hypo-articulation is really caused by shortening per se, rather than by a reduction in articulatory effort (and here you should revise the variance). Moreover, this paper is merely a correlational study, and mistaking correlation for causation is quite metaphysical. On another note, I noticed that the statistical analysis in this paper is based on a single speaker—and even so, the rules it proposes are far more complex than a simple 45‑ms threshold. What you have done is to simplify a highly speaker‑specific rule and then apply it at a general, universal level. Ever since the release of GPT‑SoVITS, Korean scholars have been anxious. In the current AI era, countries like China and the US naturally have an advantage and need not worry; Japan and South Korea have a long history of speech synthesis research, so they are also safe. But for many languages that lack even basic research, there is a real risk that they will simply vanish in this age. I think their concern is justified. In the Chinese community, people are accustomed to using unified annotation standards and leveraging model context to achieve extremely high flexibility. In fact, Synthesizer V works exactly that way. For modern speech synthesis, you need to get at least one of two things right: sufficient precision or sufficient scale. |
|
Okay, I’ve found a really hilarious problem. Your implementation seems to be wrong (DiffSingerRefinedPhonemizer.cs). You are not converting phonemes with a duration under 45 ms—you are converting those under 45 ticks. I find it hard to imagine how you could have arrived at the conclusion that your experiments were “better.” I think we can end this topic here. In summary, we have witnessed a rather amusing exchange: taking a single sentence out of context from a paper that is thirty years old—and was already outdated by twenty years even back then—trying to derive a universal method from a very local, correlational finding, and then, through a faulty implementation, hallucinating an improvement. On another note, I find it quite boring to frame opposing views as “discrimination.” For example, I see that among the supporters of the opposing view there are also advocates for Palestine—so by that logic, could I just as well call you accomplices of Netanyahu? |
|
The comparison to the Israel situation seems really irrelevant and in poor taste, but that aside, I'm sorry but Diffsinger is currently just not good enough at context to be able to handle all those cases you mentioned. It just isn't and it's delusional to pretend it is. The comparison with ACE studio is actually spot on, because they use the exact same approach you're mentioning and all langauges that aren't Chinese sound awful. Funnily enough, before the implementation of reflow Diffsinger used to be a lot better at handling that kind of context, but it's not worth it to switch back given the performance boost. Lastly, dismissing the entire PR and LAUGHING AT IT because you don't agree with the timing aspect when the rest of the PR included many changes that would be extremely useful not just for Portuguese but for all other languages is childish at best. Users in this thread have provided many examples to support the usefulness of the multiple replacements and those have just been ignored and dismissed. To frame how stupid this conversation is, you're asking people to label (in the case of Portuguese specifically):
The community has experimented with this type of labeling with English (t and dx, ah and ax, etc...) and I can assure you it only works properly about 50% of the time. I hope you can rethink this, denying this PR just because you personally don't understand it is close minded. And I'm saying this as someone who doesn't speak Portuguese and has no interest in Portuguese Diffsinger, this change would help ALL languages that aren't based on a very simple C-V structure. |
OK, you can't just call it inappropriate only when you're being labeled "racist." First, you still can't explain why that 45‑ms threshold rule is not stupid. Its origin is pedantic and outdated, your understanding of it is skewed—if not outright wrong—and even the tests you ran were flawed! It has never worked! That alone should be enough for any reviewer to question the professionalism of the person who proposed it. On the other hand, when models become more powerful yet you run into problems, why not suspect that the issue lies with your own approach? For example, after introducing Muon, many users encountered various problems with their models, and in the end, cleaning the data resolved them. Moreover, I've noticed that very few people try to solve problems heuristically. For instance, last year we introduced a stretch embed to improve the clarity of consecutive vowels. If you find that the current model's contextual capability is insufficient, why not think about improving the model itself? In short, this change will not be accepted. Perhaps in the future there will be ways to implement similar mechanisms in OpenUtau, but for now, this pull request will not be merged. |
|
I understand that everyone has different expectations. However, please keep the discussion constructive and respectful. Bringing emotions and accusations into a public discussion doesn't help. If anyone has concerns, please provide specific and actionable feedback. General criticism without concrete examples won't help resolve the issue or move the discussion forward. |
I never said anything about racism and I personally don't agree with the 45 ms thing, but I do wholeheartedly agree with all the other aspects of the PR. Why do you think people use the DIFFS EN+ phonemizer? If context really worked as well as you claim there would be no need for it at all. Also, like previously mentioned, you claiming that these are issues stemming from a lack of data or wrong labeling completely ignores regional dialects/language variations that cannot be properly accounted for without specific labeling. There were very clear examples provided by multiple people, the approach that synthV uses which aligns with this PR and has provided good results, and the approach that ACE studio uses which aligns with your vision and has provided really bad results (except for Chinese and Japanese because of how simple they are phonetically speaking). Also again, this PR is useful not just for Brazilian Portuguese, but also for other languages and to fix some odd behaviors of the G2P which in OpenUTAU is really outdated and doesn't provide sentence context. For example, in Italian the G2P has some oddities, LL and JJ phonemes get doubled for no apparent reason, and with this PR I'd be able to just set up a replacement to fix it. To bring up another example with Italian since it's what I'm most familiar with, people have different approaches to gemination (which can NOT be derived from context, it's impossibile and varies based on the spelling of the word) and implementing this PR can give people further choice. Other people have provided relevant examples for English too, the DIFFS EN+ phonemizer is really useful but it uses phonemes that not everyone has implemented and there's no fallback to fix this ("tr" and "dr" are often not présent in models and are "t r" and "d r" instead). I think you're misunderstanding the point of this PR, which isn't to fix and issue with DiffSinger, it's to fix an issue that's present in the OpenUTAU G2P. |
|
@KakaruHayate Thank you so much for pointing it our the flaw of my code! I actually forgot to add And I never said that you are discriminating. In every moment we tried to make good exchanges, but in all exchanges you guys didn't want to listen to our view, you are just saying that the way to do is the right way and everything else is wrong. There wasn't any "why you guys did like that?", "is the (put function here) is not working for you guys?", "can you explain better?", there wasn't any actual dialogue. If you say "discriminating" is because you are acting that way. If you still don't approve, that's alright. We are not here to change minds, just pointing out our point of view of who we did this. |
Commercial projects are not as transparent as open-source ones. Without clear evidence of how ACE Studio and Synthesizer V prepared and labeled their data, I won't attribute the bad results to phoneme system design alone. And, it's subjective to judge if they are good or bad - it's partially based on personal preferences.
Still, sentence context isn't lost, the model knows it, it's the model's responsibility to resolve it. Adding sentence-level context to G2P won't change real thing.
Though not related to this PR, this issue should be fixed in the G2P itself. Adding a downstream patch doesn't address the root cause.
Two things to clarify:
Some other words to supporters of this PR: I realized some of you didn't come here to discuss from the very beginning. I start with statements from technical perspective, then I received dozens of thumbs down from a bunch of organized people. It's some of you who didn't listen to the comments at all; any disagreement from non-Portugese speakers were regarded as disrespect and underestimation; and some of you even labeled opponents as discriminative or racist. Not a single thing will proceed given this much non-technical factors, bias and emotions. The first thing to remember: this is deep learning. The whole thing is the language + deep learning. You speak the language but I don't. however, the rules (which weren't even implemented correctly) showed that you have zero common sense in deep learning. The second thing to remember: this is open-source. You cannot had a wish and just expect the devs to fully accept it; you should get used to rejections. Yes, we appriciate the expectations and understanding from the community, but we don't take this as our jobs, we don't owe you anything, and we have our principles. We build together if we agree, you build your own if we don't - all common in this world. |
You don't really need to know how Synthesizer V labeled their data because you can just test their portuguese implementation on the newest technical preview. We also gave you examples of how their phonemizer is doing the exact same thing we're trying to implement here (automatically changing the sandhi phonemes instead of letting the model decide what's best). Here's an example again. synthv_phonemizer_example.mp4Also, they're using the carioca accent (from Rio de Janeiro) as the default. They don't have an accent switcher, but if someone wants their singer to sing in a paulista accent (from São Paulo), they can do it just fine by manually changing the phonemes. If we don't have to "find a voicebank from São Paulo", as you suggested on discord, with commercial synths... why do we have to do it with open source ones? carioca_to_paulista.mp4 |
For your first example: this is only one hand showing things will work with explicit sandhi symbols. Evidence of things not working without the symbols still lacks. For your second example: I later said in Discord that accent switching IS (partially) acceptable if you are generally willing to proceed with friendly manner; and I have proposed a possible better solution (accent embedding), and suggested that someone can start experiments immediately using language tags and phoneme merging, without having to modify code. |
Wouldn't that be harder for both the developers and the users? If we were having this conversation at the beginning of the project, we'd probably be more open to your way of doing things, but we've been doing our labels this way for years. Not only Team BRAPA but individuals that are making portuguese banks outside of Team Brapa do their labels the same way. This is just what it feels right for the language. From what I can see, labeling based on context is the default in chinese voicebanks, but nowhere in the Best Practices document says that this is the only way we have to do stuff to get accepted as legit. The only thing I can see that using # in the labels is not recommended, but since we label our vocalfry as /vf/, using it on the phonemizer seemed fine. I think the main reason we (both Team BRAPA and the Brazilian community as a whole) had such a strong negative reaction about this change is that we felt like we were being "punished" for something we had no idea of knowing it was the "intended way" of doing things. Changing it at this point is not only going to halt the progress of our banks in development (because we'd have to relabel everything, old and new), but it'll also be a problem for every single big western group. Since they all use our labels, everyone would need to retrain their voicebanks. Such a big change would take ages for everyone to adapt to something no one really wants. We're already used to this method, and you were the first one to say negative things about it. And since we have a commercial engine doing things the exact same way, it proves that our method is, at least, worth of being considered, even if it was not your inicial intention with the DiffSinger project. We were even willing to give up 3 out of our 4 proposals, letting just the sandhi stuff, since this is the most important part of our phonemizer for our users. It's a shame it looks like this is the thing you disagree the most. |
Hey okay this is me, I do not know why my support for palestine is being mentioned in a discussion about phonemiser development. In general can we stick to the topic and not get into topics like this? I am also not one who is in support of timing based rules as it deprives the model from learning that vowels should be destressed contextually. I have also spoken to @yqzhishen personally about my opinions on this matter so I do not appreciate my profile being mentioned like this as if I am apart of this brigade against the diffsinger development team. To keep this prompt. Can you all stop misusing these channels to discuss personal attacks against each other and stick purely to discussions about implementation? This is not the place to be hurling insults or attacks based on political opinions, or to be mentioning the personal politics of those who are involved in this discussion. You are free to continue discussing the implementation and implementation only. Thanks. |
|
For the purposes of dealing with the disputes under this PR, I will be reopening this PR and I will be adding stakira as a reviewer of the PR, as discussed on the OpenUTAU discord server by members of the BRAPA teams and the openvpi diffsinger teams. Please keep any further discussion about this topic civil and be cognisant of any language and cultural barriers that we may have between each other, alongside differences in implementational philosophy. |
Let me reiterate my position: you can't just decide it's inappropriate only when I bring up politics. Which group was it that first started talking about racism on Discord? That's exactly what frustrates me. If only native speakers or language users are allowed to discuss these issues, then why are we all wasting so much time here debating at all? |
Kindly this matter should stay on discord, it should not be brought up on this forum. I understand your frustration but it is not a productive one when it comes to the decisions of implementing this phonemiser or not, and it is something that you should discuss outside of the channels of github. My specific problem was also that I was targeted in your message when I am impartial to the changes being proposed, and am not representative of any 'side' or 'group' to this discussion. Please keep any discussion of personal/political attacks on discord rather than on GitHub, this is not a place to hash out personal or political drama, this is only a place to discuss implementation decisions. |
|
@KakaruHayate (and openvpi team in general)
openvpi does not have full authority over which diffsinger related prs get merged, please stop saying this. several of them have been merged without direct approval from your team. |
Updating the timing calculation to proper use millisecond positions instead of tick.
I trust the "aesthetic" of other reviewers. |
Could you elaborate on what you mean here? Are you requesting more reviewers on this PR? |
There was a problem hiding this comment.
shouldn't this be on the api's pr and not the phonemizer's pr?
There was a problem hiding this comment.
Since this PR should be merged after PR #1840 and this one also contains the Refined Phonemizer, my fear is that if I put the update there and this one is merged after, this fix would be lost. I'm not well versed in this PR merge logic. I'll add the fix in the other PR too
11 participants
Add DiffSinger BRAPA Phonemizer
Overview
This PR adds
DiffSingerBrapaPhonemizer, an advanced Portuguese phonemizer for the DiffSinger engine that provides sophisticated phonetic processing and accent support beyond the basic Portuguese phonemizer.Key Features
Advanced Phonetic Processing
Extends
DiffSingerRefinedPhonemizerfor advanced phonetic rule applicationDuration-Based Rules
Implements smart phoneme substitutions for short notes:
a→axfor duration ≤ 45msi→i0for duration ≤ 45msu→u0for duration ≤ 45msWord Boundary Rules
s9→zbefore vowels,s9→z9before voiced consonantsr9→rbefore vowelscl→ngwhen preceded by nasal vowels and followed by voiced plosivesℹ️Rules application can be skipped if the lyrics has phonetic hint
[ ]or it starts with/Technical Implementation
dsdict-brapa.yamlfor Portuguese-specific phonetic rulesBrapaG2pclass for grapheme-to-phoneme conversionBRAPA G2P Model
Overview
This PR also introduces the BRAPA (Brazilian Phonetic Alphabet) G2P model, a specialized grapheme-to-phoneme converter optimized for Brazilian Portuguese with enhanced accent handling and extended phonetic features.
Key Features
Enhanced Brazilian Portuguese Support
New Dummy Phonemes for Phonetic Precision
Added specialized placeholder phonemes for accurate phonetic representation:
s9: Represents S-sound before consonants or at word endings. Can be changed intosorshz9: Represents Z-sound before voiced consonants. Can be changed intozorjh9: Represents R-sound before vowels. Can be changed intoh,hrorxr9: Generic placeholder for rhotic sounds. Can be changed intoh,hr,r,rworxExtended Phonetic Support
Added extra phonemes for better accent handling
ah: Represents the default vowel phoneme in stressed syllables where the letter “a” precedes a nasal consonant. Can be orthographically represented as "a" or "â". Examples:ng: Default phoneme for nasal vowel +ginteraction. Can be changed intogwn: Default phoneme for nasal vowel +winteraction. Can be changed intowyn: Default phoneme for nasal vowel +yinteraction. Can be changed intoyAdvanced Lyric Processing Features
New markup system for vocal processing in lyrics:
cl(clousure/glottal stop)vf(vocal fry)Usage
For models / voicebanks that has Portuguese capabilities with BRAPA, this replacements are recommended:
Techninical Features
Dictionary Entries Total:
7517026766154701646716467Vowels:
Consonants
Drawbacks
RNN-T Models is not good with complex languages, so OOV entries can give wrong results. If that happens, try writing how the word is spoken instead of written