Skip to content

Fix/affricates - #19

Open
arunasrivastava wants to merge 7 commits into
mainfrom
fix/affricates
Open

Fix/affricates#19
arunasrivastava wants to merge 7 commits into
mainfrom
fix/affricates

Conversation

@arunasrivastava

Copy link
Copy Markdown
Collaborator

Fixes L2Arctic affricate label handling by preserving phoneme-token boundaries when available.

Examples for review:
data/ExamplesWithComments/l2arctic_affricates/*.wav
Each wav has a matching .txt showing the corrected token transcription vs the old joined parse. To my very biased ears, the fixes sound correct but feel free to verify. specifically verify you agree the "and jargon" one should be ʒ "zsh" not dʒ "dzh"

Updated:

  • L2Arctic HF export now includes ipa_tokens from TextGrid phone boundaries.
  • Training uses ipa_tokens directly when present, so t + ʃ / d + ʒ are not accidentally merged into / .
  • Joined IPA strings still fall back to the old tokenizer path for datasets without ipa_tokens.

After this PR:

  • Regenerate and push only the L2Arctic HF datasets.
  • Later, migrate other datasets to expose ipa_tokens in a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant