more languages

[rohlik-hu]

Thanks a lot for the impressive tool. How can additional languages be included? It seems that the sentence-transformers library supports many more …

[bfsujason]

Thank you for your interest in Bertalign! The LaBSE model supports over 100 languages. However, Bertalign relies on sentence-splitter for sentence segmentation, which currently supports only 25 languages.

If you need to align other languages, you might consider using alternative sentence segmentation tools such as pySBD and Ersatz. These tools offer broader language support and may better suit your needs.

[rvwfels]

Hi, my data has already undergone sentence splitting. How would I simply skip the sentence-splitting step?

Thanks for your work!!

[george-hawkins]

@rvwfels - I'm not associated with this repo, but I just wanted to point out that you can use the is_split argument:

aligner = Bertalign(src, tgt, is_split=True)

This disables the automatic segmenter and instead assumes you've already done the segmentation with one sentence per line.

---

@rohlik-hu - I forked this repo and, among other things, replaced the sentence splitter with Segment any Text, which supports 85 languages (including Japanese, which is what I was interested in). This also eliminated any dependency on Google Translate to detect the language.

If you're interested, you can find my fork here.

[speedyk-005]

@rohlik-hu Use yasbd-lib which is a high-accuracy rule-based sentence boundary detector that is faster and more accurate than pysbd. It supports 39 languages out of the box.

Benchmarks here: https://github.com/speedyk-005/yasbd-lib/tree/main/benchmarks