Conversation
`NormalizerConfig::Lowercase` is already parsed but has no runtime
implementation, so tokenizers declaring `{"type": "Lowercase"}` fail to load
with "unsupported normalizer type: Lowercase". Add the Lowercase normalizer
and wire it into `Normalizer`. This unlocks standalone `Lowercase` and the
CLIP / sentence-transformers shape `Sequence[NFC, Replace, Lowercase]`;
uncased BERT still needs `NFD` + `StripAccents`.
Lowercasing is done per character with `char::to_lowercase`, mirroring HF's
`NormalizedString::lowercase`. This is deliberately not `str::to_lowercase`:
the std method additionally applies the Greek final-sigma rule, rendering
`ὈΔΥΣΣΕΎΣ` as `ὀδυσσεύς` where HF produces `ὀδυσσεύσ`. U+03A3 is the only
character the two disagree on. The "would this character change?" predicate
compares against the mapping itself rather than `char::is_uppercase()`, which
is false for titlecase letters such as `Dž` U+01C5 and would produce wrong
output.
Normalization runs single-threaded over the whole document before rayon
parallelism starts at BPE, so both the scan and the transform take a SWAR fast
path over ASCII, mirroring `ascii_lower_run_end` in `pre_tokenizers::scan`. The
transform is branch-free: `0x80 >> 2 == 0x20` is exactly the ASCII case bit.
Measured against a straightforward `chars().flat_map(char::to_lowercase)` on
~100 KiB inputs: 68x faster on already-lowercase ASCII, 45x on mixed case and
on ALL CAPS, and 0.95x on CJK (both dominated by std's LOWERCASE_TABLE binary
search, which HF pays too).
Tests cover the Cow::Borrowed guarantee, a capital at every one of 28 byte
offsets to pin the SWAR lane indexing and window boundary, titlecase letters,
the 1->2 expansion of `İ` U+0130, and final sigma — the last asserting
inequality with `str::to_lowercase` so a refactor towards it fails loudly.
`local_tests::lowercase_normalizer_matches_huggingface` diffs against the
`tokenizers` crate directly.
Also re-exports `Prepend` at the crate root, which was missed when it landed
in edccd31 (`Nfc` and `Replace` are both there).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the
Lowercasenormalizer. Tokenizers declaring{"type": "Lowercase"}previously failed to load with
unsupported normalizer type: Lowercase— theconfig was parsed but had no runtime implementation.
Notes
Parity. Lowercasing is per character via
char::to_lowercase, mirroring HF'sNormalizedString::lowercase. Deliberately notstr::to_lowercase, which alsoapplies the Greek final-sigma rule —
ὈΔΥΣΣΕΎΣ→ὀδυσσεύςwhere HF givesὀδυσσεύσ. U+03A3 is the only character the two disagree on.Testing
9 unit tests covering the
Cow::Borrowedguarantee, titlecase letters, the 1→2expansion of
İU+0130, final sigma, and a capital at every one of 28 byteoffsets to pin the SWAR lane indexing. Plus
local_tests::lowercase_normalizer_matches_huggingface, which diffs directlyagainst the
tokenizerscrate. Full suite passes;fmtandclippy -D warningsclean.
Also re-exports
Prependat the crate root, missed when it landed inedccd31.