Fix Metaspace decoder dropping internal spaces in the first token#2212
Open
sohumt123 wants to merge 1 commit into
Open
Fix Metaspace decoder dropping internal spaces in the first token#2212sohumt123 wants to merge 1 commit into
sohumt123 wants to merge 1 commit into
Conversation
Metaspace::decode_chain mapped every replacement char in the first token to None when prepend_scheme != Never, instead of stripping only the single leading char that pre-tokenization prepended. Any first token containing more than one metaspace char (e.g. a BPE merge spanning a space like "_in_the", or any multi-word token when split=false) silently lost its internal word-boundary spaces: decoding ["_in_the", "_house"] produced "inthe house" instead of "in the house". Only drop the replacement char at position 0 of the first token; all other occurrences decode to a space, matching the SentencePiece convention of stripping exactly the one prepended prefix space.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the
Metaspacedecoder silently dropping internal word-boundary spaces in the first decoded token.Bug
Metaspace::decode_chainis supposed to strip the single leading replacement char (▁) that pre-tokenization prepended (whenprepend_scheme != Never) and turn every other replacement char into a space. Instead, for tokeni == 0the closure mapped every occurrence of the replacement char toNone:So any first token containing more than one metaspace char loses its internal spaces:
split=true, with a BPE-merged token spanning a space:decode_chain(["▁in▁the", "▁house"])→"inthe house"instead of"in the house"split=falseon multi-word input:decode_chain(["▁Hey▁friend"])→"Heyfriend"instead of"Hey friend"Fix
Track the char position within the token and drop the replacement char only at position 0 of the first token; every other occurrence decodes to a space. This matches the SentencePiece convention (replace all
▁with spaces, strip exactly the one prepended prefix space) and thetransformersslow-tokenizer behavior.How tested
decode_first_token_with_internal_replacementcoveringsplit=true,split=false, andPrependScheme::Never(unchanged: nothing stripped). The test fails onmain(left: ["inthe", " house"], right: ["in the", " house"]) and passes with the fix.cargo test --manifest-path ./tokenizers/Cargo.toml --lib→ 202 passed, 0 failed.cargo clippy --all-targets --all-features -- -D warningsandcargo fmt --checkclean.Always/Neverschemes (single leading▁first tokens) are unaffected, as are the Python-bindingMetaspacedecode expectations.