Skip to content

Fix Metaspace decoder dropping internal spaces in the first token#2212

Open
sohumt123 wants to merge 1 commit into
huggingface:mainfrom
sohumt123:fix-metaspace-decode-first-token
Open

Fix Metaspace decoder dropping internal spaces in the first token#2212
sohumt123 wants to merge 1 commit into
huggingface:mainfrom
sohumt123:fix-metaspace-decode-first-token

Conversation

@sohumt123

Copy link
Copy Markdown

What does this PR do?

Fixes the Metaspace decoder silently dropping internal word-boundary spaces in the first decoded token.

Bug

Metaspace::decode_chain is supposed to strip the single leading replacement char () that pre-tokenization prepended (when prepend_scheme != Never) and turn every other replacement char into a space. Instead, for token i == 0 the closure mapped every occurrence of the replacement char to None:

if c == self.replacement {
    if i == 0 && self.prepend_scheme != PrependScheme::Never {
        None          // deletes ALL ▁ in the first token, not just the prepended one
    } else {
        Some(' ')
    }
}

So any first token containing more than one metaspace char loses its internal spaces:

  • default split=true, with a BPE-merged token spanning a space: decode_chain(["▁in▁the", "▁house"])"inthe house" instead of "in the house"
  • split=false on multi-word input: decode_chain(["▁Hey▁friend"])"Heyfriend" instead of "Hey friend"

Fix

Track the char position within the token and drop the replacement char only at position 0 of the first token; every other occurrence decodes to a space. This matches the SentencePiece convention (replace all with spaces, strip exactly the one prepended prefix space) and the transformers slow-tokenizer behavior.

How tested

  • Added decode_first_token_with_internal_replacement covering split=true, split=false, and PrependScheme::Never (unchanged: nothing stripped). The test fails on main (left: ["inthe", " house"], right: ["in the", " house"]) and passes with the fix.
  • Full rust suite: cargo test --manifest-path ./tokenizers/Cargo.toml --lib → 202 passed, 0 failed. cargo clippy --all-targets --all-features -- -D warnings and cargo fmt --check clean.
  • Existing decode tests for Always/Never schemes (single leading first tokens) are unaffected, as are the Python-binding Metaspace decode expectations.

Metaspace::decode_chain mapped every replacement char in the first
token to None when prepend_scheme != Never, instead of stripping only
the single leading char that pre-tokenization prepended. Any first
token containing more than one metaspace char (e.g. a BPE merge
spanning a space like "_in_the", or any multi-word token when
split=false) silently lost its internal word-boundary spaces:
decoding ["_in_the", "_house"] produced "inthe house" instead of
"in the house".

Only drop the replacement char at position 0 of the first token; all
other occurrences decode to a space, matching the SentencePiece
convention of stripping exactly the one prepended prefix space.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant