Skip to content

Clarify byte-level pretokenizer behavior#2195

Open
michaelalbada wants to merge 2 commits into
huggingface:mainfrom
michaelalbada:feature/clarify_bytelevel_pretokenizer_docs
Open

Clarify byte-level pretokenizer behavior#2195
michaelalbada wants to merge 2 commits into
huggingface:mainfrom
michaelalbada:feature/clarify_bytelevel_pretokenizer_docs

Conversation

@michaelalbada

Copy link
Copy Markdown

Clarifies the ByteLevel pre-tokenizer docs around raw byte input.

ByteLevel operates on Unicode strings and maps their UTF-8 bytes to visible byte-level characters. Passing arbitrary bytes decoded as Latin-1 can therefore re-encode bytes above 127 as UTF-8 and produce unexpected tokens.

This PR:

  • documents that limitation in the ByteLevel docs
  • adds a short raw-byte encoding example using the GPT-2 byte-to-Unicode mapping
  • explains that already-mapped byte-level text should skip normalization and pre-tokenization

Designed to address [[https://github.com//issues/1877]]

@michaelalbada

Copy link
Copy Markdown
Author

Hi @Pierrci, would you mind taking a look at this PR?

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty for the writeup! we are kinda gonna remove it soon, as performance wise this really dose not make a lot of sense to have!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants