Skip to content
jus1dPublic

About

Simple BPE tokenizer for LLMs

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

shitoken

Quick start

Generate a BPE table

$ shitoken generate --corpus ./shakespeare.txt --output ./shakespeare.bpe --merges 500

Tokenize something

$ shitoken tokenize --table data/shakespeare.bpe --raw "Tokenize me"
84:"T" 111:"o" 107:"k" 101:"e" 110:"n" 105:"i" 122:"z" 101:"e" 32:" " 109:"m" 101:"e"

Usage example

image

About

Simple BPE tokenizer for LLMs

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages