toktok

A fast, exact BPE tokenizer for OpenAI encodings — Rust core, Python bindings. Token ids are byte-identical to tiktoken.

The Pile, 25 MB, cl100k_base, single thread, GitHub runners. Ids verified identical before timing.

Python 3.11

encoder	throughput	CPU per MB	p99 per doc
toktok	211.8 MB/s	0.0054 s	17.4 µs	15.4×
tiktoken	13.8 MB/s	0.0752 s	187.3 µs	1×

Python 3.14

encoder	throughput	CPU per MB	p99 per doc
toktok	187.0 MB/s	0.0058 s	21.4 µs	16.6×
tiktoken	11.3 MB/s	0.0931 s	235.6 µs	1×

Python 3.14t · free-threaded

encoder	throughput	CPU per MB	p99 per doc
toktok	174.7 MB/s	0.0064 s	21.0 µs	15.2×
tiktoken	11.5 MB/s	0.0901 s	227.8 µs	1×

Rust

encoder	throughput	CPU per MB	p99 per doc
toktok	249.8 MB/s	0.0040 s	4.1 µs	26.6×
bpe-openai	36.4 MB/s	0.0274 s	32.4 µs	3.9×
tiktoken-rs	9.4 MB/s	0.1066 s	132.4 µs	1×

_{Each table is one machine; the Python ones ran on separate runners, so compare within a table. Method and full results: docs/BENCHMARKS.md.}

Encodings

Pass an encoding name — not a model name. The bundled encodings are compiled into the binary, so there are no data files, downloads or runtime dependencies.

encoding	used by
cl100k_base	GPT-3.5, GPT-4, GPT-4 Turbo, text-embedding-3-*, text-embedding-ada-002
o200k_base	GPT-4o, GPT-4o mini, GPT-4.1, GPT-5, o1 / o3 / o4-mini
o200k_harmony	GPT-OSS (o200k_base's merge ranks plus the harmony special tokens)
qwen3.5	Qwen3.5's vocabulary, split the way bigpickle does — exact ids; see below
jev	TypeSafe's Jev — an upper-bound estimate, not exact ids; see below

An unknown name raises KeyError listing the valid ones.

qwen3.5

Exact ids, checked token for token against Hugging Face tokenizers on 1.1M single characters, 130k random strings and the three 25 MB benchmark corpora, and against a bigpickle server's own counts (202 texts, 145k tokens).

• It follows bigpickle's tokenizer.json: Qwen3.5's vocabulary and merges with Qwen2's split pattern. Qwen's own tokenizer.json keeps combining marks with their letters, so Hindi, Tamil, Thai and similar text counts a few percent lower there. tools/gen_qwen35.py builds the data from either file.
• Text is NFC-normalized first, as Qwen's tokenizer does. Python does it for you; truncate then returns a prefix of the NFC text. From Rust, pass NFC text: Tokenizer::unnormalized finds the few texts in a batch that aren't.
• A server that renders special tokens counts <|im_start|> or <think> as one token. Pass with_special=True to match it; the default counts them as text, which only ever counts more.
• Speed: within 10–25% of cl100k_base on English and code, about half on multilingual web text, and 70–250× faster than tokenizers (numbers).

jev — an upper bound, not an exact count

TypeSafe doesn't publish Jev's tokenizer. jev gives a safe overestimate instead, meant for splitting a payload under Jev's context limits: batch_count, truncate and batch_truncate take it in Python, and toktok::jev::{count, count_batch, truncate, truncate_batch} in Rust. It is a count only: there are no ids to encode or decode.

Checked against ~1,570 texts counted by the API (English, code, 22 languages, JSON, Markdown, logs, CSV, HTML, emoji and odd Unicode; jev-1.13 and 1.14 count the same):

text	over Jev, on average
English and code	1.19×
other languages	1.19×
JSON, Markdown, logs, CSV, HTML	1.13×
emoji and odd Unicode	1.31×

None came out under. On text the fit hadn't seen, about 1 chunk in 500 came out under, by less than 8%. Leave that much headroom at the limit, or re-split when Jev answers max_tokens_exceeded.

It splits text the way Jev does (cl100k's rules with one token per digit), then charges each piece by type: words by their cl100k tokens, with extra for rare ones, and other scripts per character. It runs at roughly half to all of plain cl100k's speed (numbers).

Things to know when you size a request:

• The question text and some fixed wrapping count too. One short question on an empty state already reports 268 input tokens.
• A JSON object state counts like json.dumps(state, indent=1). Count that text, not your compact JSON.
• Jev NFC-normalizes text and the estimate doesn't, so decomposed text (e.g. macOS filenames) only counts higher.
• Short texts carry a fixed cost of a few tokens each, so many tiny strings overcount far more than one payload does.

Python

uv add toktok-rs        # or: pip install toktok-rs

One function. Give it your texts and an encoding, get a token count per text:

import toktok

toktok.batch_count(["hello world", "how many tokens is this?"])
# [2, 6]

toktok.batch_count(docs, "o200k_base")                # GPT-4o and friends
toktok.batch_count(docs, "cl100k_base", threads=4)    # 0 = every core (default)
toktok.batch_count(["a<|endoftext|>b"], with_special=True)   # [3]

batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)

Returns list[int] — one count per text, in the order given.

parameter	default	what it does
texts	—	any iterable of str. An empty iterable returns []; an empty string counts as 0.
encoding	"cl100k_base"	which tokenizer to count with — one of the encodings. Model names are not accepted; look yours up in that table. An unknown name raises KeyError.
threads	0	worker threads. 0 uses every core; 1 counts on the calling thread; any other number caps the pool. Counting releases the GIL, so this scales — including on free-threaded builds. Threads are only worth it for large batches; for a handful of short strings the pool costs more than it saves.
with_special	False	how to treat special-token strings. False counts <|endoftext|> as the ordinary text it looks like (7 tokens); True counts it as the single special token it is (1). Use True when your text already contains rendered chat/control markup and you want the count the model will see.

Ids are never built: one scratch buffer per thread is reused across texts, so a batch of any size allocates O(threads), not O(tokens).

Wheels cover CPython 3.11–3.14 plus free-threaded 3.14t, on Linux (x86_64, aarch64), macOS and Windows.

truncate(text, max_tokens, encoding="cl100k_base")

Cut text to a token budget. Returns (text, total_tokens), where total_tokens counts the whole input — truncated or not — so you can report how much was dropped.

toktok.truncate("hello world", 1)          # ('hello', 2)
toktok.truncate("hello world", 50)         # ('hello world', 2) — same object back

toktok.batch_truncate(docs, 8192, "o200k_base", threads=8)   # [(text, total), ...]

One pass, no ids built, nothing decoded — the cut is a byte offset into the string you passed in. About 3× faster than decode(encode(text)[:n]), and ~6× with the batch form.

It also cuts on a character boundary. Byte-level BPE can split one character across tokens ("🧠" is three), so a token-boundary cut can leave a partial character that decodes to U+FFFD — decode(enc.encode("hello 🧠")[:2]) gives 'hello \ufffd'. truncate gives 'hello '.

Need ids, decoding or offsets? toktok._encoding(name) returns the full tokenizer — see docs/PYTHON.md.

Rust

cargo add toktok-rs     # the crate is toktok-rs, the library is toktok

let tok = toktok::Tokenizer::builtin("cl100k_base")?;   // nothing to download

let ids = tok.encode(b"hello world");
let text = tok.decode(&ids);                 // lossless, even on invalid UTF-8
let n = tok.count(b"how many tokens is this?");

// batch APIs are parallel; 0 threads means every core
let docs: Vec<&[u8]> = vec![b"first", b"second"];
let counts = tok.count_batch(&docs, 0, false);   // no ids materialized
let batches = tok.encode_batch(&docs, 0, false);

let (ids, spans) = tok.encode_with_offsets(b"per-token byte spans");

// truncation without building ids or decoding: a byte offset back
let t = tok.truncate(text, 8192);
let kept = &text[..t.bytes];        // always a character boundary
t.total_tokens;                     // the whole input, truncated or not

Every batch method takes the same two knobs as the Python API:

argument	meaning
threads: usize	0 uses every core; 1 runs on the calling thread; anything else caps the pool.
with_special: bool	false counts/encodes special strings as ordinary text; true maps them to their single special id.

Tokenizer is Send + Sync — load it once and share it. Full API on docs.rs/toktok-rs.

Cargo features

feature	default	what it does
embedded-data	on	compiles the vocabularies into the binary (~4 MB of .rodata), so Tokenizer::builtin() needs no files at runtime. Turn it off (default-features = false) to drop them and load from a directory with Tokenizer::load_dir(dir, name) instead.

More

• docs/BENCHMARKS.md — full results, memory and latency profile, how to reproduce
• docs/PYTHON.md — the tokenizer behind batch_count
• docs/DEVELOPMENT.md — layout, tests, how it's fast
• RELEASING.md — scripts/release.sh patch|minor|major|X.Y.Z, and the registry setup

License

MIT — see LICENSE. Ported from quicktok (MIT); vocabulary data derives from tiktoken (MIT). See NOTICE.