dirty: thread-cache

[SBrandeis]

PipelineTokenizer benchmark

3 / 4 models supported · geomean ×3.03 across supported · 22 fixtures each — ~10 kB inputs · single thread

89a5fe763 · 2026-07-08 16:08 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 48 cores

Pipeline stage mix (mean share of encode time): model 45%, pre-tokenize 38%, normalize 14%, added-token 2%. Per-model decomposition charts below.

Not yet supported: t5-base (Unigram · WhitespaceSplit+Metaspace) — roadmap cards below.

Not yet supported

Per-fixture results

bert-base-uncased — WordPiece · Bert

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	7.4	20.9	×2.81	1.22	27.83	9.01	9.54	match
arb_Arab	lang	3.8	8.4	×2.21	1.17	27.67	13.27	77.37	match
ben_Beng	lang	5.6	12.7	×2.28	1.16	19.38	8.74	49.06	match
cmn_Hani	lang	3.5	14.2	×4.06	1.17	37.79	10.88	21.15	match
ell_Grek	lang	3.5	7.1	×2.05	1.16	28.78	13.52	96.65	match
eng_Latn	lang	4.2	15.1	×3.61	2.54	39.73	3.43	19.96	match
heb_Hebr	lang	3.8	8.1	×2.14	1.17	37.80	13.29	70.76	match
hin_Deva	lang	6.0	14.7	×2.47	1.16	27.12	7.82	31.47	match
jpn_Jpan	lang	4.0	11.2	×2.82	1.17	23.43	10.78	52.47	match
kat_Geor	lang	5.8	11.2	×1.92	1.16	26.62	9.30	51.21	match
kor_Hang	lang	2.3	4.1	×1.81	1.19	35.33	21.35	185.65	match
rus_Cyrl	lang	3.3	6.4	×1.93	1.16	27.58	13.62	112.23	match
tam_Taml	lang	6.6	15.0	×2.25	1.15	18.82	8.20	37.84	match
tha_Thai	lang	8.2	14.0	×1.70	1.17	25.24	7.37	36.06	match
added_normalized_dense	modalities	6.2	21.2	×3.41	1.31	41.79	1.10	4.23	match
added_normalized_sparse	modalities	4.9	18.0	×3.65	1.93	41.02	2.32	10.99	match
added_special_dense	modalities	4.9	48.2	×9.88	5.12	9.31	2.26	3.39	match
added_special_sparse	modalities	3.8	21.4	×5.69	3.70	28.31	2.83	11.55	match
agentic-traces	modalities	3.6	13.4	×3.77	2.33	39.55	3.87	29.66	match
agentic_swe	modalities	3.8	13.7	×3.59	1.87	39.65	3.08	29.08	match
code_mixed	modalities	3.7	12.4	×3.37	2.13	39.60	3.44	36.84	match
math_latex	modalities	3.7	13.7	×3.69	2.46	39.64	3.51	27.67	match

deepseek-v4 — BPE · Split+ByteLevel

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	4.0	16.2	×4.05	0.66	0.00	42.93	16.10	match
arb_Arab	lang	3.5	8.8	×2.50	0.59	0.00	51.20	60.06	match
ben_Beng	lang	4.4	10.6	×2.43	0.59	0.00	36.32	57.93	match
cmn_Hani	lang	3.3	7.8	×2.37	0.76	0.00	63.87	61.89	match
ell_Grek	lang	3.7	9.5	×2.58	0.60	0.00	47.26	58.05	match
eng_Latn	lang	2.8	7.3	×2.61	1.99	0.00	67.77	68.23	match
heb_Hebr	lang	3.2	7.7	×2.39	0.61	0.00	51.06	79.53	match
hin_Deva	lang	4.2	11.6	×2.74	0.60	0.00	39.44	44.70	match
jpn_Jpan	lang	3.8	8.7	×2.32	0.66	0.00	55.75	56.13	match
kat_Geor	lang	4.5	11.2	×2.49	0.59	0.00	33.15	55.51	match
kor_Hang	lang	3.3	9.4	×2.80	0.61	0.00	53.13	50.11	match
rus_Cyrl	lang	3.5	8.6	×2.46	0.60	0.00	47.27	68.86	match
tam_Taml	lang	4.6	11.2	×2.44	0.59	0.00	32.63	54.99	match
tha_Thai	lang	5.5	10.7	×1.94	0.61	0.00	27.77	64.32	match
added_normalized_dense	modalities	4.0	11.5	×2.90	0.73	0.00	44.01	42.28	match
added_normalized_sparse	modalities	3.8	10.5	×2.77	1.33	0.00	49.47	45.33	match
added_special_dense	modalities	3.2	7.3	×2.28	6.73	0.26	112.29	10.89	match
added_special_sparse	modalities	3.0	7.9	×2.63	4.00	0.06	80.65	42.12	match
agentic-traces	modalities	2.5	6.1	×2.46	1.79	0.00	93.68	63.11	match
agentic_swe	modalities	2.5	5.7	×2.31	1.29	0.01	107.81	79.12	match
code_mixed	modalities	2.6	6.8	×2.62	1.54	0.00	78.54	63.64	match
math_latex	modalities	2.5	6.5	×2.62	1.90	0.00	82.57	68.92	match

llama-3 — BPE · Split+ByteLevel

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	3.5	23.1	×6.54	0.66	0.00	28.99	10.84	match
arb_Arab	lang	3.7	11.4	×3.05	0.59	0.00	33.41	52.10	match
ben_Beng	lang	2.9	12.0	×4.17	0.60	0.00	45.13	38.18	match
cmn_Hani	lang	4.1	12.5	×3.09	0.61	0.00	20.22	57.89	match
ell_Grek	lang	4.0	12.6	×3.17	0.60	0.00	29.55	47.34	match
eng_Latn	lang	3.4	14.2	×4.19	2.06	0.02	46.49	17.00	match
heb_Hebr	lang	3.3	13.6	×4.09	0.63	0.00	32.36	38.12	match
hin_Deva	lang	4.1	16.5	×4.02	0.60	0.00	50.01	6.59	match
jpn_Jpan	lang	4.6	12.8	×2.80	0.60	0.00	18.95	56.84	match
kat_Geor	lang	4.0	22.6	×5.60	0.59	0.00	18.30	24.00	match
kor_Hang	lang	3.1	11.4	×3.62	0.61	0.00	33.88	54.19	match
rus_Cyrl	lang	4.0	11.7	×2.90	0.60	0.00	26.77	57.02	match
tam_Taml	lang	3.2	12.4	×3.87	0.60	0.00	46.14	30.77	match
tha_Thai	lang	4.4	12.8	×2.90	0.61	0.00	29.81	45.41	match
added_normalized_dense	modalities	4.1	15.4	×3.74	0.74	0.00	25.85	36.51	match
added_normalized_sparse	modalities	4.2	17.3	×4.11	1.30	0.00	34.26	19.81	match
added_special_dense	modalities	4.0	12.5	×3.15	5.23	0.15	70.71	1.34	match
added_special_sparse	modalities	4.2	15.1	×3.62	3.32	0.00	56.26	4.22	match
agentic-traces	modalities	3.1	12.4	×3.98	1.83	0.00	53.05	26.47	match
agentic_swe	modalities	3.4	10.7	×3.10	1.28	0.00	58.22	27.98	match
code_mixed	modalities	3.6	13.7	×3.81	1.56	0.00	52.83	18.66	match
math_latex	modalities	3.3	12.8	×3.92	1.96	0.00	53.78	16.77	match

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SBrandeis]

Superceded by #2223 , closing