Skip to content

feat(pipeline) plug post-processors#2227

Open
SBrandeis wants to merge 4 commits into
feat/train_encode_splitfrom
feat/post-processor
Open

feat(pipeline) plug post-processors#2227
SBrandeis wants to merge 4 commits into
feat/train_encode_splitfrom
feat/post-processor

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

TL;DR

Supercedes #2182

Add support for post-processors in PipelineTokenizer

PipelineTokenizer benchmark

7 / 8 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · ~10 kB inputs · single thread + 1/2/4/8/max-thread sweep

6f56bb1fa · 2026-07-21 20:49 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 32 cores

Per-model encode throughput vs latest release

vs base branch (35b31244b) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.

Per-model encode throughput vs base branch Per-model memory footprint Minimal encode binary size
bert-base-uncased — normalizer-heavy WordPiece · ×4.91 vs v0.23.1 · ×0.98 vs base bert-base-uncased speedup bert-base-uncased stage decomposition bert-base-uncased thread scaling

Memory (RSS MB, load+encode): v0.23.1 11+0 (peak 12) · Pipeline 8+0 (peak 17)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 7.6 27.4 ×3.59 ×1.00 3% (1.2) 76% (27.5) 13% (4.6) 8% (2.8) 0% (0.0) match
arb_Arab lang 4.2 24.6 ×5.89 ×0.99 3% (1.2) 68% (27.6) 8% (3.2) 20% (8.1) 1% (0.3) match
ben_Beng lang 5.9 34.4 ×5.79 ×0.99 4% (1.2) 67% (19.2) 10% (2.8) 19% (5.5) 1% (0.2) match
cmn_Hani lang 3.8 18.2 ×4.80 ×0.97 2% (1.2) 71% (37.6) 11% (5.8) 16% (8.5) 0% (0.0) match
ell_Grek lang 3.7 23.3 ×6.33 ×0.98 3% (1.2) 68% (28.8) 8% (3.3) 21% (8.8) 1% (0.4) match
eng_Latn lang 4.3 17.7 ×4.07 ×0.97 5% (2.6) 71% (38.5) 8% (4.3) 16% (8.9) 0% (0.0) match
heb_Hebr lang 4.2 19.9 ×4.76 ×0.99 2% (1.2) 74% (37.3) 8% (3.8) 15% (7.7) 1% (0.4) match
hin_Deva lang 6.5 27.3 ×4.23 ×0.99 3% (1.2) 74% (26.9) 9% (3.1) 14% (5.1) 1% (0.2) match
jpn_Jpan lang 4.2 27.0 ×6.37 ×1.00 3% (1.2) 64% (23.6) 12% (4.5) 20% (7.5) 0% (0.0) match
kat_Geor lang 6.1 28.0 ×4.61 ×1.00 3% (1.2) 75% (26.6) 8% (2.7) 13% (4.7) 0% (0.1) match
kor_Hang lang 2.4 17.5 ×7.31 ×0.99 2% (1.2) 62% (35.3) 13% (7.4) 23% (12.8) 0% (0.2) match
rus_Cyrl lang 3.6 23.5 ×6.49 ×0.99 3% (1.2) 65% (27.5) 7% (3.1) 24% (10.2) 1% (0.2) match
tam_Taml lang 6.8 38.0 ×5.54 ×0.99 4% (1.2) 72% (18.7) 9% (2.5) 14% (3.8) 0% (0.0) match
tha_Thai lang 8.1 33.4 ×4.10 ×1.00 4% (1.2) 83% (25.1) 6% (1.8) 7% (2.0) 0% (0.1) match
added_normalized_dense modalities 6.7 19.0 ×2.85 ×0.97 3% (1.3) 81% (40.5) 14% (7.2) 2% (1.1) 0% (0.2) match
added_normalized_sparse modalities 5.3 17.8 ×3.38 ×0.98 4% (1.9) 74% (39.7) 13% (7.1) 9% (4.8) 0% (0.1) match
added_special_dense modalities 5.3 38.6 ×7.31 ×0.98 21% (5.2) 37% (9.2) 36% (9.0) 6% (1.4) 1% (0.2) match
added_special_sparse modalities 4.0 21.1 ×5.30 ×0.99 8% (3.6) 61% (28.0) 19% (8.7) 12% (5.7) 0% (0.1) match
agentic-traces modalities 3.8 17.5 ×4.59 ×0.98 4% (2.3) 70% (38.4) 9% (4.9) 17% (9.3) 0% (0.0) match
agentic_swe modalities 4.0 18.7 ×4.65 ×0.97 4% (1.9) 75% (38.6) 7% (3.6) 14% (7.3) 0% (0.0) match
code_mixed modalities 3.9 18.3 ×4.72 ×0.97 4% (2.1) 73% (38.5) 8% (4.2) 15% (7.8) 0% (0.0) match
math_latex modalities 3.9 17.4 ×4.41 ×0.96 4% (2.5) 70% (38.5) 9% (4.7) 17% (9.2) 0% (0.0) match
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×4.54 vs v0.23.1 · ×0.97 vs base deepseek-v4 speedup deepseek-v4 stage decomposition deepseek-v4 thread scaling

Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 68) · Pipeline 96+0 (peak 96)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 3.7 33.5 ×9.08 ×0.99 2% (0.7) 0% (0.0) 17% (4.8) 79% (22.3) 2% (0.5) match
arb_Arab lang 3.3 15.8 ×4.74 ×0.94 1% (0.6) 0% (0.0) 6% (3.5) 93% (56.0) 0% (0.0) match
ben_Beng lang 5.9 17.8 ×3.02 ×0.99 1% (0.6) 0% (0.0) 6% (3.2) 93% (51.8) 0% (0.0) match
cmn_Hani lang 3.1 15.5 ×4.95 ×0.91 1% (0.8) 0% (0.0) 6% (3.2) 94% (54.4) 0% (0.0) match
ell_Grek lang 3.8 17.9 ×4.72 ×1.00 1% (0.6) 0% (0.0) 6% (3.5) 92% (51.0) 0% (0.2) match
eng_Latn lang 2.2 12.2 ×5.45 ×0.95 2% (2.0) 0% (0.0) 7% (5.1) 91% (71.6) 0% (0.1) match
heb_Hebr lang 3.1 12.8 ×4.10 ×0.98 1% (0.6) 0% (0.0) 5% (3.7) 94% (71.8) 0% (0.3) match
hin_Deva lang 5.5 21.4 ×3.92 ×0.98 1% (0.6) 0% (0.0) 7% (3.5) 91% (42.6) 0% (0.1) match
jpn_Jpan lang 3.8 17.9 ×4.70 ×0.97 1% (0.7) 0% (0.0) 6% (3.0) 93% (50.7) 0% (0.1) match
kat_Geor lang 5.7 18.0 ×3.14 ×0.98 1% (0.6) 0% (0.0) 5% (3.0) 93% (51.0) 0% (0.2) match
kor_Hang lang 2.4 19.5 ×8.15 ×0.97 1% (0.6) 0% (0.0) 7% (3.8) 91% (46.3) 0% (0.2) match
rus_Cyrl lang 4.1 14.3 ×3.51 ×0.98 1% (0.6) 0% (0.0) 5% (3.3) 94% (64.4) 0% (0.1) match
tam_Taml lang 6.2 17.3 ×2.77 ×0.97 1% (0.6) 0% (0.0) 5% (2.8) 94% (53.0) 0% (0.0) match
tha_Thai lang 6.4 14.4 ×2.25 ×0.96 1% (0.6) 0% (0.0) 3% (2.3) 96% (64.9) 0% (0.0) match
added_normalized_dense modalities 6.2 23.4 ×3.80 ×1.00 2% (0.7) 0% (0.0) 7% (2.9) 91% (37.7) 0% (0.0) match
added_normalized_sparse modalities 5.5 19.4 ×3.53 ×0.99 3% (1.3) 0% (0.0) 8% (3.9) 90% (44.8) 0% (0.0) match
added_special_dense modalities 3.7 34.9 ×9.39 ×0.96 24% (6.3) 1% (0.4) 24% (6.2) 51% (13.2) 0% (0.0) match
added_special_sparse modalities 4.0 20.0 ×5.00 ×0.99 8% (3.8) 0% (0.2) 14% (6.4) 79% (37.4) 0% (0.0) match
agentic-traces modalities 2.8 13.9 ×4.91 ×0.98 2% (1.8) 0% (0.0) 7% (5.4) 90% (65.6) 0% (0.2) match
agentic_swe modalities 2.4 14.4 ×6.03 ×0.96 2% (1.3) 0% (0.0) 6% (3.9) 92% (62.0) 0% (0.0) match
code_mixed modalities 3.3 15.0 ×4.51 ×0.98 2% (1.6) 0% (0.0) 7% (4.6) 91% (58.4) 0% (0.0) match
math_latex modalities 2.9 13.8 ×4.74 ×1.00 3% (1.9) 0% (0.0) 7% (5.4) 90% (65.4) 0% (0.3) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.62 3.47 4.79 5.64 42.3 20.1 9.1 — 8.8× / 7.5× 4.2× / 3.6× 1.9× / 1.6× —
arb_Arab 1.00 3.06 3.49 5.55 49.4 23.0 10.1 — 14.1× / 8.9× 6.6× / 4.1× 2.9× / 1.8× —
ben_Beng 1.46 2.96 3.21 4.71 36.0 16.1 7.5 — 11.2× / 7.6× 5.0× / 3.4× 2.3× / 1.6× —
cmn_Hani 1.13 2.35 3.24 4.46 61.4 33.5 14.1 — 19.0× / 13.8× 10.3× / 7.5× 4.4× / 3.2× —
ell_Grek 0.58 3.02 3.47 5.91 47.3 20.9 9.8 — 13.6× / 8.0× 6.0× / 3.5× 2.8× / 1.7× —
eng_Latn 0.09 1.74 5.12 6.76 66.7 39.3 16.4 — 13.0× / 9.9× 7.7× / 5.8× 3.2× / 2.4× —
heb_Hebr 1.00 3.11 3.67 5.77 51.6 24.6 10.8 — 14.1× / 8.9× 6.7× / 4.3× 3.0× / 1.9× —
hin_Deva 1.37 3.12 3.46 5.22 38.2 18.9 8.8 — 11.0× / 7.3× 5.4× / 3.6× 2.5× / 1.7× —
jpn_Jpan 1.56 3.53 3.05 5.01 54.1 27.1 11.9 — 17.8× / 10.8× 8.9× / 5.4× 3.9× / 2.4× —
kat_Geor 1.39 2.55 3.01 4.17 32.7 15.3 7.3 — 10.9× / 7.8× 5.1× / 3.7× 2.4× / 1.7× —
kor_Hang 1.09 2.79 3.76 5.46 50.9 27.3 11.9 — 13.5× / 9.3× 7.3× / 5.0× 3.2× / 2.2× —
rus_Cyrl 1.01 2.97 3.33 5.28 47.6 21.0 9.5 — 14.3× / 9.0× 6.3× / 4.0× 2.9× / 1.8× —
tam_Taml 0.93 2.96 2.76 4.80 32.2 13.6 6.6 — 11.7× / 6.7× 4.9× / 2.8× 2.4× / 1.4× —
tha_Thai 1.37 2.59 2.35 3.57 26.8 10.1 5.2 — 11.4× / 7.5× 4.3× / 2.8× 2.2× / 1.5× —
added_normalized_dense 0.06 1.77 2.86 4.57 42.5 20.3 8.9 — 14.9× / 9.3× 7.1× / 4.4× 3.1× / 1.9× —
added_normalized_sparse 0.06 1.77 3.94 5.65 49.4 25.9 11.1 — 12.5× / 8.7× 6.6× / 4.6× 2.8× / 2.0× —
added_special_dense 0.06 1.77 6.18 7.89 178.6 96.7 38.0 — 28.9× / 22.6× 15.6× / 12.3× 6.1× / 4.8× —
added_special_sparse 0.06 1.77 6.42 8.13 105.0 56.7 23.9 — 16.3× / 12.9× 8.8× / 7.0× 3.7× / 2.9× —
agentic-traces 0.57 1.76 5.40 6.59 81.7 53.9 20.1 — 15.1× / 12.4× 10.0× / 8.2× 3.7× / 3.0× —
agentic_swe 0.51 1.73 3.89 5.12 97.4 65.3 23.5 — 25.0× / 19.0× 16.8× / 12.8× 6.0× / 4.6× —
code_mixed 0.07 1.73 4.62 6.28 74.4 53.9 18.7 — 16.1× / 11.8× 11.7× / 8.6× 4.0× / 3.0× —
math_latex 0.56 1.78 5.36 6.58 79.7 47.8 19.1 — 14.9× / 12.1× 8.9× / 7.3× 3.6× / 2.9× —
gpt2 — gpt2 ByteLevel regex · ×7.44 vs v0.23.1 · ×1.02 vs base gpt2 speedup gpt2 stage decomposition gpt2 thread scaling

Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 30+0 (peak 29)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.4 52.3 ×11.90 ×1.13 4% (0.7) 0% (0.0) 20% (3.6) 77% (14.1) 0% (0.0) match
arb_Arab lang 4.3 25.8 ×6.08 ×0.97 2% (0.6) 0% (0.0) 6% (2.3) 92% (35.3) 1% (0.2) match
ben_Beng lang 3.2 46.2 ×14.35 ×1.07 3% (0.6) 0% (0.0) 14% (3.0) 83% (17.4) 0% (0.1) match
cmn_Hani lang 4.0 29.1 ×7.26 ×0.98 2% (0.6) 0% (0.0) 7% (2.3) 91% (30.8) 1% (0.2) match
ell_Grek lang 4.6 30.4 ×6.57 ×1.08 2% (0.6) 0% (0.0) 7% (2.2) 91% (29.7) 0% (0.1) match
eng_Latn lang 3.8 13.8 ×3.65 ×1.00 3% (2.0) 0% (0.0) 4% (3.0) 93% (65.4) 0% (0.1) match
heb_Hebr lang 4.1 29.8 ×7.24 ×1.02 2% (0.6) 0% (0.0) 7% (2.3) 91% (30.2) 1% (0.2) match
hin_Deva lang 3.3 40.4 ×12.38 ×1.01 2% (0.6) 0% (0.0) 13% (3.1) 84% (20.5) 0% (0.1) match
jpn_Jpan lang 4.6 21.8 ×4.72 ×0.97 1% (0.6) 0% (0.0) 5% (2.1) 94% (42.5) 0% (0.2) match
kat_Geor lang 4.9 72.2 ×14.81 ×1.14 4% (0.6) 0% (0.0) 15% (2.1) 80% (10.7) 0% (0.0) match
kor_Hang lang 3.5 42.1 ×12.01 ×1.00 3% (0.6) 0% (0.0) 11% (2.4) 86% (20.0) 1% (0.1) match
rus_Cyrl lang 4.4 27.5 ×6.32 ×1.02 2% (0.6) 0% (0.0) 6% (2.1) 92% (33.4) 0% (0.1) match
tam_Taml lang 2.9 67.2 ×23.41 ×0.99 4% (0.6) 0% (0.0) 18% (2.6) 77% (11.3) 0% (0.0) match
tha_Thai lang 3.7 36.1 ×9.66 ×1.01 2% (0.6) 0% (0.0) 9% (2.5) 88% (24.0) 0% (0.1) match
added_normalized_dense modalities 6.2 26.8 ×4.30 ×1.03 2% (0.7) 0% (0.0) 3% (1.1) 95% (34.7) 0% (0.2) match
added_normalized_sparse modalities 5.5 22.1 ×4.03 ×1.01 3% (1.3) 0% (0.0) 5% (2.0) 93% (41.4) 0% (0.0) match
added_special_dense modalities 4.4 47.0 ×10.60 ×1.02 24% (5.0) 1% (0.1) 21% (4.3) 53% (10.9) 1% (0.3) match
added_special_sparse modalities 4.5 23.8 ×5.27 ×1.01 8% (3.2) 0% (0.1) 11% (4.6) 80% (33.1) 0% (0.2) match
agentic-traces modalities 3.4 16.3 ×4.85 ×1.01 3% (1.8) 0% (0.0) 6% (3.5) 91% (55.6) 0% (0.1) match
agentic_swe modalities 3.7 24.4 ×6.57 ×1.00 3% (1.3) 0% (0.0) 6% (2.4) 91% (36.5) 0% (0.1) match
code_mixed modalities 3.7 20.6 ×5.55 ×1.01 3% (1.5) 0% (0.0) 6% (2.9) 91% (43.7) 0% (0.0) match
math_latex modalities 3.4 15.2 ×4.47 ×1.01 3% (1.9) 0% (0.0) 5% (3.3) 92% (59.6) 0% (0.1) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.64 3.51 3.65 4.51 27.0 21.1 5.7 4.8 7.4× / 6.0× 5.8× / 4.7× 1.6× / 1.3× 1.3× / 1.1×
arb_Arab 1.00 3.06 2.26 4.33 31.6 25.9 6.7 5.2 13.9× / 7.3× 11.4× / 6.0× 3.0× / 1.5× 2.3× / 1.2×
ben_Beng 1.46 2.95 2.95 4.44 65.2 55.7 13.7 4.0 22.1× / 14.7× 18.9× / 12.5× 4.6× / 3.1× 1.4× / 0.9×
cmn_Hani 1.12 2.35 2.30 3.53 25.9 21.5 5.8 2.4 11.3× / 7.4× 9.4× / 6.1× 2.5× / 1.7× 1.0× / 0.7×
ell_Grek 0.58 3.10 2.16 4.67 28.0 21.7 5.8 4.6 13.0× / 6.0× 10.1× / 4.7× 2.7× / 1.2× 2.2× / 1.0×
eng_Latn 0.10 1.74 3.03 4.68 43.6 42.1 11.6 3.8 14.4× / 9.3× 13.9× / 9.0× 3.8× / 2.5× 1.3× / 0.8×
heb_Hebr 0.99 3.16 2.27 4.44 30.8 26.9 6.8 3.0 13.5× / 6.9× 11.9× / 6.1× 3.0× / 1.5× 1.3× / 0.7×
hin_Deva 1.36 3.13 3.07 4.84 61.2 55.9 13.6 4.2 19.9× / 12.6× 18.2× / 11.6× 4.4× / 2.8× 1.4× / 0.9×
jpn_Jpan 1.55 3.51 2.11 4.07 23.2 17.5 4.9 3.9 11.0× / 5.7× 8.3× / 4.3× 2.3× / 1.2× 1.8× / 1.0×
kat_Geor 1.39 2.49 2.06 3.15 16.8 14.2 4.1 2.0 8.2× / 5.3× 6.9× / 4.5× 2.0× / 1.3× 1.0× / 0.6×
kor_Hang 1.08 2.78 2.44 4.13 31.3 27.9 7.3 3.7 12.9× / 7.6× 11.5× / 6.8× 3.0× / 1.8× 1.5× / 0.9×
rus_Cyrl 1.01 3.04 2.10 4.13 27.1 21.1 5.7 2.5 12.9× / 6.6× 10.1× / 5.1× 2.7× / 1.4× 1.2× / 0.6×
tam_Taml 0.92 2.96 2.64 4.68 68.3 58.9 14.2 3.8 25.8× / 14.6× 22.3× / 12.6× 5.4× / 3.0× 1.4× / 0.8×
tha_Thai 1.37 2.57 2.48 3.68 39.4 31.8 8.6 3.2 15.9× / 10.7× 12.8× / 8.6× 3.5× / 2.3× 1.3× / 0.9×
added_normalized_dense 0.06 1.77 1.05 2.76 23.1 23.2 6.2 1.9 21.9× / 8.4× 22.0× / 8.4× 5.9× / 2.2× 1.8× / 0.7×
added_normalized_sparse 0.06 1.77 2.02 3.72 30.8 32.1 8.4 2.7 15.3× / 8.3× 15.9× / 8.6× 4.2× / 2.3× 1.3× / 0.7×
added_special_dense 0.06 1.77 4.32 6.02 92.9 101.2 19.7 3.0 21.5× / 15.4× 23.4× / 16.8× 4.6× / 3.3× 0.7× / 0.5×
added_special_sparse 0.06 1.77 4.62 6.32 60.1 62.1 14.4 3.4 13.0× / 9.5× 13.5× / 9.8× 3.1× / 2.3× 0.7× / 0.5×
agentic-traces 0.57 1.76 3.53 4.72 57.8 61.3 15.0 4.7 16.4× / 12.3× 17.4× / 13.0× 4.2× / 3.2× 1.3× / 1.0×
agentic_swe 0.51 1.73 2.38 3.60 58.7 70.8 14.2 3.6 24.6× / 16.3× 29.7× / 19.7× 6.0× / 3.9× 1.5× / 1.0×
code_mixed 0.07 1.73 2.93 4.60 57.5 67.4 14.7 4.2 19.6× / 12.5× 23.0× / 14.7× 5.0× / 3.2× 1.4× / 0.9×
math_latex 0.57 1.78 3.32 4.53 51.4 51.3 13.5 4.2 15.5× / 11.4× 15.4× / 11.3× 4.1× / 3.0× 1.3× / 0.9×
gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×7.57 vs v0.23.1 · ×1.01 vs base gpt-oss speedup gpt-oss stage decomposition gpt-oss thread scaling

Memory (RSS MB, load+encode): v0.23.1 4+11 (peak 16) · Pipeline 6+0 (peak 6)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 3.9 61.6 ×15.67 ×1.07 4% (0.7) 0% (0.0) 30% (4.6) 66% (10.4) 0% (0.0) match
arb_Arab lang 4.2 25.7 ×6.05 ×1.02 1% (0.6) 0% (0.0) 8% (3.2) 91% (37.8) 0% (0.2) match
ben_Beng lang 5.6 28.2 ×5.06 ×1.00 2% (0.6) 0% (0.0) 10% (3.6) 88% (30.8) 0% (0.1) match
cmn_Hani lang 4.3 37.8 ×8.81 ×0.98 2% (0.6) 0% (0.0) 13% (3.3) 85% (22.0) 0% (0.1) match
ell_Grek lang 4.1 31.4 ×7.75 ×0.97 2% (0.6) 0% (0.0) 10% (3.2) 88% (27.7) 0% (0.1) match
eng_Latn lang 3.3 21.6 ×6.62 ×1.02 4% (2.0) 0% (0.0) 10% (4.4) 86% (39.7) 0% (0.2) match
heb_Hebr lang 4.0 28.3 ×7.12 ×1.02 2% (0.6) 0% (0.0) 9% (3.2) 89% (31.3) 1% (0.2) match
hin_Deva lang 5.2 24.3 ×4.68 ×1.03 1% (0.6) 0% (0.0) 9% (3.7) 89% (36.2) 0% (0.1) match
jpn_Jpan lang 4.8 36.9 ×7.72 ×0.97 2% (0.6) 0% (0.0) 12% (3.1) 86% (22.7) 0% (0.1) match
kat_Geor lang 5.5 25.5 ×4.60 ×0.96 2% (0.6) 0% (0.0) 8% (2.9) 91% (35.0) 0% (0.1) match
kor_Hang lang 3.4 46.2 ×13.40 ×0.99 3% (0.6) 0% (0.0) 17% (3.6) 80% (17.0) 1% (0.1) match
rus_Cyrl lang 4.4 22.5 ×5.10 ×1.05 1% (0.6) 0% (0.0) 7% (3.1) 91% (40.5) 0% (0.2) match
tam_Taml lang 5.5 30.6 ×5.55 ×1.07 2% (0.6) 0% (0.0) 10% (3.2) 88% (28.7) 0% (0.0) match
tha_Thai lang 6.2 27.1 ×4.36 ×0.99 2% (0.6) 0% (0.0) 9% (3.2) 90% (33.3) 0% (0.0) match
added_normalized_dense modalities 4.3 46.9 ×10.94 ×1.02 4% (0.7) 0% (0.0) 10% (2.1) 86% (17.7) 1% (0.1) match
added_normalized_sparse modalities 3.9 37.3 ×9.54 ×1.02 5% (1.3) 0% (0.0) 12% (3.0) 83% (21.8) 1% (0.2) match
added_special_dense modalities 3.3 56.9 ×17.17 ×1.03 29% (4.9) 1% (0.2) 31% (5.3) 38% (6.5) 1% (0.2) match
added_special_sparse modalities 3.5 34.6 ×10.04 ×1.02 11% (3.2) 0% (0.0) 21% (6.0) 66% (18.7) 1% (0.3) match
agentic-traces modalities 3.1 22.8 ×7.46 ×1.01 4% (1.8) 0% (0.0) 11% (4.9) 84% (36.6) 1% (0.2) match
agentic_swe modalities 3.5 21.9 ×6.20 ×1.00 3% (1.3) 0% (0.0) 8% (3.7) 89% (40.3) 0% (0.0) match
code_mixed modalities 3.5 30.0 ×8.61 ×1.02 5% (1.5) 0% (0.0) 13% (4.3) 82% (27.2) 1% (0.2) match
math_latex modalities 3.2 22.9 ×7.15 ×1.02 4% (1.9) 0% (0.0) 11% (4.7) 84% (36.7) 0% (0.1) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.61 3.51 4.65 5.55 29.3 13.7 6.8 4.7 6.3× / 5.3× 3.0× / 2.5× 1.5× / 1.2× 1.0× / 0.8×
arb_Arab 1.00 3.04 3.17 5.21 33.6 15.7 7.4 5.1 10.6× / 6.4× 4.9× / 3.0× 2.3× / 1.4× 1.6× / 1.0×
ben_Beng 1.46 2.97 3.63 5.15 23.5 10.8 5.3 2.8 6.5× / 4.6× 3.0× / 2.1× 1.5× / 1.0× 0.8× / 0.5×
cmn_Hani 1.13 2.35 3.30 4.52 21.6 10.7 5.4 2.5 6.5× / 4.8× 3.3× / 2.4× 1.6× / 1.2× 0.8× / 0.6×
ell_Grek 0.58 3.06 3.19 5.66 29.8 15.4 6.7 5.1 9.3× / 5.3× 4.8× / 2.7× 2.1× / 1.2× 1.6× / 0.9×
eng_Latn 0.10 1.74 4.45 6.09 42.1 28.6 13.3 4.2 9.5× / 6.9× 6.4× / 4.7× 3.0× / 2.2× 0.9× / 0.7×
heb_Hebr 0.99 3.14 3.24 5.38 32.9 17.2 7.9 2.8 10.1× / 6.1× 5.3× / 3.2× 2.4× / 1.5× 0.9× / 0.5×
hin_Deva 1.37 3.13 3.71 5.47 25.6 12.4 6.3 3.1 6.9× / 4.7× 3.4× / 2.3× 1.7× / 1.1× 0.8× / 0.6×
jpn_Jpan 1.56 3.51 3.06 5.01 20.1 9.0 4.7 3.7 6.5× / 4.0× 2.9× / 1.8× 1.5× / 0.9× 1.2× / 0.7×
kat_Geor 1.39 2.47 2.91 3.98 18.9 10.0 4.5 2.0 6.5× / 4.7× 3.4× / 2.5× 1.6× / 1.1× 0.7× / 0.5×
kor_Hang 1.08 2.85 3.61 5.38 33.1 18.3 8.6 3.5 9.2× / 6.1× 5.1× / 3.4× 2.4× / 1.6× 1.0× / 0.7×
rus_Cyrl 1.01 3.03 3.09 5.11 28.3 14.7 6.5 4.9 9.2× / 5.5× 4.8× / 2.9× 2.1× / 1.3× 1.6× / 1.0×
tam_Taml 0.92 2.98 3.18 5.23 18.9 8.5 4.2 2.9 5.9× / 3.6× 2.7× / 1.6× 1.3× / 0.8× 0.9× / 0.6×
tha_Thai 1.36 2.58 3.17 4.39 13.1 5.7 2.7 2.2 4.1× / 3.0× 1.8× / 1.3× 0.9× / 0.6× 0.7× / 0.5×
added_normalized_dense 0.06 1.77 2.12 3.83 31.9 17.8 11.4 2.0 15.1× / 8.3× 8.4× / 4.6× 5.4× / 3.0× 0.9× / 0.5×
added_normalized_sparse 0.06 1.77 3.05 4.76 34.6 20.5 11.7 2.8 11.3× / 7.3× 6.7× / 4.3× 3.8× / 2.5× 0.9× / 0.6×
added_special_dense 0.06 1.77 5.30 7.00 83.3 67.5 23.4 3.3 15.7× / 11.9× 12.8× / 9.6× 4.4× / 3.3× 0.6× / 0.5×
added_special_sparse 0.06 1.77 6.04 7.75 55.3 41.5 16.5 3.7 9.1× / 7.1× 6.9× / 5.3× 2.7× / 2.1× 0.6× / 0.5×
agentic-traces 0.57 1.78 4.90 6.11 50.7 40.1 16.4 4.9 10.3× / 8.3× 8.2× / 6.6× 3.3× / 2.7× 1.0× / 0.8×
agentic_swe 0.53 1.74 3.67 4.88 51.0 48.5 16.7 3.5 13.9× / 10.5× 13.2× / 9.9× 4.6× / 3.4× 1.0× / 0.7×
code_mixed 0.07 1.75 4.29 5.96 50.7 46.5 16.7 4.3 11.8× / 8.5× 10.9× / 7.8× 3.9× / 2.8× 1.0× / 0.7×
math_latex 0.56 1.78 4.73 5.94 48.4 35.0 15.4 4.5 10.2× / 8.1× 7.4× / 5.9× 3.3× / 2.6× 0.9× / 0.8×
glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×12.42 vs v0.23.1 · ×0.99 vs base glm-5.2 speedup glm-5.2 stage decomposition glm-5.2 thread scaling

Memory (RSS MB, load+encode): v0.23.1 4+11 (peak 16) · Pipeline 5+0 (peak 5)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.7 62.7 ×13.27 ×0.97 8% (1.2) 0% (0.0) 24% (3.7) 68% (10.6) 0% (0.0) match
arb_Arab lang 4.2 71.6 ×17.09 ×0.97 9% (1.2) 0% (0.0) 17% (2.3) 74% (10.0) 0% (0.0) match
ben_Beng lang 3.5 69.6 ×19.69 ×0.99 8% (1.2) 0% (0.0) 23% (3.1) 69% (9.5) 0% (0.0) match
cmn_Hani lang 4.8 67.8 ×14.18 ×1.01 8% (1.2) 0% (0.0) 16% (2.3) 75% (10.7) 0% (0.1) match
ell_Grek lang 4.3 69.6 ×16.31 ×0.97 8% (1.2) 0% (0.0) 16% (2.2) 75% (10.4) 0% (0.0) match
eng_Latn lang 3.7 23.0 ×6.27 ×1.01 6% (2.5) 0% (0.0) 7% (3.0) 86% (37.8) 1% (0.4) match
heb_Hebr lang 4.0 70.8 ×17.79 ×0.97 9% (1.2) 0% (0.0) 17% (2.3) 75% (10.3) 0% (0.0) match
hin_Deva lang 3.2 66.1 ×20.47 ×0.99 8% (1.2) 0% (0.0) 22% (3.2) 70% (10.3) 0% (0.0) match
jpn_Jpan lang 4.9 74.2 ×15.19 ×0.98 9% (1.2) 0% (0.0) 17% (2.2) 73% (9.6) 1% (0.1) match
kat_Geor lang 4.7 79.9 ×17.04 ×0.94 10% (1.2) 0% (0.0) 17% (2.1) 73% (8.7) 0% (0.0) match
kor_Hang lang 3.6 64.5 ×17.81 ×0.97 8% (1.2) 0% (0.0) 17% (2.5) 75% (11.1) 0% (0.0) match
rus_Cyrl lang 4.1 39.1 ×9.46 ×0.98 5% (1.2) 0% (0.0) 8% (2.1) 87% (22.3) 0% (0.1) match
tam_Taml lang 3.3 71.4 ×21.40 ×1.01 9% (1.2) 0% (0.0) 20% (2.7) 72% (9.9) 0% (0.0) match
tha_Thai lang 4.0 71.3 ×17.69 ×0.97 9% (1.2) 0% (0.0) 19% (2.5) 73% (9.9) 0% (0.0) match
added_normalized_dense modalities 4.8 45.3 ×9.48 ×0.96 6% (1.3) 0% (0.0) 5% (1.1) 88% (18.3) 1% (0.2) match
added_normalized_sparse modalities 4.3 39.0 ×9.11 ×0.96 8% (1.9) 0% (0.0) 8% (2.0) 84% (20.5) 1% (0.2) match
added_special_dense modalities 3.5 41.5 ×12.00 ×1.01 51% (11.9) 0% (0.0) 23% (5.2) 25% (5.9) 1% (0.2) match
added_special_sparse modalities 3.6 33.8 ×9.46 ×1.01 23% (6.6) 0% (0.0) 19% (5.3) 58% (16.5) 1% (0.2) match
agentic-traces modalities 3.2 24.0 ×7.42 ×1.01 6% (2.4) 0% (0.0) 9% (3.7) 85% (35.5) 0% (0.1) match
agentic_swe modalities 3.5 22.1 ×6.32 ×0.99 4% (1.9) 0% (0.0) 6% (2.9) 89% (41.2) 0% (0.1) match
code_mixed modalities 3.6 31.2 ×8.71 ×1.03 7% (2.2) 0% (0.0) 10% (3.3) 82% (26.2) 1% (0.2) match
math_latex modalities 3.2 24.4 ×7.55 ×1.01 6% (2.5) 0% (0.0) 8% (3.4) 85% (34.8) 1% (0.3) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.62 3.53 3.65 4.56 27.5 16.2 5.9 4.8 7.5× / 6.0× 4.4× / 3.5× 1.6× / 1.3× 1.3× / 1.0×
arb_Arab 1.00 3.03 2.32 4.35 31.1 19.5 6.8 5.2 13.4× / 7.2× 8.4× / 4.5× 3.0× / 1.6× 2.2× / 1.2×
ben_Beng 1.46 2.93 3.10 4.58 45.4 27.1 10.4 3.7 14.6× / 9.9× 8.7× / 5.9× 3.3× / 2.3× 1.2× / 0.8×
cmn_Hani 1.12 2.35 2.34 3.57 20.0 11.7 4.7 2.4 8.5× / 5.6× 5.0× / 3.3× 2.0× / 1.3× 1.0× / 0.7×
ell_Grek 0.58 3.06 2.23 4.71 28.6 17.0 6.1 4.7 12.8× / 6.1× 7.6× / 3.6× 2.8× / 1.3× 2.1× / 1.0×
eng_Latn 0.12 1.73 3.02 4.63 44.3 32.7 12.2 3.7 14.7× / 9.6× 10.8× / 7.1× 4.1× / 2.6× 1.2× / 0.8×
heb_Hebr 0.99 3.12 2.34 4.46 30.8 20.1 7.2 3.0 13.2× / 6.9× 8.6× / 4.5× 3.1× / 1.6× 1.3× / 0.7×
hin_Deva 1.36 3.09 3.20 4.93 47.9 30.2 11.2 4.1 14.9× / 9.7× 9.4× / 6.1× 3.5× / 2.3× 1.3× / 0.8×
jpn_Jpan 1.55 3.53 2.16 4.14 18.6 9.9 4.1 3.7 8.6× / 4.5× 4.6× / 2.4× 1.9× / 1.0× 1.7× / 0.9×
kat_Geor 1.39 2.51 2.09 3.21 17.0 11.1 4.2 2.1 8.1× / 5.3× 5.3× / 3.4× 2.0× / 1.3× 1.0× / 0.6×
kor_Hang 1.08 2.89 2.50 4.31 31.3 21.6 7.4 3.7 12.5× / 7.3× 8.6× / 5.0× 2.9× / 1.7× 1.5× / 0.9×
rus_Cyrl 1.01 3.04 2.14 4.18 27.2 16.8 6.0 2.5 12.7× / 6.5× 7.9× / 4.0× 2.8× / 1.4× 1.2× / 0.6×
tam_Taml 0.92 2.95 2.67 4.70 44.8 26.4 9.8 3.5 16.8× / 9.5× 9.9× / 5.6× 3.7× / 2.1× 1.3× / 0.7×
tha_Thai 1.37 2.56 2.53 3.72 28.3 15.9 6.7 3.0 11.2× / 7.6× 6.3× / 4.3× 2.6× / 1.8× 1.2× / 0.8×
added_normalized_dense 0.06 1.77 1.11 2.81 23.6 16.9 6.5 1.9 21.3× / 8.4× 15.3× / 6.0× 5.9× / 2.3× 1.7× / 0.7×
added_normalized_sparse 0.06 1.77 1.97 3.67 31.6 22.5 8.6 2.6 16.1× / 8.6× 11.4× / 6.1× 4.4× / 2.3× 1.3× / 0.7×
added_special_dense 0.06 1.77 5.23 6.94 95.9 71.8 21.7 3.2 18.3× / 13.8× 13.7× / 10.3× 4.2× / 3.1× 0.6× / 0.5×
added_special_sparse 0.06 1.77 5.26 6.97 62.0 46.0 15.1 3.6 11.8× / 8.9× 8.7× / 6.6× 2.9× / 2.2× 0.7× / 0.5×
agentic-traces 0.58 1.76 3.69 4.86 53.2 44.0 14.9 4.6 14.4× / 10.9× 11.9× / 9.0× 4.0× / 3.1× 1.3× / 1.0×
agentic_swe 0.54 1.73 2.86 4.05 55.2 51.0 15.5 3.4 19.3× / 13.6× 17.8× / 12.6× 5.4× / 3.8× 1.2× / 0.8×
code_mixed 0.07 1.73 3.27 4.93 52.9 48.6 15.0 4.0 16.2× / 10.7× 14.9× / 9.9× 4.6× / 3.1× 1.2× / 0.8×
math_latex 0.58 1.77 3.42 4.60 51.3 39.2 14.0 4.1 15.0× / 11.2× 11.5× / 8.5× 4.1× / 3.0× 1.2× / 0.9×
llama-2 — model-bounded BPE, no pre-tokenizer · ×3.21 vs v0.23.1 · ×1.01 vs base llama-2 speedup llama-2 stage decomposition llama-2 thread scaling

Memory (RSS MB, load+encode): v0.23.1 18+0 (peak 23) · Pipeline 22+0 (peak 22)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 5.8 43.8 ×7.57 ×0.96 0% (0.0) 12% (2.7) 0% (0.0) 87% (19.1) 0% (0.0) match
arb_Arab lang 12.8 53.3 ×4.15 ×1.01 0% (0.0) 17% (3.2) 0% (0.0) 82% (15.2) 0% (0.0) match
ben_Beng lang 13.8 89.8 ×6.50 ×1.00 0% (0.0) 20% (2.1) 0% (0.0) 79% (8.3) 0% (0.0) match
cmn_Hani lang 11.6 73.2 ×6.30 ×1.04 0% (0.1) 3% (0.4) 0% (0.0) 96% (12.3) 0% (0.0) match
ell_Grek lang 13.1 60.5 ×4.64 ×0.95 0% (0.0) 20% (3.1) 0% (0.0) 80% (12.4) 0% (0.0) match
eng_Latn lang 4.7 6.6 ×1.41 ×1.02 0% (0.1) 5% (6.8) 0% (0.1) 95% (140.8) 0% (0.0) match
heb_Hebr lang 12.8 64.4 ×5.02 ×1.00 0% (0.0) 23% (3.3) 0% (0.0) 77% (11.2) 0% (0.0) match
hin_Deva lang 15.3 84.3 ×5.51 ×1.01 0% (0.0) 25% (2.8) 0% (0.0) 75% (8.3) 0% (0.0) match
jpn_Jpan lang 16.7 94.4 ×5.65 ×1.04 1% (0.1) 4% (0.4) 0% (0.0) 96% (9.5) 0% (0.0) match
kat_Geor lang 18.2 98.4 ×5.41 ×1.01 1% (0.1) 19% (1.9) 0% (0.0) 80% (7.8) 0% (0.0) match
kor_Hang lang 8.9 56.7 ×6.39 ×1.03 0% (0.1) 19% (3.3) 0% (0.0) 80% (13.5) 0% (0.0) match
rus_Cyrl lang 9.5 16.8 ×1.78 ×1.02 0% (0.0) 5% (2.8) 0% (0.0) 95% (55.4) 0% (0.0) match
tam_Taml lang 15.8 97.2 ×6.14 ×1.03 0% (0.0) 10% (1.7) 0% (0.0) 44% (7.7) 46% (8.1) match
tha_Thai lang 20.3 92.9 ×4.57 ×1.02 0% (0.0) 9% (1.0) 0% (0.0) 91% (9.2) 0% (0.0) match
added_normalized_dense modalities 6.3 8.8 ×1.39 ×0.99 0% (0.1) 3% (3.7) 0% (0.0) 96% (109.0) 0% (0.5) match
added_normalized_sparse modalities 5.3 7.6 ×1.42 ×1.00 0% (0.0) 5% (6.1) 0% (0.0) 94% (120.1) 1% (1.6) match
added_special_dense modalities 5.4 20.2 ×3.72 ×1.00 11% (5.2) 33% (15.6) 3% (1.4) 54% (25.5) 0% (0.0) match
added_special_sparse modalities 7.7 10.5 ×1.36 ×1.00 2% (2.1) 14% (13.4) 1% (0.5) 83% (81.8) 0% (0.2) match
agentic-traces modalities 5.0 7.5 ×1.50 ×1.03 0% (0.1) 5% (6.1) 0% (0.0) 95% (124.7) 0% (0.4) match
agentic_swe modalities 4.6 7.8 ×1.72 ×1.04 0% (0.0) 8% (9.6) 0% (0.0) 92% (117.0) 0% (0.3) match
code_mixed modalities 4.7 7.5 ×1.61 ×1.03 0% (0.0) 6% (7.9) 0% (0.0) 94% (122.6) 0% (0.0) match
math_latex modalities 4.9 7.1 ×1.45 ×1.03 0% (0.1) 5% (6.3) 0% (0.0) 96% (132.6) 0% (0.0) match
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×7.05 vs v0.23.1 · ×0.99 vs base llama-3 speedup llama-3 stage decomposition llama-3 thread scaling

Memory (RSS MB, load+encode): v0.23.1 143+0 (peak 199) · Pipeline 145+0 (peak 200)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.3 48.4 ×11.14 ×0.95 3% (0.7) 0% (0.0) 19% (3.6) 77% (14.8) 0% (0.0) match
arb_Arab lang 4.2 18.3 ×4.35 ×1.01 1% (0.6) 0% (0.0) 4% (2.3) 95% (50.5) 0% (0.0) match
ben_Beng lang 4.1 31.9 ×7.85 ×0.98 2% (0.6) 0% (0.0) 10% (3.1) 87% (26.2) 0% (0.1) match
cmn_Hani lang 4.7 16.7 ×3.55 ×0.97 1% (0.6) 0% (0.0) 4% (2.3) 94% (53.8) 1% (0.7) match
ell_Grek lang 4.9 19.6 ×3.96 ×0.99 1% (0.6) 0% (0.0) 4% (2.2) 95% (46.5) 0% (0.0) match
eng_Latn lang 4.1 39.9 ×9.80 ×0.99 9% (2.0) 0% (0.0) 13% (3.0) 77% (17.5) 1% (0.3) match
heb_Hebr lang 4.0 27.1 ×6.72 ×0.98 2% (0.6) 0% (0.0) 6% (2.3) 92% (32.9) 0% (0.1) match
hin_Deva lang 4.2 76.9 ×18.40 ×1.00 5% (0.6) 0% (0.0) 27% (3.2) 68% (8.0) 0% (0.0) match
jpn_Jpan lang 5.5 16.3 ×2.96 ×0.98 1% (0.6) 0% (0.0) 4% (2.2) 95% (55.8) 0% (0.0) match
kat_Geor lang 5.5 37.0 ×6.77 ×0.99 2% (0.6) 0% (0.0) 8% (2.1) 89% (23.2) 0% (0.1) match
kor_Hang lang 3.7 19.0 ×5.18 ×0.97 1% (0.6) 0% (0.0) 5% (2.5) 94% (46.6) 0% (0.0) match
rus_Cyrl lang 4.8 16.4 ×3.42 ×0.97 1% (0.6) 0% (0.0) 4% (2.1) 95% (54.7) 0% (0.3) match
tam_Taml lang 4.0 35.5 ×8.90 ×0.97 2% (0.6) 0% (0.0) 10% (2.7) 87% (23.5) 0% (0.1) match
tha_Thai lang 5.1 20.4 ×4.00 ×0.97 1% (0.6) 0% (0.0) 5% (2.5) 94% (44.0) 0% (0.0) match
added_normalized_dense modalities 6.3 27.5 ×4.38 ×0.97 2% (0.7) 0% (0.0) 3% (1.1) 95% (33.0) 0% (0.0) match
added_normalized_sparse modalities 5.7 43.5 ×7.61 ×0.99 6% (1.3) 0% (0.0) 9% (1.9) 85% (18.6) 1% (0.1) match
added_special_dense modalities 4.4 74.1 ×16.71 ×1.02 39% (4.9) 1% (0.1) 40% (5.0) 16% (2.0) 3% (0.4) match
added_special_sparse modalities 4.6 72.6 ×15.76 ×1.03 25% (3.2) 0% (0.0) 39% (4.9) 34% (4.3) 2% (0.3) match
agentic-traces modalities 3.7 33.8 ×9.19 ×0.98 7% (1.8) 0% (0.0) 14% (3.7) 79% (21.3) 1% (0.2) match
agentic_swe modalities 4.1 26.8 ×6.55 ×0.98 4% (1.3) 0% (0.0) 8% (2.9) 89% (31.2) 0% (0.0) match
code_mixed modalities 4.2 43.5 ×10.42 ×0.99 7% (1.5) 0% (0.0) 16% (3.3) 78% (16.0) 0% (0.0) match
math_latex modalities 3.7 37.0 ×10.05 ×1.01 7% (1.9) 0% (0.0) 14% (3.5) 78% (19.7) 1% (0.3) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.62 3.54 3.65 4.56 26.6 16.2 6.0 4.8 7.3× / 5.8× 4.4× / 3.5× 1.6× / 1.3× 1.3× / 1.0×
arb_Arab 1.00 2.96 2.30 4.27 30.1 19.2 7.1 5.2 13.1× / 7.1× 8.3× / 4.5× 3.1× / 1.7× 2.3× / 1.2×
ben_Beng 1.46 2.94 3.09 4.57 44.1 27.5 10.6 3.8 14.3× / 9.6× 8.9× / 6.0× 3.4× / 2.3× 1.2× / 0.8×
cmn_Hani 1.13 2.34 2.33 3.54 19.4 11.5 4.8 2.4 8.3× / 5.5× 4.9× / 3.2× 2.0× / 1.3× 1.0× / 0.7×
ell_Grek 0.58 3.09 2.19 4.70 27.6 17.1 6.2 4.8 12.6× / 5.9× 7.8× / 3.6× 2.8× / 1.3× 2.2× / 1.0×
eng_Latn 0.12 1.74 2.97 4.60 42.8 33.1 12.6 3.7 14.4× / 9.3× 11.1× / 7.2× 4.2× / 2.7× 1.3× / 0.8×
heb_Hebr 0.99 3.12 2.29 4.41 29.7 20.2 7.0 3.1 13.0× / 6.7× 8.8× / 4.6× 3.1× / 1.6× 1.3× / 0.7×
hin_Deva 1.36 3.12 3.19 4.95 46.4 30.1 11.5 4.1 14.5× / 9.4× 9.4× / 6.1× 3.6× / 2.3× 1.3× / 0.8×
jpn_Jpan 1.56 3.47 2.15 4.07 18.0 9.8 4.1 3.7 8.4× / 4.4× 4.6× / 2.4× 1.9× / 1.0× 1.7× / 0.9×
kat_Geor 1.39 2.50 2.11 3.22 16.4 11.2 4.2 2.1 7.8× / 5.1× 5.3× / 3.5× 2.0× / 1.3× 1.0× / 0.7×
kor_Hang 1.08 2.83 2.49 4.23 30.6 21.7 7.7 3.7 12.3× / 7.2× 8.7× / 5.1× 3.1× / 1.8× 1.5× / 0.9×
rus_Cyrl 1.01 3.02 2.14 4.14 26.6 16.5 6.0 2.6 12.4× / 6.4× 7.7× / 4.0× 2.8× / 1.4× 1.2× / 0.6×
tam_Taml 0.93 2.95 2.67 4.69 43.5 26.3 10.3 3.4 16.3× / 9.3× 9.8× / 5.6× 3.9× / 2.2× 1.3× / 0.7×
tha_Thai 1.36 2.60 2.50 3.73 27.3 16.2 6.9 3.0 10.9× / 7.3× 6.5× / 4.3× 2.8× / 1.8× 1.2× / 0.8×
added_normalized_dense 0.06 1.77 1.10 2.81 22.5 17.0 6.6 1.9 20.4× / 8.0× 15.5× / 6.1× 5.9× / 2.3× 1.7× / 0.7×
added_normalized_sparse 0.06 1.77 1.92 3.63 30.1 23.8 8.7 2.6 15.7× / 8.3× 12.4× / 6.6× 4.5× / 2.4× 1.4× / 0.7×
added_special_dense 0.06 1.77 4.97 6.68 91.4 73.6 21.5 3.2 18.4× / 13.7× 14.8× / 11.0× 4.3× / 3.2× 0.6× / 0.5×
added_special_sparse 0.06 1.77 4.91 6.62 58.6 45.6 15.3 3.6 11.9× / 8.8× 9.3× / 6.9× 3.1× / 2.3× 0.7× / 0.5×
agentic-traces 0.58 1.76 3.68 4.85 51.2 45.9 15.7 4.7 13.9× / 10.5× 12.5× / 9.4× 4.3× / 3.2× 1.3× / 1.0×
agentic_swe 0.51 1.73 2.86 4.08 53.4 53.7 16.3 3.4 18.7× / 13.1× 18.8× / 13.2× 5.7× / 4.0× 1.2× / 0.8×
code_mixed 0.07 1.73 3.26 4.92 51.0 51.4 15.7 4.0 15.7× / 10.4× 15.8× / 10.5× 4.8× / 3.2× 1.2× / 0.8×
math_latex 0.58 1.77 3.46 4.64 49.4 39.3 14.5 4.2 14.3× / 10.6× 11.4× / 8.5× 4.2× / 3.1× 1.2× / 0.9×
Not yet supported: t5-base
t5-base not supported

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Jul 21, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis
SBrandeis changed the base branch from main to feat/train_encode_split July 21, 2026 19:30
@SBrandeis
SBrandeis force-pushed the feat/post-processor branch from c3c90e1 to fe553ff Compare July 21, 2026 19:43
@SBrandeis
SBrandeis marked this pull request as ready for review July 21, 2026 19:58
SBrandeis
SBrandeis commented Jul 22, 2026
View reviewed changes
Comment thread tokenizers/tk-encode/src/processors/template.rs Outdated
SBrandeis
SBrandeis commented Jul 22, 2026
View reviewed changes
Comment thread tokenizers/tk-encode/src/processors/template.rs Outdated
SBrandeis
SBrandeis commented Jul 22, 2026
View reviewed changes
Comment thread tokenizers/tk-encode/src/tokenizer/pipeline.rs Outdated
@SBrandeis
SBrandeis force-pushed the feat/post-processor branch from 6f56bb1 to bf761c7 Compare July 22, 2026 07:35
SBrandeis added 2 commits July 22, 2026 09:35
@SBrandeis
Apply suggestions from code review
a200253 
Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>
@SBrandeis
fmt
8cf013a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@McPatate McPatate Awaiting requested review from McPatate
@ArthurZucker ArthurZucker Awaiting requested review from ArthurZucker

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SBrandeis @HuggingFaceDocBuilderDev