Atomnorm#2209
Open
ArthurZucker wants to merge 34 commits into
Open
Conversation
Ports the self-contained pure-Rust normalization port from feat/fast-normalization: nfd.rs + nfd_tables/nfkd_tables/compose_tables, wired additively into lib.rs. No change to atomsplit's classify/fsm/simd core. Differential tests vs unicode-normalization (dev-only) pass byte-exact.
Rewire the unicode normalizers off the unicode-normalization crate onto the pure-Rust atomsplit::nfd port. Byte-exact (normalizers::unicode tests pass).
Make classify/classify_scalar + the NEON kernel generic over <const CONT, MB, CJK_TAG> + a &Tables arg (no trait — a different Tables instance is a different scheme; CJK_TAG=NO_CJK compiles out the CJK lead-byte shortcut). Pretok Atom path is a thin behaviour-identical wrapper (parity/fsm stay byte-exact). Add the norm scheme: NORM_TABLES + norm_classify::classify, with a SIMD==scalar parity test.
Thread <const CONT, MB, CJK_TAG> + &Tables through the AVX-512/SSE x86_body! macro (via macro args, C/M/CJ generics to avoid the const-in-pattern collision) and the wasm SIMD128 kernel, matching the NEON generic-ization. Cross-checked: x86_64-unknown-linux-gnu + wasm32-unknown-unknown build clean; aarch64 unchanged.
…split::norm_classify Consume the landed norm_classify SIMD engine: one classify pass tags every byte with the NormClass bitmask, then copy INERT runs verbatim and touch only flagged chars (clean_text/CJK/lowercase/strip_accents). New tagged.rs helper (tag_driven); Bert fast path + Strip/StripAccents + Lowercase rewired onto it. Added a scalar norm_classify::classify_scalar wrapper for the differential test. Byte-exact vs the legacy normalizers (tag_driven_bert_matches_legacy over all flag combos + strip/lowercase differential tests). Out-of-scope Sequence metaspace-fusion from the PR was NOT ported (needs fancy-regex + diverged).
…+ fix NFD 2-byte skip panic Adds `benches/normalize.rs`: NFD/NFKD/NFC/NFKC throughput for atomsplit's `nfd` against `unicode-normalization` (Rust reference / test oracle) and, behind `--features xxutf`, the dzfrias/xxUTF C+SIMD normalizer (a feature-gated build script downloads + compiles the pinned MIT amalgamation; default builds stay C-free — no build script, no `cc`, no network). Same corpora as the regex bench; min-of-7 ns/byte + a byte-exactness gate vs unicode-normalization. The bench immediately surfaced a panic: `skip_clear_2byte` (the vld2q stable-run skip) advanced `i` by 64 per fully-clear 32-byte window — a stray `i += 32` after the `match` on top of the `None => i += 32` arm — while verifying only 32, desyncing the byte cursor from char boundaries when the unverified half held a width-≠2 char, so `&input[i..ns]` in `decompose` sliced mid-codepoint and panicked (Cyrillic/Greek NFD/NFKD). The 3-byte twin `skip_clear_3byte` is correct (single `None => i += 48`, no trailing add); dropped the extra increment to match. nfd differential tests still pass (byte-exact vs unicode-normalization).
…ose only mark-clusters `compose` recomposed the whole string once ANY composition-relevant char appeared. Instead, stream: copy every STABLE run verbatim (SIMD-skipped by `next_set::<R>`) and decompose+recompose only the ACTIVE window around each relevant char — `[starter-before-rel, next-not-in-R-char)`. A char not in R is ccc-0 & QC=Yes: it neither composes backward nor is reached by a preceding composition, so it's a safe segment boundary. Byte-exact (nfd differential tests + the normalize bench's exactness gate on 11 corpora × 4 forms). Turns NFC/NFKC on marked scripts from whole-document recompose into ~80%-copy: NFC Russian 7.9→0.58 ns/B, Hebrew 7.7→0.75, Arabic 8.3→0.85, Korean 3.8→0.91, Greek 11.6→0.58. vs xxUTF (C+SIMD), NFC now ties on Russian/Greek, wins on Chinese/Japanese/French, and is within ~1.2× on Hebrew/Arabic (dense Devanagari/ Thai marks remain ~0.4× — those and all NFD/NFKD need a SIMD decompose kernel). No change to NFD/NFKD.
…atch or beat xxUTF Three layers, all byte-exact vs unicode-normalization (exhaustive unit gates: every cp × mark suffixes × 4 forms through BOTH kernel paths, plus 11-corpus bench exactness): 1. Byte-form decomposition tables (generated, derived 1:1 from the committed (ccc,char) tries — `gen_byte_tables`): each unstable cp's UTF-8 decomposition blob + [first_ccc, last_ccc, mark_run_off] headers. The owned path emits via ONE unaligned 16-byte copy per char (arithmetic direct-store jamo for Hangul) instead of char-encode pushes; NFKC additionally bakes composed blobs for neighbour-inert compat starters (fullwidth punctuation → one copy, no window). 2. A dedicated check-tag classify scheme (`check_tag`, generated into NFD_CHECK_TABLES via bitmap_gen, which now derives it from atomsplit's own tables): per-cp tag = ccc RANK (order-preserving, 6 bits) | QC-Maybe flag (0x40) | compat-change (0x3C/0x3D) | canonical-change split by composition-relevance (0x7E stable / 0x7D relevant). Both decompose checks and the NFC/NFKC quick-check run on pure tag arithmetic — no decode, no table gathers; composed é/ά/Hangul are excluded from the compose scan entirely. 3. Strategy dispatch by a 64-byte content sample: the tag kernel for marked scripts (flat whole-buffer scan, candidates verified in-register), the bitset kernel for ASCII-dominant and CJK-block text — with fused ASCII scan+copy, a byte-compare Hangul-or-ASCII run skip (Korean NFC 13.4 → 0.19 ns/B, 3× FASTER than xxUTF), verify-as-you-go owned rebuilds (in-order marks stay in the gap memcpy), and callless small-copies. vs xxUTF (C+SIMD, "fastest open source"), 44 cells (4 forms × 11 scripts): 38 tie-or-beat (NFC/NFKC sweep almost everything at 1.2-3×; English 1.7-1.9×, Russian/Greek/Hebrew/Arabic/Hindi 1.0-1.4×, CJK NFC 1.4-3×). Remaining 0.7-0.95×: Thai/French/Korean NFD-side + NFKC Chinese — bounded by the classify pass on those scripts. vs unicode-normalization: 6-500×.
…match or beat xxUTF Byte-exact throughout (exhaustive per-cp gates through every kernel path + 11-corpus exactness), 0 warnings. Final bench vs xxUTF (C+SIMD): min 1.0×, max 3.6×, mean 1.4×; vs unicode-normalization 4-500×. The last mile: - 0x7D/0x7E tag split (composition-relevant vs composition-stable canonical-decomposers): composed é/ά/Hangul become tag-only compose skips; scan excludes 0x7E entirely (Korean NFC 13.4 → 0.19 ns/B — 3× faster than xxUTF; Greek/French NFC to 1.2-1.3×). - Baked NFKC blobs for neighbour-inert compat starters, emitted in both compose paths (NFKC Chinese 1.82 → 0.44, fullwidth punctuation is one 16-byte copy). - Hangul-or-ASCII byte-class run skip (16-byte chunks, pair-constrained lead ranges — no decode). - Streaming in-register check for single-block 3-byte scripts (Thai/Lao): the block's own fast3 check-tag table probed via vqtbl4 with vectorized break + ccc-rank-order tests, block-cached across ASCII holes, plus its verify-as-you-go owned twin (Thai NFD 0.96 → 0.47, tie) and a compose-predicate variant (Thai NFC 0.61 → 0.41, 1.5×). A scalar referee using the identical tag semantics keeps every bail byte-exact. - Write-through skips: the fused ASCII loop stores its test register and movemask-jumps to the first non-ASCII byte (French NFD/NFKD → 1.0-1.1×); the 3-byte skip vst3q's its own deinterleaved registers back and forwards the decoded SET codepoint it stops at (Japanese NFD/NFKD 0.63 → 0.45, 1.3×; lifted Chinese NFKD and Korean too). - Hangul syllable-run subloop in the emit (direct 3-byte decode, no generic continuation glue) — Korean NFD/NFKD to 1.1-1.2×.
…xUTF
Thread a `const SIMD: bool` through the drivers (dead-code eliminated — zero cost on the shipping
path) and expose `#[doc(hidden)] atomnorm::scalar::{nfd,nfkd,nfc,nfkc}`. The bench now shows all
four columns and the exactness gate covers BOTH paths; the exhaustive tests differentially verify
the scalar path too.
Notable: the scalar path alone runs 0.27-1.6 ns/B (3-20x over unicode-normalization) and even beats
the SIMD path on hit-dense scripts (Greek/Hindi) — the layered-skip design carries most of the win,
SIMD is the cherry on top for clean-run scripts (English/French/Russian 5-10x further).
…in atomnorm Restore atomsplit/ and bitmap_gen/ byte-for-byte to the PR base (feat/train_encode_split): drop norm_classify.rs, norm_tables.rs, and the classify_with genericization of the classify engine. Revert the tk-encode tagged-normalizer fast paths (bert/strip/utils/tagged) that depended on them; that work stays on feat/fast-normalization. tk-encode keeps only the atomnorm integration (unicode.rs NFD/NFKD/NFC/NFKC).
…, test the NEON path on an arm runner instead of SDE/wasm jobs atomnorm has no code for
…fused bert (2-134x) One new primitive covers all four: a layered skip-scan over per-rule property bitmaps (runtime-OR'd once per config, lead mask derived), with an in-register ASCII lane — the PR #2036 |0x20 port plus bert/nmt whitespace folds — and scalar-first streak escalation into the NEON kernel so hit-dense scripts never pay kernel round-trips. Property sets are generated from the SAME crates the legacy tk-encode normalizers use (std to_lowercase, unicode_categories, unicode-normalization-alignments), so they are bug-compatible by construction; bert's strip fixup routes unstable clusters through the existing NFD machinery (cross-char reorder + clean-removed transparency stay exact). tk-encode pipeline impls repointed to atomnorm; legacy NormalizedString paths untouched. Exhaustive per-codepoint tests (all 1.1M cps in context, both paths) + all 16 bert flag combos; 297 tk-encode tests green. Wikipedia bench (ns/B, simd vs legacy char-iterator): bert english 15.9->0.12 (134x), chinese 17.9->2.5 (7.3x), all cells 2.1-134x; nmt 0.11-0.17 on alphabetic scripts (12-17x); lowercase 1.3-48x; strip_accents 0.5-39x (mark-dense scripts stay ~legacy speed, <=2 ns/B absolute — known ceiling).
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…8-35x, byte-exact) atomnorm grows a public runtime Scanner — the parameterized skip-scan over a caller-provided set (with an ASCII-membership lane the built-ins don't need). tk-encode wraps spm_precompiled (serde-transparent) and derives at model load: scan-hot = single-char keys ∪ multi-char-key tails (own DoubleArray DFS over the charsmap blob), cluster-class = probed from the same unicode_segmentation that runs the exact walk. Scan skips cold runs; a hit walks back over class chars to a provable grapheme boundary and runs the exact walk locally. The reachability invariant (every multi-char key tail is cluster-class) is verified per charsmap — violation falls back to the plain walk. albert nmt_nfkc on Wikipedia corpora: english 16.6->0.64 ns/B (26x), russian 12.3->0.35 (35x), korean 10.7->1.01 (10.6x), chinese 8.2->2.9 (2.8x, genuine fullwidth remapping). spm_precompiled stays the exact engine.
…-35x, bert 5-25x on cased scripts) Source table = SCAN_REG2 (88 cps: 2-byte uppercase mapping to exactly cp+0x20 — Latin-1/Greek/Cyrillic capitals), masked per config against every other enabled rule (under bert strip, decomposing capitals like Й stay scalar fixups). Target is arithmetic: continuation +0x20 with the UTF-8 carry (-0x40, lead +1) so Р D0A0 -> р D180 in-register. New write-only scan2_case NEON kernel (skip2-style dual 2048-bit probes: SET stops, REG2 transforms; trailing leads deferred so pairs never straddle chunks) + the scalar twin; next_hit escalates into it when suspect-2-byte probes dominate, clean terrain keeps the light 32B kernel. Also fixes lead-mask over-coverage: lead E0 valid cps start at 0x800, so Latin-1 capitals no longer taint the whole Devanagari/Thai range (lower hindi 2.0->0.07). lower ns/B vs legacy: russian 0.43 (8.8x, was 1.9x), greek 0.41 (9.0x), hindi 0.071 (31x), thai 0.065 (29x); bert: russian 0.89 (14.1x, was 5.5x), hebrew 0.49 (24.8x), arabic 1.22 (9.7x). All exhaustive per-cp suites green, both paths.
…er (forms, lower, strip, nmt, bert, spm-precompiled) vs its legacy implementation on the Wikipedia corpora
… normalizer benchmark
…ed walk simd_norm: delete the dead vld2q experiment in skip3 (if false block), drop the per-body cfg(aarch64) (the module is already gated in lib.rs), and factor the shared pieces — first_lane (POW movemask), class2, load2048/probe2048 (2048-bit in-register probe), upper_mask, ascii_policy (the bert/nmt fold+removal masks, previously duplicated verbatim in scan_prefix and scan2_case). skip2_ascii stays deliberately hand-inlined: the helper-factored variant (array-of-tables) measured +25% on NFD Russian — with one table the four vqtbl4 groups must stay individual SSA values; noted in a comment. precompiled: factor transform_grapheme — the one copy of the exact-walk step both normalize_scan and normalize_walk share (was duplicated with subtle prefix-materialization logic). Byte-exactness suites green (both paths), full benchmark unchanged within noise (NFD Russian 0.33, bert English 0.12, spm Russian 0.35).
…ncode README The Precompiled wrapper dropped spm_precompiled's inherent `from(&[u8])` constructor, breaking the Python/Node bindings (and every wheel job) through the tokenizers crate's re-export — restored as a delegation. The README drift predates this PR: cargo-readme strips `# `-hidden doctest lines, so README.md must not carry them.
…C and the feature only feeds the bench comparator (windows clippy runs --all-features)
Drop the ~160 lines of reimplemented legacy normalizers: the baseline is now the real released crate (tokenizers-release, bench-baseline feature) driven through its NormalizedString path — what users actually ran. All 99 cells byte-exact. Highlights vs 0.23.1: bert 41-64 ns/B -> 0.12-6.5 (10-345x), forms up to 488x, spm 4-60x; worst cell on the board is strip hindi at 3.3x. run: cargo bench -p tk-encode --bench normalize --features bench-baseline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PipelineTokenizer benchmark
7 / 8 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) · ~10 kB inputs · single thread + 1/2/4/8/max-thread sweep141e001f2 · 2026-07-16 12:13 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 32 coresvs base branch (
2c4f6100b) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.bert-base-uncased — normalizer-heavy WordPiece · ×15.76 vs v0.23.1 · ×2.81 vs base
Memory (RSS MB, load+encode): v0.23.1 2+0 (peak 6) · Pipeline 7+0 (peak 7)
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×5.19 vs v0.23.1 · ×1.01 vs base
Memory (RSS MB, load+encode): v0.23.1 52+0 (peak 58) · Pipeline 85+0 (peak 85)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt2 — gpt2 ByteLevel regex · ×9.80 vs v0.23.1 · ×1.00 vs base
Memory (RSS MB, load+encode): v0.23.1 17+1 (peak 19) · Pipeline 20+0 (peak 20)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×7.85 vs v0.23.1 · ×0.99 vs base
Memory (RSS MB, load+encode): v0.23.1 2+4 (peak 6) · Pipeline 3+0 (peak 6)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×13.12 vs v0.23.1 · ×0.99 vs base
Memory (RSS MB, load+encode): v0.23.1 2+4 (peak 6) · Pipeline 3+0 (peak 6)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).llama-2 — model-bounded BPE, no pre-tokenizer · ×4.12 vs v0.23.1 · ×1.02 vs base
Memory (RSS MB, load+encode): v0.23.1 15+0 (peak 15) · Pipeline 14+0 (peak 14)
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×9.09 vs v0.23.1 · ×1.02 vs base
Memory (RSS MB, load+encode): v0.23.1 129+0 (peak 189) · Pipeline 130+0 (peak 189)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).Not yet supported:
t5-base