Pipeline-only tk-encode + tk-train on the pipeline (thin deps)#2200
Closed
ArthurZucker wants to merge 158 commits into
Closed
Pipeline-only tk-encode + tk-train on the pipeline (thin deps)#2200ArthurZucker wants to merge 158 commits into
ArthurZucker wants to merge 158 commits into
Conversation
…rst, classify first then FSM simd when possible, pure scalar for known patterns!
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
commit bd7e173 Author: Arthur <48595927+ArthurZucker@users.noreply.github.com> Date: Mon Jul 13 17:57:47 2026 +0200 pipeline-bench: show wins vs the base branch (#2192) * pipeline-bench: add "vs base branch" comparison The pipeline benchmark only compared PipelineTokenizer against the latest released crate. Add a "vs base branch" overview + per-fixture "Δ base" column so a PR's own wins/regressions against feat/train_encode_split are visible (regressions in red). Baseline is cached, not recomputed per PR: - a push to feat/train_encode_split uploads its merged bench JSON to the HF Hub dataset as baselines/pipeline-<sha>.json and force-moves the lightweight `pipeline-baseline` git tag to that commit (only after the id check passes, so a broken base never becomes the baseline); - a PR run resolves the tag, downloads the cached JSON, and diffs against it. No tag / no cached JSON yet -> the base overview is skipped and the release charts render as before. render_pipeline_bench.py: generalize overview_svg/scale over a speedups fn; new base_speedup helpers; --base-bench/--base-ref; base overview (red on regression) + Δ base table column, both gated on base data being present. * pipeline-bench: fall back to base-branch artifact when no baseline tag The "vs base branch" section never showed on the PR that introduces it: the `pipeline-baseline` tag is only created by a base-branch push running this workflow, which can't happen until the workflow is merged — so the feature was invisible during its own review (chicken-and-egg). Add a fallback: when the tag (or its HF Hub JSON) is absent, resolve the baseline from the newest successful base-branch bench artifact via `gh run download`. That artifact is produced by every base-branch run, so the comparison shows up immediately — no manual seeding, no merge required. The tag + HF Hub JSON stays the durable primary (survives the 30-day artifact retention). Needs `actions: read` to download the artifact. commit 93650e3 Author: Luc Georges <luc.sydney.georges@gmail.com> Date: Mon Jul 13 14:58:09 2026 +0200 fix: account for `FastPtrHash` no remap sparse layout Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> commit 9f9b2ec Author: Luc Georges <luc.sydney.georges@gmail.com> Date: Mon Jul 13 11:57:23 2026 +0200 feat: bump `ptr_hash` to `2.0.1` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> Merge branch 'feat/train_encode_split' into remove-char-repr-bpe
Add tk_encode pipeline::TrainingPretokenizer (normalize + atomsplit split, with the byte-level alphabet remap the encode-path TryFrom drops). Retarget TokenizerTrainExt from the legacy generic TokenizerImpl<M,N,PT,PP,D> (which drove training through the legacy Normalizer/PreTokenizer trait runtime) onto the concrete config wrappers, and feed the trainer from TrainingPretokenizer instead of pre_tokenize_for_training.
Remove the legacy inference path so tk-encode is pipeline-first:
- delete decoders/ and processors/ modules, tokenizer/encoding.rs, utils/{padding,truncation}.rs
- collapse TokenizerImpl<M,N,PT,PP,D> -> <M,N,PT>; Tokenizer keeps only loading
(from_file/from_bytes/from_pretrained/from_str), config accessors, add_tokens,
get_vocab and save; drop encode*/decode*/DecodeStream/post_process/padding/truncation
- slim AddedVocabulary to a data store: drop the daachorse matcher + regex word-boundary
logic (find_matches/split_with_indices/extract_and_normalize); keep add_tokens id
assignment, get_added_tokens_decoder, is_special_token and serde
- drop the legacy Decoder/PostProcessor traits + their component impls (ByteLevel,
Metaspace, Replace); keep the pipeline (Cow/span) impls
- serialization: emit/parse the pipeline fields only; ignore truncation/padding/
post_processor/decoder sections on load
- tk-train: target the collapsed TokenizerImpl<M,NormalizerWrapper,PreTokenizerWrapper>
Umbrella crate + python/node bindings intentionally break here (follow-up).
Remove daachorse (legacy AddedVocabulary matcher), monostate + derive_builder + itertools (decoders/processors), and compact_str (unused). regex stays for now — it is still anchored by the kept legacy Whitespace/Split pretokenizers + Replace normalizer; moving those onto SysRegex/atomsplit is a follow-up.
- fixture_bench: the pipeline correctness gate now compares against the released tokenizers crate (the in-tree Tokenizer no longer encodes) - move daachorse to dev-dependencies (buckets special-token-matcher comparison test) - drop the metaspace decode() unit test (Decoder impl removed)
…code Drop the decoders/processors re-exports (modules removed) and update the crate doc example to the PipelineTokenizer load+encode path.
Rewire the Python Tokenizer onto the thin tk-encode: encode/encode_batch return token ids via PipelineTokenizer (built from the tokenizer.json serde round-trip); keep from_file/from_str/from_buffer/from_pretrained/save, get_vocab, token_to_id, id_to_token, add_tokens/add_special_tokens, and model/normalizer/pre_tokenizer get+set. Drop the removed surface: decoders.rs, processors.rs, encoding.rs, utils/pretokenization.rs, decode/padding/truncation/post_process, and Encoding/ offsets. TokenizerImpl generic collapses to <PyModel,PyNormalizer,PyPreTokenizer>.
opt-level=z + codegen-units=1 + panic=abort + strip (inherits fat LTO). For on-device footprint testing: cargo build --profile thin. Minimal encode binary: 1.90 MB -> 1.27 MB (-33%) vs release.
…ile) Upstream ptr_hash 2.0.1 forces colored, mem_dbg (+ its mem_dbg-derive -> syn/quote/ proc-macro2 chain) and tempfile as NON-optional deps, though they only serve CLI/ debug/on-disk-sharding paths we never hit. Vendor a copy under vendor/ptr_hash with those three removed (MemSize derives stripped, colored .bold() dropped, a std TempDir shim for the unused disk-sharding path) and [patch.crates-io] to it. tk-encode normal deps 85 -> 74. MPHF build/query unchanged: vocab tests 25/25 pass.
Collaborator
Author
|
Closing in favor of #2207 — the base branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes
tk-encodepipeline-first with thin dependencies, rewirestk-trainonto the pipeline, and (new) rebuilds the python binding as a pipeline-only inference extension — now that benches compare against the releasedtokenizerscrate rather than an in-tree legacyTokenizer.tk-encodesrc −7.3k lines · prod deps 31 → 26 · python wheel −12% download.Changes
tk-traintrains through the pipeline — newtokenizer::pipeline::TrainingPretokenizer(normalize + atomsplit split + GPT-2 byte-level remap) replacespre_tokenize_for_training;TokenizerTrainExtretargeted off the legacyNormalizer/PreTokenizertrait runtime onto the concrete config wrappers.tk-encode—encode*/decode*/DecodeStream/post_process/do_tokenize,decoders/,processors/,encoding.rs,padding.rs,truncation.rs.TokenizerImpl<M,N,PT,PP,D>→<M,N,PT>.AddedVocabularyslimmed to a data store (daachorse matcher gone).Tokenizerkeeps loading + config accessors +add_tokens/get_vocab/save.daachorse(→dev),monostate,derive_builder,itertools,compact_str.tokenizers— trimmed re-exports to the pipeline-only surface (dropsdecoders/processors).bindings/python→ pipeline-only inference — PyTokenizernow wrapstk_encode::Tokenizer+PipelineTokenizer. Kept:from_file/from_str/from_buffer/from_pretrained/save,encode/encode_batch→ token ids,get_vocab/get_vocab_size/token_to_id/id_to_token/get_added_tokens_decoder,add_tokens/add_special_tokens,model/normalizer/pre_tokenizerget+set. Removeddecoders.rs/processors.rs/encoding.rs/utils/pretokenization.rs.Binary size (python extension
.so, this platform).sostripped (on disk)Green
cargo build -p tk-encode -p tk-train -p tokenizers✅ ·cargo test -p tk-train✅ (10) ·cargo test -p tk-encode --lib✅ 197 pass (1 env failure: gitignored precompiled fixture) ·cd bindings/python && cargo build --release✅ (cdylib).Known regressions / follow-ups
encoderebuilds the pipeline per call (via a tokenizer.json serde round-trip) — correct but slow; needs a mutation-invalidatedPipelineTokenizercache before this is usable for throughput.train/train_from_iteratordropped —TokenizerTrainExtneeds concrete wrappers; the Py tokenizer usesPy*wrappers. (Rust-sidetk-traintraining is unaffected.)decode, offsets/word_ids, padding, truncation, post-processing; custom Python pre-tokenizers/normalizers error (their ref-mut offset machinery was removed)..py/.pyistubs are now stale.regexnot yet dropped — still anchored by theReplacenormalizer +Split/Whitespacepretokenizers.