Pipeline-only tk-encode + tk-train on the pipeline (thin deps) — replaces #2200#2207
Draft
ArthurZucker wants to merge 8 commits into
Draft
Pipeline-only tk-encode + tk-train on the pipeline (thin deps) — replaces #2200#2207ArthurZucker wants to merge 8 commits into
ArthurZucker wants to merge 8 commits into
Conversation
opt-level=z + codegen-units=1 + panic=abort + strip (inherits fat LTO). For on-device footprint testing: cargo build --profile thin. Minimal encode binary: 1.90 MB -> 1.27 MB (-33%) vs release.
…ile) Upstream ptr_hash 2.0.1 forces colored, mem_dbg (+ its mem_dbg-derive -> syn/quote/ proc-macro2 chain) and tempfile as NON-optional deps, though they only serve CLI/ debug/on-disk-sharding paths we never hit. Vendor a copy under vendor/ptr_hash with those three removed (MemSize derives stripped, colored .bold() dropped, a std TempDir shim for the unused disk-sharding path) and [patch.crates-io] to it. tk-encode normal deps 85 -> 74. MPHF build/query unchanged: vocab tests 25/25 pass.
Add tk_encode pipeline::TrainingPretokenizer (normalize + atomsplit split, with the byte-level alphabet remap the encode-path TryFrom drops). Retarget TokenizerTrainExt from the legacy generic TokenizerImpl<M,N,PT,PP,D> (which drove training through the legacy Normalizer/PreTokenizer trait runtime) onto the concrete config wrappers, and feed the trainer from TrainingPretokenizer instead of pre_tokenize_for_training.
Remove the legacy inference path so tk-encode is pipeline-first:
- delete decoders/ and processors/ modules, tokenizer/encoding.rs, utils/{padding,truncation}.rs
- collapse TokenizerImpl<M,N,PT,PP,D> -> <M,N,PT>; Tokenizer keeps only loading
(from_file/from_bytes/from_pretrained/from_str), config accessors, add_tokens,
get_vocab and save; drop encode*/decode*/DecodeStream/post_process/padding/truncation
- slim AddedVocabulary to a data store: drop the daachorse matcher + regex word-boundary
logic (find_matches/split_with_indices/extract_and_normalize); keep add_tokens id
assignment, get_added_tokens_decoder, is_special_token and serde
- drop the legacy Decoder/PostProcessor traits + their component impls (ByteLevel,
Metaspace, Replace); keep the pipeline (Cow/span) impls
- serialization: emit/parse the pipeline fields only; ignore truncation/padding/
post_processor/decoder sections on load
- tk-train: target the collapsed TokenizerImpl<M,NormalizerWrapper,PreTokenizerWrapper>
Umbrella crate + python/node bindings intentionally break here (follow-up).
Remove daachorse (legacy AddedVocabulary matcher), monostate + derive_builder + itertools (decoders/processors), and compact_str (unused). regex stays for now — it is still anchored by the kept legacy Whitespace/Split pretokenizers + Replace normalizer; moving those onto SysRegex/atomsplit is a follow-up.
- fixture_bench: the pipeline correctness gate now compares against the released tokenizers crate (the in-tree Tokenizer no longer encodes) - move daachorse to dev-dependencies (buckets special-token-matcher comparison test) - drop the metaspace decode() unit test (Decoder impl removed)
…code Drop the decoders/processors re-exports (modules removed) and update the crate doc example to the PipelineTokenizer load+encode path.
Rewire the Python Tokenizer onto the thin tk-encode: encode/encode_batch return token ids via PipelineTokenizer (built from the tokenizer.json serde round-trip); keep from_file/from_str/from_buffer/from_pretrained/save, get_vocab, token_to_id, id_to_token, add_tokens/add_special_tokens, and model/normalizer/pre_tokenizer get+set. Drop the removed surface: decoders.rs, processors.rs, encoding.rs, utils/pretokenization.rs, decode/padding/truncation/post_process, and Encoding/ offsets. TokenizerImpl generic collapses to <PyModel,PyNormalizer,PyPreTokenizer>.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pipeline-only, thin-deps
tk-encode+ a pipeline-basedtk-train+ a pipeline-only python binding. This is the re-port of #2200 ontofeat/train_encode_split— #2200's original base (remove-char-repr-bpe) was merged & deleted, so that PR retargeted into an unmergeable 100-commit state; this replaces it, re-authored cleanly on the current base (only one conflict the whole way).Changes
tk-traintrains through the pipeline —tokenizer::pipeline::TrainingPretokenizer(normalize + atomsplit split + GPT-2 byte-level remap);TokenizerTrainExtretargeted off the legacyNormalizer/PreTokenizertrait runtime onto the concrete config wrappers.tk-encode— theencode*/decode*/DecodeStream/post_processengine,decoders/,processors/,tokenizer/encoding.rs,utils/{padding,truncation}.rs.TokenizerImpl<M,N,PT,PP,D>→<M,N,PT>.AddedVocabularyslimmed to a data store (daachorse matcher gone).Tokenizerkeeps loading + config accessors +add_tokens/get_vocab/save. Models (incl. no alloc model, follow-ups #2175's scratchtokenize_pipeline+ feat: other data structure for WordPiece (yada) #2183 yada) and normalizers (pipelineCowpath) are kept.tokenizers— re-exports trimmed to the pipeline-only surface.bindings/python→ pipeline-only inference — PyTokenizerwrapstk_encode::Tokenizer+PipelineTokenizer; keepsfrom_file/from_str/from_pretrained/save,encode/encode_batch→ ids,get_vocab/token_to_id/id_to_token/add_tokens, component get/set. Removeddecoders.rs/processors.rs/encoding.rs/utils/pretokenization.rs.ptr_hash—tokenizers/vendor/ptr_hashwith the non-optionalcolored/mem_dbg/tempfiledeps stripped,[patch.crates-io]-ed. MPHF byte-exact (vocab tests pass).thinrelease profile —opt-level=z+codegen-units=1+panic=abort+ strip, for on-device footprint testing (cargo build --profile thin).Deps dropped
daachorse,monostate,derive_builder,compact_str, pluscolored/mem_dbg/tempfile(via the vendoredptr_hash).Green
cargo build -p tk-encode -p tk-train -p tokenizers✅ ·cargo test -p tk-train✅ 10/10 ·cargo test -p tk-encode --lib197 pass (1 env failure: gitignored precompiled fixture) ·bindings/pythoncdylib builds (5.68 MB stripped).Intentionally deferred (follow-up)
regex/unicode-*/spm/yadadeps stay — they're anchored by the kept normalizers/models, not the removed encode path (dropping them needs the pipeline to grow the fullEncoding/offset contract first).Replaces #2200.