Skip to content

Pull requests: huggingface/tokenizers

New pull request New
132 Open 922 Closed
132 Open 922 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

feat(pipeline) plug post-processors
#2227 opened Jul 21, 2026 by SBrandeis Contributor Loading…
1
chore(deps): bump shell-quote from 1.8.1 to 1.10.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2224 opened Jul 20, 2026 by dependabot Bot Loading…
1
feat(bpe): ScratchPool pattern + BPE word cache
#2223 opened Jul 20, 2026 by SBrandeis Contributor • Draft
10
chore(deps): bump brace-expansion from 1.1.13 to 1.1.16 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2222 opened Jul 20, 2026 by dependabot Bot Loading…
1
bench(pipeline): add sebpop/upstream as a second reference series
#2221 opened Jul 20, 2026 by SBrandeis Contributor • Draft
1
Fix malformed BPE merges panicking during build
#2219 opened Jul 19, 2026 by 0xzr Loading…
1
Preserve backslashes in generated Python stub docstrings
#2218 opened Jul 18, 2026 by cupkk Loading…
1
5
Add a ParityBpeTrainer example and list it in the trainers API docs
#2217 opened Jul 18, 2026 by cimeister Contributor Loading…
Fix XSS in EncodingVisualizer HTML attributes
#2216 opened Jul 18, 2026 by varneeth01 Loading…
1
1
Fix dead wikitext-103 S3 link in quicktour.mdx
#2215 opened Jul 18, 2026 by RudrenduPaul Loading…
1
Fix token_to_sequence returning Some(0) for out-of-bounds index
#2214 opened Jul 17, 2026 by Osamaali313 Loading…
1
feat: multi threaded pipeline
#2213 opened Jul 17, 2026 by McPatate Member • Draft
1
Fix Metaspace decoder dropping internal spaces in the first token
#2212 opened Jul 17, 2026 by sohumt123 Loading…
Add missing return type hints to BaseTokenizer.save, save_model, to_str
#2211 opened Jul 17, 2026 by RudrenduPaul Loading…
Atomnorm
#2209 opened Jul 16, 2026 by ArthurZucker Collaborator Loading…
1
chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2208 opened Jul 15, 2026 by dependabot Bot Loading…
1
Pipeline-only tk-encode + tk-train on the pipeline (thin deps) — replaces #2200
#2207 opened Jul 15, 2026 by ArthurZucker Collaborator • Draft
fix: skip pre-tokenizer when is_pretokenized=True
#2204 opened Jul 15, 2026 by pctablet505 • Draft
1
Upgrade hf-hub to 1.0.0
#2203 opened Jul 15, 2026 by assafvayner • Draft
1
perf(atomsplit): fuse o200k letter case-split + 32-wide ASCII classify
#2201 opened Jul 15, 2026 by ArthurZucker Collaborator Loading…
1
perf: reduce BPE tokenizer load-time memory
#2199 opened Jul 15, 2026 by xrl1 Loading…
perf: reduce BPE tokenizer load-time memory
#2197 opened Jul 14, 2026 by xrl1 Loading…
2
Upgrade hf-hub to 1.0
#2196 opened Jul 14, 2026 by assafvayner Loading…
1
Clarify byte-level pretokenizer behavior
#2195 opened Jul 14, 2026 by michaelalbada Loading…
2
(wip) faster bpe run-pipeline-bench
#2194 opened Jul 13, 2026 by SBrandeis Contributor • Draft
ProTip! Filter pull requests by the default branch with base:main.