fix(tokenizers): add qwen2 + qwen35 pre-tokenizer pipelines; complete the #373/#387 pre-tokenizer work

[jamesburton]

Closes #397
Closes #373
Closes #387

What was already on dev, and what this adds

Two thirds of the umbrella work had already landed on dev and simply never closed their issues (the merged PR targeted dev, not the default branch):

Issue	Status before this PR
#373 loud gate + pixtral	Implemented in 8abf2281 (already on dev)
#387 tekken	Implemented in cc0ee1e0 (already on dev)
#397 qwen2 / qwen35	Absent — this PR

So #387's claim that Nemotron-Nano-9B-v2 was "unloadable since #373's loud gate" was a branch-local truth, not a false report. Nothing existing was rewritten; this builds on top of both commits and completes their remaining acceptance criteria (the docs table, and #373's explicit negative control).

The fix

qwen2 and qwen35 were missing from TiktokenPreTokenizer. Since the loud gate landed, every Qwen2/Qwen3 GGUF throws at load; before it, they ran with no pre-tokenization at all, so BPE merges crossed the digit boundaries the model was trained to respect. #237 measured tokenizer drift alone at +3.46% PPL, which is why this gates any quality measurement on Qwen3.8-27B or the Bonsai family.

Both expressions are the original tokenizer.json patterns that llama.cpp quotes verbatim in the comments above its own copies (src/llama-vocab.cpp, LLAMA_VOCAB_PRE_TYPE_QWEN2 / _QWEN35, fetched from master — the local checkout predates qwen35). llama.cpp spells the contractions out as '[sS]|'[tT]|… only because std::regex has no (?i:…) group; .NET supports the original form directly, matching what the existing Llama3Pipeline already does.

• qwen2 — the llama3 expression with a bare \p{N} (one digit per segment) instead of \p{N}{1,3}.
  Aliases llama.cpp routes to LLAMA_VOCAB_PRE_TYPE_QWEN2: deepseek-r1-qwen, kormo, f2llmv2, megrez.
  Also mapped: stablelm2, hunyuan, solar-open — distinct enum values that share the same regex_exprs block by case fall-through, exactly as the file already does for the StarCoder eight.
• qwen35 — qwen2 with combining marks folded into the letter run ([\p{L}\p{M}]+) and \p{M} excluded from the punctuation class.
• minerva-7b — llama.cpp's actual spelling of the value the table listed only as minerva. minerva is kept for compatibility.

What I verified

Discriminating unit tests (PreTokenizerPolicyTests, synthetic 256-byte vocab + one merge, so a single merge fires or is blocked purely by segment placement):

• QwenPipelines_SplitEveryDigit_UnlikeGpt2AndLlama3 — "34" with a 3 4 merge is 2 tokens under every Qwen alias, 1 under gpt2 ( ?\p{N}+) and 1 under llama3 (\p{N}{1,3}). Fails if either is mis-routed.
• Qwen35_KeepsCombiningMarksInTheLetterRun_UnlikeQwen2Gpt2AndLlama3 — the load-bearing one, since the digit test cannot separate qwen35 from qwen2. A decomposed e + U+0301 is one segment under qwen35 (2 tokens via the byte-level e+0xCC merge) and two under qwen2/llama3/gpt2 (3 tokens). Literals are built from code points so an editor NFC-composing the file cannot defeat it.
• NegativeControl_TheOldSilentNoPreTokenizationPath_LoadsQuietlyAndMisTokenizes — Unknown tokenizer.ggml.pre silently disables pre-tokenization entirely (e.g. pixtral) #373's explicit negative control. The pre-fix behaviour (unknown pre → null regex → no pre-tokenization) is reconstructed via CreateTiktokenWithRegex(..., preRegex: null) and shown to load without complaint and produce 1 token where qwen2 gives 2, 3 where llama3 gives 4.

Real-GGUF token-id parity against llama.cpp — both pre values covered, neither skipped
(QwenPreTokenizerParityTests, Integration; fixtures resolved through KnownTestFixtures, no weights in the repo):

pre	Fixture	Result
qwen2	Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q8_0.gguf	exact id match, 34 ids
qwen35	prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf	exact id match, 32 ids

Reference ids from the official ggml-org/llama.cpp release b10434 Windows CPU build: llama-tokenize -m <gguf> -f <fixture> --ids --no-bos.

Correction (was wrong in the first revision of this PR). This section previously claimed the fixture "exercises every alternative the pipelines disagree on". That was false as measured, and review caught it. On these two vocabularies the original fixture was byte-identical under qwen2, qwen35, llama3 and tekken — only a fall-back to gpt2 was detectable, and only via the trailing \n\tdone. The parity test would therefore have passed unchanged with qwen35 routed to qwen2. See "Discrimination" below for the fix.

Caveat, stated plainly: upstream llama.cpp cannot open the Bonsai Q2_0 tensors (non-upstream quantization type 42). Its qwen35 ids were therefore produced from a tensor-less copy of that file's header + KV section — identical vocabulary, merges and tokenizer.ggml.pre, tensor_count patched to 0, no tensor data. That is a faithful vocab source, but it is not a full-model load; the scratch file is not checked in and the test's XML doc records how to regenerate it. Everything else about the parity run is a normal end-to-end GgufFile.Open → GgufBpeTokenizerFactory.Load → Encode on the real GGUF.

Test runs (Release, --no-build): PreTokenizerPolicyTests 18 passed / 0 failed; QwenPreTokenizerParityTests 2 passed / 0 skipped. (Full DotLLM.Tests.Unit.Tokenizers namespace was green at 369 passed / 12 pre-existing HF-fixture skips before the two tests added in review.)

Discrimination — added after review, proved by mutation

Parity and discrimination are two different properties; the first revision only had the first. The suite now asserts both.

Amended fixture (reference ids regenerated from llama.cpp b10434, not from dotLLM — this stays a parity test):

"It's １０ ที่ 2026: cafe\u0301 1234.56 — nai\u0308ve/OK\n\tdone"

Two measured vocabulary properties made the original fixture blind, and the two added characters defeat each:

Blindness	Cause (measured)	Discriminator added
\p{N} vs \p{N}{1,3} invisible → qwen2/qwen35 ≡ llama3	Neither vocab contains any ASCII multi-digit token, so no amount of ASCII digits can expose it	Fullwidth １０ (U+FF11 U+FF10) — the only multi-digit tokens in either vocab
[\p{L}\p{M}]+ vs \p{L}+ invisible → qwen35 ≡ qwen2	Latin decomposed marks never merge with letter bytes here (café, naïve, Zürich all identical)	Thai ที่ (U+0E17 U+0E35 U+0E48) — base + vowel sign + tone mark, which does merge

New AssertDiscriminates re-tokenizes the same real vocabulary and merge table under a forced tokenizer.ggml.pre and requires every wrong pipeline — gpt2, llama3, starcoder, gpt-4o, and the other Qwen — to produce a different id stream. Its failure message explains how the discriminators were chosen, so a future fixture edit that reintroduces blindness fails loudly instead of silently.

Mutation-tested, not assumed. With the two discriminators removed (i.e. the original fixture) and the parity asserts suppressed so property 2 is what reports, the new assertion fires:

Fixture cannot discriminate 'qwen35' from 'llama3' on this vocabulary — the parity assertion
would pass even if the pre value were routed to the wrong pipeline. ...
Fixture cannot discriminate 'qwen2' from 'llama3' on this vocabulary — ...
Failed! - Failed: 2, Passed: 0

Restored to the amended fixture: 2 passed, 0 skipped. So the assertion is load-bearing rather than vacuously true.

qwen35 vs tekken (review MEDIUM). These agree exactly on both real vocabularies even with the amended fixture, because neither contains a merge crossing an upper/lower-case boundary — the only place their expressions differ. Asserting it at integration level would be vacuous, so per the review's own guidance it is proved at unit level instead, on a vocabulary built to contain such a merge: Qwen35_KeepsCaseBoundariesInsideTheLetterRun_UnlikeTekken — "aB" with an a B merge is 1 token under qwen35's [\p{L}\p{M}]+ and 2 under tekken's case-split pair. tekken is explicitly excluded from the integration WrongPipelines list with a comment saying why.

Route taken: the strong one — amended fixture with regenerated llama.cpp reference ids, not the fall-back of freezing the old parity test and bolting on separate unit tests. The b10434 release binary was already fetched during the first pass, so no llama.cpp build was needed. (The qwen35 reference still comes from the tensor-less header+KV copy of the Bonsai GGUF, since upstream cannot parse quant type 42 — unchanged, and noted above.)

Note on stablelm2 (review LOW, not fixed by design). It matches qwen2 at regex level, but llama.cpp gives it clean_spaces = true where qwen2/hunyuan/solar-open get false. dotLLM models no clean_spaces concept at all, so there is no divergence today; it is a pre-existing decode-side gap, not something this PR introduces.

Audit — pre values llama.cpp knows that dotLLM still does not

Requested by #397. All of these currently hit the loud gate. The first group is nearly free — each maps onto a pipeline this file already has:

• → existing GPT-2 pipeline: gpt-2, phi-2, mpt, olmo, jais, trillion, granite-docling, jina-es, jina-de, jina-v2-es, jina-v2-de, gigachat, a.x-4.0, mellum, modern-bert
• → existing Llama-3 pipeline: dbrx, smaug-bpe, glm4, chatglm-bpe
• → existing GPT-4o pipeline: kanana2, talkie, minimax-m2

The rest need new expressions: deepseek-v3, youtu, falcon, jais-2, poro-chat, bloom, gpt3-finnish, viking, chameleon, superbpe, seed-coder, hunyuan-dense, joyai-llm, kimi-k2, grok-2, afmoe, laguna, exaone4, exaone-moe, bailingmoe, bailingmoe2, llada-moe, sarvam-moe, tiny_aya, cohere2moe, granite-embed-multi-97m, granite-embed-multi-311m, minicpm5, jina-v1-en, jina-v2-code, roberta-bpe, whitespace. (gemma4 is handled by its own model path, not this table.)

I did not add these — out of scope for #397, and each wants its own verification rather than a bulk paste.

Docs

docs/TOKENIZERS.md now tabulates every supported pre value per pipeline, including tekken, which #387 shipped but never documented. The unknown-value policy and the DOTLLM_ALLOW_UNKNOWN_PRETOKENIZER=1 escape hatch were already documented by #373 and are unchanged.

[Copilot]

Pull request overview

Adds missing Qwen2/Qwen3 (qwen2) and Qwen3.5 (qwen35) tiktoken pre-tokenizer regex pipelines to align GGUF tokenization with llama.cpp, completing the remaining “loud gate + pre-tokenizer coverage” work for the tokenizer.ggml.pre table.

Changes:

• Add qwen2 and qwen35 regex pipelines (plus routing aliases) to TiktokenPreTokenizer.
• Add discriminating unit tests for Qwen routing/behavior and integration parity tests against llama.cpp token IDs on real GGUF fixtures.
• Document supported tokenizer.ggml.pre values per pipeline in docs/TOKENIZERS.md, and add fixture resolvers for the new integration tests.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/DotLLM.Tests.Unit/Tokenizers/PreTokenizerPolicyTests.cs	Adds discriminating unit tests for Qwen2/Qwen35 segmentation differences and a negative control for the old silent no-pre-tokenization path.
tests/DotLLM.Tests.Integration/Tokenizers/QwenPreTokenizerParityTests.cs	Adds real-GGUF integration tests asserting token-id parity with llama.cpp for qwen2 and qwen35.
tests/DotLLM.Tests.Integration/Fixtures/KnownTestFixtures.cs	Adds named fixture resolvers for the Qwen2.5 and Bonsai GGUFs used by parity tests.
src/DotLLM.Tokenizers/Bpe/TiktokenPreTokenizer.cs	Implements Qwen2 and Qwen3.5 pre-tokenizer regex pipelines and adds routing for relevant aliases.
docs/TOKENIZERS.md	Updates documentation to tabulate supported tokenizer.ggml.pre values per pipeline, including Qwen and tekken.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jamesburton]

CI triage: the Integration Tests failure is #392, not this PR

Check	Result
Build (Debug)	SUCCESS
Build (Release)	SUCCESS
Unit Tests	SUCCESS
Integration Tests	FAILURE — pre-existing, unrelated

All three failures are Xunit.SkipException being counted as a failure:

Failed RealHfSafetensorsEndToEndTests.Mistral7B_LoadsAndForwardsEndToEnd_WhenCheckpointPresent
  Xunit.SkipException : Mistral-7B checkpoint not found. Set DOTLLM_MISTRAL_7B_CHECKPOINT_PATH ...
Failed RealHfSafetensorsEndToEndTests.Qwen15MoeA27B_LoadsAndForwardsEndToEnd_WhenCheckpointPresent
Failed RealHfSafetensorsEndToEndTests.Mixtral8x7B_LoadsAndForwardsEndToEnd_WhenCheckpointPresent

Failed! - Failed: 3, Passed: 145, Skipped: 314, Total: 462

That is exactly #392 — "Three RealHfSafetensorsEndToEndTests methods use [Fact] instead of [SkippableFact] — SkipException counted as FAIL" — same three methods, same mechanism. The runner has no gated safetensors checkpoints, so the tests correctly want to skip and the wrong attribute turns each skip into a failure.

Nothing in this PR touches safetensors loading; the diff is TiktokenPreTokenizer.cs, two test files, a fixtures registry entry, and docs/TOKENIZERS.md.

Consequence worth acting on

#392 makes the Integration Tests job red on every PR, which destroys its value as a merge gate — a genuinely broken integration test would be indistinguishable from this noise. That is the same "silent skip / false signal" failure class this repo keeps finding in its verification apparatus, and it is a two-line fix. Prioritising it.

This PR is otherwise green and MERGEABLE.