Implement unk_token, fuse_unk, byte_fallback and ignore_merges

[bjo4]

Refs #1

Stacked on #2. Both branches touch models/bpe.py. Please
merge that one first; this diff will shrink to just this PR's changes once it
lands. Happy to rebase onto main instead if you would rather review them
independently.

These four options were stored and serialized but never affected tokenization,
so any model relying on them was silently mis-tokenized.

Every expectation here was measured against huggingface/tokenizers==0.23.1,
not derived from reading its source. Two of the findings are ones I got wrong on
the first attempt, so they are worth stating explicitly.

Resolution runs before merging

This is the part that is not an implementation detail. HuggingFace resolves each
character onto vocabulary tokens first, and the tokens that resolution produces
then take part in merges:

V = {"a":0,"<unk>":1,"<0xE4>":2,"<0xB8>":3,"<0xAD>":4,"<0xE4><0xB8>":5}
m = models.BPE(vocab=V, merges=[("<0xE4>","<0xB8>")],
               byte_fallback=True, unk_token="<unk>")
m.tokenize("中")   # -> ['<0xE4><0xB8>', '<0xAD>']

Generated unks merge too: with <unk><unk> in vocab and a merge over
("<unk>","<unk>"), 'azza' gives ['a', '<unk><unk>', 'a'].

So tokenize() now splits into _resolve_symbols() followed by
_apply_merges(), and _apply_merges()'s signature changes from a string to a
sequence of already-resolved symbols. The merge loop body itself is unchanged —
the only edits inside it are the signature line and tuple(piece) →
tuple(symbols).

Behaviour implemented

Option	Behaviour
unk_token	out-of-vocabulary characters become this token. Without one they are dropped, which is what HuggingFace does; previously they reached Tokenizer._encode_single and became id -1
fuse_unk	consecutive unresolvable characters collapse into a single unk
byte_fallback	an out-of-vocabulary character becomes its UTF-8 bytes as <0xXX> tokens, uppercase and zero-padded. All-or-nothing per character: one missing byte token sends the whole character to unk rather than emitting a partial expansion
ignore_merges	a piece already present in the vocabulary is emitted whole, skipping merges

Setting unk_token to a string absent from the vocabulary stays legal at
construction and raises ModelError only when unresolvable input actually
arrives, matching HuggingFace's laziness.

Two things I want to flag rather than bury

A pending unk is not flushed by a successful byte expansion. With <0xE4>
missing, '中é' gives ['<0xC3>', '<0xA9>', '<unk>'] — the unk for 中 lands
after the bytes of é. The consistent rule is a single pending-unk slot that an
in-vocab character flushes, a byte-fallback hit does not, a second unk flushes
unless fuse_unk coalesces, and end-of-piece flushes.

This is probably an upstream quirk. I reproduced it because byte-parity with
HuggingFace is what this package is for, and deviating silently would be worse
than either choice made openly. It has a comment at the implementation site and
fixtures pinning it. If you would rather emit unks in position, say so — the
fixtures make either choice easy to lock down, but they cannot both hold.

byte_fallback is encode-only. ByteLevel.decode maps each character through
the byte table, so it renders <0xE4> as the six ASCII characters of its own
name:

tok.encode("中").tokens   # ['<0xE4>', '<0xB8>', '<0xAD>']
tok.decode(ids)           # '<0xE4><0xB8><0xAD>'   not '中'

Round-tripping needs a ByteFallback decoder, which does not exist yet. I would
rather add it in a follow-up than grow this PR; flagging it so the limitation is
not a surprise.

Known divergence

ignore_merges diverges from HuggingFace here, because this repo inserts added
tokens into model.vocab (tokenizer.py:140, :518) where HuggingFace does not.
An added token can therefore spuriously trigger the whole-piece short-circuit.
Fixing it needs a model/Tokenizer contract change, so I left it documented
rather than in scope. Tell me if you would rather it be fixed here.

Relatedly, the -1 fallback at tokenizer.py:305 becomes dead code for every
path except unk_token set but missing from the vocabulary, which now raises
ModelError. I left the line alone to keep this diff focused.

Testing

Ground truth lives in tests/fixtures/hf_bpe_flags.json — 17 cases, 91 inputs —
recorded by scripts/gen_hf_flag_fixtures.py against a pinned tokenizers.
tests/test_bpe_flags.py reads it using only the standard library, so the
suite still runs with no third-party package installed. I verified that by
hiding tokenizers via PYTHONPATH and re-running: 174 passed either way.

The generator is deterministic (same SHA-256 across runs) and is the sole
consumer of the new dev extra. dependencies = [] is untouched. A missing
fixture file is a hard failure rather than a skip, so the parity coverage cannot
silently disappear.

Cases include deliberate gate-keepers: a merge_over_byte_tokens_rank_order case
whose output only differs if resolution precedes merging; an
ignore_merges_off_control / ignore_merges_discriminating pair over a vocab
where merges cannot reach the whole word, plus a test asserting the two disagree
so the coverage cannot become vacuous; the full pending-unk slot table; and low
bytes (\t, \n, \x01, \x0f), which are the only inputs that catch
uppercase-but-not-zero-padded hex.

Full suite: 174 passed (53 before this work, 25 from PR 1, 96 here). No
existing test changed.