Add exact batched all-pairs Levenshtein distances

[ESultanik]

What this adds

A new module, graphtage/batch_distance.py, that computes exact Levenshtein distances for a whole
collection of string pairs at once, behind a registry of interchangeable backends.

all_pairs(from_strings, to_strings, *, backend=None) -> np.ndarray   # 2-D cross product
all_flat(a, b, *, backend=None) -> np.ndarray                        # 1-D, position by position
available_backends() -> tuple[str, ...]                              # fastest first

Nothing in Graphtage calls this module. It is a pure addition, and reviewing it needs no context
beyond the file itself. Wiring it into the matching engine is a separate change.

Two backends ship:

• python wraps the existing graphtage.levenshtein.levenshtein_distance. It is the oracle the
  other backends are checked against and the fallback for batches below any threshold.
• numpy is the default above 32 pairs. numpy>=1.26 is already a dependency, so this adds none.

GRAPHTAGE_BATCH_BACKEND pins the choice, which is what makes the benchmark below and the parity
tests possible. An explicit backend= argument wins over the environment variable, which wins over
the automatic choice.

How the numpy backend works

The horizontal dependency cur[j] = min(base[j], cur[j - 1] + 1) is what normally stops a row of
the Levenshtein matrix from being vectorized. Subtracting j from both sides turns it into a
running minimum of base[j] - j, which np.minimum.accumulate computes in one pass. A batch then
costs as many array passes as its strings are long, rather than as many interpreter steps as it has
matrix cells. Each pair is read out in the row where i reaches the length of its first string, at
the column given by the length of its second, so pairs of different lengths share one batch.

Three things happen before the kernel sees a batch:

1. Each pair is oriented shorter string first, which is free because the metric is symmetric, and
   gives the fewest and widest array passes.
2. str and bytes pairs are separated, and a pair that mixes the two is rejected rather than
   answered. ord('a') and b'a'[0] are both 97 while 'a' == b'a' is False, so there is no
   answer that satisfies both readings. str encodes to one uint32 per code point and bytes to
   one uint16 per byte, so the two cannot be concatenated without an explicit cast.
3. Pairs are sorted by length and split into chunks that fit a working set budget. Without this, one
   long string among short ones pads every row of the batch to its length.

Pairs are deduplicated on both sides and indexed back through two dictionaries, and pairs that are
equal or have an empty side are answered before the kernel.

Measured throughput

Apple M3 Max, macOS 26.6.2, Python 3.14.0, numpy 2.5.3. Random lowercase strings of 15 to 30
characters, timed through all_pairs end to end, including deduplication, bucketing and encoding.

pairs	DP cells	numpy	python	speedup
57,600	29.3 M	0.161 s (357,000 pairs/s)	2.39 s (24,100 pairs/s)	14.8x
160,000	83.2 M	0.451 s (355,000 pairs/s)	6.76 s (23,700 pairs/s)	15.0x

Where the two backends meet depends on string length: numpy overtakes python at about 9 pairs of
8 to 25 character strings, but not until about 50 pairs of 2 to 5 character ones. The threshold is
set at 32 pairs, which covers the shorter case; either backend answers a batch that small in well
under a millisecond.

The chunk budget matters more than it looks. At 8 MiB the rows stop fitting in cache and throughput
drops by about a third, so it is set to 256 KiB.

What the tests pin

test/test_batch_distance.py asserts these properties:

• Every backend returns exactly what levenshtein_distance returns, over a corpus of empty strings,
  single characters, repeated strings, a 200 character string and a near copy of it, non-ASCII text
  with a combining mark, and characters outside the Basic Multilingual Plane.
• A character outside the Basic Multilingual Plane costs one edit, matching Python's own iteration
  by code point and the per-character lattice the engine already builds.
• Each pair is read out at its own lengths, not one column early and not at the padded width of its
  chunk.
• One 4 KB string among 5 character ones stays bucketed. The bound is relative to the python
  backend measured in the same run, so it reports a regression without depending on machine speed.
• all_pairs([], []), all_pairs(['a'], []) and all_flat([], []) keep their zero-length axes.
• Every backend, forced through GRAPHTAGE_BATCH_BACKEND, gives the same answer, and a batch that
  would go to numpy goes to python when the environment says so.
• A str cannot be compared to a bytes, and a call holding both kinds gives each the answer it
  would get on its own.
• Deduplication maps every repeated string and every repeated pair back to all of its positions.

Each test names the specific failure it prevents, and each was confirmed to fail against a
deliberately broken module before being kept. Fifteen mutations were tried and all fifteen are
caught, including the silent ones: a substitution cost of 2, an off-by-one in the readout column,
reading at the chunk's padded width, UTF-16 instead of UTF-32 encoding, no bucketing, no
shorter-string-first orientation, no str against bytes check, a misplaced scatter out of a
kind's sub-batch, an empty string short-circuiting to 0, either deduplication index dropped, the
environment variable ignored, and a zero-length axis squeezed out of the result.

Checks

• pytest -q: 211 passed, 34 subtests passed.
• ruff check graphtage test docs bindist: clean.
• make -C docs html SPHINXOPTS="-W --keep-going": clean. batch_distance is in the
  from . import ... line in graphtage/__init__.py, so docs/build_api.py picks it up. The
  generated .rst files are covered by docs/.gitignore, so there is nothing to commit for them.
• The suite also passes on Python 3.10.20 with numpy 2.2.6, the floor in the CI matrix.

🤖 Generated with Claude Code