Skip to content

Pin the Levenshtein edit script output contract - #196

Merged
ESultanik merged 4 commits into
masterfrom
pin-levenshtein-edit-script-contract
Sep 15, 2026
Merged

ESultanik merged 4 commits into
masterfrom
pin-levenshtein-edit-script-contract

Conversation

@ESultanik

Copy link
Copy Markdown
Collaborator

Test-only, except for one comment. No behavior changes.

An upcoming change makes StringEdit carry an exact integer cost and builds the per-character
EditDistance lattice lazily, only for the edits that get rendered. That should preserve output
exactly, because the edit script still comes from the same unchanged EditDistance. This PR turns
that claim into an executable contract, so the next diff can be reviewed against tests instead of
prose.

Contracts pinned

1. The tie-break ordering (test_edit_script_tie_break_is_stable).
EditDistance._best_match orders candidate predecessors by accumulated cost, then by the number of
edits on the path, then by a fixed direction order: diagonal, border insertion, border removal. Its
docstring calls that order part of the output contract, but nothing checked it. The existing tests
assert only that the reconstructed script is an optimal alignment, never which one.

The test pins the exact edit sequence for eight pairs, including three chosen to isolate one key
each:

pair what it isolates
'aabc' -> 'bcb' the path-length key: two substitutions and a removal tie on cost with a removal, two matches and an insertion
'aba' -> 'bab' the two border directions tie on both keys, so the removal has to come out before the insertion
'caccda' -> 'bcddcb' six alignments tie on both keys; this pair changes under all five reorderings of the three directions

2. The shared prefix strip is output-visible (test_shared_prefix_biases_the_alignment).
EditDistance.__init__ strips a common prefix and a common reversed suffix before building the
matrix. The prefix strip is not just a speedup: it forces the leading elements to be matched
diagonally, whereas backward reconstruction otherwise reaches the origin by a border move. 14% of
all pairs over abc with a combined length of 7 or less, and 9.8% of 2000 random pairs over abcd
with lengths up to 12, produce a different script when the strip is removed. The total cost and the
number of edits are identical in every one of those cases, which is exactly why no existing test
notices. A comment at the block records this, which is the only production change here.

3. The script realizes the cost the edit reports (test_edit_script_realizes_the_reported_cost).
A randomized differential over small-alphabet pairs replays the reconstructed script and compares
its cost to both levenshtein_distance and the edit's own bounds(). Nothing checked this before:
test_string_edit_distance_reconstruction checks that the script rebuilds both strings, and
test_string_edit_distance_is_levenshtein checks the reported distance, but a cost computed
anywhere other than from the script itself passes both. This is the strongest invariant available
for the next PR to lean on. It adds about 0.2 seconds, so it runs in the default suite.

4. Rendered snapshots (test/test_json.py::TestRenderedDiffs). Four rendered diffs, using the
render helper that module already has: a dict in which every key and value changes, a list of
similar strings, a string nested two levels deep, and a list item inserted inside two dicts.

Each expectation in the first two tests is checked against an independent dynamic program before it
is compared, so the test fails if the pinned script is not optimal on both cost and path length. A
snapshot of a regression cannot pass itself off as a contract.

Break verification

Each break was applied to graphtage/, pytest test/test_levenshtein.py test/test_json.py was run,
and the source was restored.

break tests that fail
candidate order diag, rem, ins tie_break
candidate order ins, diag, rem tie_break
candidate order ins, rem, diag tie_break, shared_prefix, dict_with_every_key_and_value_changed, nested_dict_with_a_changed_string_and_number
candidate order rem, diag, ins same four
candidate order rem, ins, diag same four
drop the path-length key tie_break, realizes_the_reported_cost
disable the prefix strip shared_prefix, list_of_similar_strings
disable the suffix strip none
_best_match picks the worst candidate six, including the two pre-existing optimality tests
bounds() reports cost + 1 realizes_the_reported_cost, the pre-existing is_levenshtein
Printer.indent_str set to two spaces all four snapshots and five pre-existing join-option tests

Disabling the prefix strip leaves the total cost and the edit count unchanged on every pair tested
(0 of 24604 exhaustive pairs and 0 of 2000 random pairs differ in either), so that break changes
which optimal alignment is rendered and nothing else. That is the point of pinning it.

test_nested_dict_with_an_appended_list_item is the one test no Levenshtein break reaches: its
document has no string-level ties, so it pins the pretty-printer rather than the edit script, and it
was verified against the indentation break instead.

Findings

The shared suffix strip is not output-visible. Disabling it changed no script in 24604
exhaustive pairs over abc (combined length 7 or less) or 2000 random pairs over abcd, and failed
no test in the suite. It appears to agree with the diagonal-first tie-break that reconstruction
already applies at the end of the matrix, so it is a pure optimization, unlike the prefix strip next
to it. This is not a bug, but it does mean nothing currently covers that half of the block. It is
stated in the test docstring and the comment rather than pinned, because it would be pinning the
absence of an effect.

graphtage/levenshtein.py:137 is a self-assignment. self.reversed_shared_suffix = self.reversed_shared_suffix does nothing. Left alone here to keep this PR test-only.

path_costs is numpy.uint16. A path can be as long as len(from_seq) + len(to_seq), so
sequences with a combined length above 65535 raise OverflowError on assignment under NumPy 2.
The matrix is O(n*m), so such an input is not reachable in practice; noting it only because the next
PR touches this accounting.

🤖 Generated with Claude Code

ESultanik and others added 4 commits September 15, 2026 15:32
EditDistance._best_match documents its candidate ordering as part of the
output contract, but nothing checked it: the existing tests only assert
that the reconstructed script is *an* optimal alignment, not which one.

Pin the exact edit sequence for pairs of strings that have more than one
optimal alignment, covering all three ordering keys: accumulated cost,
path length, and the direction order of diagonal, border insertion, and
border removal. Each expectation is checked against an independent
dynamic program before it is compared, so a regression cannot be
snapshotted in as a contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stripping a common prefix before building the Levenshtein matrix reads as
a pure optimization, but it is output-visible: it forces the leading
elements to be matched diagonally, whereas backward reconstruction
otherwise reaches the origin by a border move. About 10% of small-alphabet
pairs come out with a different, equally optimal script when the strip is
removed.

Pin four of those scripts and record the effect in a comment, so the
block is not mistaken for a speedup that can be replaced freely. The
shared-suffix strip has no such effect, because it agrees with the
diagonal-first tie-break that reconstruction already applies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The suite checks that the reconstructed script rebuilds both strings and,
separately, that EditDistance reports the canonical Levenshtein distance.
Nothing checked that the script a user sees adds up to the cost the edit
reports, so a cost computed anywhere other than from the script itself
passes both existing tests.

Add a randomized differential over small-alphabet pairs that replays the
script and compares its cost to both levenshtein_distance and the edit's
own bounds. It runs in well under a second, so it stays in the default
suite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The edit script contracts are pinned edit by edit in test_levenshtein.py,
but nothing pinned what a user actually sees. Add four snapshots of
rendered JSON diffs: a dict in which every key and value changes, a list
of similar strings, a string nested two levels deep, and a list item
inserted inside two dicts.

Three of the four change if the candidate ordering in _best_match is
permuted or the shared prefix strip is removed; the fourth pins the
pretty-printer's indentation and insertion markers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ESultanik
ESultanik merged commit 3f0e748 into master Sep 15, 2026
12 checks passed
@ESultanik
ESultanik deleted the pin-levenshtein-edit-script-contract branch September 15, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant