Pin the Levenshtein edit script output contract - #196
Merged
Merged
Conversation
EditDistance._best_match documents its candidate ordering as part of the output contract, but nothing checked it: the existing tests only assert that the reconstructed script is *an* optimal alignment, not which one. Pin the exact edit sequence for pairs of strings that have more than one optimal alignment, covering all three ordering keys: accumulated cost, path length, and the direction order of diagonal, border insertion, and border removal. Each expectation is checked against an independent dynamic program before it is compared, so a regression cannot be snapshotted in as a contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stripping a common prefix before building the Levenshtein matrix reads as a pure optimization, but it is output-visible: it forces the leading elements to be matched diagonally, whereas backward reconstruction otherwise reaches the origin by a border move. About 10% of small-alphabet pairs come out with a different, equally optimal script when the strip is removed. Pin four of those scripts and record the effect in a comment, so the block is not mistaken for a speedup that can be replaced freely. The shared-suffix strip has no such effect, because it agrees with the diagonal-first tie-break that reconstruction already applies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The suite checks that the reconstructed script rebuilds both strings and, separately, that EditDistance reports the canonical Levenshtein distance. Nothing checked that the script a user sees adds up to the cost the edit reports, so a cost computed anywhere other than from the script itself passes both existing tests. Add a randomized differential over small-alphabet pairs that replays the script and compares its cost to both levenshtein_distance and the edit's own bounds. It runs in well under a second, so it stays in the default suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The edit script contracts are pinned edit by edit in test_levenshtein.py, but nothing pinned what a user actually sees. Add four snapshots of rendered JSON diffs: a dict in which every key and value changes, a list of similar strings, a string nested two levels deep, and a list item inserted inside two dicts. Three of the four change if the candidate ordering in _best_match is permuted or the shared prefix strip is removed; the fourth pins the pretty-printer's indentation and insertion markers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test-only, except for one comment. No behavior changes.
An upcoming change makes
StringEditcarry an exact integer cost and builds the per-characterEditDistancelattice lazily, only for the edits that get rendered. That should preserve outputexactly, because the edit script still comes from the same unchanged
EditDistance. This PR turnsthat claim into an executable contract, so the next diff can be reviewed against tests instead of
prose.
Contracts pinned
1. The tie-break ordering (
test_edit_script_tie_break_is_stable).EditDistance._best_matchorders candidate predecessors by accumulated cost, then by the number ofedits on the path, then by a fixed direction order: diagonal, border insertion, border removal. Its
docstring calls that order part of the output contract, but nothing checked it. The existing tests
assert only that the reconstructed script is an optimal alignment, never which one.
The test pins the exact edit sequence for eight pairs, including three chosen to isolate one key
each:
'aabc' -> 'bcb''aba' -> 'bab''caccda' -> 'bcddcb'2. The shared prefix strip is output-visible (
test_shared_prefix_biases_the_alignment).EditDistance.__init__strips a common prefix and a common reversed suffix before building thematrix. The prefix strip is not just a speedup: it forces the leading elements to be matched
diagonally, whereas backward reconstruction otherwise reaches the origin by a border move. 14% of
all pairs over
abcwith a combined length of 7 or less, and 9.8% of 2000 random pairs overabcdwith lengths up to 12, produce a different script when the strip is removed. The total cost and the
number of edits are identical in every one of those cases, which is exactly why no existing test
notices. A comment at the block records this, which is the only production change here.
3. The script realizes the cost the edit reports (
test_edit_script_realizes_the_reported_cost).A randomized differential over small-alphabet pairs replays the reconstructed script and compares
its cost to both
levenshtein_distanceand the edit's ownbounds(). Nothing checked this before:test_string_edit_distance_reconstructionchecks that the script rebuilds both strings, andtest_string_edit_distance_is_levenshteinchecks the reported distance, but a cost computedanywhere other than from the script itself passes both. This is the strongest invariant available
for the next PR to lean on. It adds about 0.2 seconds, so it runs in the default suite.
4. Rendered snapshots (
test/test_json.py::TestRenderedDiffs). Four rendered diffs, using therenderhelper that module already has: a dict in which every key and value changes, a list ofsimilar strings, a string nested two levels deep, and a list item inserted inside two dicts.
Each expectation in the first two tests is checked against an independent dynamic program before it
is compared, so the test fails if the pinned script is not optimal on both cost and path length. A
snapshot of a regression cannot pass itself off as a contract.
Break verification
Each break was applied to
graphtage/,pytest test/test_levenshtein.py test/test_json.pywas run,and the source was restored.
diag, rem, insins, diag, remins, rem, diagrem, diag, insrem, ins, diag_best_matchpicks the worst candidatebounds()reports cost + 1Printer.indent_strset to two spacesDisabling the prefix strip leaves the total cost and the edit count unchanged on every pair tested
(0 of 24604 exhaustive pairs and 0 of 2000 random pairs differ in either), so that break changes
which optimal alignment is rendered and nothing else. That is the point of pinning it.
test_nested_dict_with_an_appended_list_itemis the one test no Levenshtein break reaches: itsdocument has no string-level ties, so it pins the pretty-printer rather than the edit script, and it
was verified against the indentation break instead.
Findings
The shared suffix strip is not output-visible. Disabling it changed no script in 24604
exhaustive pairs over
abc(combined length 7 or less) or 2000 random pairs overabcd, and failedno test in the suite. It appears to agree with the diagonal-first tie-break that reconstruction
already applies at the end of the matrix, so it is a pure optimization, unlike the prefix strip next
to it. This is not a bug, but it does mean nothing currently covers that half of the block. It is
stated in the test docstring and the comment rather than pinned, because it would be pinning the
absence of an effect.
graphtage/levenshtein.py:137is a self-assignment.self.reversed_shared_suffix = self.reversed_shared_suffixdoes nothing. Left alone here to keep this PR test-only.path_costsisnumpy.uint16. A path can be as long aslen(from_seq) + len(to_seq), sosequences with a combined length above 65535 raise
OverflowErroron assignment under NumPy 2.The matrix is O(n*m), so such an input is not reachable in practice; noting it only because the next
PR touches this accounting.
🤖 Generated with Claude Code