Build the string edit lattice only when it is rendered - #197
Merged
Merged
Conversation
Computing a plain Levenshtein distance through EditDistance materializes one TreeNode per character and one live Edit per matrix cell. Callers that only need the cost have no way to ask for it. Add a helper that answers the same question arithmetically: equal strings cost nothing, an empty operand costs the length of the other, and a shared prefix and suffix are stripped before the canonical dynamic program runs on what is left. It accepts str and bytes in either combination, matching what StringNode wraps, and is the single seam where a batch or cached backend can later be substituted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
StringEdit.__init__ called string_edit_distance eagerly, so every candidate pair of strings a diff considered paid for a full lazy EditDistance lattice. WeightedBipartiteMatcher.edges is dense, so an N-by-M match built N*M lattices although only min(N, M) of them are ever printed. Derive the cost from exact_string_distance instead and make StringEdit a ConstantCostEdit, whose bounds are definitive on construction and whose tighten_bounds returns False. Move the lattice behind a lazy property that keeps the public edit_distance name, so StringFormatter.print_StringEdit and YAMLStringFormatter build it for the edits they render and nothing else does. The edit script still comes from the same unchanged EditDistance, so the tie-break contract that test/test_levenshtein.py and test/test_json.py pin is preserved by construction. Point LeafNode.edits at the same helper so non-string leaves share the seam. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The change that makes StringEdit a ConstantCostEdit has three separable ways to go wrong, and nothing in the suite covered any of them. TestExactStringDistance cross-checks exact_string_distance against levenshtein_distance over randomized str, bytes, and non-ASCII corpora, plus the cases its short-circuits decide: equal operands, empty operands, a prefix that runs into the suffix, and a mixed str/bytes pair. TestStringEdit pins that the bounds are definitive on construction, that tighten_bounds returns False, that the lattice stays unbuilt until edit_distance is read, that its final cost equals the reported cost, and that neither printing a StringNode nor costing a dense 8-by-8 match builds more lattices than the diff renders. A 60-key dict diff inside run_with_time_limit guards the speedup itself: it takes over 30 seconds when StringEdit prices itself through the lattice and under a second when it does not. TestMatchingOptimality solves the assignment problem a second time with scipy over a cost matrix built directly from levenshtein_distance, and asserts that the pairs WeightedBipartiteMatcher chose cost the same. That is an absolute statement about the matching, so it needs no copy of the old algorithm to compare against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F2sHz5c5TvMs9tFn2HhwaC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
StringNode.editsreturned aStringEditwhose__init__eagerly calledstring_edit_distance. That built aListNodeof oneTreeNodeper character and ran the full lazyEditDistancelattice, an (n+1)×(m+1) matrix oflive
Editobjects, to arrive at a plain integer Levenshtein distance.WeightedBipartiteMatcher.edgesis dense, so an N-by-M match paid forN*Mlattices although it renders at mostmin(N, M)of them. The laziness never earned its cost either:EditDistance._best_matchprices every cellthrough
_exact_cost, whosewhile not definitive and tighten_bounds()loop runs to completion, so nestingforces full evaluation rather than enabling pruning.
What changed
StringEditderives from the existingConstantCostEditinstead ofAbstractEdit. Its cost comes from a newexact_string_distancehelper, so its bounds are definitive on construction andtighten_boundsreturnsFalse.StringEdit.boundsandStringEdit.tighten_boundsare gone.StringEdit.edit_distanceis now a lazy property. The name is unchanged, and its two readers,StringFormatter.print_StringEditandYAMLStringFormatter.write_start_quote, are unchanged. The lattice isbuilt for the edits a diff renders and for nothing else.
graphtage.levenshtein.exact_string_distanceanswers the same question arithmetically: equal operands costnothing, an empty operand costs the length of the other, and a shared prefix and suffix are stripped before
levenshtein_distanceruns on the remainder. It acceptsstrandbytesin either combination, which is whatStringNodewraps. It is a single seam, so a batch or cached backend can be substituted behind it later.LeafNode.editsuses the same helper, so non-string leaves share that seam.The edit script still comes from the same unchanged
EditDistance, so the tie-break contract that #196 pinned ispreserved by construction.
test_edit_script_tie_break_is_stable,test_shared_prefix_biases_the_alignment,test_edit_script_realizes_the_reported_cost, andTestRenderedDiffspass unmodified.Performance
Wall clock for
pydiff.build_tree(a).diff(pydiff.build_tree(b)).edited_cost(), Python 3.14 on an Apple laptop.Each shape uses random lowercase strings with a fixed seed.
Counted directly on an 8-by-8 dict diff in which no key survives: master builds 128 lattices, this branch builds
16, one per string it renders.
pytest -qgoes from 231 to 250 passing.Differential
The offline corpus is 24 cases: the three shapes above, nested containers, dicts whose values are mutations of
each other, mixed leaf types,
bytesvalues, non-ASCII strings, shared affixes, empty and single-characterstrings, a small-alphabet corpus that maximizes ties, four unordered shapes that route through
MultiSetEdit,and one document in each of JSON, JSON5, YAML, XML, HTML, CSV, TOML, INI, and plist. Each case records the total
edit cost and the rendered output.
near-miss strings, a set of 12 tuples, and a dict against a set.
The four are exactly the cases that reach
WeightedBipartiteMatcher. It prices its edges withbounds().upper_boundafter only partial tightening throughmake_distinct, so master handsscipy.optimize.linear_sum_assignmenta weight matrix with some entries over-stated. With exact weights, scipyselects a different assignment among the ones that tie. Solving the assignment problem independently over a cost
matrix built from
levenshtein_distanceconfirms that master's matchings on these shapes were already optimal intotal, which is why the cost did not move; only which optimum is chosen did.
Ordered containers and all nine file formats render byte-identically.
The corpus is not committed. Keeping the old algorithm alive in the suite to compare against would be worse than
having no comparison at all.
Tests
test/test_levenshtein.py,TestExactStringDistance: cross-checksexact_string_distanceagainstlevenshtein_distanceover randomizedstr,bytes, and non-ASCII corpora, and pins the cases itsshort-circuits decide.
test/test_graphtage.py,TestStringEdit: bounds definitive on construction,tighten_boundsreturnsFalse,the lattice stays unbuilt until
edit_distanceis read, the lattice's final cost equals the reported cost,printing a
StringNodebuilds no lattice, and a dense 8-by-8 match builds no more lattices than it renders. Plustest_a_sixty_key_dict_diff_is_fast, which runs a 60-key diff with every key and value different insiderun_with_time_limit(seconds=30).test/test_multiset.py,TestMatchingOptimality: solves the assignment problem a second time withscipy.optimize.linear_sum_assignmentover a cost matrix built directly fromlevenshtein_distance, and assertsthat the pairs
MultiSetEdit._matcher.matchingchose cost the same. A matching optimal under exact costs is byconstruction no worse than anything the old path could produce, so this needs no copy of the old algorithm.
Break verification
Each test was verified against a deliberately broken tree.
exact_string_distancereturnsmax(len(s), len(t))on the remainderTestExactStringDistancemethods, 4TestMatchingOptimalitymethods, 3TestStringEditmethods, and the pre-existingtest_string_diff_printingtest_shared_prefix_and_suffix_do_not_overlapand 3 randomized agreement testsexact_string_distancecoerces its operands withstr()test_agrees_on_random_bytes,test_str_never_equals_bytes,test_the_lattice_agrees_with_the_reported_cost, and the pre-existingtest_str_versus_bytesStringEditbuilds its lattice in__init__againtest_the_lattice_is_not_built_until_it_is_read,test_printing_a_string_node_builds_no_lattice,test_a_diff_builds_a_lattice_only_for_what_it_renders(128 lattices against a bound of 32)StringEdit.boundsandtighten_boundsdelegate to the lattice, as on mastertest_bounds_are_definitive_on_construction,test_tighten_bounds_returns_false, andtest_a_sixty_key_dict_diff_is_fastThe time limit was also checked against the unmodified
origin/mastersources directly: the 60-key diff tripsthe 30-second limit there and finishes in 0.57 s here.
TestMatchingOptimalitydoes not fail onorigin/master, because master's matchings on these corpora, thoughdifferent, were already cost-optimal. It is verified by the first break in the table rather than by the
before/after comparison, and it is written as an absolute invariant for that reason.
🤖 Generated with Claude Code
https://claude.ai/code/session_01F2sHz5c5TvMs9tFn2HhwaC