Fix the crash on diffing two differing bytes values - #194
Merged
Merged
Conversation
StringNode.edits took a shortcut for single-character strings that called
len() on the wrapped object. The character-level lattice that
string_edit_distance builds wraps each element of the input in its own
StringNode, and iterating bytes yields int, so comparing two differing
bytes values reached StringNode(int).edits(StringNode(int)) and raised
"TypeError: object of type 'int' has no len()".
A byte node stands for one character, so _num_characters reports 1 for an
int. StringNode.calculate_total_size now uses the same count, which makes
inserting or removing a byte cost 1 rather than the 1 to 3 that
len(str(byte_value)) charged, and makes StringNode(b"hello").total_size 5
rather than the 8 of len("b'hello'").
The pickle filetype is the user-visible case: every leaf built from a
bytes constant is a StringNode wrapping bytes, so diffing two pickles
whose binary payloads differ aborted with a traceback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
StringFormatter.write_end_quote wrote the `b` prefix of a bytes literal a second time, so a bytes node rendered as b"hello" followed by a stray b, and a diff of two bytes values ended in b" instead of ". Only the prefix in write_start_quote belongs in the output. The JSON, YAML, and pydiff formatters override both quote methods, so the stray character reached output through StringNode.print and through any formatter that inherits the default quoting, such as XML, TOML, and INI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Diffing two differing
bytesvalues raisesTypeError:Bounds are computed lazily, so the exception surfaces on the first
tighten_bounds()call ratherthan in
edits().Mechanism
StringNode.editstakes a shortcut for single-character strings:StringEditprices a pair of strings by callingstring_edit_distance, which builds acharacter-level lattice by wrapping each element of the input in its own
StringNode. Iteratingover
bytesyieldsintbyte values, not length-onebytes, so the lattice holdsStringNode(int)nodes.graphtage/levenshtein.pythen callsStringNode(int).edits(StringNode(int))for each cell,
inthas no__len__, and the diff aborts.Equal values return
Match(self, node, 0)before the shortcut is reached, so only differing valuescrash.
User-visible impact
The
picklefiletype is the reachable case.graphtage.builder.BasicBuilder.build_stris registeredfor both
strandbytes, andgraphtage.pydiff.ASTBuilderextends it, so every leaf built from abytesconstant is aStringNodewrappingbytes. Diffing two pickles whose binary payloads differends in a traceback:
The same applies to the
graphtage.pydiff.diffandgraphtage.pydiff.build_treelibrary entrypoints, and to any downstream
Buildersubclass that reachesbuild_strwithbytes.The other filetypes are unaffected, because
graphtage.json.build_treecallspython_obj.decode('utf-8')onbytesand stores astr. That is why plist<data>and YAML!!binaryvalues do not reach this code path. See the follow-up note below.After the fix:
Semantics
bytesvalues diff byte by byte, and one differing byte costs 1, matching what one differingcharacter costs for
str. Two things in the existing code fix that model:StringFormatter.write_charalready acceptsc: str | intand renders anintas a printablecharacter or as
\xNN, with the comment "we are printing a bytes object, not a str". The latticenode for a byte is meant to wrap an
int.StringNode.editscharges 1 for a mismatched pair of single-character nodes, andgraphtage.edits.InsertandRemovechargeto_insert.total_size + penalty, which is 1 percharacter for
strbecauseLeafNode.calculate_total_sizereturnslen(str(self.object)).So a byte node has to report a length of 1 and a total size of 1.
_num_charactersreports 1 for anintandlenotherwise, andStringNode.editsand the newStringNode.calculate_total_sizeboth use it.
The
total_sizeoverride matters on its own.LeafNode.calculate_total_sizereturnslen(str(self.object)), which for a byte node islen(str(111)), or 2. That made removing a bytecost 2 or 3 depending on its value, so
b""againstb"abc"cost 6 where""against"abc"costs 3. The override also brings
StringNode(b"hello").total_sizeto 5, from the 8 oflen("b'hello'"), which is what the same value costs as astr.stragainstbyteskeeps the behavior the code already implied:"hello"never equalsb"hello", each position is a substitution, and the distance is 5. The tests pin that rather thanchanging it.
Rendering
StringFormatter.write_end_quotewrote thebprefix of a bytes literal a second time, soStringNode(b"hello")rendered asb"hellob"and a diff of two bytes values ended inb"insteadof
". The second commit drops the two lines; only the prefix inwrite_start_quotebelongs in theoutput. The JSON, YAML, and pydiff formatters override both quote methods, so the stray character
reached output through
StringNode.printand through any formatter that inherits the defaultquoting, which includes XML, TOML, and INI.
Tests
test/test_graphtage.pygainsTestBytesStringNode, covering the substitution case, equal values,single-byte values, empty values on either side, insertion and removal cost,
stragainstbytes,and the two rendering cases.
test/test_pickle.pyis new and covers the end-to-end filetype diff.Each test was checked against the unfixed code: reverting the
editsshortcut fails thesubstitution,
stragainstbytes, rendering, and pickle tests with the originalTypeError;reverting
calculate_total_sizefails the empty-value and insertion tests on cost; restoring theduplicated
bfails the two rendering tests.pytest -qpasses, 202 tests.ruff check graphtage test docs bindistis clean.Follow-up, not in this PR
graphtage.json.build_treedecodesbyteswithpython_obj.decode('utf-8'), so a plist<data>element that is not valid UTF-8 fails before any diff starts:
YAML
!!binarytakes the same route. That is a separate defect with a different mechanism, andfixing it means deciding whether those filetypes should build
StringNode(bytes)and how eachformatter should write one. This PR makes that option viable but does not take it.
🤖 Generated with Claude Code