Repository navigation
Frontier walks, typed walks, parents, locations and subtree hashes in Rust - #7
Merged
Merged
Conversation
Return only the descendants a NodeVisitor would dispatch on, in generic_visit order, so a visitor with a few visit_<kind> methods can skip every other node without a Python loop. The edges variant carries the parent slot each match was read from, for a NodeTransformer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hv7iDZqYjqWUscP5LcdwfB
All three, and the frontier walks, read each child by its _fields name rather than by dict position, so they also handle nodes a transformer built with keyword arguments in another order. subtree_hashes mixes in each leaf value's type, so 1, 1.0 and True hash apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hv7iDZqYjqWUscP5LcdwfB
- walk_of_types(node, kinds, prune) stops below nodes of the pruned types. - set_parents sets child.parent across a tree. - subtree_hash returns the root's structural hash without the per-node dict. - walk_frontier_events reports enter/leave around bracketed types, for a visitor that does work around its descent. - A field a constructor left out (ast.Constant(value=1) has no kind entry) is read through attribute lookup, as ast.iter_fields reads it, so it hashes like the parsed node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hv7iDZqYjqWUscP5LcdwfB
A field read can run Python code -- a property, a __getattr__, a leaf's __hash__ or __repr__ -- that drops the tree's own references, and the walks held borrowed pointers across it: a property computing a node returned a pointer to a freed object. Every pushed node, edge and event is now an owned reference, and a list is re-read item by item. - TypeInfo holds its type, so a freed type's address is not read with its _fields by a new type allocated there; the cache is filled without a borrow held across the Python calls that read it. - is_ast recognises a subclass at any depth, not only two levels down. - A node with no instance dict reads its fields by attribute. - Leaf values other than str, bytes, int, bool, None and Ellipsis hash by repr, as ast.dump compares them: NaNs together, -0.0 apart from 0.0, and an unhashable constant no longer raises. A child a property computes afresh is hashed on the spot rather than panicking. - set_parents reads each node's fields once. - walk_frontier and walk_frontier_events fill their lists directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hv7iDZqYjqWUscP5LcdwfB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hv7iDZqYjqWUscP5LcdwfB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the traversals flexgen's guardrails linter needs to stop recursing once per tree level and to move its hot walks out of Python (ENG-13126). A long attribute chain in generated code was raising
RecursionErrorout of the linter.New functions
walk_frontier(node, kinds)/walk_frontier_edges(node, kinds): the descendants anast.NodeVisitorwould dispatch tovisit_<kind>methods. The walk goes in depth-first pre-order and does not go below a match. The edges form also returns(parent, field, index), so aNodeTransformercan put a replacement back.walk_frontier_events(node, kinds, bracket): the same walk as(event, node)pairs, with enter/leave events aroundbrackettypes, for visitors that do work around the generic descent.walk_of_types(node, kinds, prune=None): every node of the given exact types, descending into matches but not below aprunetype.set_parents(node):child.parent = parentfor the whole tree.fix_missing_locations(node):ast.fix_missing_locationswithout recursion.subtree_hashes(node)/subtree_hash(node): structural hashes where equal hash means equalast.dump. Leaf value types are mixed in, so1,1.0andTruehash apart.How children are read
These functions live in
src/fields.rs. They read each node's children by_fieldsname instead of by instance-dict position, because a transformer can build nodes whose dict is in a different order. A field missing from the instance dict falls back togetattr, so a node built asast.Constant(value="x")gets the class default forkindand hashes the same as the parsed node. A walk that only reports some types skips childless nodes it could never report.Tests
tests/test_frontier.pyandtests/test_fields.pycheck each function against a pure-Python reference over a broad source sample. They also cover nodes built with a different attribute order, 50,000-deep attribute chains, and refcount neutrality. 242 tests pass locally on CPython 3.14.Release
flexgen falls back to Python when these functions are missing, so merging is safe on its own. To use them, it needs a release with them and a
uv.lockbump.Cargo.tomlis still0.2.2andv0.2.2is already tagged, so the release needs a version bump first.🤖 Generated with Claude Code
https://claude.ai/code/session_01Hv7iDZqYjqWUscP5LcdwfB
Generated by Claude Code