Rust core: linguist + index + reads/writes with differential proof of equivalence - #22
Open
aaliboyev wants to merge 21 commits into
Open
Rust core: linguist + index + reads/writes with differential proof of equivalence#22aaliboyev wants to merge 21 commits into
aaliboyev wants to merge 21 commits into
Conversation
Faithful port of shared/linguist/ — strategy chain, Blob semantics (errors=ignore decode, py-splitlines boundaries, 5MiB cap, stat size), Language registry, vendor-pruned walk, LanguageStatistics — plus the heuristics.yml strategy Python never wired in (upstream position, after Manpage; --no-heuristics reproduces the Python chain bit-exactly) and a persisted classification cache (.symbol/linguist.msgpack.zst, msgpack+zstd, mtime+size invalidation, atomic writes). Parity gate: scripts/linguist_parity.py — exact integer match vs 'symbol loc --format json' on the symbol repo and pallets/flask with --no-heuristics; heuristics-on delta documented in rust/linguist/README.md. 26 tests.
- symbol-core crate: v6 byte-compatible SymbolIndex (Python and Rust read each other's .symbol/symbol_index.msgpack.zst), reads (search/code/ outline/callers), writes (patch/delete/replace/rename tier-1/undo) with Python-compatible transactions and session read cache. - pyscan daemon: PythonAstAdapter behind the go-scan JSON-RPC stdio contract — Python analyzed by Python ast, Go by go-scan, Rust core orchestrates. Adapter registry with env-based daemon discovery. - Verified cross-interop: Rust-built index queried by Python, Python-built index served by Rust, Rust-committed rename undone by Python symbol undo. - Divergences documented in rust/symbol/README.md (tier-1 rename for all kinds, insert-symbol/audit/map/MCP not ported).
scripts/diff_runner.py: dataset-driven differential harness — executes a JSONL ops file (authored externally) against both implementations, diffs canonical payloads for reads and byte-exact file trees for writes (with undo-restores-pristine checks), classifies match/known_divergence/MISMATCH, reports timing medians. --index-check does a full structural index diff (every qpath row, refs, imports) with no dataset needed. Results so far: fixture 19/19, flask index 1569/1569 exact. gin initially 3201 mismatches — the runner caught the missing Go module_prefix; ported GoAstAdapter.module_prefix (go.mod walk + module directive parse, per-dir cache) into the Rust adapter → gin 1614/1614 exact.
The k8s index diff caught the gap: Python's full build applies spec-level [checker] excludes (**/build/** etc.) via filter_paths; Rust didn't. Ported a minimal shim (global data/spec.toml + target symbol.toml / pyproject [tool.symbol.checker]); per-package spec excludes still pending. Differential index-check results after fix — zero mismatches: kubernetes 141,404/141,404 (index build: rust 10.6s vs python 31.6s) flask 1,569/1,569 gin 1,614/1,614 scripts/ops_dataset_spec.md documents the JSONL ops schema + coverage guidance for model-generated test datasets.
pyscan gains rename_member/rename_module_binding: loads the shared on-disk index (Rust-written, byte-compatible), builds RenamerIndexQuery, runs the real receiver-aware analysis. Rust renamer routes python engine kinds through it (candidate files = declaring + refs + import aliases, per-file analysis, byte splices, tx commit) and keeps tier-1 for everything else. Differential result: engine-ops set 9/9 match (was known_divergence) — decoy same-leaf method correctly skipped with resolved_to_qpath, strings untouched, trees byte-identical to Python's engine. Rename median 373ms (py) vs 148ms (rs).
ScannedSymbol.bases (wire + schema), scanner capture (Subscript unwrapped, keywords ignored), bases_of parallel table mirroring refs_of (ingest/ tombstone/payload/load/accessor) in Python and Rust, INDEX_VERSION 6→7, RenamerIndexQuery.class_bases implemented (same-module outward walk + import resolution, unresolvables dropped), diff_runner index snapshot includes bases. Gates: cargo test green; pytest 443 passed; index-check fixture 12 / flask 1569 / gin 1614 — zero mismatch; interop verified (Rust-built v7 loaded by Python, dotted bases correct). Implemented by Opus subagent; reviewed.
rename_member careful-mode bucketing: exact owner equality → inclusive ancestor-set membership (_ancestor_set: cycle-safe walk over index.class_bases, size-capped, phantom bases terminal-not-expanded). Subclass call sites now rewrite correctly. Plain super() resolves statically to the enclosing class's ancestors minus itself via a dedicated resolve_super_receiver pre-pass; two-arg super stays unresolved. Unresolvable/external bases keep skipped_mismatch (not unresolved) — rationale in _ancestor_set docstring. Gates: pytest 451 passed (8 new MRO tests incl. diamond, cycle guard, decoy, external base, super both directions); twin-tree parity vs Rust byte-identical with the subclass site rewritten by both engines. Implemented by Opus subagent; reviewed.
Mirror go-scan's project-wide rename RPC: one call, per-file slicing, three buckets aggregated with file tags, affected_interfaces lifted from the declaring file's entry (deduped). Routing: go method→member, function/type/var/const→module_binding; tier-1 stays for the rest. Wire fix: Go marshals nil slices as JSON null — null_as_default serde helper on all RenameFileAnalysis vectors. Dev discovery now also finds adapters/go_ast/daemon/go-scan (bundled bin/ binaries predate the rename RPC — CI rebuild needed, out of scope). DECISION RECORDED: daemon results are intersected with the index-derived candidate set to stay in lockstep with the Python oracle — this caps go/types' cross-package reach (gin: ginS/gins.go finding dropped, 30 refs matching oracle vs 31 from the daemon). When Python's oracle role ends, drop the intersection and trust go/types fully. Gates: cargo green; gin twin-tree diffs empty for method (+interface surfaced), type, const, function renames; engine=v2 on all. Implemented by Opus subagent; reviewed.
…ly cap Per Abror: if a higher-capability analysis offers correct findings, take them — the tier label bounds what we CLAIM, not what we apply. Rust drops the candidate-set intersection on the Go path (all daemon-reported files applied). The Python oracle mirrors via a new optional adapter hook pop_project_wide_analyses consumed by the neutral renamer after the candidate loop — go/types findings for files the index heuristic missed now land on both sides. Proof: gin RouterGroup rename now 31 refs / 3 files on BOTH engines (previously oracle-capped at 30, dropping ginS/gins.go); twin trees byte-identical. pytest 451 passed.
Four anchor positions with exact insert_symbol.py byte math (end == after offset, body indent = anchor + 4; before hops above decorators via new pyscan first_decorator_line RPC, gated to python+has_body kinds), content normalization honoring --no-reindent, zero-width splice through the patch engine, daemon blank-gap normalization post-apply. CLI subcommand + diff_runner insert-symbol op (py --position/--content flags vs rs positionals). 8 new integration tests. Gates: cargo green (8 new cli tests), pytest 451, twin-tree differentials empty for after / before-on-decorated / start / end. Implemented by Opus subagent; reviewed.
…index v8) Scanner emits module-level UPPER_CASE constants and class fields (class-body targets + __init__ self.x, deduped class-body-wins, parented to the class row). rename_field engine method sharing the attribute walker with rename_member (_classify_attr_sites extraction — member behavior byte-unchanged), full plumbing: protocol, pyscan RPC, Rust rename_analysis(method) + FIELD_KINDS routing, renamer dispatch. via_engine gate in rename_symbol.py extended with field/attribute — without it fields fell to tier-1 regex (necessary out-of-scope edit, justified). Index content changes → v8 both sides. Gates: pytest 458; cargo green; index-check fixture 16 / flask 1787 (+222 new rows, all matched) / gin 1614 — zero mismatch; twin-tree field and cross-file constant renames byte-identical, decoy untouched. Implemented by Opus subagent; reviewed.
Engine rename_parameter: def arg + in-scope body uses (nested-scope shadowing honored incl. defaults/annotations evaluating in enclosing scope) + keyword arguments at call sites statically resolved to the target (method params via receiver+MRO discrimination, module functions via name tables); same-leaf-different-func → skipped_mismatch, unresolvable callee → unresolved. Language isolation preserved: param collision checks via new adapter method function_parameters (no ast in neutral layers). Full plumbing: protocol, pyscan, SymbolRenamer entry reusing _run_per_file, Python CLI command, Rust orchestration + subcommand, diff_runner op. Gates: pytest 468; cargo green; index counts unchanged (re-verified 16/1787); twin trees identical for method-param w/ discrimination, cross-file module-param, collision refusal both sides. Implemented by Opus subagent; reviewed.
…ts fixed Datasets (scripts/datasets/, every target from real index dumps, corpus SHAs in its README): flask 426 ops, gin 383, self 385 + regeneration scripts. Reports in scripts/datasets/reports/. Final sweep: flask 588/588, gin 469/469, self 495/495 dataset verdicts; index checks 1787/1614/2284/16 — zero MISMATCH, zero known_divergence anywhere; engine-kind tag never fired. The sweep's 229 initial mismatches root-caused to 7 defects — none in the program's feature work (WP1-6 all held): Go signature wire-key + missing go-only fallback, search limit-truncation ordering (first-seen row order), two runner CLI-arg bugs (patch/replace options), runner undo assertion on non-tx ops, CPython-exact st_mtime arithmetic (last-ulp mismatch caused spurious refreshes), and an ORACLE bug fixed in both implementations: the incremental-refresh path applied no vendor filter, so refreshed indexes smuggled testdata/ back in. Also: Go bare-snippet package-stub prelude ported to Rust scan_content/validate_syntax (byte/line shift-back), go-scan scan_content mapped to scan_file. Gates: cargo 8 suites green, pytest 468, all sweeps re-run to zero. Datasets + sweep by Opus subagent; runner and core fixes by reviewer.
loc logic moves from the linguist binary's main into symbol_linguist::loc (shared library); the standalone symbol-linguist binary becomes a thin wrapper (kept — parity harness references it), and symbol-rs grows 'loc' so one binary covers the entire ported surface. No behavior change: linguist parity gate still exact on flask; 8 cargo suites green.
Design correction (Abror): the standalone rename-parameter command is removed from both CLIs. replace-symbol now detects parameter renames in the new content — conservative rule: equal arity, positional pairing, new name never present in the old list and vice versa (swaps/reorders refuse) — and propagates keyword-argument call sites (receiver/MRO discriminated) in the SAME transaction. Ambiguous or arity-changing edits propagate nothing and report signature_changed with both param lists. Result carries parameter_renames buckets. Engine gains kwargs_only + source-override (post-splice analysis); WP4's engine core reused intact. Datasets converted (rename-parameter ops → equivalent replace-symbol ops), spec + runner updated, tests rewritten as test_replace_signature.py. Gates: pytest 465; cargo green; index-checks 16/1787/1614 unchanged; twin trees byte-identical incl. cross-file kwarg propagation + pristine undo both sides; converted ops 63/63 match. Reviewer end-to-end: kwarg site rewritten through resolvable receiver, factory-bound receiver honestly bucketed at the tier-2 boundary. Implemented by Opus subagent; reviewed.
symbol-dataset mines/composes/verifies differential task datasets from real corpora; symbol-harness is a black-box scenario runner driving the symbol-rs CLI. scripts/feature_cases holds the local python_core fixture and the pinned real-world corpora manifests they consume. Register both crates in the workspace and move [profile.release] to the workspace root — profiles in member manifests are ignored by cargo.
map_or(true, ..) → is_none_or, match → if-let, reflow long signatures. No behavior change. index doc notes the persisted format is v8; rename doc reflects the live Go v2 engine rather than a tier-1-only fallback.
apply_multi_patch applies several non-overlapping edits to one file in a single atomic write: range-addressed edits reuse the patch read-cache preflight, exact-old edits must match exactly once and prove their own context. Wire it up as the MultiPatch subcommand. Also on the write path: resolve relative file paths against the project root, tolerate sub-ulp served-mtime differences in preflight, refresh restored files explicitly after undo instead of trusting mtime, and derive read-cache byte ranges from the served start/end lines.
Add test-rust / test-features / test-real-world targets and invoke them from CI behind a stable Rust toolchain, uploading the harness reports. ops_dataset_spec drops the removed rename-parameter refusal in favour of invalid-replacement-content.
CLAUDE.md drops the directory tree, capability table, roadmap, and local path, keeping only the never-violate rules (language isolation, architecture layering, conventions). README describes symbol-rs — the index, read and write surfaces, loc, and the native-AST daemon architecture — and the Python-to-Rust transition; build-from-source until a packaged distribution lands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Rust port of symbol's core, built as the beachhead of the full Python→Rust migration:
rust/linguist(symbol-linguist) — faithful port of the GitHub-Linguist subsystem, parity-gated exact againstsymbol loc --format json, plus theheuristics.ymlstrategy Python never wired in and a persisted classification cache.rust/symbol(symbol-rs) — the language-neutral core: v8 byte-compatible symbol index (either implementation reads the other's), reads (search/code/outline/callers), writes (patch, delete/replace/insert/rename-symbol, rename-parameter, undo) with Python-compatible transactions.pyscandaemon (~200-line socket aroundPythonAstAdapter) and the unchangedgo-scan.super()in the rename engine,field/constantas real symbol kinds with receiver-awarerename_field,rename-parameter(def + scope + statically-resolved kwarg call sites), Go renames at fullgo/typesreach withaffected_interfaces.scripts/diff_runner.py+ model-authored datasets inscripts/datasets/): executes every op against both implementations and diffs canonical payloads / byte-exact trees / undo round-trips.Proof
Final sweep (corpus SHAs in
scripts/datasets/README.md, full table inrust/symbol/README.md):The sweep's initial findings root-caused to 7 defects — none in the ported feature work; notably one oracle bug fixed in both implementations (the incremental-refresh path applied no vendor filter, so index content depended on which path built it).
Kubernetes-scale numbers: index build 31.6s (Python) → 10.6s (Rust); linguist classification of linux (41M lines) 58s → 1.0s cold / 0.34s warm.
Not in this PR (tracked for follow-up programs)
MCP server (rmcp), in-process ruff adapter (pyscan stays the oracle until it passes the same diff gate), audit/map migration, CI cross-compile + wheel bundling of
symbol-rs/rebuiltgo-scanbinaries, Go parameter rename.python-legacypreserves the pre-Rust state ofmain.