perf(graph): memoise the CHA-lite inheritance cone — warm llvm --grep 159.7s to 9.2s, default map 248s to 10s by joyful-ii-V-I · Pull Request #83 · redhat-et/ripwire

joyful-ii-V-I · 2026-09-09T18:52:07Z

The warm --grep floor was never the scan. It was the graph.

Measured on llvm-project (182,555 files), warm: --grep 159.7 s, --callers=main 152.9 s — while --help-task, which does the same crawl, cache and model work but builds no graph, is 3.8 s. Inside buildGraph, 143 s of 154 s was one thing: the CHA-lite inheritance cone — {type} ∪ ancestors ∪ descendants, two BFS walks over the class-name graph with an O(n²) std::find dedup — recomputed on every still-ambiguous receiver-typed call. 86,667 cones for 2,984 distinct receiver types, mean cone 1,075 names, 1.65 ms each.

Each receiver type's cone is now computed once and membership answered from it:

warm, llvm-project	before	after
`--grep`	159.7 s	9.2 s
`--callers`	152.9 s	8.6 s
default map	248 s	10 s

test/chaconecheck.sh pins the two behaviours a re-implementation silently changes. The cone is keyed on the RECEIVER TYPE, not the callee name — g1/g2 (Hound) and g3 (Lynx) call the same vocalize and must get their own cones, so arm 3 uses the Lamp case where the answers differ, because Hound and Lynx both contain Creature and would mask a callee-keyed bug. Arm 5 pins the 4096 cap's exact shape: the check is at the outer loop only, so the adjacency list that crosses the cap is pushed whole and nothing after it expands.

Arm 6 pins the failure this class actually has, which is not staleness — the memo is a local of one buildGraph call and its inputs are immutable for that lifetime — but a HIT that returns the wrong hierarchy. A third hierarchy (Machine/Droid, cone excluding Creature) is filled after Hound's entry exists, then Hound is asked for again; the answer must be Creature::vocalize and never Machine::vocalize. Mutation control: coneFor returning &cones_.back() on a hit reds exactly arm 6, naming the Machine line, every other arm still green.

The handle that mutation described is gone too — Cone is now an index resolved inside contains(), so no pointer is handed out and the "valid until the next coneFor()" invariant is unstatable rather than merely documented.

Whole-corpus evidence: default maps byte-identical pre/post on go and llvm-project.

Two repo-wide fixes ride along, both found while trying to land this.

test/maxfilesizecheck.sh asserted skipped_oversize=15 on the unstated premise that no non-JSON indexed file exceeds 1 MiB. docs/EVALS.md crossed it (1,042,372 B → over 1,048,576 B, cumulative across four lanes each appending a section). The tool reported 16 and was right; the arm's premise was stale and it blamed the tool. The expectation is now derived from the tool's own rows — every non-JSON oversize row at the 1M ceiling must name a file that really is over 1,048,576 B on disk — so there is no list to maintain and nothing to go stale.

.github/workflows/ci.yml — the ubuntu-24.04 runner image ships a Google Chrome apt source. When Google's mirror is mid-publish, sudo apt-get update exits 100 with Hash Sum mismatch, the toolchain never installs, and every ubuntu job on every branch dies in step 3 — which reads exactly like a build failure from the job names. It took out main's own run, three lanes, and two runs of this one. The removal now runs before the first apt call at all four Linux install sites, globbed google-chrome* rather than *.list: Ubuntu 24.04 ships apt sources in deb822 format as google-chrome.sources, and the narrower glob left 5 of 18 jobs still dying. Confirmed against the still-flaky mirror: 18 of 18 ubuntu jobs cleared step 3, read from each job's step list.

CI: run 34386912480 — 26/26 green on b7a1767f, the first completed run on this lane after five cancellations and two infrastructure kills. Gate count 566 → 567.

🤖 Generated with Claude Code

…puted per call — memoised, byte-identical The tgrep head-to-head left warm --grep at 40 us/file up to 15,865 files and 938 us/file at 182,555, and named the experiment: stub the graph build, re-time. --help-task IS that arm (returns before buildGraph on the same lean cache blob): on llvm-project warm it takes 3.8 s where --grep takes 159.7 s and --callers=main 152.9 s, all at ~5.9 GB RSS. The floor is the graph; the cache load and per-file validation are linear (16 us/file on llvm vs 28 on go); memory is the ingest's tables and does not move with the graph. Profiled to one operation. New PROFILE_SCOPE scopes on the ingest path (parse-pool join + merge, the two corpus-wide model post-passes) and inside buildGraph (the resolve loop and the five passes after it); a scratch six-span split inside the loop put 145.1 s of 153.2 s in "CHA-lite cone + arity + locality", and a second split put arity at 0.015 s and locality at 0.27 s. The cone — {type} ∪ ancestors ∪ descendants over the class-name graph, two BFS walks with an O(n²) std::find dedup, capped at 4096 — was rebuilt for every still-ambiguous receiver-typed call: 86,667 rebuilds for 2,984 distinct receiver types, mean cone 1,075 names, 1.65 ms each. The obvious suspect, the five linear passes over same-name candidates, visited 1.23 billion candidates and cost 3.3 s. ChaConeMemo (src/graph.h) computes each receiver type's cone once, over class names interned in byte-sorted order, with the per-call walk's exact seed, discovery order and outer-loop-only cap, and answers membership by binary search. Warm llvm: --grep 159.7 s -> 9.2 s, --callers 152.9 s -> 8.6 s, the default map 248 s -> 10 s; --help-task unchanged. Default maps at --top-k=100000 are byte-identical pre/post on go (10,415,057 B) and llvm (21,802,319 B). Gate test/chaconecheck.sh (+ test/chaconefix/): the cone keyed on the receiver type not the callee (Dog and Cat on one speak), the memo hit from a second file, a receiver with no inheritance facts degrading rather than emptying the tier, a parameter-receiver control, and the cap's own shape on a generated 4,097-class corpus (B never expanded, B1::m outside the cone: count=4095, amb="1"). Every arm passes against the pre-fix binary. Gate count 564 -> 565 at every spelled site. Ledger in bench/PROFILE.md, the evidence chain in docs/EVALS.md, a CHANGELOG entry. No timing assertion anywhere. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…rows — docs/EVALS.md crossed 1 MiB The arm asserted skipped_oversize == the count of tracked .json over the 256 K JSON-lane ceiling, i.e. "nothing but the JSON lane drops at --max-file-size=1M". That is a fact about this tree's shape, not about the tool, and it expired on 2026-09-09 when docs/EVALS.md reached 1,059,956 B on a landing tip: 1M is 1024*1024 (cli.h parseByteSize), ripwire drops the file as oversize, the header says 16, the arm expects 15, and every shard carrying the gate went red — read as a libstdc++-vs-libc++ split only because the macOS Release leg had been cancelled. main is at 1,042,372 B and this branch at 1,048,058 B; the next append by anyone crosses. Three arms replace it, each an invariant of the tool: the JSON-lane rows at 1M are exactly the BIGJSON files; every other oversize row names a file really over 1,048,576 B on disk (a ceiling never drops a file under it); and skipped_oversize= equals the rows itemized. Verified on three trees: this branch (15 + 0), the tgrep lane merged onto it with EVALS.md at 1,065,642 B (15 + 1, was red), and the tgrep lane tip alone (15 + 0). A tree with no non-JSON file over 1 MiB passes the new arms exactly where the old one did, so nothing is weakened. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…aptured — two shard reds from the first CI run recallevalcheck pins `--for=Robot` over the whole repository to test/chafix/cha.cpp's Robot at rank 1; the new fixture had duplicated chafix's Animal/Dog/Cat/Robot/speak, so name-exact routing had two candidates and four shard-1 jobs went red. The fixture is now Creature/Hound/Lynx/Automaton/Lamp with vocalize(), the same shape under names no other gate pins; chaconecheck, recallevalcheck and manifestcheck pass. readmeexamplecheck pins README's `--callers=rankGraphTeleport` example rows to live paths; the ChaConeMemo insertion moved rankGraph to src/graph.h:3099 and anchoredLexicalRank to :3648, so the two rows are re-captured from the live command (two shard-2 jobs). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ter's own cone — arm 6, mutation-proven The memo cannot go stale in the invalidation sense: it lives inside one buildGraph call and its inputs (chaUp/chaDown) are immutable for that lifetime. The failure this class of change actually has is a hit that returns the WRONG hierarchy — a cached cone pointer, an index that goes wrong once the cone vector reallocates, a key collision. Arm 6 discriminates it: a third hierarchy (Machine, Droid : Machine) whose cone excludes Creature is filled AFTER Hound's entry exists, then Hound is asked for again; the answer must be Creature::vocalize and never Machine::vocalize. Mutating coneFor to return the newest cone on a hit (`&cones_.back()`) turns exactly that arm red (observed: count=1, the Machine line); the revert is green. The Lamp and parameter-receiver arms now see a 3-way split, which is the same DEGRADE and control under the larger fixture. docs/EVALS.md gains the correctness finding the per-line regex timing surfaced — `^#include` on llvm-project: 1,487 hits on main, 289,646 on the tgrep lane's line-anchored scan, and the per-line shape faster because it bounds `.*` to a line — stated with attribution to that lane's fix. The file is now 1,049,181 B, past the 1M ceiling; test/maxfilesizecheck.sh's derived arm passes on this tip with 15 JSON-lane + 1 ceiling rows, so the fix in the previous commit is exercised by the real tree, not only by a merged copy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…owing table Cone held `const std::vector<std::uint32_t>*` into cones_, a vector that grows on every first-time fill, so the handle was valid only until the next coneFor() — an invariant stated in a comment and enforced by nothing. The one call site used it immediately, so it was safe today; the next call site, or a hoist out of the loop, would have been a use-after-free in graph construction that ASan catches only when the reallocation happens to move the buffer. The handle is now the cone's index, resolved inside contains() on every membership test, so no handle can dangle for the memo's lifetime. Output unchanged: default map on go byte-identical to the previous build; chaconecheck, chacheck, callerscheck, nsfiltercheck, localitycheck green. README's --callers example rows re-captured for the two-line shift (the same line-number pin that fired on the first CI run; a documentation gate pinned to source line numbers is its own fragility, noted in the round's report). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ndle shift (the previous commit's edit did not apply) The re-capture in 8bea449 split the live rows on whitespace and matched nothing, so README kept the old line numbers and readmeexamplecheck stayed red; this is the same two-row re-capture, done from the live command and verified green before the commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…te — a Hash Sum mismatch there killed every Linux job At 17:51–17:56 UTC on 2026-09-09 every ubuntu-24.04 job on this repository — main's own run 34385457240, three lanes' runs, 18 jobs each — died in "Install tooling (Linux)" before compiling anything: Err:29 https://dl.google.com/linux/chrome-stable/deb stable/main amd64 Packages E: Failed to fetch .../binary-amd64/Packages.gz Hash Sum mismatch ##[error]Process completed with exit code 100. The ubuntu-24.04 runner image ships /etc/apt/sources.list.d/google-chrome.list; nothing here installs from it, but `apt-get update` exits 100 when any configured index fails, and apt.llvm.org's llvm.sh runs its own `apt-get update`, so the style job dies one line later at `clang-format --version`. Removing that source as the first line of each Linux install step (four sites: style, the release matrix, the fallback emitter, asan) makes the workflow independent of a mirror it never uses. No gate, no build flag, no version changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… match google-chrome* and list the directory The previous line removed google-chrome*.list and the Chrome index was still fetched on run 34386505159: the 24.04 image carries /etc/apt/sources.list.d/google-chrome.sources. Six of eight sampled jobs cleared the install step only because the mirror happened to answer consistently; five did not. The glob now matches both spellings, and the step prints the directory first so the next miss names itself in the log. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

coderabbitai · 2026-09-09T18:52:29Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 81edc9a3-7daf-4ec4-849e-4dcd048b53ce

📥 Commits

Reviewing files that changed from the base of the PR and between 597df08 and b7a1767.

📒 Files selected for processing (15)

.github/workflows/ci.yml
CHANGELOG.md
README.md
bench/PROFILE.md
docs/EVALS.md
present/deck5_ripwire_build.js
src/graph.h
src/ingest.cpp
src/ingest_parsepool.h
test/chaconecheck.sh
test/chaconefix/a.cpp
test/chaconefix/b.cpp
test/chaconefix/zoo.h
test/maxfilesizecheck.sh
test/regression.sh

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📝 Summary

Summary by CodeRabbit

Performance
- Improved graph-building performance for --grep, --callers, and default map generation, especially on warm runs.
- Preserved byte-identical default map output while reducing repeated inheritance analysis.
Bug Fixes
- Improved handling of ambiguous receiver-typed calls and inheritance resolution.
Documentation
- Updated performance measurements, examples, regression-gate counts, and release documentation.
Tests
- Added coverage for inheritance resolution, memoization behavior, determinism, skipped-file reporting, and an expanded regression gate suite.

Walkthrough

The graph resolver now memoizes CHA-lite inheritance cones. New gates validate cone behavior and performance-related documentation records the results. Profiling scopes cover graph and ingestion phases. CI, gate checks, and published gate counts are updated.

Changes

CHA memoization

Layer / File(s)	Summary
Memoized cone resolution `src/graph.h`	`ChaConeMemo` computes receiver-type cones once and uses binary-search membership during candidate filtering.
Cone fixtures and validation `test/chaconefix/*`, `test/chaconecheck.sh`	Fixtures and the gate test memo hits, distinct cones, degradation, memo growth, determinism, and the 4096-entry cap.
Performance records `CHANGELOG.md`, `bench/PROFILE.md`, `docs/EVALS.md`	Documentation records the warm-query investigation, measured improvements, and correctness checks.

Profiling coverage

Layer / File(s)	Summary
Graph phase instrumentation `src/graph.h`	Profiling scopes cover graph resolution and subsequent edge-building phases.
Ingestion phase instrumentation `src/ingest.cpp`, `src/ingest_parsepool.h`	Profiling scopes cover ingestion post-passes, fact merging, workers, and joining.

Maintenance updates

Layer / File(s)	Summary
CI and gate maintenance `.github/workflows/ci.yml`, `test/maxfilesizecheck.sh`, `test/regression.sh`	CI removes the Chrome APT source, skipped-file accounting is derived from rows, and `churnjoincheck` is included in the regression loop.
Published gate counts `README.md`, `present/deck5_ripwire_build.js`, `docs/EVALS.md`	Published gate counts change from 566 to 567, and README source references are refreshed.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GraphBuilder
  participant ChaConeMemo
  participant ResolveFilter
  GraphBuilder->>ChaConeMemo: Request cone for receiver type
  ChaConeMemo-->>GraphBuilder: Return cached or computed cone
  GraphBuilder->>ResolveFilter: Check candidate scope membership
  ResolveFilter-->>GraphBuilder: Keep or reject candidate

Suggested reviewers: andriytyurnikov

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/zealous-thompson-4b5f32

_{Comment @coderabbitai help to get the list of available commands.}

…three honesty stories, a --quality-delta examples slot The deck grows from 29 slides to 33. Every figure on the new slides is quoted, with its origin, in that slide's speaker notes. Everything here is merged on main 766913d; claims about redhat-et#135, redhat-et#136 and redhat-et#126 are in the next commit. - 0.6.0 at a glance: Dart, the 23rd grammar (6f91fed), and the six hand-sized language arrays its landing caught; Rip'n Fast (redhat-et#83 warm --grep 159.7 -> 9.2 s; redhat-et#127 llvm cold parse -19.9% CPU, go --pack-task -27.6%); honest where it counts (redhat-et#134 2,107 -> 3; redhat-et#127 per-kind --quality-delta 12/12 -> 8/12, --help-task 13/25 -> 0); fewer tokens, nothing hidden (redhat-et#92 --help 46,385 -> 4,473 tokens; redhat-et#127 --handoff 6 -> 50, listing verbs page). Footer: the x86-64-v3 upgrade note from redhat-et#127's body. - The scale rung: the O(C^2) child walks (redhat-et#127, redhat-et#130), the inheritance cone (redhat-et#83) and the cache that evicted itself (redhat-et#127), each with its llvm-project number; a per-corpus bar row from redhat-et#127's table. The title says "found at llvm-project scale", because only the child walk is sourced as invisible on the standard corpora. One callout lists what is built on or with LLVM. Each item was checked against a primary page, and the URL and quote are in the notes. XLA was dropped: no fetched openxla.org page says it is MLIR-based. The PS4 is the only console named, because only Sony's 2013 PS4 toolchain slides were read. - Honesty stories: std::move's 2,107 callers (redhat-et#134); the probe that lied (b4ebf0b corrects 150fb6d's reason: clang folded the one-add probe to a scalar addb, and the gate now disassembles its own probe); the audit's instrument was wrong twice (the redhat-et#127 brief, 209f97a, 5723b2c, redhat-et#127 lane H's 59/195 -> 64/151). - What --quality-delta catches: a marked PLACEHOLDER. qdExamples() renders QD_EXAMPLES, a data-only array at the top of the generator, as 1-3 wide or 4-6 two-column before/after cards in the mono face. The array is empty until docs/QUALITY_DELTA_CATALOG.md lands, and the helper refuses more than 6 entries or an empty field. The two drawing helpers (storyCards, qdExamples) take a slide their caller added. That keeps one literal addSlide call per slide, which is what test/deckclaimcheck.sh counts. A first cut added the slide inside the helper: the count still read 33, but only by coincidence, and a second call to the helper would have drifted from the pages while the gate stayed green. The cover and its wave are 680a0a3's, unchanged apart from notes that source its 23-grammar count. The slide count moves in README.md (two places), present/README.md and the re-derive row. The pptx and PDF are regenerated (pptxgenjs 4.0.1; LibreOffice 26.2 impress_pdf_Export; 33 pages). Gates on this tree with a main-tip binary (built_from=766913d02): deckcheck ALL PASS (0 bad values, 0 stale), deckclaimcheck ALL PASS (179 long flags, 33 slides), readmedriftcheck ALL PASS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

joyful-ii-V-I and others added 8 commits September 9, 2026 13:52

joyful-ii-V-I merged commit 599684c into main Sep 9, 2026
52 of 53 checks passed

Navigation Menu

Uh oh!

perf(graph): memoise the CHA-lite inheritance cone — warm llvm --grep 159.7s to 9.2s, default map 248s to 10s - #83

perf(graph): memoise the CHA-lite inheritance cone — warm llvm --grep 159.7s to 9.2s, default map 248s to 10s#83
joyful-ii-V-I merged 8 commits into
mainfrom
claude/zealous-thompson-4b5f32

joyful-ii-V-I commented Sep 9, 2026

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 9, 2026 •

edited

Loading

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

joyful-ii-V-I commented Sep 9, 2026

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Sep 9, 2026 •

edited

Loading