harvest(tgrep): --regex anchors are LINE anchors — 1,487 hits become 289,646 on llvm-project, and std::regex::multiline faults in libc++ by joyful-ii-V-I · Pull Request #85 · redhat-et/ripwire

joyful-ii-V-I · 2026-09-09T20:34:44Z

Harvest round, lane A (tgrep). Three things: a correctness fix to --regex, a crash removed rather than worked around, and the head-to-head that prompted both.

--regex anchors are LINE anchors now. Handing the whole file to one sregex_iterator made ECMAScript's ^ match only at offset 0 and $ only at end-of-file — in a verb whose every answer is a single line of CDATA. Measured on llvm-project: --regex='^#include' returned 1,487 hits where the line-oriented scan returns 289,646. That is 288,159 matches silently missing from a confidently-reported count, and it is the reason this lane exists. grep, rg, tgrep and every editor's find box read those anchors per line; so does this now.

It is also faster — 3.40 s against 4.48 s warm on that corpus — which reads as a measurement error until you see the mechanism: the per-line bound stops .* running across lines, so it does strictly less work per match attempt.

Disclosed narrowing, because it is real: a match may no longer span lines. . never could (ECMAScript's dot excludes line terminators), but [\s\S]* could and now cannot — the same line-oriented contract rg's default has. A trailing \r sits outside every line's range, so $ behaves on CRLF input the way rg --crlf does rather than never matching.

std::regex::multiline is the obvious fix and it is unusable. Apple libc++'s __l_anchor_multiline<char>::__exec reads *std::prev(__s.__current_) before testing whether the match position is the first character — so at offset 0 it reads one byte before the buffer. Content-dependent, because a byte before a heap buffer is usually readable, which is why it presented as 8 crashes in 10 runs and five nondeterministic shard reds rather than one clean failure. A 40-line standalone with no ripwire code in it faults on 74 of this repository's ~130 headers (Apple clang 21.0.0, macOS 26.5.1). A gate cannot defend against a standard-library out-of-bounds read; not using the node can. The tree passes std::regex::multiline nowhere, and test/grepanchorcheck.sh arm I is the crash regression — green on the plain build and on both ASan legs, which is what proves it on libstdc++ as well as libc++.

A second crash class, also removed: std::regex_error (error_complexity/error_space) thrown mid-scan by catastrophic backtracking. Only regex construction was guarded before, so an uncaught throw during matching reached std::terminate and killed the entire run over one bad file. It now degrades — keeps that file's hits and moves on.

corpus_pruned_dirs= on the grep root, and the tgrep head-to-head: Q* priced on five rungs to llvm-project at 182,555 files, with both halves of the resident margin refuted.

The top rung's ripwire-warm column was superseded the same day and says so. #83's ChaConeMemo took warm --grep on that corpus from 159.7 s to 9.2 s, so the 176–233 s column this lane measured is no longer what the tool does. The section cites the post-fix re-timing (8.99–9.23 s) with attribution rather than copying it into the measured table — the table records what this harness measured on that binary — and re-derives Q* ≈ 1.7 from its own formula. The harness's own re-run of the llvm rung on the merged tree is registered as owed.

Squash-merged rather than merged: d1aac586, f48a0e04, 5743fa37 and 8732d612 carry the private validation corpus's name (135/143/1/143 references) behind a405fcd5's forward scrub. The tree is clean — canyonraid 0, qgames 0 — but the history is not, so squashing keeps it off main's history and the lane ref is deleted with it.

Verified before landing: tip contains current main; CI run 34392166925 26/26 green on the exact tip; gate count re-derived on the tip from regression.sh's own loop (567 → 568) rather than carried — main and this lane had both independently and correctly held 567, which merges clean into a wrong number and is precisely what re-derivation exists to catch; printf_parity.manifest diff exactly one row with the live ripwire --help | sha256 on a --clean-first build matching the pinned value; qschemetripcheck green so untouched.

🤖 Generated with Claude Code

--regex hands a whole file's bytes to ONE std::sregex_iterator built with ECMAScript|optimize, so `^` matched only at offset 0 of that buffer and `$` only at its very end: the anchors were FILE anchors in a verb whose answer is LINE-shaped. Measured before the fix on this repository's own src/ at 4c10be9: `--regex='^#include'` reported hits="1" where `rg -n '^#include' src` reported 1648; on canyonraid48, 26 against 6171. Not a zero (which the legend governs) — a confident 0.06% of the truth, in the one syntax nobody re-reads the manual for. test/regexcheck.sh has carried `'^int '` in its battery since it was written, commented "an anchored line start" — and its independent `grep -lE` oracle arm runs a SHORTER pattern list that omits exactly that pattern, so the divergence sat inside the suite's blind spot rather than outside its scope. This gate compares (file,line) hit sets against `grep -nE` EXACTLY (an anchor that over-matches is as wrong as one that under-matches), re-checks prefilter soundness with an anchor in the pattern, and pins that `.` still does not cross a newline — with a [\s\S]* mutation control proving that arm is not vacuous. RED against 4c10be9: 6 of 8 assertions fail, all three mutation controls fire. Gate count 563→564. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…one shared syntax constant Turns test/grepanchorcheck.sh green: `--regex='^#include'` over src/ now reports 1648, byte-for-byte the count `rg -n '^#include' src` reports. The flag set was spelled out three times in this file (the compile probe, the indexed-file worker, the unindexed-aux scan); a pattern that COMPILES under one set and MATCHES under another is a defect with no symptom, so the three sites now share kGrepRegexSyntax and cannot drift apart. Nothing else moves. ECMAScript's `.` excludes line terminators with or without multiline, so a `.*` still cannot cross a newline (arm C, with its own mutation control). The Russ-Cox prefilter treats an anchor as ε (riAnchor), which is sound under either reading, so no candidate set narrows on one — arm E re-checks prefiltered == full-scan with an anchor in the pattern. libc++ verified directly before the change; CI carries the libstdc++ leg. Found by the tgrep head-to-head (bench/tgrep-h2h/): tgrep and rg agreed on the hit set for 15 of 16 frozen queries at every rung of the corpus ladder, and every disagreement with ripwire on the regex side was this one anchor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…btraction --grep never named --grep already names two of the three ways a file leaves the corpus: corpus_excluded= for an --exclude= hit, corpus_oversize= for the size ceiling. Neither fires for rw::kCrawlSkipDirs — vendor, third_party, build, dist, out, target, node_modules, captures — which prunes those subtrees WHOLE and increments a directory counter that only the skipped verb ever reported. So a grep answer could carry complete="1" over a corpus that had silently lost entire trees. Measured on this repository at 4c10be9: `--grep='malloc('` served 33 hits with complete="1" where `rg -F 'malloc(' .` found 78 matching lines. The 45 missing are exactly the lines under third_party/ — 58% of the truth, behind a completeness claim. A DIRECTORY count is the honest cheap unit: files under a pruned subtree are never stat'd, so a file count would cost a second walk to report a number nothing else needs. Same convention as its two siblings (present only when non-zero, absent means zero), same spelling the skipped verb already uses, and the same condition in the MCP dialect so the two cannot disagree about what was searched. The legend's own claim that corpus_excluded= covers the "built-in crawl policy" was false and is corrected; the complete= clause now names the largest subtraction on the root itself instead of deferring all of it to another verb. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… rungs of the corpus ladder Prices the question P3 did NOT answer. P3 (2026-07-27) removed a PER-INVOCATION trigram index because building it cost more than the one scan it replaced; that verdict stands and src/search.h states it. tgrep's index is RESIDENT — built once, held by a server, reused every query — so this harness asks Q* = B / (q_scan - q_index): how many queries into a session a persisted index has paid for itself. Five arms, each with its delivery posture written down, because a comparison that mixes them is not a comparison: ripwire cold (fresh TMPDIR) and warm, ripwire warm with --limit (the only arm whose emission is comparable to rg's, and the one the agreement matrix reads), tgrep against a resident server, tgrep --no-index (the control that separates "the index helps" from "the Rust scanner is faster"), and rg --sort path as the floor. 16 queries frozen before any arm was timed, with each one's selectivity and purpose recorded next to it. Raw output is written OUTSIDE the checkout: an untracked file anywhere in this tree makes `git status --porcelain` dirty, and every stamped verb reads that command for the `+dirty` half of its at= anchor, so a raw/ beside this script would flip every determinism arm running in parallel with the harness. results.json is scrubbed of absolute paths on write. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…he losses first The crossover, corrected: P3 priced a PER-INVOCATION index and its verdict stands; this prices a RESIDENT one. Q* = B / (q_scan - q_index) is 10 queries against rg and 2 against ripwire-warm at 2,240 files, and 5 / 1 at 15,865 — against a median of 26 grep-class commands per grepping session, measured from the substitution meter's own class field (14,872 events, 96 of 522 sessions, p75 198). The crossover has already flipped at the smallest realistic repository size. Losses before wins, and both fixed ones are in this round's commits. The third bucket and the opposite-direction gitignore defect are recorded unfixed with their locations, as is the part of tgrep's advantage that a one-shot CLI cannot have: on a pattern whose plan tgrep itself reports as MatchAll, its server still beats rg 15x, which is a resident content cache and not the index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…reign flags out of prose Category (b) of the allowlist's own header: a different tool's flag quoted verbatim inside a worked example. --stats and --no-index name the two tgrep arms whose output IS the evidence in the head-to- head section, so precision is worth the row. --release and --files were not: cargo's build profile and ripgrep's file listing read the same in prose, and a generic token like --files in the allowlist would mask a fabricated ripwire flag of that name forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The pre/post agreement counts alone read as "nothing improved" on two rungs. What moved is the scale of the disagreement, not always its existence: ^#include went 29 -> 2,149 of rg's 2,828 on this tree (every residual under third_party/) and 26 -> 6,171 against rg's 6,171 on canyonraid48, where what is left is a +-4 symmetric difference from two different, named classes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…:multiline is unusable here The previous commit reached for std::regex::multiline. That flag is the obvious repair and it CRASHES. Apple libc++'s __l_anchor_multiline<char>::__exec reads *std::prev(__s.__current_) before it tests whether the position IS the first character, so at offset 0 it reads one byte BEFORE the buffer. Measured: `--regex='^'` over src/ faulted 8 of 10 runs (EXC_BAD_ACCESS one byte low, SIGBUS when the string's buffer starts a page), and a 40-line standalone with no ripwire code — default-constructed regex, assigned, sregex_iterator per file, single-threaded — faulted on 74 of this repository's ~130 headers. Content-dependent, because a byte before a heap buffer is usually readable, which is why the gate suite saw it as five nondeterministic shard failures (CI 34357046881) rather than one red arm. So the anchors are line anchors for a better reason: grepScanText now searches each LINE as its own range, which is what grep, rg and tgrep do and what this verb's line-shaped answer always implied. `--regex='^#include'` over src/ returns 1648, exactly rg's count, with no standard-library node that can read out of bounds. A trailing \r sits outside the range, so `$` behaves as rg's --crlf does. The narrowing is real and is stated in the code: a match may no longer SPAN lines. `.` never could (ECMAScript's dot excludes line terminators); `[\s\S]*` could and now cannot — the same line-oriented contract rg has, where crossing lines is an opt-in mode this verb does not offer. Gate: test/grepanchorcheck.sh arm C now asserts the contract in BOTH directions with a WITHIN-line control (the same two patterns must match when the halves share a line, so the arm measures the line boundary and not a pattern that could never match). New arm I is the crash regression itself — ten bare-`^` runs over real source, plus an unanchored control so a red I means the anchor and not the machine. It is the arm that goes red the day someone reintroduces the flag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…d its EVALS section test/ripwirepubliccheck.sh arm 1 is zero-tolerance about the private development tree's name and CI run 34357046881 caught it in the committed doc; the harness then added 141 more places, because results.json records the corpus key and every argv. The rung is now `privcpp` and every path in results.json is a $RIPWIRE_TREE / $PRIV_CORPUS / $RW_H2H_HOME placeholder; EVALS names it the way the other private-corpus entries in that file already do ("a private C++/ObjC++ tree, named in the local ledger"). The harness README now says the scrub is a precondition of committing a re-run, since the one thing a future re-run reliably regenerates is the leak. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…nerated from The anchors changed meaning; --help is the flag catalog and docs/COMMANDS.md is generated from it, so a semantic that only lived in a source comment would reach nobody. Two clauses on the --grep/--regex entry: each line is its own search range, `^` and `$` are line anchors, no match spans a newline, and a trailing CR sits outside the range. COMMANDS.md regenerated from the same committed capture, so the docscommandscheck regeneration-parity arm stays byte-exact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…egin an empty one The line-oriented scan walked one segment too far: after consuming the final `\n` it searched the empty range at end-of-file, so a zero-width pattern gained one PHANTOM hit per file, on a line number no reader could open. An empty file had the same defect with no lines at all. Both are now the rule grep states: `grep -c '^'` counts one per REAL line and zero in an empty file, and ripwire's hit count now equals it on all four shapes — trailing newline, no trailing newline, empty, and one lone newline. Not caught by arm D, because none of its anchored patterns match the empty string; found by reading the loop's edges against grep rather than against the fixture. New arm J is that comparison, run against `grep -c '^'` itself so the oracle is not this gate's own arithmetic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…h is part of the finding The section still credited std::regex::multiline, which is the fix that had to be withdrawn. Records what actually shipped and, more usefully, why the obvious repair is not available: a libc++ out-of-bounds read in the multiline anchor node, its measurement, and the two facts that make it worth writing down — it surfaced as five NONDETERMINISTIC gate-suite shard failures rather than one red arm, and twenty targeted plain-build gates passed on the crashing binary while one ASan run on the touched command would have named it. Also records the second defect the rewrite's first cut had (a phantom hit per file from the empty range after the final newline) and the gate arm that pins it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ting them The section claimed tgrep and rg agreed "on every query at every rung". True of the nine queries the frozen set declares as the agreement subset; NOT true of three `go` cells outside it, and an unqualified claim there is the kind of overclaim this file exists not to make. Both causes traced, and `tgrep --no-index` reproduces both, so neither is the trigram index: R2 misses four lines in a `.dat` file because `.dat` is one of the ~65 extensions tgrep's walker rejects as binary before reading it (tgrep-core/src/walker.rs), which ripgrep does not do; L4 and L5 differ by a byte per line on CRLF files because tgrep always strips a trailing CR. Both are in tgrep's README and neither is disclosed on the ANSWER — which is tgrep's mirror image of ripwire's own unindexed-extension bucket, and sharpens rather than softens the disclosure comparison below it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…t nothing — measured The section proposed a persisted postings index as the portable half of tgrep's advantage and had the cost ratios to show it is affordable. One experiment refutes the value: time a verb that builds the same graph and scans NO text (--callers=main) against --grep on the same warm cache, interleaved. 0.09 s vs 0.09 s at 2,240 files; ~0.62 s vs ~0.65 s at 15,865. startGrepScanPrefetch already runs the scan on its own thread concurrent with the graph build, so the warm per-query cost is max(ingest, scan) and the ingest wins at every rung. On llvm-project the absent LITERAL costs 176.4 s and the prefilter-defeating zero-hit REGEX — a full std::regex verification of 2.9 GB — costs 171.2 s: two completely different scan workloads, the same wall time. So P3's removal note is right for a second reason it does not state: not only is the build more work than the one scan it saves, the scan is already free behind the ingest. What the numbers point at is the warm-ingest floor (~171 s at 182,555 files, paid to annotate at most 100 printed hits with in=), and the one experiment that would decide whether that floor is the graph or the cache load is named rather than guessed at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…us/file at 183k The floor the previous commit identified is flat per file up to 15,865 files (40.2 us at 2,240, 39.1 us at 15,865) and then 937.8 us at 182,555: the corpus grew 11.5x and the floor grew 276x, a 24x per-file regression on a tree whose cold peak RSS is 6.45 GB. Recorded because nothing in the gate suite exercises a corpus large enough to see it, and because it turns "the ingest floor is the lever" from a design opinion into a bug report with a shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ng was measured "One machine, shared" understates it. At the end of the llvm rung the 18-core host was at a 1-minute load average of 38.8 with 62 concurrent ripwire processes belonging to other work; the earlier cells were taken under materially lighter load. The conclusions are all ratios or same-arm comparisons taken minutes apart, so none of them moves — but a single llvm absolute is an order of magnitude and not a precise figure, and the limits paragraph now says so instead of implying a controlled machine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ne number "returns exactly 1,648" was one pattern and could be a coincidence of that pattern. The hit SET now equals `rg -n`'s exactly on five anchored shapes over this tree's src/: ^#include 1,648, ^int 18, ^\s*// 41,005, h>$ 42 and ^$ 9,936 — the last two being the zero-width cases the trailing-newline rule decides, which is where the rewrite's own first cut was wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… one query The top rung lands. ripwire cold 252.7 s / 6.01 GB peak / 542 MB of cache; tgrep's index builds in 11.1 s / 541 MB peak / 1,037 MB on disk — 0.04x of ripwire's own cold ingest and 1.91x of its cache blob. Per query, warm: ripwire 176-233 s against tgrep's 0.011-11.2 s and rg's 4.1-13.5 s. Q* is 2.5 queries against rg and 0.1 against ripwire-warm, i.e. a resident index would have paid for itself before the first --grep on that tree finished. Two things the readout now does honestly that it did not: the ripwire arms ran a DECLARED six-query subset on this rung, so Q* vs ripwire-warm is computed against tgrep's mean over those same six rather than over all sixteen (two different workloads), and the table prints which set each column used. run.py's scrub now covers every corpus ROOT from the ladder, longest-first, because this run proved that a re-run regenerates the private tree's name into results.json even after the file was cleaned. Also recorded: L4 `int` is the one cell where rg beats tgrep (6.90 s against 10.43 s, 563 MB of output), which is the failure mode tgrep's own BENCHMARKS.md names — their model predicting their own losing cell is the reason to trust the rest of their table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…named Two things a re-runner would otherwise rediscover the hard way. The scrub: results.json records the corpus key and every argv, so a re-run regenerates the private tree's name even into a file that was cleaned — run.py now scrubs every ladder root to a $CORPUS_<NAME> placeholder on write, and the README says to check it rather than assume it. The load: "one machine, shared" understated a 1-minute load average of 38.8 with 62 concurrent ripwire processes at the end of the top rung, so the limits paragraph now says a single llvm absolute is an order of magnitude and not a precise figure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…arden's --doctor paragraph plus tgrep's line-oriented --regex clause The lane's own re-pin (7053904) dropped out of the replay by design: its hash was computed against a --help that lacked githarden's paragraph, and the batch's hash lacked tgrep's clause. Regenerated from the --clean-first build at 7653cc45; exactly one row moved against the batch's manifest (help stdout), the stderr digest and the other 11 rows are byte-identical, and the live `ripwire --help | sha256` on this binary equals the pinned value. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…the same day — cite the post-fix numbers, re-derive Q* The profiling lane (bc38d41, ChaConeMemo) answered this section's own "one experiment away": the super-linear warm floor was the CHA-lite cone rebuilt per call, and warm llvm --grep went 159.7 s -> 9.2 s. Its re-timing of this harness's six frozen queries (8.99-9.23 s) is CITED with attribution, not copied into the measured table; by this section's own formula Q* against ripwire-warm at 182,555 files is ~1.7, not 0.1, and the sentence says so. This harness's own re-run of the llvm rung on the merged tree is registered as owed. The measurement stands as the record of what was measured; the conclusion it pointed at is now named and removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…568, never carried Replayed onto 599684c (the ChaConeMemo lane, chaconecheck, 566 -> 567). The eight published sites merged CLEANLY at 567 from both sides — the silent form of the collision, caught by re-derivation — and were set once to the number test/manifestcheck.sh derives from the tip's own absorb line, which now carries hermesinstallcheck, githardencheck, textdocscheck, chaconecheck and grepanchorcheck. The lane's own maxfilesizecheck repair was dropped before this replay in favour of 709159e underneath. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

coderabbitai · 2026-09-09T20:35:07Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: f9dc1e20-cefe-40ee-91a1-0d9b105f926b

📥 Commits

Reviewing files that changed from the base of the PR and between 599684c and c610e77.

📒 Files selected for processing (18)

README.md
bench/tgrep-h2h/README.md
bench/tgrep-h2h/arms.py
bench/tgrep-h2h/queries.json
bench/tgrep-h2h/readout.py
bench/tgrep-h2h/results.json
bench/tgrep-h2h/run.py
docs/COMMANDS.md
docs/EVALS.md
present/deck5_ripwire_build.js
src/cli.h
src/mcpverbs.h
src/search.h
src/verbs_grep.h
test/deckcheck_allowlist.txt
test/grepanchorcheck.sh
test/printf_parity.manifest
test/regression.sh

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📝 Summary

Summary by CodeRabbit

Bug Fixes
- Regex searches now treat each line independently: ^ and $ anchor lines, matches cannot cross newlines, and trailing carriage returns are excluded.
- Search reports now identify directories omitted by the built-in crawl policy.
Documentation
- Clarified regex behavior in help and command documentation.
- Added head-to-head benchmarking documentation comparing search performance and index trade-offs.
- Updated test-suite references from 567 to 568 gate scripts.
Tests
- Added coverage for line-oriented regex anchors, deterministic results, output formats, and edge cases.

Walkthrough

The change makes regex matching line-oriented, reports directories removed by the crawl denylist, adds tests for anchor behavior, and introduces a ripwire/tgrep/ripgrep benchmark harness with reporting and documentation updates.

Changes

Grep semantics and reporting

Layer / File(s)	Summary
Line-oriented regex matching `src/search.h`, `src/cli.h`, `docs/COMMANDS.md`, `test/grepanchorcheck.sh`, `test/printf_parity.manifest`	Regex matching now processes each line independently. Anchors apply per line, matches cannot cross newlines, and tests cover output, crashes, prefiltering, and line-edge cases.
Crawl-prune reporting `src/verbs_grep.h`, `src/mcpverbs.h`	CLI and JSON output now report whole-subtree pruning through `corpus_pruned_dirs`.
Gate-count consistency `README.md`, `docs/EVALS.md`, `present/deck5_ripwire_build.js`, `test/regression.sh`	References now use 568 gate scripts. The regression-loop edit is whitespace-only.

Benchmark evaluation

Layer / File(s)	Summary
tgrep comparison harness `bench/tgrep-h2h/*`, `docs/EVALS.md`, `test/deckcheck_allowlist.txt`	The harness defines frozen queries, runs ripwire, tgrep, and ripgrep across corpus sizes, records scrubbed results, and reports timing, Q*, agreement, and index-cost comparisons.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant run.py
  participant tgrep
  participant ripwire
  participant rg
  run.py->>tgrep: Build index and start server
  run.py->>ripwire: Run cold and warm queries
  run.py->>tgrep: Run indexed and no-index queries
  run.py->>rg: Run sorted baseline queries
  run.py->>run.py: Write scrubbed results.json

Suggested reviewers: quaterniondrift, andriytyurnikov

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch lane/harvest-tgrep-2026-09-09

_{Comment @coderabbitai help to get the list of available commands.}

…— disclose it `ripwire . --for=Q --detail=30 --max-tokens=300` printed `max_tokens="300"` on the <ctx> root and delivered `est_tokens="2640"` — 8.8x the ceiling it named — with no over_ceiling= anywhere. Reported by YogevKr as #61, reproduced verbatim on 2026-09-09, the third of three honesty-class issues from that reporter. The flag was never inert: it shapes the body count, and `<bodies capped="1">` disclosed THAT cut honestly right beside the silence. What --max-tokens does not do is bound the document — verbs_for.h turns it into detailBodyBudget, a budget over the BODIES alone, so the header, signatures, legend and symbol table are never charged against it. METHODOLOGY §9 #6 states the defect in one sentence: "a ceiling attribute names the ceiling actually applied." DISCLOSURE, NOT ENFORCEMENT — and the argument, because this is a §9 decision. §9 #2: "when a ceiling would cut something above the cliff, compress first, move prose into attributes second, and if it still does not fit, exceed the ceiling with over_ceiling="1" rather than drop the row that would have terminated the search." Thirty small functions totalling ~3.4K tokens, complete, ARE the terminating answer. A rung that trimmed them to fit 300 tokens would make the tool worse and would still have been perfectly honest, so no rung was added. The ladder was applied first and came up empty on this shape: the reporter measured --legend=compact at 8,761 B / est 2,974, still far past 300. What was missing is the VERDICT the default map has computed since §F5 (main.cpp, maxTokensFit.isOverCeiling) — the "as the default map does" the reporter's own Expected behavior cites. THE OPEN QUESTION, decided explicitly: --max-tokens is NOT converged onto the whole bundle. The tool already has a flag that means "bound the document" (--token-budget, the reporter's verified workaround: est 902 at --token-budget=1000) and one that means "shape the map" (--max-tokens). Convergence is allowed under one-step-smart-defaults and new-tool-no-compat-debt, but it is a DEFAULT change whose effect is to CUT rows, which is the direction §9 #1 says the data does not support — the budget flag "trims a ranking from the tail and cannot know which row would have ended the search". The two meanings stay, and both are now documented on --max-tokens and --detail in --help. WHAT CHANGED - src/verbs_for.h forLensOverCeiling: the XML dialect's over-ceiling predicate, a free function beside its JSON twin (forLensJsonOverCeiling) for the same reason that one is — runForLens is one of the largest bodies in the file and this is a contract of its own. Same rule, same unit, same attribute budget_tokens already answers to (packtask.h F2): over_ceiling="1" whenever est_tokens exceeds a ceiling the root states. Reused, not re-derived. The label is decided INSIDE the existing est_tokens fixpoint, so its own 17 bytes and its legend clause are charged — a disclosure that made est_tokens wrong is the one place that error matters most. - src/serialize.h: the legend sentence for a max_tokens-keyed verdict, beside the budget_tokens one, plus overCeilingLegendFor so no surface picks the wording by hand. Keyed on which ceilings the ROOT CARRIES, not on which one fired, so the choice cannot be made stale by the fixpoint it rides inside. A budget-only document is byte-identical to before. - src/cli.h / docs/COMMANDS.md (regenerated): --max-tokens and --detail=N now state what the flag bounds, what it does not, and which flag bounds the document. MEASURED, this repo's fixture of 30 tiny TS functions: --max-tokens=300 goes 9,647 -> 9,714 B (the 67-byte disclosure) and now reads `max_tokens="300" est_tokens="3589" over_ceiling="1"`; --max-tokens=8000 fits and is byte-identical at 14,968 B with no attribute. GATE FIRST (non-negotiable #1). test/formaxtokenscheck.sh, written before the code and red on the pre-fix binary at 6 of 9 band points plus the named reproduction. Arms: (A) the biconditional est_tokens > ceiling <=> over_ceiling="1", swept across a band that provably contains BOTH states, with non-vacuity asserted on each half; (B) no --max-tokens => no ceiling attribute and no verdict; (C) est_tokens re-derived exactly from the delivered bytes at both rates, at every point, which is what catches an emitted-but-uncharged disclosure; (D) the legend defines the attribute against the ceiling actually on that root; (E) --token-budget alone and beside --max-tokens; (F) determinism; (G) five mutation controls, each re-running the SAME judge over a deliberately corrupted real document. Presence tests read the root element through an XML parser, never a text grep — the legend DEFINES over_ceiling= (verbs_for.h:719). test/shapingflagcheck.sh (A) re-pinned 20 -> 21 --max-tokens read sites: the new site is a DISCLOSURE of a budget --for --detail=N already honored, so kShapingVerbs' honorsMaxTokens column is unchanged. Gate count 568 -> 569 in all 8 published sites, DERIVED from regression.sh's own loop on this tip. Worth recording how that number was nearly wrong: the previous revision of this branch published 568, and the lane that landed underneath it (#85, tgrep) had itself bumped 567 -> 568. So README.md, docs/EVALS.md and the deck did NOT conflict on rebase — the two lanes had written IDENTICAL text — and git auto-merged them to a tree publishing 568 while this branch's own loop names 569. That is the silent-merge failure this repo keeps re-learning, and the only thing that catches it is re-deriving from the loop rather than trusting a clean merge. A gate-count bump that merges CLEAN onto a main which has landed a gate since you branched is the failure, not the success; regression.sh conflicted loudly and the three prose sites did not. test/printf_parity.manifest: the `help` hash re-pinned, 72b76cbc... -> d9b77634..., because this commit edits --help on purpose. printffmtparitycheck is a byte-parity fence over 12 labels and `help` is one of them; its FAIL text ("any file whose conversion moved these bytes must be reverted") is written for the printf -> std::print conversion case, where the whole claim is that the bytes must NOT move. There is no conversion here — the two new --help blocks are the change, so the bytes moved deliberately and the pin is what needs updating, not the prose. Reviewed rather than rubber-stamped: the manifest diff is EXACTLY ONE LINE, `help` STDOUT; the other 11 labels are byte-identical and help's own STDERR hash is unchanged (still e3b0c442..., the empty-string digest). The baseline it moves FROM is main's, re-read on this rebase tip rather than carried from the branch's pre-rebase value — a hash computed against an older main would pin bytes no binary in this history produces. Pre-flighted against the WIDENED fence lane/stdprint-conversion brings (40 labels, purely additive — it rewrites none of the 12): 39 pass, 1 fail, and the one is `help`. So this change moves exactly one label out of forty, and the one-line invariant survives that landing whichever order the two lanes take. --test-gate does not name printffmtparitycheck for a help edit: it routes by call edges, and script-to-binary is not one — it discloses that as script_gates_unmodelled= rather than implying the list is complete. If you touch printUsage, run that gate by hand. Local, on this rebase tip after a --clean-first rebuild (five landings under this branch, and an incremental build across a branch switch can produce a binary that exists at no single commit — CLAUDE.md documents that failure at length): formaxtokenscheck, printffmtparity, manifest, shapingflag, docscommands, deck, deckclaim, readmedrift, readmeexample, fordisclosure, w3fixbudget and legendcoverage all green; determinism + golden + xmllint clean; --quality-delta exit 0 (its one gating row, short-horizon-churn churn=self on runForLens, acked with its reason). Every pinned number here was RE-DERIVED on this tip — the gate count from regression.sh's own loop, the read-site count from the gate's own grep expression, the parity hash from the rebuilt binary — none carried forward from the pre-rebase branch. docs/COMMANDS.md is regenerated from that same binary. Closes #61 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

joyful-ii-V-I and others added 22 commits September 9, 2026 14:53

joyful-ii-V-I merged commit a297317 into main Sep 9, 2026
77 of 79 checks passed

joyful-ii-V-I deleted the lane/harvest-tgrep-2026-09-09 branch September 9, 2026 20:34

Navigation Menu

Uh oh!

harvest(tgrep): --regex anchors are LINE anchors — 1,487 hits become 289,646 on llvm-project, and std::regex::multiline faults in libc++ - #85

harvest(tgrep): --regex anchors are LINE anchors — 1,487 hits become 289,646 on llvm-project, and std::regex::multiline faults in libc++#85
joyful-ii-V-I merged 22 commits into
mainfrom
lane/harvest-tgrep-2026-09-09

joyful-ii-V-I commented Sep 9, 2026

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 9, 2026 •

edited

Loading

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

joyful-ii-V-I commented Sep 9, 2026

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Sep 9, 2026 •

edited

Loading