fix(crawl): --grep no longer scans gitignored files the indexer skips — the leak was in the crawl, not the verb

README.md

@@ -1800,9 +1800,9 @@ wrong, and it has. These are the results that say so, all in-tree, all published
 ### In the tests
 
 <details>
-<summary><b>569 gate scripts</b>, five contracts no unit test can hold, and the house rule: write the gate before the code it measures</summary>
+<summary><b>570 gate scripts</b>, five contracts no unit test can hold, and the house rule: write the gate before the code it measures</summary>
 
-`test/regression.sh` names **569 gate scripts** and is the authoritative list;
+`test/regression.sh` names **570 gate scripts** and is the authoritative list;
 `python3 test/pargates.py . ./build/ripwire -j 6` runs the same set in parallel. On top of them sit the
 contracts that do not fit a unit test: two runs byte-identical, warm output identical to cold, output
 that pipes clean through `xmllint --noout`, a sanitizer build with `-fno-sanitize-recover=all`, and a

docs/ARCHITECTURE.md

@@ -46,12 +46,15 @@ runs. The parse itself runs one tree-sitter parser per worker thread and merges
 lists afterwards, which is safe precisely because the definitions and references are re-sorted before
 they are used — collection order never reaches the output.
 
-**`.gitignore` is not consulted.** Skipping is a fixed, committed denylist (`kCrawlSkipDirs[]` in
-`src/ingest.h`, shared with the CMake walk in `darkflags.h` so the two crawlers cannot disagree about
-what counts as source), not a per-repository ignore file. That is a real difference from a
-`.gitignore`-aware tool in both directions: a build directory this repository happens not to ignore is
-still pruned, and a directory a project ignores but that is not on the list is still indexed. What is
-skipped:
+**`.gitignore` is consulted, after the denylist.** Skipping starts from a fixed, committed denylist
+(`kCrawlSkipDirs[]` in `src/ingest.h`, shared with the CMake walk in `darkflags.h` so the two crawlers
+cannot disagree about what counts as source). In a git work tree the crawl then also honours git's own
+ignore verdict — one `git ls-files --others --ignored --exclude-standard --directory` fork per root, so
+the answer is git's and never a re-implemented matcher — and `--no-ignore` turns that half off. The
+denylist still prunes a build directory the repository happens not to ignore; what the repository
+ignores leaves the map and is disclosed as `ignored_files=` / `ignored_dirs=`, and the `--grep`
+unindexed scan reads none of it either (a gitignored file of an unindexed extension is in no class at
+all, the same treatment an `--exclude`'d one gets). What the denylist skips:
 
 - **directories by NAME:** `.git`, `.claude`, `.hg`, `.svn`, `node_modules`, `vendor`, `third_party`,
   `.cache`, `build`, `dist`, `out`, `target`, `.venv`, `venv`, `__pycache__`, `.idea`, `.vscode`,

docs/EVALS.md

@@ -21,7 +21,7 @@ section, and it is not an afterthought.
 | **Co-change / known-item evals** | `--eval`, `--eval-retrieval` (see `bench/ANSWERQUALITY.md`) | Whether the tool surfaces the other files a real historical commit touched; and known-item retrieval across four rankers. |
 | **Ensemble calibration harness** | `bench/ensemblecal/` | Whether `--ensemble`'s four evidence families are actually orthogonal, how often each fires, how stable each is across commits — and the preset ladder derived from that (§9). |
 | **Differential argv harness** | `test/argvdiffcheck.sh` | That a refactor changed *nothing observable*: two binaries, every argv vector, stdout + stderr + exit code byte-identical. |
-| **The gate suite** | `test/regression.sh`, `test/pargates.py` | 569 gate scripts plus the determinism, cache-transparency and golden contracts. |
+| **The gate suite** | `test/regression.sh`, `test/pargates.py` | 570 gate scripts plus the determinism, cache-transparency and golden contracts. |
 | **`--quality-delta`** | `src/quality.h` | Ten measured code-quality failure modes, reported only where a change made them worse. |
 
 ### The labeling protocol (why the held-out eval is allowed to disagree with the ranker)
@@ -5579,7 +5579,7 @@ copy here would be exactly the dialect divergence that gate exists to catch. Com
 tags, wrap, stable-order defaults), seven individually invoked standalone gates (`g1freshcheck`,
 `skillscan`, `htmlexport`, `compresscheck`, `handoffcheck`, `releaseinstallcheck`,
 `taskroutecheck`), and a single loop
-naming **569 gate scripts**, all of which exist on disk.
+naming **570 gate scripts**, all of which exist on disk.
 
 `python3 test/pargates.py . ./build/ripwire -j 6` runs the same scripts in parallel so a full
 verification fits in one sitting. It does not modify `regression.sh`.
@@ -6491,7 +6491,7 @@ Listed because the reason is more useful than the silence.
   shipped**. See `bench/locbench/anchorhop_calib.json`. The mention anchor's reproducible numbers are
   the ablations in §4.
 - **A single round gate-count.** Two in-tree numbers disagree (`test/pargates.py`'s docstring says
-  ~210; `test/argvdiffcheck.sh` says 200+), while the loop in `test/regression.sh` names 569. The
+  ~210; `test/argvdiffcheck.sh` says 200+), while the loop in `test/regression.sh` names 570. The
   loop is the authority; the stale docstrings are a known drift. `test/manifestcheck.sh` asserts this
   very number against the loop's actual length, so it cannot go stale silently again.
 - **"282 argv vectors."** The gate asserts a floor of ≥250 assembled from five sources; 282 was a

present/deck5_ripwire_build.js

@@ -708,7 +708,7 @@ function row(s, y, h, cols, opts={}){
   kicker(s, "// how it stays true", AMBER);
   title(s, "Proven, not promised");
   const cards = [
-    ["569 gate scripts", "the suite runs on every push — plus determinism, cache-transparency and golden contracts; the gate count itself is gated against the runner's own loop"],
+    ["570 gate scripts", "the suite runs on every push — plus determinism, cache-transparency and golden contracts; the gate count itself is gated against the runner's own loop"],
     ["byte-identical, always", "two runs over the same tree produce the same bytes; warm equals cold. Enforced in CI, twice — Release AND a plain flavour, because NDEBUG once blinded a whole class of checks"],
     ["differential refactoring", "a refactor must prove it changed nothing observable: two binaries, hundreds of argv vectors, stdout + stderr + exit codes byte-identical"],
     ["held-out labels, authored blind", "eval labels were written by reading source before the ranker ever ran on them — so the eval is allowed to say the ranker is wrong. It has."],
@@ -732,7 +732,7 @@ function row(s, y, h, cols, opts={}){
   title(s, "Claims you can trust, because we publish what failed", { size: 32 });
 
   card(s, MX, 1.72, 3.86, 1.72);
-  stat(s, "569", "gate scripts named by test/regression.sh — and the COUNT itself is gated against the runner's own loop, so it cannot go stale quietly",
+  stat(s, "570", "gate scripts named by test/regression.sh — and the COUNT itself is gated against the runner's own loop, so it cannot go stale quietly",
     MX+0.15, 1.86, 3.56, CYAN, { bsize: 42, bh: 0.66, lsize: 9.5 });
   card(s, 4.68, 1.72, 3.86, 1.72, CARD2);
   stat(s, "8", "registered NEGATIVES — changes built, gated green, measured against a band written before the code, and reverted rather than tuned",
@@ -971,7 +971,7 @@ function row(s, y, h, cols, opts={}){
     ["179 long flags · 29 slides",        "bash test/deckclaimcheck.sh"],
     ["every --flag named here exists",    "bash test/deckcheck.sh"],
     ["74.7% fewer element bytes",         "bash test/showcasecapturecheck.sh"],
-    ["569 gate scripts",                  "bash test/manifestcheck.sh"],
+    ["570 gate scripts",                  "bash test/manifestcheck.sh"],
     ["46 repos · 69 papers · 237 surveyed","bash test/readmedriftcheck.sh"],
     ["the ten moments, any row",          "ripwire . --callers=SYM | wc -c"],
     ["the head-to-head table",            "bench/headtohead/r4-2026-08-06/"],

src/ingest_crawl.h

@@ -828,16 +828,34 @@ void recordCrawlDrop( std::vector<SkippedFile>& rows, std::uint64_t& exactCount,
 // the user did NOT ask to hide (an --exclude'd .ml is requested absence, not a language this build cannot
 // read). Swap the two and both classes start lying.
 //
+// THE THIRD TEST, FOR ONE CLASS ONLY (§N6-C, closed 2026-09-09). The unsupported-ext population is not
+// merely reported — grep's aux scan (search.h grepCollectAux) READS it and SERVES its hits — so it must
+// hold only files the REPOSITORY did not ask to hide either, on exactly the rule it already applies to
+// --exclude. Before this test existed the crawl asked the ignore set only about files that survived
+// here, so a file that was both gitignored and of an unindexed extension was rowed as unsupported-ext:
+// measured, --regex='^#include' served four hits from a `.cpp.bak` beside its source that the
+// repository's own .gitignore names (rg does not open it), while unindexed_files_scanned= and
+// unsupported_ext= both counted it, so nothing disclosed that an ignored file had been read.
+// test/grepignorecheck.sh pins the fix. `ignored` is a LAZY predicate for the same reason fullPath is:
+// the lookup stringifies the path, so it is paid only once the two cheaper tests have already admitted
+// the file to this class — never for a binary asset or an --exclude'd file, and never for an indexable
+// file, which takes the crawl's own ignore test after this returns false. A file dropped here is in NO
+// class — neither this one nor ignored=, exactly as an --exclude'd unsupported-ext file is in neither
+// this one nor excluded=: ignored= describes only what would OTHERWISE HAVE BEEN INDEXED (the number the
+// map header's accounting invariant carries), and a language this build cannot read that the repository
+// hid is not a disclosure the reader is owed. --no-ignore makes the predicate false, so the escape hatch
+// restores the row and both counts with it.
+//
 // `fullPath` is the caller's LAZY path materializer, taken as a template parameter rather than a
 // std::string: a monorepo crawl walks far more non-source files than source ones, and stringifying every
 // one of them to record the handful that are reportable would be a real per-file cost for nothing.
-template< typename PathFn >
+template< typename PathFn, typename IgnoredFn >
 bool recordPreSizeDrop( CrawlSkips& skips, HashMap<std::string, std::uint64_t>& extTally,
-                        const std::string& ext, bool excluded, const fs::directory_entry& entry, PathFn&& fullPath )
+                        const std::string& ext, bool excluded, const fs::directory_entry& entry, PathFn&& fullPath, IgnoredFn&& ignored )
 {
     if( lookupLang( ext ) == nullptr && !docparse::isDocExtension( ext ) )
     {
-        if( !excluded && !isNonTextExtension( ext ) )
+        if( !excluded && !isNonTextExtension( ext ) && !ignored() )
         {
             ++extTally[ ext ];
             recordCrawlDrop( skips.unsupported, skips.unsupportedFiles, fullPath(), ext, entry );
@@ -1002,8 +1020,11 @@ GitIgnoreSet collectGitIgnored( const char* rootDir )
 // extension classification and the --exclude match (the same reason recordPreSizeDrop's header gives for
 // its own two): `ignored` then only ever describes a file that would OTHERWISE HAVE BEEN INDEXED, which is
 // what lets the header's accounting invariant carry it — indexed= + oversize= + excluded= + ignored= = the
-// population the crawl enumerated — and keeps unsupported_ext=/unindexed= meaning exactly what they meant
-// before this lane. The DIRECTORY test runs after the built-in denylist for the mirror reason: ignoredDirs=
+// population the crawl enumerated. The ONE earlier consult is recordPreSizeDrop's unsupported-ext branch,
+// which asks the same predicate before it records a row and records NOTHING when the answer is yes: that
+// class is served by grep's aux scan, so unsupported_ext=/unindexed= describe the population grep actually
+// reads, and an ignored file of an unindexed extension is counted in neither class (its header has the
+// measured leak). The DIRECTORY test runs after the built-in denylist for the mirror reason: ignoredDirs=
 // then counts only the subtrees no rule this build already carried had pruned.
 bool pathInIgnoreSet( const std::vector<std::string>& sorted, std::string_view rel ) noexcept
 {
@@ -1220,16 +1241,26 @@ CrawlResult collectSources( const char* rootDir, const std::vector<std::string>&
         // doc post-pass instead). Use the filename here so rejected regular files do not pay to stringify the
         // full path; materialize the full path only after the extension survives.
         //
+        // §N6-C: the repository's own verdict on this file, ONE lazy predicate shared by the two sites that
+        // ask it — the lookup stringifies the path (relForHash over fullPath), so it is evaluated only where
+        // a class actually consults it, never for a binary asset or an --exclude'd file. False under
+        // --no-ignore, on a non-git root, and when git could not answer (ignoreSet.available).
+        const auto ignored = [ & ]() -> bool
+        {
+            return ignoreSet.available && pathInIgnoreSet( ignoreSet.files, relForHash( fullPath(), rootDir ) );
+        };
+
         // §L1: the two NON-SIZE drops are classified and recorded together (recordPreSizeDrop) — see its
-        // header for why the two tests must run in that order, and why they are not written inline here.
+        // header for why the tests must run in that order, why the unsupported-ext class alone consults the
+        // ignore verdict BEFORE it records a row, and why none of it is written inline here.
         const std::string ext = lowerExtensionOf( name );
-        if( recordPreSizeDrop( skips, extTally, ext, excluded, *it, fullPath ) )
+        if( recordPreSizeDrop( skips, extTally, ext, excluded, *it, fullPath, ignored ) )
         {
             continue;
         }
 
         // §N6-C: AFTER the extension and the --exclude match — see pathInIgnoreSet's header for the ordering.
-        if( ignoreSet.available && pathInIgnoreSet( ignoreSet.files, relForHash( fullPath(), rootDir ) ) )
+        if( ignored() )
         {
             recordCrawlDrop( skips.ignored, skips.ignoredFiles, fullPath(), ext, *it );
             continue;

src/model.h

@@ -821,7 +821,10 @@ struct CrawlSkips
     // extension classification, the --exclude match and the built-in denylist, so every existing counter
     // keeps exactly the meaning it had: ignoredFiles counts files that would OTHERWISE HAVE BEEN INDEXED
     // (which is what makes it the number the header's accounting invariant can carry), and ignoredDirs
-    // counts only the subtrees no other rule had already pruned.
+    // counts only the subtrees no other rule had already pruned. The one class that consults the verdict
+    // EARLIER is `unsupported` above: grep serves that population, so a gitignored file of an unindexed
+    // extension is not rowed there either — it is in no class at all, exactly as an --exclude'd one
+    // already was (ingest_crawl.h recordPreSizeDrop's header).
     std::vector<SkippedFile>  ignored;              // capped rows, path-sorted — the individual ignored files
     std::vector<SkippedFile>  ignoredDirRows;       // capped rows, path-sorted — the pruned subtrees (bytes 0, ext "")
     std::uint64_t             ignoredFiles    = 0;  // EXACT count (rows may be fewer)

src/verbs_report.h

@@ -1652,7 +1652,9 @@ constexpr const char* kSkippedLegend =
                  " HEADER: indexed= is files= on the map; the ACCOUNTING INVARIANT is indexed= + oversize= + excluded= = the candidate"
                  " population the crawl ENUMERATED, at every ceiling and exclude setting. unsupported_ext= counts source/text-looking files"
                  " outside that population (binary/asset extensions are deliberately not counted — an unindexed .png is a picture, not a"
-                 " language this build failed to read); its per-extension breakdown is the <e x= files=/> rows, which the map header rolls"
+                 " language this build failed to read — and neither is a file an exclude or the repository's own ignore rules hid:"
+                 " requested absence, not an unread language; the grep verb's unindexed scan reads exactly this class, nothing hidden);"
+                 " its per-extension breakdown is the <e x= files=/> rows, which the map header rolls"
                  " up as unindexed= — a TOP-6 list, and the map's unindexed_exts= beside it names how many DISTINCT such"
                  " extensions exist, present exactly when that list was cut and absent when it is complete."
                  " excluded_dirs= counts SUBTREES an exclude pruned: the walk stopped at the directory, so how many files"