fix(procedures): bound the scout's search to the record stores by drewdrewthis · Pull Request #9 · drewdrewthis/claude-plugins

drewdrewthis · 2026-08-07T08:15:52Z

Closes #15

What

/how-do-i was slow when the corpus had no match. This bounds the procedure-scout agent's search surface and makes a miss a finished answer instead of a reason to keep grepping.

Why

The script is not the cost. Measured:

$ time query-records.sh --keyword "zzqxnonexistentqqzz12345"   # miss
real 0m0.259s
$ time query-records.sh --keyword "autonomy"                    # hit
real 0m0.168s

Both sub-300ms. The latency is the agent's own follow-on searching.

Nothing instructs it to fan out — I grepped for it. The cause is an absent boundary: procedure-scout declares no tools: restriction (full Bash/Grep/Glob), step 3 says "err inclusive", and Boundaries forbade acting, inventing, omitting a trap, and flattening a contradiction — never widening. With full tools, an instruction to err inclusive, and no stop sign, an empty result reads as "try one more grep". The NOT FOUND output section already existed; nothing routed to it.

Changes

Boundaries — names the allowed surface by pointing at the discovery mechanism ("the stores query-records.sh covers, plus mistakes.jsonl") rather than freezing a prose copy of the list, and names step 2's synonym expansion as the only remaining widening lever.
Step 4 carries the mistakes.jsonl grep it previously only described, with a reported cap — the original tail -5 dropped traps silently, against the file's own boundary.
Step 5 — a miss emits the NOT FOUND section including its /create-new line, and stops.
SKILL.md trimmed to a pointer at the agent's Boundaries.

Human verification (for if you really don't trust me)

cd plugins/procedures && bats hooks/tests — 141 pass, unchanged from main (this PR touches no script). Note this proves nothing about the diff: no test asserts anything about procedure-scout.md or SKILL.md, so the suite is green with the whole change reverted.
grep -n 'tail' plugins/procedures/agents/procedure-scout.md — one hit, the sweep's tail -20; confirm the count line above it and the STANDING NOTES slot in the Output block, so truncation is reported rather than silent.
Read agents/procedure-scout.md Boundaries and step 5: confirm the stop rule has one canonical home and the NOT FOUND instruction is not reducible to a bare token.
Against issue /how-do-i is slow when the corpus has no match #15's ACs: confirm each of the five is addressed by the diff.

How I can prove I was successful

Suite is green at this head — proven, and non-load-bearing.
```
$ cd plugins/procedures && bats hooks/tests
ok=141  notok=0
```
Stated for completeness only: nothing in hooks/tests covers the two prose files this PR changes.
The stop rule has one home — proven. Grepped widen|Nothing searches|every pass you get|NOT FOUND across both files: step 5's fused warning is the only stop statement; the Boundaries bullet is now the scope rule, and SKILL.md is a pointer.
The agent actually behaves differently on a miss — proven (n=2). Installed this branch and origin/main into throwaway CLAUDE_CONFIG_DIRs and ran the same guaranteed-miss query against each, twice:

run branch main

miss query A 79.4s, returned NOT FOUND 98.8s, returned NOT FOUND

miss query B 79.1s, returned NOT FOUND 103.9s, did not return NOT FOUND

Measured at 0a4a7ad (artifacts written 09:58-10:03Z; that commit landed 09:56:45Z), i.e. against the capped sweep this PR ships. Round 4 (00ae1ea) changed prose only — the store list, the SKILL.md surface and the STANDING NOTES slot — not the search shape.

Read this as weaker than it looks. Two different queries, one run per arm each — so within-query n=1, and no condition was repeated. Arms ran sequentially, so arm is confounded with time. And the cost is identical (~$0.45/run) in both arms, which argues against the stated mechanism: fewer follow-on greps should mean fewer tool calls and fewer tokens, hence lower cost. Equal cost with lower wall-clock is at least as consistent with API-load drift as with a behaviour change. The one observation that is not merely timing is the qualitative divergence below.

The arms are not emitting identical output, and the difference is the point rather than a confound. Main's run B improvised its own verdict header instead of the canonical section, and narrated searching outside the corpus:

"I searched every record store and found zero matches ... not in references/**, not in plans/, not in mistakes.jsonl, not in skills/."

skills/ is not a record store and is not in the allowed surface — that is the widening this PR exists to stop, visible in the baseline arm. Branch runs returned the ## NOT FOUND section with its /create-new routing both times. So the wall-clock gap and the format divergence share one cause.
The recall sweep is capped at 20 and the cap is reported — proven. Round 1 removed tail -5 outright; the re-review showed that made the reported problem worse, because on the live corpus grep -icE 'pr|review' ~/.claude/mistakes.jsonl matches all 428 lines / 440851 bytes — a broad term set would pull ~110k tokens. The sweep now counts first, reads the 20 most recent, and reports the count in STANDING NOTES when it exceeds what was read:
```
$ grep -n 'tail' plugins/procedures/agents/procedure-scout.md
93:   grep -iE  '<term set>' "$M" | tail -20   # the 20 most recent
```
Known limit, not hidden: a term set broad enough to match every line makes tail -20 "the last 20 mistakes" rather than the 20 most relevant. The count is what stops that from being silent — it does not make the selection good. Narrowing recall selection is out of scope here.

Measurement path — stated, not glossed. The runs above go through claude -p, which exercises the skill and the procedure-scout dispatch (the scout output above is real). It does not exercise this plugin's hooks: I confirmed separately that no plugin hook fires in print mode. That does not affect this PR, which changes no hook — but the measured path is print-mode skill dispatch, not an interactive session, and I have not shown the two are identical.

Backend-only: this diff is two markdown files that instruct a subagent. There is no UI, no rendered surface, and no running app to screenshot; the terminal output above is the whole observable behaviour.

Review

Reviewed by the standard own-PR fan-out across three rounds. Round 1 raised four blocking threads (fixed in 90d4aef). The scoped re-review found that two of those fixes did not hold — the stop-rule collapse had replaced one rationale tail with another, and the boundary pointed at a set the scout could not enumerate (fixed in 517f7ae). proof-reviewer then found that removing the tail cap made the reported latency worse, not better (fixed in 0a4a7ad). See the verdict comment.

Acceptance criteria (issue #15)

AC	Status	Evidence
1. Allowed surface named explicitly, outside forbidden	✅	Boundaries now inline the seven stores + `mistakes.jsonl` (`00ae1ea`); earlier revisions pointed at a file the scout could not read without breaching the boundary
2. Explicit last pass; miss returns the NOT FOUND section incl. `/create-new`	✅	step 4 is terminal; step 5 emits the section
3. Stop rule has exactly one canonical home	✅	step 5's fused warning; Boundaries hold the scope rule only; SKILL.md points at each correctly
4. No trap silently dropped; any cap reportable	✅	`tail -20` + count + a STANDING NOTES slot in the Output block. The cap's selection quality on a very broad term set is a known limit, stated above
5. SKILL.md describes the same surface, including `plans/`	✅	restored in `00ae1ea` after round 2 regressed it; list verified against `scripts/lib/stores.sh`

Reporter's case — "/how-do-i is slow when the corpus has no match": PARTIALLY VERIFIED. The qualitative divergence is real and reproducible in direction (main's scout searched skills/, outside the corpus, and improvised its verdict format). The latency claim is weaker than the table suggests, for the reasons stated in that section — equal cost across arms is the disconfirming datum and I have not explained it away.

/how-do-i was slow on a miss. Nothing instructed the scout to fan out, but nothing stopped it either: procedure-scout runs with all tools, step 3 says "err inclusive", and Boundaries forbade acting, inventing, omitting, and flattening — never widening. With no stop condition, an empty result reads as a reason to keep grepping rather than an answer. query-records.sh is not the cost: a miss measures 0.259s against 0.168s for a hit. The latency is the agent's own follow-on search. - Boundaries: name the search surface (references/**, plans/, mistakes.jsonl) and forbid everything outside it. - Step 4 is the last pass, and now carries the mistakes.jsonl command it only described (skills/log/SKILL.md:47 had it; the scout had to improvise). - Step 5: a miss returns NOT FOUND immediately. - SKILL.md states the same contract for the caller. Prose only — no script behavior changes, so no bats test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

coderabbitai · 2026-08-07T08:16:00Z

Warning

Review limit reached

@drewdrewthis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ed6621bc-2ff2-442b-94ff-f03bb6efa321

📥 Commits

Reviewing files that changed from the base of the PR and between 1135439 and 6e14654.

📒 Files selected for processing (4)

plugins/procedures/.claude-plugin/plugin.json
plugins/procedures/agents/procedure-scout.md
plugins/procedures/hooks/tests/store-list-drift.bats
plugins/procedures/skills/how-do-i/SKILL.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…y to reads use-case-reviewer, two findings: - SKILL.md listed references/** and mistakes.jsonl but not plans/, and the new "those stores are the whole search surface" sentence hardened that omission into an authoritative claim. query-records.sh scans plans/ on every call (scripts/lib/stores.sh) and the agent's own boundary allows it. - The boundary read literally forbids touching anything outside CODEX_ROOT, but step 2 runs query-records.sh from ${CLAUDE_PLUGIN_ROOT}. Say the restriction scopes what is read, not where the tool lives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

drewdrewthis

Own-PR review fan-out: principles, hygiene, security. Personas skipped under the prose/config-only guard — the diff is two .md files, no code surface. Verdict comment follows.

drewdrewthis · 2026-08-07T09:25:30Z

Review verdict: READY

Reviewed at: 6e14654 · Run: review (own-PR) · four rounds

No blocking concerns. All review threads resolved at this SHA, and the final scoped gate at this head returned NO BLOCKING DEFECTS.

Scope note: READY here means no unresolved blocking review threads. It is not merge-readiness — see the residuals below, which are declared rather than fixed.

What the rounds actually found

Worth recording, because three of the four rounds found defects in the fixes, not the original diff:

Round 1 fixed the four original threads.
Round 2 — fix procedures plugin is missing the respond (reply-first/ack) gate — third invariant of ADR-016 not ported #4 had not held: one rationale tail was deleted and another added in the same commit, and the new boundary pointed at stores.sh, a file the scout would have to breach the boundary to read.
Round 3 — the round-1 fix made the reported problem worse. Removing | tail -5 left an uncapped grep matching 428/428 lines of mistakes.jsonl (~110k tokens) on a broad term set, and issue /how-do-i is slow when the corpus has no match #15's symptom is latency. Replaced with count-then-read-20 plus a reported overflow.
Round 4 — restored issue /how-do-i is slow when the corpus has no match #15 AC5 (plans/), which an earlier fix had silently dropped from SKILL.md, and gave the STANDING NOTES overflow report a slot in the Output block.

The PR body also carried a false claim ("the recall sweep is uncapped") with a quoted transcript that no longer existed at HEAD; corrected at all four assertion sites and re-verified against the published body.

Non-blocking (Decide / New Issue)

[proof-reviewer] tail -20 degenerates on a very broad term set: if every line matches, it selects the 20 most recent rather than the 20 most relevant. The count makes that visible instead of silent, which is what AC4 requires — improving recall selection is a separate problem. (Decide)
[proof-reviewer] The latency measurement is weaker than a table suggests: two different queries, one run per arm each, run sequentially, and equal cost across arms argues against the stated mechanism. Stated in the body rather than smoothed over. The qualitative divergence — main's scout searched skills/, outside the corpus — is the stronger evidence. (Decide)
[principles] SKILL.md invokes the script via ${CLAUDE_SKILL_DIR}, the agent via ${CLAUDE_PLUGIN_ROOT}. Two idioms for one script, both pre-existing on main. (New Issue)
[principles] references/adrs/ is a codex record store but is absent from stores.sh, so ADRs are invisible to both the query tool and the scout's surface. Pre-existing; deserves a decision rather than an accident. (New Issue)
[synthesizer] Both open PRs bump plugin.json from 0.2.0 — this one to 0.2.1, feat(procedures): one off-switch per gate, on by default, recorded when used #10 to 0.3.0. Whichever merges second conflicts on that line and needs a rebase and re-bump. (Decide)

Evidence at this SHA

bats hooks/tests → 145 ok / 0 not ok.
New hooks/tests/store-list-drift.bats pins the inlined store list to stores.sh in both directions, with a non-empty guard first; falsified on a copy (dropping a store reddens it).
All five issue /how-do-i is slow when the corpus has no match #15 ACs re-verified at this head, not carried forward.

Verdict is prose, not a GitHub approval. Scope: review findings only — READY means no unresolved blocking review threads at this SHA. It is not a merge-readiness signal; that is pr-ready-check.sh (8 criteria, of which this verdict is C3).

… rule Addresses all four blocking review threads on #9. - Drop `| tail -5` from the mistakes.jsonl sweep. It silently discarded matches against this file's own boundary ("never omit a trap because it looks unlikely"), and unlike query-records.sh's --full cap, tail prints no truncation notice — so neither scout nor caller could tell. Replaced with a fused warning at the trap site. - Step 5 now says "emit only the NOT FOUND section, including its /create-new line". "Return NOT FOUND now" read as "emit the bare token", which would drop the instruction the Invariants make mandatory — the whole value of a miss. - Step 4's "Nothing searches after this step" forbade reading the records that step had just located (it produces paths; the batch-read lives in step 3). Rescoped to "No new terms and no new stores after this step — reading the records this sweep named is part of it." - Collapse the stop rule to one home. It was stated four times across two files (agent-facing-docs.md rule 1). Step 5's fused warning is the template-sanctioned form and survives; the Boundaries bullet is now the scope rule only, with its exiled rationale tail cut (rule 7). - Boundaries now point at the discovery mechanism ("the stores query-records.sh covers") instead of freezing a third prose copy of the store list (rule 2), and name step 2's synonym expansion as the only remaining widening lever — the design tension a bounded search creates. - SKILL.md trimmed to a pointer at the agent's Boundaries, and its "so the caller can write the missing procedure" corrected to "improvise and draft the procedure afterward" — it inverted the agent's own invariant that a procedure written before it has succeeded once is a guess with an id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…le tail Three Must-Fixes from the scoped re-review of 90d4aef. - The boundary said "the stores `query-records.sh` covers" — a predicate the scout cannot resolve without reading two scripts, in a doc whose point is context economy. Rule 2 wants a discovery mechanism, not an unresolvable reference. Now names the SSOT by path: the STORES list in scripts/lib/stores.sh. - SKILL.md enumerated `references/**` + `plans/` while the boundary named the seven STORES dirs — not the same set (adrs/, policies/, scenarios/ and loose references/*.md fall between them), and the skill then said "the scout's Boundaries hold the rule" having just stated a different one. Contradictions are bugs (rule 5). Enumeration deleted; the agent file is the one home. - Fix 4 from round 1 did not hold. I deleted the old rationale tail and added a new one in the same commit ("a bounded search makes a false miss a keyword problem, so spend the breadth there") — the same defect class, reintroduced by its own fix. Cut. The bullet now ends at "are not the answer surface." Also: step 4's "reading the records this sweep named is part of it" needed a read command or the scout improvises a per-file loop — the context bloat step 3 exists to prevent. Points at step 3's awk batch-read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…bump proof-reviewer found that round 1's fix made the reported problem worse. Removing `| tail -5` left an uncapped `grep -iE` over mistakes.jsonl. Measured against the live file: $ wc -lc ~/.claude/mistakes.jsonl 428 440851 $ grep -icE 'pr|review' ~/.claude/mistakes.jsonl 428 A term set of the shape step 2 encourages ("expand synonyms", "the capability as well as the identifier") returns the whole 430KB file — ~110k tokens — into the scout's context. `grep -i` is unanchored, so it matches inside paths and URLs too. Issue #15's symptom IS latency, and this was the one command in the diff whose cost scales with corpus size: AC4 satisfied to the letter while working against the goal. "Uncapped" was never the only alternative to a silent cap. Now: count first, read the 20 most recent, and report in STANDING NOTES when the count exceeds what was read. A cap is allowed; a silent one is not. Also bumps 0.2.0 -> 0.2.1. Without it the fix does not reach version-pinned caches (precedent: 03830b2), so the change could not have shipped at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…t the count proof-reviewer, three defects introduced by the earlier fix rounds. - The boundary pointed at `scripts/lib/stores.sh` — the only unqualified path in the file, matching two real files on this box, and reading either is itself outside the allowed surface. To obey the boundary the scout had to breach it. That is round 2's "set the scout could not enumerate" relocated, not removed. The seven stores are now inline, with the SSOT named (qualified with ${CLAUDE_PLUGIN_ROOT}) for anyone who needs to confirm. - Issue #15 AC5 regressed: SKILL.md named plans/ at c815903 and, after round 2 deleted the enumeration to fix a contradiction, named no surface at all. It now describes exactly what query-records.sh searches, plus mistakes.jsonl as the store it does NOT cover — verified against stores.sh. - The STANDING NOTES overflow report had no slot in the Output block, and "emit only the NOT FOUND section" on a miss contradicted reporting it. Added the line to the schema and made it the one stated exception, since an unreported truncation is the silent drop the cap is only permitted because it reports. Also corrects SKILL.md's pointer: Boundaries hold the scope rule, step 5 holds the stop rule. It previously attributed both to Boundaries and then restated the stop rule anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…list Final gate returned NO BLOCKING DEFECTS; these are its Should-Improves. - Drop the "stores.sh is the SSOT if you ever need to confirm" half of the parenthetical. The carve-out one sentence later exempts RUNNING query-records.sh, not READING files under CLAUDE_PLUGIN_ROOT, so it invited a read the surrounding rule forbids — round 3's defect, softened but not gone. It was inert anyway: the scout is bound by the inlined list whatever stores.sh says. - Collapse the four-line rationale tail added by round 4 into the slot comment it was explaining. 517f7ae was literally "drop the new rationale tail"; round 4 added another one. Third time this file has grown a tail while removing one. - SKILL.md now uses ${CODEX_ROOT:-~/.claude} to match query-records.sh's own resolution, which the agent's Boundaries already got right. NEW: hooks/tests/store-list-drift.bats. Inlining the store list was the right call — the pointer it replaced named a file the scout could not read without breaching the boundary that names it — but it traded auto-freshness for silent drift with nothing guarding it. Four cases, both directions, with a non-empty assertion first so none can pass vacuously. Falsified on a copy: removing one store from the agent's list reddens case 2. The reverse check uses a negative lookbehind on `/` because `skills/create-new/references/create-procedure.procedure.md` otherwise reports a store nobody claimed. Suite 145 ok / 0 not ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

drewdrewthis commented Aug 7, 2026

View reviewed changes

Comment thread plugins/procedures/agents/procedure-scout.md Outdated

Comment thread plugins/procedures/agents/procedure-scout.md Outdated

Comment thread plugins/procedures/agents/procedure-scout.md Outdated

Comment thread plugins/procedures/agents/procedure-scout.md Outdated

drewdrewthis commented Aug 7, 2026

View reviewed changes

Comment thread plugins/procedures/agents/procedure-scout.md Outdated

drewdrewthis mentioned this pull request Aug 7, 2026

CodeRabbit skips draft PRs — drafts have no automated reviewer #13

Open

claude added 5 commits August 7, 2026 09:49

drewdrewthis mentioned this pull request Aug 7, 2026

feat(procedures): one off-switch per gate, on by default, recorded when used #10

Open

drewdrewthis marked this pull request as ready for review August 7, 2026 10:38

drewdrewthis self-assigned this Aug 7, 2026

drewdrewthis merged commit 4cb4071 into main Aug 7, 2026
1 check passed

drewdrewthis deleted the fix/scout-bounded-search branch August 7, 2026 11:46

github-actions Bot mentioned this pull request Aug 7, 2026

chore(main): release procedures 0.2.3 #31

Merged

Navigation Menu

fix(procedures): bound the scout's search to the record stores - #9

fix(procedures): bound the scout's search to the record stores#9
drewdrewthis merged 7 commits into
mainfrom
fix/scout-bounded-search

drewdrewthis commented Aug 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

Review limit reached

Uh oh!

drewdrewthis left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drewdrewthis commented Aug 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

run	branch	main
miss query A	79.4s, returned NOT FOUND	98.8s, returned NOT FOUND
miss query B	79.1s, returned NOT FOUND	103.9s, did not return NOT FOUND

Conversation

drewdrewthis commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Changes

Human verification (for if you really don't trust me)

How I can prove I was successful

Review

Acceptance criteria (issue #15)

Uh oh!

coderabbitai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

drewdrewthis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drewdrewthis commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review verdict: READY

What the rounds actually found

Non-blocking (Decide / New Issue)

Evidence at this SHA

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

drewdrewthis commented Aug 7, 2026 •

edited

Loading

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

drewdrewthis commented Aug 7, 2026 •

edited

Loading