Skip to content

Make the README witness check capable of failing - #360

Merged
sroussey merged 1 commit into
mainfrom
claude/p2-sec-350-example-coverage
Sep 8, 2026
Merged

Make the README witness check capable of failing#360
sroussey merged 1 commit into
mainfrom
claude/p2-sec-350-example-coverage

Conversation

@sroussey

@sroussey sroussey commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Closes #350.

The problem

sourceFiles collected every .ts under src/, tests included — and the witness table is a set of string literals in this very file. The corpus therefore contained the strings it was searching for, so every witness matched itself and the check could not fail for any reason.

Three changes, each verified by breaking the tree

I checked each guard by making the regression it exists to catch and confirming the right test went red, rather than by reading the code.

1. Test files are out of the corpus, and a self-check holds them out. Excluding them is one line in sourceFiles; without an assertion the next refactor of that function restores the loop with nothing failing. Reverting the line now fails:

AssertionError: expected [ …(836) ] to not include 'src/exampleCoverage.test.ts'

It also asserts the corpus is non-empty — an empty one passes every check below it.

2. A witness must appear on an import, not anywhere in the file. RateLimiter occurs eleven times outside tests:

  • once as real evidence — import type { RateLimiter } from "workglow"
  • ten times in prose comments, or as a substring of a longer local name like secFetchRateLimiterTableNames

Ten of the eleven prove nothing, and a substring search cannot tell them from the one that does. Verified by putting a witness string in a comment only and watching the package be reported unproven.

The extraction strips comments first and matches whole statements, not lines. That detail is load-bearing: these imports are routinely multi-line, and createStandardKbStrategy sits on a continuation line of one — a line-wise filter reported @workglow/ai as missing, which is how I found it.

3. The README's commands resolve against the registered command tree. For the reason commandsBoot.test.ts gives: help output is prose, and a name appearing in some description satisfies a substring check without being registered at all. It walks the leading bare words of each backticked `sec …` and stops at the first flag or placeholder, since only the subcommand path is resolvable.

One correction to the issue

The issue says this check "has already found one defect (sec show companies does not exist) before being written". That string is not in the README todaygrep finds nothing, and all fifteen backticked invocations resolve. So the check lands green rather than red; it is a ratchet against the next one, not a fix for a live break. Adding `sec show companies` to the README does fail it, which is how I confirmed it is not vacuous.

Verification

format, lint, typecheck all clean. exampleCoverage.test.ts 4/4 and commandsBoot.test.ts 2/2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LWp6Z6wvAPDaDCjAFcTSj6


Generated by Claude Code

`sourceFiles` collected every `.ts` under `src/`, tests included — and the
witness table is a set of string literals in this very file. The corpus
therefore contained the strings it was searching for, so every witness matched
itself and the check could not fail for any reason.

Three changes, each verified by breaking the tree and watching the right test go
red rather than by reading the code:

Test files are out of the corpus, and a self-check holds them out. Excluding
them is one line, and without an assertion the next refactor of `sourceFiles`
restores the loop with nothing failing. Reverting that line now fails with
`expected [ …(836) ] to not include 'src/exampleCoverage.test.ts'`. It also
asserts the corpus is non-empty, since an empty one passes everything below it.

A witness must appear on an IMPORT, not anywhere in the file. `RateLimiter`
occurs eleven times outside tests: once as a real
`import type { RateLimiter } from "workglow"`, and ten times in prose or as a
substring of a longer local name like `secFetchRateLimiterTableNames`. Ten of
the eleven prove nothing and a substring search cannot tell them apart. The
extraction strips comments first and matches whole statements rather than
lines — these imports are routinely multi-line, and `createStandardKbStrategy`
sits on a continuation line of one, which a line-wise filter reported as a
missing package.

The README's commands are resolved against the registered command tree, for the
reason `commandsBoot.test.ts` gives: help text is prose, and a name appearing in
some description satisfies a substring check without being registered. It walks
the leading bare words of each backticked `sec …` and stops at the first flag or
placeholder. Fifteen invocations resolve today; adding one for a command that
does not exist fails it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWp6Z6wvAPDaDCjAFcTSj6
@sroussey
sroussey merged commit de00b86 into main Sep 8, 2026
1 check passed
@sroussey
sroussey deleted the claude/p2-sec-350-example-coverage branch September 8, 2026 17:29
sroussey added a commit that referenced this pull request Sep 8, 2026
### Features

- implement ITabularStorage.join for @WorkGlow 0.5.0 (#362)
- implement CLI signal teardown and resource shutdown

### Bug Fixes

- declare the embedding width beside the model instead of assuming it (#359)

#### release

- declare the runtime floor, and derive the release bump (#357)

#### kb

- stop --dry-run creating the index tables, and report them in db stats
- bound `ask`'s implicit index, pin the embedding model, and let `db reset` drop the index

#### ask

- refuse rather than answer from model memory when nothing is retrieved

#### adv

- scope each Form ADV archive to its own folder and make ingest idempotent

### Performance

#### status

- read the newest ADV snapshot with one indexed row

### Tests

#### coverage

- make the README witness check capable of failing (#360)

### Chores

- update deps
- update dependencies to latest versions
- migrate from Prettier to oxfmt for code formatting

#### deps

- upgrade Vitest to 5 (#342)

### Updated Dependencies

- `@workglow/cli`: 0.5.0
- `typebox`: 1.3.29
- `workglow`: 0.5.0
- `@types/bun`: 1.4.2
- `bunset`: 1.1.1
- `oxlint`: ^1.82.0
- `vitest`: ^5.0.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

exampleCoverage.test.ts's package-witness check cannot fail: the witness table is inside the corpus it searches

2 participants