From 66733075823504aca73aec1e23306898b844b4a7 Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:55:30 -0600 Subject: [PATCH 01/11] docs: refresh README, api reference, and cover for v4.5.0 surface The README and api reference still claimed APIs that were removed in v4.5.0 and described type unions that no longer match src/types.ts. The README's opening also led with prose instead of the artifact, and the cover image referenced in the assets path was missing. Specific changes: - Add assets/cover.svg (was missing, referenced by README header) - Rewrite README header to show cover + concrete 2-sentence description - Drop the "What's New in v4.5.0" section (duplicates CHANGELOG.md) - Trim 110 lines of inline CLI deep-dive (lives in docs/cli-reference.md) - Add the tree-sitter engine row to How It Works (re-added in v4.5.0) - docs/api-reference.md: remove the Agent Invocation section (buildAgentConfig, invokeAgent, isAgentSdkAvailable, hasAgentOutput, watchForCompletion, countCodeFiles were all removed in v4.5.0 and no longer exist in src/) - docs/api-reference.md: correct RuleCategory from 15 stale members to the 8 in src/types.ts; correct VerifierType to 4 members - docs/api-reference.md: replace the ruleprobe/semantic subpath import example with a note (package.json has no exports field) Closes: docs-cover-missing, docs-stale-agent-invocation, docs-rulecategory-mismatch, docs-verifiertype-mismatch, docs-semantic-subpath Co-found: docs-reviewer --- README.md | 240 ++++++++++++------------------------------ assets/cover.svg | 81 ++++++++++++++ docs/api-reference.md | 45 ++------ 3 files changed, 157 insertions(+), 209 deletions(-) create mode 100644 assets/cover.svg diff --git a/README.md b/README.md index 0288041..f38a1ee 100644 --- a/README.md +++ b/README.md @@ -1,108 +1,94 @@

-

RuleProbe

-

- Translate instruction files into ESLint configs, detect drift, and extract rules. -

-

- npm version - build status - license - TypeScript - Node.js >= 18 - GitHub stars -

+ RuleProbe cover

-## Why +# RuleProbe -AI coding agents read your `CLAUDE.md`, `AGENTS.md`, or `.cursorrules`, but nothing checks whether your ESLint config actually enforces the same rules. Drift accumulates: the instruction file says "use camelCase" but ESLint never checks it, or ESLint enforces rules the instruction file never mentioned. +When your `CLAUDE.md` says "use camelCase" but ESLint doesn't enforce it, drift has already happened. RuleProbe reads an instruction file and translates it to an ESLint config. It detects what each enforces but the other misses, or converts ESLint rules back to instruction prose. -RuleProbe closes that gap. It reads your instruction file, extracts machine-verifiable rules, and bridges them to your tooling in three ways: +[![npm version](https://img.shields.io/npm/v/ruleprobe?style=flat-square)](https://www.npmjs.com/package/ruleprobe) +[![build](https://img.shields.io/github/actions/workflow/status/moonrunnerkc/ruleprobe/self-check.yml?style=flat-square&label=build)](https://github.com/moonrunnerkc/ruleprobe/actions/workflows/self-check.yml) +[![license](https://img.shields.io/github/license/moonrunnerkc/ruleprobe?style=flat-square)](https://github.com/moonrunnerkc/ruleprobe/blob/main/LICENSE) +[![TypeScript](https://img.shields.io/badge/language-TypeScript-3178c6?style=flat-square)](https://www.typescriptlang.org) +[![Node.js >= 18](https://img.shields.io/badge/node-%3E%3D18-339933?style=flat-square)](https://nodejs.org) +[![stars](https://img.shields.io/github/stars/moonrunnerkc/ruleprobe?style=flat-square)](https://github.com/moonrunnerkc/ruleprobe/stargazers) -**Translate.** Generate an ESLint config from your instruction file. `ruleprobe lint-config CLAUDE.md` produces a flat or legacy config you can drop into your project immediately. +## Installation -**Detect drift.** Compare your instruction file against an existing ESLint config. `ruleprobe drift CLAUDE.md .eslintrc.json` reports rules present in one but missing from the other, severity mismatches, and config argument differences. - -**Extract.** Pull a rules section out of an ESLint config for pasting into an instruction file. `ruleprobe extract .eslintrc.json` converts ESLint rules back into human-readable instruction prose. - -## Quick Start +Requires Node.js 18 or later. ```bash npm install -g ruleprobe ``` -Or run it directly: +Or try it without installing: ```bash npx ruleprobe --help ``` -**Translate an instruction file to ESLint config:** +Confirm it's working: + +```bash +ruleprobe --version +# 4.5.0 +``` + +## Usage + +For maintainers who use AI coding agents and ESLint. Works with any instruction file format your agents read. + +**Translate an instruction file to an ESLint config:** ```bash ruleprobe lint-config CLAUDE.md ruleprobe lint-config AGENTS.md --format legacy --output .eslintrc.json ``` -**Detect drift between instructions and ESLint:** +**Detect drift between an instruction file and an existing ESLint config:** ```bash ruleprobe drift CLAUDE.md .eslintrc.json ruleprobe drift CLAUDE.md .eslintrc.json --format markdown ``` -**Extract rules from an ESLint config:** +**Convert ESLint rules back to instruction prose:** ```bash ruleprobe extract .eslintrc.json ruleprobe extract .eslintrc.json --output rules-section.md ``` -**Parse an instruction file** to see what rules RuleProbe can extract: +**See what rules RuleProbe can parse from an instruction file:** ```bash -ruleprobe parse CLAUDE.md -ruleprobe parse AGENTS.md --show-unparseable +ruleprobe parse CLAUDE.md --show-unparseable ``` -**Verify agent output** against extracted rules (legacy mode). Valid formats: `text`, `json`, `markdown`, `rdjson`, `summary`, `detailed`, `ci`. +**Check code against extracted rules (legacy verify mode):** ```bash -ruleprobe verify CLAUDE.md ./agent-output --format text ruleprobe verify AGENTS.md ./src --changed-since origin/main ``` -## What It Does +**Discover and cross-reference all instruction files in a project:** -**Translate.** Reads 7 instruction file formats (CLAUDE.md, AGENTS.md, .cursorrules, copilot-instructions.md, GEMINI.md, .windsurfrules, .rules) and emits an ESLint config. Flat config is the default; pass `--format legacy` for `.eslintrc.json` output. Each extractable rule maps to an ESLint rule with appropriate severity and options. Rules that have no ESLint equivalent appear as comments in the output so you know what wasn't covered. - -**Detect drift.** Compares parsed rules against an existing ESLint config. Reports rules in the instruction file but missing from ESLint (you're not enforcing what you said), rules in ESLint but not in the instructions (you're enforcing what you never stated), severity mismatches, and argument differences. Use `--format markdown` for PR-ready output. - -**Extract.** Parses an ESLint config and generates a markdown rules section you can paste into an instruction file. Stylistic rules (semicolons, quotes) are reported but skipped from the output by default since they don't carry meaningful instruction content. - -**Parse.** Extracts machine-verifiable rules from instruction files with qualifiers (`always`, `prefer`, `when-possible`, `avoid-unless`, `try-to`, `never`) and section context. Subjective lines like "write clean code" are reported as unparseable. - -**Verify.** Runs extracted rules against a directory of code. Deterministic by default, optional semantic analysis available. The original mode, still supported but no longer the primary focus. +```bash +ruleprobe analyze ./my-project --format json +``` -**Analyze.** Discovers all instruction files in a project, parses each, and cross-references them for conflicts and redundancies. Pass `--semantic` for structural pattern analysis. +Seven instruction file formats are supported: `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `copilot-instructions.md`, `GEMINI.md`, `.windsurfrules`, `.rules`. Full flag reference: [docs/cli-reference.md](docs/cli-reference.md) ## Configuration -RuleProbe auto-discovers a config file in the working directory (or any parent). You can also pass `--config ` explicitly. Supported file names, in priority order: - -- `ruleprobe.config.ts` -- `ruleprobe.config.js` -- `ruleprobe.config.json` -- `.ruleproberc.json` - -A config file lets you add custom rules, override extracted rules, or exclude rules entirely: +RuleProbe auto-discovers a config file in the working directory or any parent. Pass `--config ` to override. Supported names, in priority order: `ruleprobe.config.ts`, `ruleprobe.config.js`, `ruleprobe.config.json`, `.ruleproberc.json`. ```typescript // ruleprobe.config.ts import { defineConfig } from 'ruleprobe'; export default defineConfig({ - // Add rules that the parser can't extract from your instruction file + // Rules the parser can't extract from your instruction file rules: [ { id: 'custom-no-lodash', @@ -113,7 +99,7 @@ export default defineConfig({ }, ], - // Change severity or expected values on extracted rules + // Change severity or thresholds on extracted rules overrides: [ { ruleId: 'naming-camelcase', severity: 'warning' }, { ruleId: 'structure-max-file-length', expected: '500' }, @@ -124,43 +110,14 @@ export default defineConfig({ }); ``` -`defineConfig()` is a no-op passthrough that provides type checking in TypeScript configs. JSON configs work without it. - -Custom rules use the same verifier types (`ast`, `regex`, `filesystem`) and pattern types as extracted rules. - -## CLI Reference - -Six commands: `parse`, `verify`, `analyze`, `lint-config`, `drift`, `extract`. - -```bash -ruleprobe parse CLAUDE.md --show-unparseable -ruleprobe verify AGENTS.md ./src --format detailed --threshold 0.9 -ruleprobe lint-config CLAUDE.md --format flat --output eslint.config.js -ruleprobe drift CLAUDE.md .eslintrc.json --format markdown -ruleprobe extract .eslintrc.json --output rules-section.md -ruleprobe analyze ./my-project --format json -``` - -Full command reference with all options: [docs/cli-reference.md](docs/cli-reference.md) - -### Incremental verification with `--changed-since` - -`ruleprobe verify` accepts `--changed-since ` to limit per-file checks to files changed between the given ref and `HEAD`. Useful on PRs where you only want to enforce rules on the new diff and not surface pre-existing violations. - -```bash -ruleprobe verify AGENTS.md ./src --changed-since origin/main -ruleprobe verify AGENTS.md ./src --changed-since $(git merge-base HEAD origin/main) -``` - -The flag runs `git diff --name-only --diff-filter=ACMR ...HEAD` (added, copied, modified, renamed; deleted files are excluded). Project-level rules (`changelog-exists`, `strict-mode`, etc.) still run regardless of the diff. Cross-file rules like `test-files-exist` see the full file list so they can find a test file even when the test itself was not changed. - -If git is not on `PATH` or the ref is invalid, `verify` exits with code 2 and a descriptive message. +`defineConfig()` is a no-op passthrough that provides TypeScript type checking. ## GitHub Action -Drop this into `.github/workflows/ruleprobe.yml`: +Add drift detection to every pull request (PR): ```yaml +# .github/workflows/ruleprobe.yml name: RuleProbe Drift on: [pull_request] jobs: @@ -178,57 +135,45 @@ jobs: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` -No API keys needed, deterministic results, runs in seconds. The action only runs drift detection when instruction files or ESLint config files change in the PR. `@v4` tracks the latest v4.x release; pin to a specific tag (e.g., `@v4.5.0`) for reproducible builds. +No API keys needed. The action runs only when instruction files or ESLint configs change in the PR. Pin to `@v4.5.0` for reproducible builds.
Full action options -```yaml -- uses: moonrunnerkc/ruleprobe@v4 - with: - mode: drift - instruction-file: CLAUDE.md - eslint-file: .eslintrc.json - regenerate-on-drift: "false" - comment-on-pr: "true" - fail-on-drift: "false" -``` - | Input | Default | Description | |-------|---------|-------------| -| `mode` | `drift` | Run mode: `drift` (default) or `verify` (legacy) | -| `instruction-file` | (required) | Path to instruction file | -| `eslint-file` | (auto-detected) | Path to ESLint config file | -| `regenerate-on-drift` | `false` | Open a follow-up PR with regenerated config when drift is detected | +| `mode` | `drift` | `drift` (default) or `verify` (legacy) | +| `instruction-file` | required | Path to instruction file | +| `eslint-file` | auto-detected | Path to ESLint config | +| `regenerate-on-drift` | `false` | Open a follow-up PR with the regenerated config | | `comment-on-pr` | `true` | Post drift results as a PR comment | | `fail-on-drift` | `false` | Fail the action if drift is detected | -| `changed-since` | (unset) | Verify mode only. Git ref to diff against; only files changed in `...HEAD` are checked per-file | +| `changed-since` | unset | Verify mode only: git ref to diff against | Drift mode outputs: `drift-count`, `has-drift`.
-### Verify mode with incremental diff - -To run `verify` on the PR diff only — surfacing new violations without flagging pre-existing ones — set `mode: verify` and pass the PR base SHA via `changed-since`: +## How It Works -```yaml -- uses: actions/checkout@v4 - with: - fetch-depth: 0 # full history so the base SHA is reachable -- uses: moonrunnerkc/ruleprobe@v4 - with: - mode: verify - instruction-file: AGENTS.md - output-dir: src - changed-since: ${{ github.event.pull_request.base.sha }} +``` +Instruction File --> Parser --> RuleSet --> Mapper --> ESLint Config +Instruction File --> Parser --> RuleSet --. +ESLint Config --> Parser --> Parsed --+--> Drift Detector --> Drift Report +ESLint Config --> Extractor --> Markdown Rules Section +Agent Output --> Verifier (verify mode, legacy) ``` -`fetch-depth: 0` is required because `actions/checkout` defaults to a shallow clone and the base SHA may not be present in a shallow history. +| Engine | What it checks | +|--------|----------------| +| AST (Abstract Syntax Tree) via ts-morph | TypeScript and JavaScript structure, naming, imports, type safety | +| Tree-sitter | Python and Go: function naming, function length | +| Regex | Line-level patterns across any text file | +| Filesystem | File existence, naming conventions, directory structure | -## Programmatic API +34 ESLint-mappable matchers across 7 categories (`naming`, `forbidden-pattern`, `structure`, `import-pattern`, `error-handling`, `type-safety`, `code-style`). Rules with no ESLint equivalent appear as comments in generated configs. Full matcher table: [docs/matchers.md](docs/matchers.md) -Core pipeline functions are exported for programmatic use: +## Programmatic API ```typescript import { parseInstructionFile, verifyOutput, generateReport, formatReport } from 'ruleprobe'; @@ -246,69 +191,14 @@ console.log(formatReport(report, 'summary')); Full API reference: [docs/api-reference.md](docs/api-reference.md) -## How It Works - -``` -Instruction File --> Rule Parser --> RuleSet --+ - +--> Verifier --> Adherence Report - Agent Output ---------+ - -Instruction File --> Rule Parser --> RuleSet --> Mapper --> ESLint Config - -Instruction File --> Rule Parser --> RuleSet --+ -ESLint Config --> Parser --> Parsed --+--> Drift Report - -ESLint Config --> Extractor --> Markdown Rules Section -``` - -The parser reads your instruction file and identifies lines that map to deterministic checks. Each rule gets a category, a verifier type, a pattern, and a qualifier (how strictly the instruction is worded). Three verifier engines handle different rule types: - -| Engine | What it checks | -|--------|---------------| -| AST (ts-morph) | Code structure, naming, type safety, imports for TypeScript/JavaScript | -| Filesystem | File existence, naming conventions, directory structure | -| Regex | Content patterns, forbidden strings | - -The mapper translates extractable rules into ESLint rule entries. The drift detector compares parsed rules against an existing ESLint config. The extractor reverses the mapping: ESLint rules become human-readable instruction prose. - -## Supported Rule Types - -34 mappable matchers across 7 categories produce ESLint config entries: - -| Category | Count | Verifier | Examples | -|----------|------:|----------|----------| -| naming | 5 | AST, Filesystem | camelCase variables, PascalCase types, kebab-case files, UPPER_CASE constants | -| forbidden-pattern | 5 | AST, Regex | no `any`, no `console.log`, no `var`, max line length, no `TODO` comments | -| structure | 4 | AST, Filesystem | named exports, JSDoc required, max file length, no unused exports | -| import-pattern | 4 | AST | no path aliases, no deep relative imports, no namespace imports, no wildcard exports | -| error-handling | 2 | AST | no empty catch, throw Error only | -| type-safety | 5 | AST, Regex | no enums, no type assertions, no non-null assertions, no implicit any, no ts directives | -| code-style | 9 | AST, Regex | prefer const, no else after return, no nested ternary, no magic numbers, semicolons, quotes, max function length, max params, no TODO comments | - -Rules that can't map to ESLint (test file requirements, project config, git conventions) are reported as unmappable so you can enforce them through other tooling. - -Full matcher details: [docs/matchers.md](docs/matchers.md) - ## Security -RuleProbe never executes scanned code, never makes network calls (unless you opt in with `--llm-extract`, `--rubric-decompose`, or `--semantic`), and never modifies files in the scanned directory. User-supplied paths are resolved and bounded to the working directory; symlinks outside the project are skipped unless you pass `--allow-symlinks`. All dependencies are pinned to exact versions. - -When `--semantic` is enabled, all analysis runs locally. The only network calls are to the Anthropic API for LLM judgment when vector similarity is ambiguous (using your own `ANTHROPIC_API_KEY`). Only numeric AST vectors, opaque sub-tree hashes, boolean flags, and rule text are sent to the LLM. No source code, variable names, comments, import paths, or file paths leave the machine. See [SECURITY.md](SECURITY.md) for the full model. +RuleProbe never executes scanned code, makes no network calls by default, and never writes to the scanned directory. The optional `--llm-extract`, `--rubric-decompose`, and `--semantic` flags call external APIs using your own keys. User-supplied paths are resolved and restricted to the working directory; symlinks outside it are skipped unless you pass `--allow-symlinks`. All runtime dependencies are pinned to exact versions; see [SECURITY.md](SECURITY.md) for the full model. ## Limitations -- **Not all rules map to ESLint.** Test file requirements, project config conventions, git workflow rules, and preference pairs don't have ESLint equivalents. These are reported as unmappable so you can enforce them through other tooling. -- **Monorepo support is limited.** Drift detection and lint-config scan from the repository root and use the first ESLint config found. Monorepos with per-package instruction files or configs need to specify paths explicitly. - -## What's New in v4.5.0 - -v4.5.0 pivots RuleProbe from "verify adherence" to "translate instruction files into ESLint configs." The core value proposition is now: translate, detect drift, extract. - -Key changes: -- **New commands**: `lint-config` (translate instructions → ESLint config), `drift` (compare instructions vs an existing ESLint config), `extract` (ESLint config → instruction prose). -- **Removed**: `compare`, `tasks`, `task`, `run`, the runner module exports (`buildAgentConfig`, `invokeAgent`, `watchForCompletion`, `countCodeFiles`). -- **Deprecated**: `verify` still works, but the primary workflow is now lint-config, drift, and extract. -- **Matcher audit**: 34 ESLint-mappable matchers remain across 7 categories (`naming`, `forbidden-pattern`, `structure`, `import-pattern`, `error-handling`, `type-safety`, `code-style`). 67 unmappable matchers and the `test-requirement`, `dependency`, `preference`, `file-structure`, `tooling`, `testing`, and `workflow` categories were removed. +- Not all rules map to ESLint. Test file requirements, git conventions, and preference pairs are reported as unmappable so you can enforce them through other tooling. +- Monorepo drift detection scans from the repo root and uses the first ESLint config found. Specify paths explicitly for per-package instruction files or configs. ## Contributing @@ -322,4 +212,4 @@ Issues and pull requests welcome at [github.com/moonrunnerkc/ruleprobe](https:// ## License -[MIT](LICENSE) \ No newline at end of file +[MIT](LICENSE) diff --git a/assets/cover.svg b/assets/cover.svg new file mode 100644 index 0000000..6020922 --- /dev/null +++ b/assets/cover.svg @@ -0,0 +1,81 @@ + + + + + + + + + RuleProbe + + + + + + Translate instruction files to ESLint. Detect drift. Extract rules. + + + + + + CLAUDE.md + + + + + + + + eslint.config.js + + + + .eslintrc.json + + + + + + + drift? + diff --git a/docs/api-reference.md b/docs/api-reference.md index 579af85..2914dcc 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -45,17 +45,6 @@ RuleProbe exports core pipeline functions, project analysis, configuration, and Semantic analysis runs entirely in-process. Raw source code never leaves the machine; only numeric AST vectors, opaque hashes, boolean flags, and rule text are sent to the Anthropic API for LLM-assisted judgments. -## Agent Invocation - -| Function | Purpose | -|----------|---------| -| `buildAgentConfig(options)` | Build an agent invocation configuration | -| `invokeAgent(config)` | Invoke an AI agent via SDK | -| `isAgentSdkAvailable()` | Check if the Claude Agent SDK is installed | -| `hasAgentOutput(dir)` | Check if a directory contains agent output | -| `watchForCompletion(options)` | Watch a directory for agent output | -| `countCodeFiles(dir)` | Count code files in a directory | - --- ## Usage Examples @@ -129,21 +118,7 @@ export default defineConfig({ ### Semantic analysis (v4.0.0+) -```typescript -import { analyzeProject } from 'ruleprobe'; -import { analyzeProjectSemantic, integrateSemanticResults } from 'ruleprobe/semantic'; -import { resolveSemanticConfig } from 'ruleprobe/semantic/config'; - -const analysis = analyzeProject('./my-project'); -const config = resolveSemanticConfig('./my-project', { anthropicKey: process.env.ANTHROPIC_API_KEY }); - -if (config) { - const rules = analysis.files.flatMap(f => f.ruleSet.rules); - const semanticResult = await analyzeProjectSemantic('./my-project', config, rules); - const enhanced = integrateSemanticResults(analysis, semanticResult); - console.log(`Semantic verdicts: ${semanticResult.report.verdicts.length}`); -} -``` +Semantic analysis runs via the `--semantic` flag on the `analyze` command. The underlying `analyzeProjectSemantic`, `integrateSemanticResults`, and `resolveSemanticConfig` functions live in `src/semantic/` but are not part of the published package's main export. Call semantic analysis through the CLI rather than the programmatic API. --- @@ -195,14 +170,16 @@ interface ProjectAnalysis { } type RuleCategory = - | 'naming' | 'forbidden-pattern' | 'structure' | 'test-requirement' - | 'import-pattern' | 'error-handling' | 'type-safety' | 'code-style' - | 'dependency' | 'preference' | 'file-structure' | 'tooling' - | 'testing' | 'workflow' | 'agent-behavior'; - -type VerifierType = - | 'ast' | 'regex' | 'filesystem' | 'treesitter' - | 'preference' | 'tooling' | 'config-file' | 'git-history'; + | 'naming' + | 'forbidden-pattern' + | 'structure' + | 'import-pattern' + | 'error-handling' + | 'type-safety' + | 'code-style' + | 'agent-behavior'; + +type VerifierType = 'ast' | 'regex' | 'filesystem' | 'treesitter'; type QualifierType = | 'always' | 'prefer' | 'when-possible' From e705a924d3d74c55a62f83e4945a349fb85bbcde Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:55:51 -0600 Subject: [PATCH 02/11] docs: refresh SECURITY.md for v4.5.0 surface and audit status Two stale claims in SECURITY.md: The list of network opt-in flags still referenced ruleprobe run, a command removed in v4.5.0. Trimmed it back to the three flags that actually trigger network calls: --llm-extract, --rubric-decompose, and --semantic. The audit status block still claimed "as of v0.1.0, 5 moderate advisories in esbuild." Current npm audit reports 7 moderate advisories across the vitest chain (vite, esbuild, postcss, brace-expansion). All dev-tooling; none reach the published package. Updated text and removed the unsupported version pin. Closes: docs-security-run-stale, docs-security-audit-stale Co-found: docs-reviewer, security-reviewer --- SECURITY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index dacfbdd..f9e6a78 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -5,7 +5,7 @@ RuleProbe reads files and produces reports. That is the entire operational scope. - **No code execution.** ts-morph parses TypeScript into ASTs for structural analysis. It never runs the TypeScript compiler's emit pipeline and never executes scanned code. -- **No network calls by default.** RuleProbe has zero runtime network dependencies. It does not phone home, fetch updates, or transmit any data. Network calls happen only when you explicitly opt in with `--llm-extract`, `--rubric-decompose`, `--semantic`, or `ruleprobe run`. +- **No network calls by default.** RuleProbe has zero runtime network dependencies. It does not phone home, fetch updates, or transmit any data. Network calls happen only when you explicitly opt in with `--llm-extract`, `--rubric-decompose`, or `--semantic`. - **No file modification.** RuleProbe never writes to the scanned directory. Output goes to stdout or to a user-specified `--output` path, nowhere else. - **No auth, no database, no state.** Each invocation is stateless. Nothing is persisted between runs. @@ -47,7 +47,7 @@ npm run audit ### Current audit status -As of v0.1.0, `npm audit` reports 5 moderate advisories in `esbuild`, a transitive dev dependency of vitest. These affect the vitest development server only and have no impact on RuleProbe's runtime behavior. esbuild is not bundled in the published package. +As of v4.5.0, `npm audit` reports 7 moderate advisories in the `vitest` dev-dependency chain (`vite`, `esbuild`, `postcss`, `brace-expansion`). All four are dev-tooling only. None are reachable at runtime and none are included in the published package; the `files` field in package.json restricts the npm artifact to `dist/`, `action.yml`, and metadata. ## Reporting Security Issues From cf763acf5886912c3dbca935529616a3b5ce5a99 Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:56:16 -0600 Subject: [PATCH 03/11] chore(gitignore): add credential extensions, editor backups, and dedupe The .gitignore was missing a handful of defense-in-depth patterns that are baseline for a Node project, and it had a duplicate entry for .ruleprobe-semantic/. No sensitive files are currently tracked as a result of the gaps, but the gaps create accident surface for future contributors. Added patterns: .env.* glob (with .env.example/.env.template exceptions), credential extensions (*.pem, *.key, *.p12, *.pfx, *.crt), editor backup patterns (*~, *.bak, *.orig, *.swp, *.swo), npm and yarn debug logs, Windows noise (Thumbs.db, desktop.ini), and the local pubprep output directory. Removed the duplicate .ruleprobe-semantic/ line. Closes: L-02-gitignore-gaps Co-found: security-reviewer --- .gitignore | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index d9792f6..989426d 100644 --- a/.gitignore +++ b/.gitignore @@ -3,9 +3,33 @@ dist/ coverage/ *.tsbuildinfo .DS_Store +Thumbs.db +desktop.ini # Environment variables .env +.env.* +!.env.example +!.env.template + +# Credential file extensions +*.pem +*.key +*.p12 +*.pfx +*.crt + +# Editor backups +*~ +*.bak +*.orig +*.swp +*.swo + +# npm and yarn debug logs +npm-debug.log* +yarn-debug.log* +yarn-error.log* # Phase 0 data collection cache and output .cache/ @@ -26,6 +50,9 @@ tests/fixtures/calibration/payloads/*-payload.json .ruleprobe/ test-treesitter*.mjs +# Publication prep tooling output (local only) +.pubprep/ + # Private project docs (keep local only) .github/copilot-instructions.md .github/ruleprobe-e2e-verification-guide.md @@ -33,4 +60,3 @@ docs/ruleprobe-build-guide.md .ruleprobe-semantic/ .codex .codex-tmp/ -.ruleprobe-semantic/ From 9ca847817ae80e97f9ae3c7ada6bdcfd4e555dab Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:56:43 -0600 Subject: [PATCH 04/11] security(ci): pin GitHub Actions to commit SHAs Both the self-check workflow and the published composite action referenced actions/checkout and actions/setup-node by mutable v4 tag. A tag redirect by a compromised maintainer account would run in CI with contents:read and pull-requests:write, exposing the workflow's GITHUB_TOKEN. Pinned actions/checkout to 11bd71901bbe5b1630ceea73d27597364c9af683 (v4.2.2) and actions/setup-node to 49933ea5288caeca8642d1e84afbd3f7d6820020 (v4.4.0). Version tags retained as adjacent comments for human review and Dependabot tag-comment updates. Closes: M-01-a03-ci-sha-pinning Co-found: security-reviewer --- .github/workflows/self-check.yml | 2 +- action.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/self-check.yml b/.github/workflows/self-check.yml index e0ab4da..1c3223e 100644 --- a/.github/workflows/self-check.yml +++ b/.github/workflows/self-check.yml @@ -10,7 +10,7 @@ jobs: contents: read pull-requests: write steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - uses: ./ with: mode: verify diff --git a/action.yml b/action.yml index ca37001..51d0f36 100644 --- a/action.yml +++ b/action.yml @@ -94,7 +94,7 @@ runs: using: "composite" steps: - name: Set up Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version: "22" From 9aa3e4ffd080fcff92110214bfd567f72a2690ee Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:58:24 -0600 Subject: [PATCH 05/11] fix(emitter): keep legacy ESLint output as valid JSON emitLegacyConfig appended a // Unmappable rules comment block after the JSON body whenever any rule from the instruction file did not map to an ESLint rule. The full output was not valid JSON, breaking the JSDoc contract on emitEslintConfig that promises "valid JSON suitable for .eslintrc.json files." The existing test acknowledged this by stripping the comment block before calling JSON.parse, which just tested around the invariant violation. emitLegacyConfig now returns strict JSON only. Unmappable rules are surfaced through a new exported helper formatUnmappableSummary(), which lint-config writes to stderr after the JSON has been emitted. The flat config path is unchanged because flat config is JavaScript and the comments are syntactically valid there. The existing test is rewritten to assert two contracts: the legacy output parses cleanly and contains no // markers, and the summary helper surfaces unmappable entries by id, reason, and source text. Closes: a3f1c8d2 Co-found: tech-debt-reviewer --- src/commands/lint-config.ts | 12 ++++++++- src/emitter/eslint.ts | 51 ++++++++++++++++++++++-------------- tests/emitter/eslint.test.ts | 36 ++++++++++++++++++------- 3 files changed, 68 insertions(+), 31 deletions(-) diff --git a/src/commands/lint-config.ts b/src/commands/lint-config.ts index 3b1b840..605e10a 100644 --- a/src/commands/lint-config.ts +++ b/src/commands/lint-config.ts @@ -8,7 +8,7 @@ import { parseInstructionFile } from '../parsers/index.js'; import { mapRuleSetToEslintConfig } from '../mapper/index.js'; -import { emitEslintConfig } from '../emitter/eslint.js'; +import { emitEslintConfig, formatUnmappableSummary } from '../emitter/eslint.js'; import type { EslintFormat } from '../mapper/types.js'; import { resolveSafePath } from '../utils/safe-path.js'; import { writeFileSync } from 'node:fs'; @@ -54,4 +54,14 @@ export async function handleLintConfig( } else { process.stdout.write(output + '\n'); } + + // Legacy JSON output cannot carry unmappable rules inline (no JSON + // comment syntax). Surface them on stderr so the user still sees + // which instructions had no ESLint equivalent. + if (format === 'legacy') { + const summary = formatUnmappableSummary(eslintConfig); + if (summary !== '') { + process.stderr.write(summary + '\n'); + } + } } \ No newline at end of file diff --git a/src/emitter/eslint.ts b/src/emitter/eslint.ts index 503b4ea..1302d65 100644 --- a/src/emitter/eslint.ts +++ b/src/emitter/eslint.ts @@ -4,12 +4,14 @@ * Serializes an EslintConfig to a runnable ESLint configuration string. * Supports flat config (default) and legacy .eslintrc format. * - * Flat config emits valid ES module JavaScript that can be imported by ESLint. - * Legacy config emits valid JSON suitable for .eslintrc.json files. + * Flat config emits valid ES module JavaScript that can be imported by + * ESLint. Unmappable rules are appended as JS comments, which remain + * valid JavaScript. * - * Unmappable rules are emitted as commented sections with the original - * instruction text and a one-line reason explaining why no ESLint rule - * can enforce them. + * Legacy config emits strictly valid JSON suitable for .eslintrc.json + * files. Unmappable rules are not included in the JSON output because + * JSON has no comment syntax; callers should surface them through a + * sidecar channel via formatUnmappableSummary(). */ import type { EslintConfig, EslintFormat, EslintRuleEntry } from '../mapper/types.js'; @@ -179,27 +181,36 @@ function emitLegacyConfig(config: EslintConfig): string { } configObj['rules'] = rulesObj; - // Serialize to JSON with indentation - const jsonStr = JSON.stringify(configObj, null, 2); + // Legacy output is strictly JSON. Unmappable rules cannot be encoded + // inside the JSON without breaking JSON.parse, so they are omitted + // here. Callers that want to surface them should use + // formatUnmappableSummary() and write to stderr or a sidecar path. + return JSON.stringify(configObj, null, 2); +} - // Append unmappable rules as a comment block after the JSON +/** + * Build a human-readable summary of unmappable rules. + * + * Returns an empty string when there are no unmappable rules. Otherwise + * returns a multi-line block listing each rule's id, reason, and the + * original instruction text. Safe to print to stderr or write alongside + * the legacy JSON output as a sidecar file. + * + * @param config - The mapped ESLint config + * @returns A summary block or empty string when nothing is unmappable + */ +export function formatUnmappableSummary(config: EslintConfig): string { if (config.unmappable.length === 0) { - return jsonStr; + return ''; } - - const commentLines: string[] = [ - '', - '// Unmappable rules (not part of the JSON config):', - '// The following rules have no direct ESLint equivalent.', - '', + const lines: string[] = [ + 'Unmappable rules (not enforceable by ESLint):', ]; for (const rule of config.unmappable) { - commentLines.push(`// [${rule.sourceRuleId}] ${rule.reason}`); - commentLines.push(`// Original: ${rule.sourceText}`); - commentLines.push(''); + lines.push(` [${rule.sourceRuleId}] ${rule.reason}`); + lines.push(` Original: ${rule.sourceText}`); } - - return jsonStr + commentLines.join('\n'); + return lines.join('\n'); } /** diff --git a/tests/emitter/eslint.test.ts b/tests/emitter/eslint.test.ts index 449eb14..2a5ebea 100644 --- a/tests/emitter/eslint.test.ts +++ b/tests/emitter/eslint.test.ts @@ -8,7 +8,7 @@ */ import { describe, it, expect } from 'vitest'; -import { emitEslintConfig } from '../../src/emitter/eslint.js'; +import { emitEslintConfig, formatUnmappableSummary } from '../../src/emitter/eslint.js'; import type { EslintConfig } from '../../src/mapper/types.js'; function makeConfig(overrides: Partial = {}): EslintConfig { @@ -269,7 +269,7 @@ describe('emitEslintConfig', () => { expect(parsed.rules['prefer-const']).toEqual(['warn', { destructuring: 'all' }]); }); - it('emits unmappable rules in a comment block after the JSON', () => { + it('keeps legacy output as pure JSON even when unmappable rules exist', () => { const config = makeConfig({ unmappable: [ { @@ -281,16 +281,32 @@ describe('emitEslintConfig', () => { }); const output = emitEslintConfig(config, 'legacy'); - // Find the JSON portion (ends before the unmappable comment block, or at end) - const commentMarker = '// Unmappable rules'; - const commentStart = output.indexOf(commentMarker); - const jsonPart = commentStart >= 0 ? output.substring(0, commentStart).trimEnd() : output; - const parsed = JSON.parse(jsonPart); + // The entire output must parse as JSON. No trailing comment block, + // no JS-style // markers anywhere. + const parsed = JSON.parse(output); expect(parsed).toHaveProperty('rules'); + expect(output).not.toContain('//'); + }); + + it('exposes unmappable rules through formatUnmappableSummary', () => { + const config = makeConfig({ + unmappable: [ + { + sourceRuleId: 'strict-mode-1', + sourceText: 'Use TypeScript strict mode', + reason: 'TypeScript strict mode is a tsconfig setting', + }, + ], + }); + const summary = formatUnmappableSummary(config); + expect(summary).toContain('strict-mode-1'); + expect(summary).toContain('TypeScript strict mode is a tsconfig setting'); + expect(summary).toContain('Use TypeScript strict mode'); + }); - // The unmappable section must appear after the JSON - expect(output).toContain('// [strict-mode-1]'); - expect(output).toContain('TypeScript strict mode is a tsconfig setting'); + it('returns an empty summary when no rules are unmappable', () => { + const config = makeConfig(); + expect(formatUnmappableSummary(config)).toBe(''); }); }); }); \ No newline at end of file From 26d35bc882503f51ad092f0060511b0337e478d9 Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:58:57 -0600 Subject: [PATCH 06/11] fix(emitter): drop implicit extends from legacy ESLint output emitLegacyConfig unconditionally added eslint:recommended plus plugin:/recommended to the extends field whenever any plugin was referenced. The flat emitter does not do this. The same EslintConfig produced a minimal targeted output in flat format and a full recommended-set output (often hundreds of extra rules) in legacy format, breaking symmetry and surprising users who asked for a targeted config. The legacy emitter now omits extends entirely. Users who want the recommended sets can add `extends` in their own root config; the generated file no longer makes that choice for them. Updated the corresponding test to assert extends is absent. Closes: e5a3c197 Co-found: tech-debt-reviewer --- src/emitter/eslint.ts | 12 ++++-------- tests/emitter/eslint.test.ts | 7 +++---- 2 files changed, 7 insertions(+), 12 deletions(-) diff --git a/src/emitter/eslint.ts b/src/emitter/eslint.ts index 1302d65..875b264 100644 --- a/src/emitter/eslint.ts +++ b/src/emitter/eslint.ts @@ -161,14 +161,10 @@ function emitLegacyConfig(config: EslintConfig): string { configObj['plugins'] = config.plugins; } - // Extends - if (config.plugins.length > 0) { - const extendsList = ['eslint:recommended']; - for (const plugin of config.plugins) { - extendsList.push(`plugin:${plugin}/recommended`); - } - configObj['extends'] = extendsList; - } + // No automatic `extends` block. The flat config emitter does not add + // one and emitting eslint:recommended plus plugin recommended sets + // here would enable hundreds of unrelated rules. Users who want + // recommended sets can extend them in their own config. // Rules as an object with severity + options const rulesObj: Record = {}; diff --git a/tests/emitter/eslint.test.ts b/tests/emitter/eslint.test.ts index 2a5ebea..e9b4ec0 100644 --- a/tests/emitter/eslint.test.ts +++ b/tests/emitter/eslint.test.ts @@ -210,10 +210,9 @@ describe('emitEslintConfig', () => { const parsed = JSON.parse(output); expect(parsed.rules['@typescript-eslint/no-explicit-any']).toBe('error'); expect(parsed.plugins).toEqual(['@typescript-eslint']); - expect(parsed.extends).toEqual([ - 'eslint:recommended', - 'plugin:@typescript-eslint/recommended', - ]); + // Legacy emitter must not auto-enable plugin recommended sets: + // it would enable hundreds of rules the flat emitter does not. + expect(parsed.extends).toBeUndefined(); }); it('emits rules with options in legacy format', () => { From 9447ce70d8ac11ce44ba4672db8a826f59684886 Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 21:59:43 -0600 Subject: [PATCH 07/11] fix(mappings): sync bidirectional table with active per-rule mapper The bidirectional mapping table in src/mappings/index.ts had two entries that did not match the canonical names used by the per-rule mappers in src/mapper/mappings/. The extractor reads this table to reverse-map ESLint rule names back to instruction prose, so the mismatch meant `ruleprobe extract` silently failed to recognize the rules the mapper now correctly emits. - no-enum: was mapped to @typescript-eslint/no-enum (a rule that does not exist in typescript-eslint). Now no-restricted-syntax, matching src/mapper/mappings/type-safety.ts which uses no-restricted-syntax with a TSEnumDeclaration selector. - no-unused-exports: was mapped to no-unused-vars (different semantics, scoped to local variable usage). Now import/no-unused-modules, matching the canonical mapper. Closes: b7e2a941 Co-found: tech-debt-reviewer --- src/mappings/index.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mappings/index.ts b/src/mappings/index.ts index dea49f7..c535279 100644 --- a/src/mappings/index.ts +++ b/src/mappings/index.ts @@ -75,11 +75,11 @@ export const MAPPINGS: MappingEntry[] = [ { patternType: 'throw-error-only', eslintRuleName: 'no-throw-literal', defaultSeverity: 'error', description: 'Only Error objects may be thrown' }, // type safety - { patternType: 'no-enum', eslintRuleName: '@typescript-eslint/no-enum', plugin: '@typescript-eslint', defaultSeverity: 'warn', description: 'Enums must not be used; prefer union types' }, + { patternType: 'no-enum', eslintRuleName: 'no-restricted-syntax', defaultSeverity: 'warn', description: 'Enums must not be used; prefer union types' }, { patternType: 'no-type-assertions', eslintRuleName: '@typescript-eslint/consistent-type-assertions', plugin: '@typescript-eslint', defaultSeverity: 'warn', description: 'Type assertions (as casts) must not be used' }, { patternType: 'no-non-null-assertions', eslintRuleName: '@typescript-eslint/no-non-null-assertion', plugin: '@typescript-eslint', defaultSeverity: 'warn', description: 'Non-null assertions (!) must not be used' }, { patternType: 'no-implicit-any', eslintRuleName: '@typescript-eslint/no-implicit-any', plugin: '@typescript-eslint', defaultSeverity: 'warn', description: 'No implicit any types' }, - { patternType: 'no-unused-exports', eslintRuleName: 'no-unused-vars', plugin: '@typescript-eslint', defaultSeverity: 'warn', description: 'Exported declarations must be imported by other files' }, + { patternType: 'no-unused-exports', eslintRuleName: 'import/no-unused-modules', plugin: 'import', defaultSeverity: 'warn', description: 'Exported declarations must be imported by other files' }, { patternType: 'no-ts-directives', eslintRuleName: '@typescript-eslint/ban-ts-comment', plugin: '@typescript-eslint', defaultSeverity: 'error', description: 'TypeScript suppression directives must not be used' }, // function limits From e3744b8f61575b78af1a789da3a09760f50b3b7d Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 22:00:39 -0600 Subject: [PATCH 08/11] refactor(llm): extract KNOWN_PATTERN_TYPES into a shared module KNOWN_PATTERN_TYPES was defined twice as a module-level const in src/llm/pipeline.ts and src/llm/rubric-pipeline.ts, each with a "keep in sync" comment and no test to enforce equality. The two lists had already diverged: the rubric version included type-aware and tree-sitter entries that the extraction version did not, so an unparseable line involving those checks would degrade the extraction prompt without warning. Moved the canonical list to src/llm/known-patterns.ts and grouped the entries by verifier (AST, regex, filesystem, type-aware, tree-sitter) so the next addition lands in one place. Both pipelines now import the same constant. Closes: c9d4b073 Co-found: tech-debt-reviewer --- src/llm/known-patterns.ts | 60 ++++++++++++++++++++++++++++++++++++++ src/llm/pipeline.ts | 26 +---------------- src/llm/rubric-pipeline.ts | 26 +---------------- 3 files changed, 62 insertions(+), 50 deletions(-) create mode 100644 src/llm/known-patterns.ts diff --git a/src/llm/known-patterns.ts b/src/llm/known-patterns.ts new file mode 100644 index 0000000..c6b713e --- /dev/null +++ b/src/llm/known-patterns.ts @@ -0,0 +1,60 @@ +/** + * Single source of truth for the pattern types LLM prompts may reference. + * + * Both the LLM extraction pipeline and the rubric decomposition + * pipeline send this list to the model so it knows which checks the + * project can actually verify. Previously each pipeline kept its own + * private list, and the two drifted; the rubric list carried type-aware + * and tree-sitter entries that the extraction list did not. Keep this + * file in sync with the verifier switch statements in + * src/verifier/ast-verifier.ts, src/verifier/regex-verifier.ts, and + * src/verifier/file-verifier.ts. + */ + +/** Pattern types verified by the AST verifier (ts-morph). */ +const AST_PATTERN_TYPES = [ + 'camelCase', 'PascalCase', 'no-any', 'no-console-log', 'named-exports', + 'jsdoc-public', 'no-path-aliases', 'no-deep-relative-imports', + 'no-empty-catch', 'no-enum', 'no-type-assertions', 'no-non-null-assertions', + 'throw-error-only', 'no-console-extended', 'no-nested-ternary', + 'no-magic-numbers', 'no-else-after-return', 'max-function-length', + 'max-params', 'no-namespace-imports', 'no-barrel-files', 'no-settimeout-in-tests', + 'no-var', 'prefer-const', 'no-wildcard-exports', +] as const; + +/** Pattern types verified by the regex verifier. */ +const REGEX_PATTERN_TYPES = [ + 'line-length', 'no-ts-directives', 'no-test-only', 'no-test-skip', + 'quote-style', 'banned-import', 'no-todo-comments', 'consistent-semicolons', +] as const; + +/** Pattern types verified by the filesystem verifier. */ +const FILESYSTEM_PATTERN_TYPES = [ + 'kebab-case', 'test-files-exist', 'max-file-length', 'test-file-naming', + 'strict-mode', 'file-exists', 'formatter-config', 'pinned-dependencies', +] as const; + +/** Pattern types verified by the type-aware checks. */ +const TYPE_AWARE_PATTERN_TYPES = [ + 'no-implicit-any', 'no-unused-exports', 'no-unresolved-imports', +] as const; + +/** Pattern types verified by the tree-sitter verifier. */ +const TREE_SITTER_PATTERN_TYPES = [ + 'python-snake-case', 'python-class-naming', 'go-naming', 'function-length', +] as const; + +/** + * The full set of known pattern types passed to LLM prompts. + * + * Spans AST, regex, filesystem, type-aware, and tree-sitter checks. + * Used by both the extraction and rubric decomposition pipelines so + * the model is told the same surface in either path. + */ +export const KNOWN_PATTERN_TYPES: string[] = [ + ...AST_PATTERN_TYPES, + ...REGEX_PATTERN_TYPES, + ...FILESYSTEM_PATTERN_TYPES, + ...TYPE_AWARE_PATTERN_TYPES, + ...TREE_SITTER_PATTERN_TYPES, +]; diff --git a/src/llm/pipeline.ts b/src/llm/pipeline.ts index 8e30af4..5150ad4 100644 --- a/src/llm/pipeline.ts +++ b/src/llm/pipeline.ts @@ -8,31 +8,7 @@ import type { Rule, RuleSet } from '../types.js'; import type { LlmExtractOptions, LlmRuleCandidate } from './types.js'; - -/** - * All known pattern types across all verifiers. - * - * This list is used in LLM prompts so the model knows which - * checks are available. Keep in sync with the verifier switch - * statements in ast-verifier.ts, regex-verifier.ts, and - * file-verifier.ts. - */ -const KNOWN_PATTERN_TYPES: string[] = [ - // AST checks - 'camelCase', 'PascalCase', 'no-any', 'no-console-log', 'named-exports', - 'jsdoc-public', 'no-path-aliases', 'no-deep-relative-imports', - 'no-empty-catch', 'no-enum', 'no-type-assertions', 'no-non-null-assertions', - 'throw-error-only', 'no-console-extended', 'no-nested-ternary', - 'no-magic-numbers', 'no-else-after-return', 'max-function-length', - 'max-params', 'no-namespace-imports', 'no-barrel-files', 'no-settimeout-in-tests', - 'no-var', 'prefer-const', 'no-wildcard-exports', - // Regex checks - 'line-length', 'no-ts-directives', 'no-test-only', 'no-test-skip', - 'quote-style', 'banned-import', 'no-todo-comments', 'consistent-semicolons', - // Filesystem checks - 'kebab-case', 'test-files-exist', 'max-file-length', 'test-file-naming', - 'strict-mode', 'file-exists', 'formatter-config', 'pinned-dependencies', -]; +import { KNOWN_PATTERN_TYPES } from './known-patterns.js'; /** * Run LLM extraction on a RuleSet's unparseable lines. diff --git a/src/llm/rubric-pipeline.ts b/src/llm/rubric-pipeline.ts index ee03931..dec93b8 100644 --- a/src/llm/rubric-pipeline.ts +++ b/src/llm/rubric-pipeline.ts @@ -9,31 +9,7 @@ import type { Rule, RuleSet } from '../types.js'; import type { RubricDecomposer, DecomposedRubric, RubricCheck } from './rubric-types.js'; - -/** - * All known pattern types that rubric checks can reference. - * Kept in sync with the verifier switch statements. - */ -const KNOWN_PATTERN_TYPES: string[] = [ - // AST checks - 'camelCase', 'PascalCase', 'no-any', 'no-console-log', 'named-exports', - 'jsdoc-public', 'no-path-aliases', 'no-deep-relative-imports', - 'no-empty-catch', 'no-enum', 'no-type-assertions', 'no-non-null-assertions', - 'throw-error-only', 'no-console-extended', 'no-nested-ternary', - 'no-magic-numbers', 'no-else-after-return', 'max-function-length', - 'max-params', 'no-namespace-imports', 'no-barrel-files', 'no-settimeout-in-tests', - 'no-var', 'prefer-const', 'no-wildcard-exports', - // Regex checks - 'line-length', 'no-ts-directives', 'no-test-only', 'no-test-skip', - 'quote-style', 'banned-import', 'no-todo-comments', 'consistent-semicolons', - // Filesystem checks - 'kebab-case', 'test-files-exist', 'max-file-length', 'test-file-naming', - 'strict-mode', 'file-exists', 'formatter-config', 'pinned-dependencies', - // Type-aware checks - 'no-implicit-any', 'no-unused-exports', 'no-unresolved-imports', - // Tree-sitter checks - 'python-snake-case', 'python-class-naming', 'go-naming', 'function-length', -]; +import { KNOWN_PATTERN_TYPES } from './known-patterns.js'; /** Options for rubric decomposition. */ export interface RubricDecomposeOptions { From 326ec38ab124905d08054f43c30cf0c26c4a1576 Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 22:02:02 -0600 Subject: [PATCH 09/11] feat(llm): retry transient 429 and 503 responses from OpenAI The OpenAI provider made a single fetch and surfaced any non-ok response as an error. A transient rate-limit (429) or upstream outage (503) silently dropped the batch of unparseable lines for that call, even though the failure mode was retriable. Static extraction always covers the same lines on the next run, so the failure went unnoticed. The provider now retries 429 and 503 responses up to maxAttempts (default 3), with exponential backoff starting at 1000ms or the Retry-After header value when present. Non-transient statuses (400, 401, 500, etc.) still surface immediately because retrying will not fix a misconfigured request. The provider config grew three injectable knobs (maxAttempts, retryBaseDelayMs, sleep, fetchImpl) so the retry path is testable without a network. New tests cover 429/503 retry, non-retryable errors, exhaustion, and Retry-After honoring. Closes: a8c5d620 Co-found: tech-debt-reviewer --- src/llm/openai-provider.ts | 117 ++++++++++++++++++------ tests/llm/openai-provider.test.ts | 145 ++++++++++++++++++++++++++++++ 2 files changed, 237 insertions(+), 25 deletions(-) create mode 100644 tests/llm/openai-provider.test.ts diff --git a/src/llm/openai-provider.ts b/src/llm/openai-provider.ts index 197402f..3cf5ecf 100644 --- a/src/llm/openai-provider.ts +++ b/src/llm/openai-provider.ts @@ -19,6 +19,47 @@ export interface OpenAiProviderConfig { baseUrl?: string; /** Request timeout in milliseconds. Defaults to 30000. */ timeoutMs?: number; + /** + * Total attempts on transient errors (429 and 503). Defaults to 3. + * Set to 1 to disable retries. The first attempt counts. + */ + maxAttempts?: number; + /** + * Initial backoff in milliseconds between retries. Doubled each + * subsequent attempt. Honored only when the response has no + * Retry-After header. Defaults to 1000. + */ + retryBaseDelayMs?: number; + /** + * Sleep implementation, overridable for tests. Defaults to + * setTimeout-based wait. + */ + sleep?: (ms: number) => Promise; + /** + * Fetch implementation, overridable for tests. Defaults to global fetch. + */ + fetchImpl?: typeof fetch; +} + +/** Parse a Retry-After header into milliseconds. Returns null if absent or invalid. */ +function parseRetryAfter(header: string | null): number | null { + if (header === null || header === '') { + return null; + } + const seconds = Number(header); + if (Number.isFinite(seconds) && seconds >= 0) { + return Math.round(seconds * 1000); + } + const date = Date.parse(header); + if (!Number.isNaN(date)) { + return Math.max(0, date - Date.now()); + } + return null; +} + +/** Default sleep implementation using setTimeout. */ +function defaultSleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); } /** @@ -42,6 +83,10 @@ export function createOpenAiProvider(config: OpenAiProviderConfig = {}): LlmProv const model = config.model ?? 'gpt-4o-mini'; const baseUrl = (config.baseUrl ?? 'https://api.openai.com/v1').replace(/\/$/, ''); const timeoutMs = config.timeoutMs ?? 30000; + const maxAttempts = Math.max(1, config.maxAttempts ?? 3); + const retryBaseDelayMs = config.retryBaseDelayMs ?? 1000; + const sleep = config.sleep ?? defaultSleep; + const fetchFn = config.fetchImpl ?? fetch; return { name: `openai/${model}`, @@ -55,33 +100,54 @@ export function createOpenAiProvider(config: OpenAiProviderConfig = {}): LlmProv } const prompt = buildExtractionPrompt(lines, knownPatternTypes); + const body = JSON.stringify({ + model, + messages: [ + { role: 'system', content: prompt.system }, + { role: 'user', content: prompt.user }, + ], + temperature: 0, + response_format: { type: 'json_object' }, + }); - const controller = new AbortController(); - const timer = setTimeout(() => controller.abort(), timeoutMs); - - try { - const response = await fetch(`${baseUrl}/chat/completions`, { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'Authorization': `Bearer ${apiKey}`, - }, - body: JSON.stringify({ - model, - messages: [ - { role: 'system', content: prompt.system }, - { role: 'user', content: prompt.user }, - ], - temperature: 0, - response_format: { type: 'json_object' }, - }), - signal: controller.signal, - }); + for (let attempt = 1; attempt <= maxAttempts; attempt++) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + + let response: Response; + try { + response = await fetchFn(`${baseUrl}/chat/completions`, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': `Bearer ${apiKey}`, + }, + body, + signal: controller.signal, + }); + } finally { + clearTimeout(timer); + } + + // Retry only on 429 (rate limit) and 503 (service unavailable). + // Other non-ok statuses surface as errors immediately because + // they indicate a config or request problem retrying cannot + // fix. + const retryable = response.status === 429 || response.status === 503; + if (!response.ok && retryable && attempt < maxAttempts) { + const retryAfter = parseRetryAfter(response.headers.get('Retry-After')); + const backoff = retryAfter ?? retryBaseDelayMs * Math.pow(2, attempt - 1); + process.stderr.write( + `OpenAI API returned ${response.status}; retrying in ${backoff}ms (attempt ${attempt}/${maxAttempts})\n`, + ); + await sleep(backoff); + continue; + } if (!response.ok) { - const body = await response.text(); + const errBody = await response.text(); throw new Error( - `OpenAI API error ${response.status}: ${body.slice(0, 200)}`, + `OpenAI API error ${response.status}: ${errBody.slice(0, 200)}`, ); } @@ -95,9 +161,10 @@ export function createOpenAiProvider(config: OpenAiProviderConfig = {}): LlmProv } return parseExtractionResponse(content, lines, knownPatternTypes); - } finally { - clearTimeout(timer); } + + // Loop exits via return or throw; unreachable in practice. + throw new Error('OpenAI API retries exhausted without a final response'); }, }; } diff --git a/tests/llm/openai-provider.test.ts b/tests/llm/openai-provider.test.ts new file mode 100644 index 0000000..d8fb337 --- /dev/null +++ b/tests/llm/openai-provider.test.ts @@ -0,0 +1,145 @@ +/** + * Tests for the OpenAI provider's retry behavior on transient errors. + * + * These tests inject a fake fetch implementation so we can drive the + * retry path deterministically without touching the network. + */ + +import { describe, it, expect } from 'vitest'; +import { createOpenAiProvider } from '../../src/llm/openai-provider.js'; + +const PROMPT_PATTERNS = ['camelCase', 'no-any']; + +/** Build a Response-like object with the given status, body, and headers. */ +function makeResponse( + status: number, + body: unknown, + headers: Record = {}, +): Response { + return new Response(typeof body === 'string' ? body : JSON.stringify(body), { + status, + headers, + }); +} + +/** Body that parses through the extractor without errors. */ +const SUCCESS_BODY = { + choices: [ + { + message: { + content: JSON.stringify({ rules: [], unparseable_lines: [] }), + }, + }, + ], +}; + +describe('createOpenAiProvider retry behavior', () => { + it('retries on 429 and succeeds on a follow-up 200', async () => { + const calls: number[] = []; + const fetchImpl = async (): Promise => { + calls.push(Date.now()); + if (calls.length === 1) { + return makeResponse(429, '', { 'Retry-After': '0' }); + } + return makeResponse(200, SUCCESS_BODY); + }; + + const provider = createOpenAiProvider({ + apiKey: 'test-key', + maxAttempts: 3, + sleep: async () => undefined, + fetchImpl, + }); + + await provider.extractRules(['some unparseable line'], PROMPT_PATTERNS); + expect(calls.length).toBe(2); + }); + + it('retries on 503 and succeeds on a follow-up 200', async () => { + const responses: Response[] = [ + makeResponse(503, ''), + makeResponse(200, SUCCESS_BODY), + ]; + let i = 0; + const fetchImpl = async (): Promise => { + const next = responses[i] ?? makeResponse(500, 'unexpected'); + i += 1; + return next; + }; + + const provider = createOpenAiProvider({ + apiKey: 'test-key', + maxAttempts: 3, + sleep: async () => undefined, + fetchImpl, + }); + + await provider.extractRules(['line'], PROMPT_PATTERNS); + expect(i).toBe(2); + }); + + it('does not retry on 400 (non-transient error)', async () => { + let calls = 0; + const fetchImpl = async (): Promise => { + calls += 1; + return makeResponse(400, 'bad request'); + }; + + const provider = createOpenAiProvider({ + apiKey: 'test-key', + maxAttempts: 3, + sleep: async () => undefined, + fetchImpl, + }); + + await expect(provider.extractRules(['line'], PROMPT_PATTERNS)).rejects.toThrow( + /OpenAI API error 400/, + ); + expect(calls).toBe(1); + }); + + it('gives up after maxAttempts when all attempts return 429', async () => { + let calls = 0; + const fetchImpl = async (): Promise => { + calls += 1; + return makeResponse(429, '', { 'Retry-After': '0' }); + }; + + const provider = createOpenAiProvider({ + apiKey: 'test-key', + maxAttempts: 2, + sleep: async () => undefined, + fetchImpl, + }); + + await expect(provider.extractRules(['line'], PROMPT_PATTERNS)).rejects.toThrow( + /OpenAI API error 429/, + ); + expect(calls).toBe(2); + }); + + it('honors the Retry-After header value when present', async () => { + const sleeps: number[] = []; + const responses: Response[] = [ + makeResponse(429, '', { 'Retry-After': '2' }), + makeResponse(200, SUCCESS_BODY), + ]; + let i = 0; + const fetchImpl = async (): Promise => { + const next = responses[i] ?? makeResponse(500, 'unexpected'); + i += 1; + return next; + }; + + const provider = createOpenAiProvider({ + apiKey: 'test-key', + maxAttempts: 3, + retryBaseDelayMs: 9999, + sleep: async (ms) => { sleeps.push(ms); }, + fetchImpl, + }); + + await provider.extractRules(['line'], PROMPT_PATTERNS); + expect(sleeps).toEqual([2000]); + }); +}); From d78571ddb5132bbf3c32fa6cfc0f91624e68527a Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 22:03:45 -0600 Subject: [PATCH 10/11] refactor(types): split types.ts into core and results modules src/types.ts sat at 293 lines, 7 short of the self-check 300-line limit. The next substantive type addition would have tripped the self-check workflow on the PR that introduced it; src/types.ts is imported by nearly every feature module, so any new type ends up here. Split into two files: - src/types-core.ts: instruction-file and rule primitives (Rule, RuleSet, VerificationPattern, RuleCategory, VerifierType, QualifierType, InstructionFileType, MarkdownSection, RuleMatcher, INSTRUCTION_FILE_NAMES) - src/types-results.ts: verification and report types (Evidence, RuleResult, ReportSummary, AdherenceReport, AgentRun, TaskTemplate, ReportFormat, CrossFileConflict, CrossFileRedundancy, FileAnalysis, ProjectAnalysis, DEFAULT_COMPLIANCE_THRESHOLD) src/types.ts is now a 39-line barrel that re-exports both. Every existing `from '../types.js'` import keeps working with no call-site changes. Closes: d2e6f318 Co-found: tech-debt-reviewer --- src/types-core.ts | 152 ++++++++++++++++++++ src/types-results.ts | 161 +++++++++++++++++++++ src/types.ts | 324 +++++-------------------------------------- 3 files changed, 348 insertions(+), 289 deletions(-) create mode 100644 src/types-core.ts create mode 100644 src/types-results.ts diff --git a/src/types-core.ts b/src/types-core.ts new file mode 100644 index 0000000..ded859b --- /dev/null +++ b/src/types-core.ts @@ -0,0 +1,152 @@ +/** + * Core type primitives for RuleProbe. + * + * These types describe instruction files, the rules extracted from + * them, and the matchers that drive extraction. Result and report + * types live in types-results.ts so that adding a new result field + * does not push the file past the 300-line self-check limit. + * + * Re-exported by types.ts; prefer importing from '../types.js' in + * most call sites so callers see a single surface. + */ + +/** Categories of machine-verifiable rules extracted from instruction files. */ +export type RuleCategory = + | 'naming' + | 'forbidden-pattern' + | 'structure' + | 'import-pattern' + | 'error-handling' + | 'type-safety' + | 'code-style' + | 'agent-behavior'; + +/** Which verification engine handles a given rule. */ +export type VerifierType = 'ast' | 'regex' | 'filesystem' | 'treesitter'; + +/** + * Qualifier describing the strength of an instruction. + * + * Detected via deterministic keyword/phrase matching on the rule text + * during extraction. Rules with no qualifier keyword default to 'always'. + */ +export type QualifierType = + | 'always' + | 'prefer' + | 'when-possible' + | 'avoid-unless' + | 'try-to' + | 'never'; + +/** Instruction file format detected from the file path. */ +export type InstructionFileType = + | 'claude.md' + | 'agents.md' + | 'cursorrules' + | 'copilot-instructions' + | 'gemini.md' + | 'windsurfrules' + | 'rules' + | 'generic-markdown' + | 'unknown'; + +/** Describes the specific check a verifier runs for a rule. */ +export interface VerificationPattern { + /** The kind of check, e.g. "camelCase", "no-any", "file-exists". */ + type: string; + /** What to check, e.g. "variables", "*.ts", "src/". */ + target: string; + /** The expected value, pattern, or boolean condition. */ + expected: string | boolean; + /** Whether the check applies per-file or across the whole project. */ + scope: 'file' | 'project'; +} + +/** A single machine-verifiable rule extracted from an instruction file. */ +export interface Rule { + /** Unique identifier, e.g. "naming-camelcase-variables". */ + id: string; + /** Which category this rule belongs to. */ + category: RuleCategory; + /** The raw text from the instruction file that produced this rule. */ + source: string; + /** Human-readable summary of what the rule checks. */ + description: string; + /** Whether a violation is an error or a warning. */ + severity: 'error' | 'warning'; + /** Which verification engine handles this rule. */ + verifier: VerifierType; + /** The specific check to run. */ + pattern: VerificationPattern; + /** Confidence level of the extraction (high = exact keyword match). */ + confidence?: 'high' | 'medium' | 'low'; + /** How this rule was extracted. */ + extractionMethod?: 'static' | 'llm' | 'rubric' | 'rubric-deterministic' | 'custom'; + /** Weight within a rubric (0-1). Only set for rubric-decomposed rules. */ + rubricWeight?: number; + /** The markdown section header this rule was found under. */ + section?: string; + /** Qualifier strength detected from the instruction text. */ + qualifier?: QualifierType; +} + +/** A complete set of rules extracted from a single instruction file. */ +export interface RuleSet { + /** Path to the instruction file that was parsed. */ + sourceFile: string; + /** Detected file format. */ + sourceType: InstructionFileType; + /** All machine-verifiable rules that were extracted. */ + rules: Rule[]; + /** Lines from the instruction file that could not be converted to rules. */ + unparseable: string[]; +} + +/** A parsed section from a markdown instruction file. */ +export interface MarkdownSection { + /** The header text (without leading # characters). */ + header: string; + /** Header depth (1 for #, 2 for ##, etc). */ + depth: number; + /** The body content under this header, as raw text. */ + body: string; + /** Lines within the body, trimmed and filtered for empties. */ + lines: string[]; +} + +/** + * A matcher definition that maps natural language patterns in instruction + * files to structured, machine-verifiable rules. + */ +export interface RuleMatcher { + /** Unique identifier prefix for rules produced by this matcher. */ + id: string; + /** Regex patterns that match instruction lines this rule covers. */ + patterns: RegExp[]; + /** The rule category. */ + category: RuleCategory; + /** Which verifier handles this rule. */ + verifier: VerifierType; + /** Human-readable description of what this rule checks. */ + description: string; + /** Default severity. */ + severity: 'error' | 'warning'; + /** Confidence level for rules produced by this matcher. */ + confidence?: 'high' | 'medium' | 'low'; + /** Build the verification pattern from the matched line. */ + buildPattern: (line: string, match: RegExpMatchArray) => VerificationPattern; +} + +/** + * Recognized instruction file names. + * Used by project-level discovery to find all instruction files in a repo. + */ +export const INSTRUCTION_FILE_NAMES = [ + 'CLAUDE.md', + 'AGENTS.md', + '.cursorrules', + '.github/copilot-instructions.md', + 'GEMINI.md', + '.windsurfrules', + '.rules', +] as const; diff --git a/src/types-results.ts b/src/types-results.ts new file mode 100644 index 0000000..0f4d0ca --- /dev/null +++ b/src/types-results.ts @@ -0,0 +1,161 @@ +/** + * Result, report, and project-analysis types for RuleProbe. + * + * Built on the primitives in types-core.ts. Split from types.ts to + * keep both files under the 300-line self-check limit. Re-exported + * by types.ts; prefer importing from '../types.js' in most call + * sites so callers see a single surface. + */ + +import type { + Rule, + RuleCategory, + RuleSet, + InstructionFileType, +} from './types-core.js'; + +/** Valid output format for adherence reports. */ +export type ReportFormat = 'text' | 'json' | 'markdown' | 'rdjson' | 'summary' | 'detailed' | 'ci'; + +/** A standardized coding task designed to exercise rule categories. */ +export interface TaskTemplate { + /** Unique identifier, e.g. "rest-endpoint". */ + id: string; + /** Human-readable name, e.g. "REST API Endpoint". */ + name: string; + /** The full prompt given to the coding agent. */ + prompt: string; + /** Files the agent output should contain. */ + expectedFiles: string[]; + /** Which rule categories this task exercises. */ + exercises: RuleCategory[]; +} + +/** Metadata about a single agent run. */ +export interface AgentRun { + /** Agent identifier, e.g. "claude-code", "copilot", "cursor". */ + agent: string; + /** Model version, e.g. "opus-4.6". */ + model: string; + /** Which task template was given to the agent. */ + taskTemplateId: string; + /** Path to the directory containing agent output files. */ + outputDir: string; + /** ISO 8601 timestamp of when the run started. */ + timestamp: string; + /** How long the agent took, or null if not measured. */ + durationSeconds: number | null; +} + +/** A piece of evidence supporting a rule result (pass or fail). */ +export interface Evidence { + /** The file where the check was performed. */ + file: string; + /** Line number of the finding, or null for file-level checks. */ + line: number | null; + /** What was actually found in the code. */ + found: string; + /** What the rule required. */ + expected: string; + /** Surrounding code for readability. */ + context: string; +} + +/** The result of checking a single rule against agent output. */ +export interface RuleResult { + /** The rule that was checked. */ + rule: Rule; + /** Whether the agent output conformed to this rule. */ + passed: boolean; + /** Compliance ratio from 0 to 1. Binary checks return 0 or 1. Pattern checks return the ratio. */ + compliance: number; + /** Evidence of what was checked and found. */ + evidence: Evidence[]; + /** Whether this rule was skipped because it has no concrete implementation. */ + skipped?: boolean; +} + +/** Per-category breakdown of pass/total counts. */ +export interface CategoryScore { + passed: number; + total: number; +} + +/** Summary statistics for an adherence report. */ +export interface ReportSummary { + /** Total number of rules checked. */ + totalRules: number; + /** Number of rules that passed. */ + passed: number; + /** Number of rules that failed. */ + failed: number; + /** Number of rules skipped (present in ruleset but excluded from verification, e.g. by severity filter). */ + skipped: number; + /** Number of warnings (failed rules with severity "warning"). */ + warnings: number; + /** Adherence score as a percentage (passed / totalRules * 100). */ + adherenceScore: number; + /** Pass/total breakdown by rule category. */ + byCategory: Record; +} + +/** A complete adherence report for a single agent run. */ +export interface AdherenceReport { + /** Metadata about the agent run. */ + run: AgentRun; + /** The rules that were checked. */ + ruleset: RuleSet; + /** Individual results for each rule. */ + results: RuleResult[]; + /** Aggregate summary. */ + summary: ReportSummary; +} + +/** A conflict between rules in different instruction files. */ +export interface CrossFileConflict { + /** Topic or pattern category the conflict relates to. */ + topic: string; + /** Rules from different files that contradict each other. */ + rules: Array<{ file: string; rule: Rule }>; + /** Description of the conflict. */ + description: string; +} + +/** A redundancy: the same instruction appearing in multiple files. */ +export interface CrossFileRedundancy { + /** Normalized text of the redundant instruction. */ + normalizedText: string; + /** Occurrences across files. */ + occurrences: Array<{ file: string; originalText: string }>; +} + +/** Per-file analysis result within a project. */ +export interface FileAnalysis { + /** Path to the instruction file. */ + filePath: string; + /** Detected file format. */ + fileType: InstructionFileType; + /** Rules extracted from this file. */ + ruleSet: RuleSet; + /** Verification results (populated after verification). */ + results: RuleResult[]; +} + +/** Complete project-level analysis across all instruction files. */ +export interface ProjectAnalysis { + /** Root directory of the project. */ + projectDir: string; + /** Per-file analysis results. */ + files: FileAnalysis[]; + /** Cross-file conflicts (same topic, different instructions). */ + conflicts: CrossFileConflict[]; + /** Cross-file redundancies (same instruction, different wording). */ + redundancies: CrossFileRedundancy[]; + /** Map of rule categories to which files contain rules in that category. */ + coverageMap: Record; + /** Aggregate summary across all files. */ + summary: ReportSummary; +} + +/** Default compliance threshold for determining pass/fail from compliance ratios. */ +export const DEFAULT_COMPLIANCE_THRESHOLD = 0.8; diff --git a/src/types.ts b/src/types.ts index 6d708f1..50d3a85 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1,293 +1,39 @@ /** - * All shared types for RuleProbe. + * Public type surface for RuleProbe. * - * These types define the data structures flowing through the core pipeline: - * instruction file parsing, rule extraction, verification, and reporting. + * Re-exports the core type primitives and the result/report types + * from their respective modules. Existing imports `from '../types.js'` + * keep working; new code may import from types-core.ts or + * types-results.ts directly when only one half is needed. */ -/** Valid output format for adherence reports. */ -export type ReportFormat = 'text' | 'json' | 'markdown' | 'rdjson' | 'summary' | 'detailed' | 'ci'; - -/** Categories of machine-verifiable rules extracted from instruction files. */ -export type RuleCategory = - | 'naming' - | 'forbidden-pattern' - | 'structure' - | 'import-pattern' - | 'error-handling' - | 'type-safety' - | 'code-style' - | 'agent-behavior'; - -/** Which verification engine handles a given rule. */ -export type VerifierType = 'ast' | 'regex' | 'filesystem' | 'treesitter'; - -/** - * Qualifier describing the strength of an instruction. - * - * Detected via deterministic keyword/phrase matching on the rule text - * during extraction. Rules with no qualifier keyword default to 'always'. - */ -export type QualifierType = - | 'always' - | 'prefer' - | 'when-possible' - | 'avoid-unless' - | 'try-to' - | 'never'; - -/** Instruction file format detected from the file path. */ -export type InstructionFileType = - | 'claude.md' - | 'agents.md' - | 'cursorrules' - | 'copilot-instructions' - | 'gemini.md' - | 'windsurfrules' - | 'rules' - | 'generic-markdown' - | 'unknown'; - -/** Describes the specific check a verifier runs for a rule. */ -export interface VerificationPattern { - /** The kind of check, e.g. "camelCase", "no-any", "file-exists". */ - type: string; - /** What to check, e.g. "variables", "*.ts", "src/". */ - target: string; - /** The expected value, pattern, or boolean condition. */ - expected: string | boolean; - /** Whether the check applies per-file or across the whole project. */ - scope: 'file' | 'project'; -} - -/** A single machine-verifiable rule extracted from an instruction file. */ -export interface Rule { - /** Unique identifier, e.g. "naming-camelcase-variables". */ - id: string; - /** Which category this rule belongs to. */ - category: RuleCategory; - /** The raw text from the instruction file that produced this rule. */ - source: string; - /** Human-readable summary of what the rule checks. */ - description: string; - /** Whether a violation is an error or a warning. */ - severity: 'error' | 'warning'; - /** Which verification engine handles this rule. */ - verifier: VerifierType; - /** The specific check to run. */ - pattern: VerificationPattern; - /** Confidence level of the extraction (high = exact keyword match). */ - confidence?: 'high' | 'medium' | 'low'; - /** How this rule was extracted. */ - extractionMethod?: 'static' | 'llm' | 'rubric' | 'rubric-deterministic' | 'custom'; - /** Weight within a rubric (0-1). Only set for rubric-decomposed rules. */ - rubricWeight?: number; - /** The markdown section header this rule was found under. */ - section?: string; - /** Qualifier strength detected from the instruction text. */ - qualifier?: QualifierType; -} - -/** A complete set of rules extracted from a single instruction file. */ -export interface RuleSet { - /** Path to the instruction file that was parsed. */ - sourceFile: string; - /** Detected file format. */ - sourceType: InstructionFileType; - /** All machine-verifiable rules that were extracted. */ - rules: Rule[]; - /** Lines from the instruction file that could not be converted to rules. */ - unparseable: string[]; -} - -/** A standardized coding task designed to exercise rule categories. */ -export interface TaskTemplate { - /** Unique identifier, e.g. "rest-endpoint". */ - id: string; - /** Human-readable name, e.g. "REST API Endpoint". */ - name: string; - /** The full prompt given to the coding agent. */ - prompt: string; - /** Files the agent output should contain. */ - expectedFiles: string[]; - /** Which rule categories this task exercises. */ - exercises: RuleCategory[]; -} - -/** Metadata about a single agent run. */ -export interface AgentRun { - /** Agent identifier, e.g. "claude-code", "copilot", "cursor". */ - agent: string; - /** Model version, e.g. "opus-4.6". */ - model: string; - /** Which task template was given to the agent. */ - taskTemplateId: string; - /** Path to the directory containing agent output files. */ - outputDir: string; - /** ISO 8601 timestamp of when the run started. */ - timestamp: string; - /** How long the agent took, or null if not measured. */ - durationSeconds: number | null; -} - -/** A piece of evidence supporting a rule result (pass or fail). */ -export interface Evidence { - /** The file where the check was performed. */ - file: string; - /** Line number of the finding, or null for file-level checks. */ - line: number | null; - /** What was actually found in the code. */ - found: string; - /** What the rule required. */ - expected: string; - /** Surrounding code for readability. */ - context: string; -} - -/** The result of checking a single rule against agent output. */ -export interface RuleResult { - /** The rule that was checked. */ - rule: Rule; - /** Whether the agent output conformed to this rule. */ - passed: boolean; - /** Compliance ratio from 0 to 1. Binary checks return 0 or 1. Pattern checks return the ratio. */ - compliance: number; - /** Evidence of what was checked and found. */ - evidence: Evidence[]; - /** Whether this rule was skipped because it has no concrete implementation. */ - skipped?: boolean; -} - -/** Per-category breakdown of pass/total counts. */ -export interface CategoryScore { - passed: number; - total: number; -} - -/** Summary statistics for an adherence report. */ -export interface ReportSummary { - /** Total number of rules checked. */ - totalRules: number; - /** Number of rules that passed. */ - passed: number; - /** Number of rules that failed. */ - failed: number; - /** Number of rules skipped (present in ruleset but excluded from verification, e.g. by severity filter). */ - skipped: number; - /** Number of warnings (failed rules with severity "warning"). */ - warnings: number; - /** Adherence score as a percentage (passed / totalRules * 100). */ - adherenceScore: number; - /** Pass/total breakdown by rule category. */ - byCategory: Record; -} - -/** A complete adherence report for a single agent run. */ -export interface AdherenceReport { - /** Metadata about the agent run. */ - run: AgentRun; - /** The rules that were checked. */ - ruleset: RuleSet; - /** Individual results for each rule. */ - results: RuleResult[]; - /** Aggregate summary. */ - summary: ReportSummary; -} - -/** A parsed section from a markdown instruction file. */ -export interface MarkdownSection { - /** The header text (without leading # characters). */ - header: string; - /** Header depth (1 for #, 2 for ##, etc). */ - depth: number; - /** The body content under this header, as raw text. */ - body: string; - /** Lines within the body, trimmed and filtered for empties. */ - lines: string[]; -} - -/** - * A matcher definition that maps natural language patterns in instruction - * files to structured, machine-verifiable rules. - */ -export interface RuleMatcher { - /** Unique identifier prefix for rules produced by this matcher. */ - id: string; - /** Regex patterns that match instruction lines this rule covers. */ - patterns: RegExp[]; - /** The rule category. */ - category: RuleCategory; - /** Which verifier handles this rule. */ - verifier: VerifierType; - /** Human-readable description of what this rule checks. */ - description: string; - /** Default severity. */ - severity: 'error' | 'warning'; - /** Confidence level for rules produced by this matcher. */ - confidence?: 'high' | 'medium' | 'low'; - /** Build the verification pattern from the matched line. */ - buildPattern: (line: string, match: RegExpMatchArray) => VerificationPattern; -} - -/** - * Recognized instruction file names. - * Used by project-level discovery to find all instruction files in a repo. - */ -export const INSTRUCTION_FILE_NAMES = [ - 'CLAUDE.md', - 'AGENTS.md', - '.cursorrules', - '.github/copilot-instructions.md', - 'GEMINI.md', - '.windsurfrules', - '.rules', -] as const; - -/** A conflict between rules in different instruction files. */ -export interface CrossFileConflict { - /** Topic or pattern category the conflict relates to. */ - topic: string; - /** Rules from different files that contradict each other. */ - rules: Array<{ file: string; rule: Rule }>; - /** Description of the conflict. */ - description: string; -} - -/** A redundancy: the same instruction appearing in multiple files. */ -export interface CrossFileRedundancy { - /** Normalized text of the redundant instruction. */ - normalizedText: string; - /** Occurrences across files. */ - occurrences: Array<{ file: string; originalText: string }>; -} - -/** Per-file analysis result within a project. */ -export interface FileAnalysis { - /** Path to the instruction file. */ - filePath: string; - /** Detected file format. */ - fileType: InstructionFileType; - /** Rules extracted from this file. */ - ruleSet: RuleSet; - /** Verification results (populated after verification). */ - results: RuleResult[]; -} - -/** Complete project-level analysis across all instruction files. */ -export interface ProjectAnalysis { - /** Root directory of the project. */ - projectDir: string; - /** Per-file analysis results. */ - files: FileAnalysis[]; - /** Cross-file conflicts (same topic, different instructions). */ - conflicts: CrossFileConflict[]; - /** Cross-file redundancies (same instruction, different wording). */ - redundancies: CrossFileRedundancy[]; - /** Map of rule categories to which files contain rules in that category. */ - coverageMap: Record; - /** Aggregate summary across all files. */ - summary: ReportSummary; -} - -/** Default compliance threshold for determining pass/fail from compliance ratios. */ -export const DEFAULT_COMPLIANCE_THRESHOLD = 0.8; +export type { + RuleCategory, + VerifierType, + QualifierType, + InstructionFileType, + VerificationPattern, + Rule, + RuleSet, + MarkdownSection, + RuleMatcher, +} from './types-core.js'; + +export { INSTRUCTION_FILE_NAMES } from './types-core.js'; + +export type { + ReportFormat, + TaskTemplate, + AgentRun, + Evidence, + RuleResult, + CategoryScore, + ReportSummary, + AdherenceReport, + CrossFileConflict, + CrossFileRedundancy, + FileAnalysis, + ProjectAnalysis, +} from './types-results.js'; + +export { DEFAULT_COMPLIANCE_THRESHOLD } from './types-results.js'; From 44464aaca63d3a3830202bf876b710741cd21434 Mon Sep 17 00:00:00 2001 From: Brad Kinnard Date: Fri, 22 May 2026 22:05:04 -0600 Subject: [PATCH 11/11] security(docs): document LLM call budgets per opt-in mode The security review flagged LLM10:2025 unbounded consumption for the opt-in LLM paths. Each path already has an effective per-invocation budget in code, but the budget model was not documented and the review treated it as absent. - --llm-extract: one OpenAI call per invocation (batch of 50 lines), 3 retry attempts on 429/503 only. - --rubric-decompose: one OpenAI call per invocation (batch of 20). - --semantic: bounded by --max-llm-calls (default 20), already enforced by the semantic engine and tested. Added a SECURITY.md section spelling this out so operators know their worst-case cost ceiling without reading the source. Closes: L-03-llm10-unbounded-calls Co-found: security-reviewer --- SECURITY.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/SECURITY.md b/SECURITY.md index f9e6a78..3ff90fb 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -23,6 +23,24 @@ What is NEVER sent: - Source code, variable names, function names, string literals - Comments, import paths, module names, file paths, scope names +## LLM Call Budgets + +External LLM calls happen only on the opt-in `--llm-extract`, +`--rubric-decompose`, and `--semantic` paths, and each has an upper +bound on calls per invocation: + +- `--llm-extract`: at most one OpenAI call per invocation. Unparseable + lines are sent as a single batch (default 50 lines). Transient 429 + and 503 responses are retried up to 3 times with exponential + backoff; non-transient errors fail immediately. +- `--rubric-decompose`: at most one OpenAI call per invocation, batch + of 20 unparseable lines. +- `--semantic`: capped by `--max-llm-calls` (default 20). The semantic + engine logs and stops escalation when the budget is hit. + +If you supply your own API key, these budgets define the maximum cost +ceiling for a single ruleprobe invocation against your account. + ## Path Traversal Protection User-supplied paths (instruction files and output directories) are resolved and bounded to the current working directory before any filesystem operation.