From 6262c46ad00dc99179ae5e13b7ba6459ab8621c1 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Mon, 29 Jun 2026 14:30:04 -0700 Subject: [PATCH 01/18] feat(warden): Add semantic review chunk planning Add a model-backed semantic chunk planner that groups atomic review chunks before scanner execution. The planner owns semantic titles and summaries, validates planned chunk membership and size limits, and feeds scanner prompts with the resulting review chunks. Wire semantic chunking through SDK and CLI execution, share planning across skill tasks, and enable the behavior in this repo's Warden config. Support semantic groups that span multiple files, align evals with the Pi/OpenRouter default, and prove cross-file semantic grouping with a live eval. Co-Authored-By: GPT-5 Codex --- .agents/skills/warden-qa/SKILL.md | 74 +++++ .github/workflows/evals.yml | 6 +- .github/workflows/warden.yml | 2 + packages/evals/eval-bug-detection.yaml | 2 +- packages/evals/eval-precision.yaml | 2 +- packages/evals/eval-security-scanning.yaml | 2 +- packages/evals/src/auth.ts | 46 +++ packages/evals/src/code-review.eval.ts | 11 +- packages/evals/src/e2e.eval.ts | 14 +- packages/evals/src/harness.ts | 8 +- packages/evals/src/judge.ts | 78 ++--- packages/evals/src/security-review.eval.ts | 11 +- packages/evals/src/semantic-chunking.eval.ts | 123 ++++++++ packages/evals/src/types.ts | 4 +- packages/evals/src/verify.eval.ts | 15 +- .../src/action/triggers/executor.test.ts | 11 +- .../warden/src/action/triggers/executor.ts | 81 +++-- .../src/action/workflow/pr-workflow.test.ts | 10 +- .../warden/src/action/workflow/pr-workflow.ts | 35 ++- .../warden/src/cli/output/ink-runner.test.tsx | 1 + packages/warden/src/cli/output/ink-runner.tsx | 7 +- packages/warden/src/cli/output/tasks.test.ts | 177 ++++++++--- packages/warden/src/cli/output/tasks.ts | 224 +++++++++++--- packages/warden/src/config/loader.test.ts | 36 +++ packages/warden/src/config/loader.ts | 11 + packages/warden/src/config/schema.ts | 22 +- packages/warden/src/diff/index.ts | 1 + packages/warden/src/diff/review-chunk.test.ts | 32 ++ packages/warden/src/diff/review-chunk.ts | 121 ++++++++ packages/warden/src/sdk/analyze.test.ts | 106 ++++++- packages/warden/src/sdk/analyze.ts | 265 +++++++++------- packages/warden/src/sdk/extract.ts | 20 +- packages/warden/src/sdk/prepare.test.ts | 72 ++++- packages/warden/src/sdk/prepare.ts | 30 +- packages/warden/src/sdk/prompt.ts | 64 +++- packages/warden/src/sdk/runner.test.ts | 17 +- packages/warden/src/sdk/runner.ts | 4 +- packages/warden/src/sdk/runtimes/types.ts | 4 +- .../src/sdk/semantic-chunk-planner.test.ts | 246 +++++++++++++++ .../warden/src/sdk/semantic-chunk-planner.ts | 253 ++++++++++++++++ packages/warden/src/sdk/types.ts | 20 +- skills/warden/references/config-schema.md | 10 +- skills/warden/references/configuration.md | 10 + specs/semantic-review-chunks.md | 283 ++++++++++++++++++ warden.toml | 9 +- 45 files changed, 2206 insertions(+), 374 deletions(-) create mode 100644 .agents/skills/warden-qa/SKILL.md create mode 100644 packages/evals/src/auth.ts create mode 100644 packages/evals/src/semantic-chunking.eval.ts create mode 100644 packages/warden/src/diff/review-chunk.test.ts create mode 100644 packages/warden/src/diff/review-chunk.ts create mode 100644 packages/warden/src/sdk/semantic-chunk-planner.test.ts create mode 100644 packages/warden/src/sdk/semantic-chunk-planner.ts create mode 100644 specs/semantic-review-chunks.md diff --git a/.agents/skills/warden-qa/SKILL.md b/.agents/skills/warden-qa/SKILL.md new file mode 100644 index 00000000..e8ad9035 --- /dev/null +++ b/.agents/skills/warden-qa/SKILL.md @@ -0,0 +1,74 @@ +--- +name: warden-qa +description: Concise local QA workflow for Warden changes. Use when asked to manually QA Warden, verify scanner behavior, exercise config or chunking changes, or sanity-check local output before finishing. +--- + +Use one targeted local run to prove the changed Warden behavior. Prefer the +smallest deterministic path that exercises the real code. + +## Choose the Probe + +- For diff parsing, chunking, trigger matching, config loading, or report + shaping, use a focused `tsx` snippet against package APIs. +- For CLI behavior, use `pnpm cli -- ...` with the narrowest command and flags + that reach the changed path. +- For model-backed scanner behavior, use one small fixture and one skill. If + credentials or network access block the run, report that and verify the + deterministic pre-model path instead. +- Do not run broad, expensive, or unrelated Warden scans for a narrow change. + +## Common Commands + +Run package API probes from `packages/warden` when they do not need a built CLI: + +```sh +pnpm --filter @sentry/warden exec tsx -e '' +``` + +Run Warden through the repo wrapper when validating operator-facing CLI behavior: + +```sh +pnpm cli -- --json +pnpm cli -- --log +``` + +For chunking changes, verify the prepared file shape directly: + +```sh +pnpm --filter @sentry/warden exec tsx -e '' +``` + +Use synthetic patches when they prove the behavior more clearly than a live PR. +Keep them tiny: one file, two or three hunks, and expected changed line ranges. + +## What to Inspect + +- The exact config that loaded. +- File count and chunk count. +- `ReviewChunk.contentMode`. +- `ReviewChunk.changedLineMap`. +- Any rendered prompt or JSON output that proves the changed path. +- Exit status and the key output, not the full transcript. + +## Failure Handling + +If the output is too broad to prove the change, narrow the fixture or command. +If a live scanner run is blocked by credentials, model gateway access, or +network restrictions, say so and show the deterministic local evidence that was +still checked. + +Do not hide uncertainty behind a passing test command. Name what local QA did +not prove. + +## Reporting + +Report: + +- pass or fail +- exact command run +- key observed output +- what behavior the output proves +- anything still unproven locally + +Keep automated validation such as `pnpm lint`, `pnpm build`, `pnpm test`, and +typechecks in a separate validation note. Manual QA is the local behavior probe. diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml index 79cd6c2e..bfd0e214 100644 --- a/.github/workflows/evals.yml +++ b/.github/workflows/evals.yml @@ -85,6 +85,8 @@ jobs: timeout-minutes: 90 env: ANTHROPIC_API_KEY: ${{ secrets.WARDEN_ANTHROPIC_API_KEY }} + WARDEN_OPENROUTER_API_KEY: ${{ secrets.WARDEN_OPENROUTER_API_KEY || secrets.OPENROUTER_API_KEY }} + OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY || secrets.WARDEN_OPENROUTER_API_KEY }} EVAL_SCORE_BASELINE: "0.75" steps: - uses: actions/checkout@v4 @@ -99,8 +101,8 @@ jobs: - name: Verify eval secret run: | - if [ -z "$ANTHROPIC_API_KEY" ]; then - echo "WARDEN_ANTHROPIC_API_KEY is required to run evals in CI" >&2 + if [ -z "$OPENROUTER_API_KEY" ] && [ -z "$WARDEN_OPENROUTER_API_KEY" ]; then + echo "OPENROUTER_API_KEY or WARDEN_OPENROUTER_API_KEY is required to run evals in CI" >&2 exit 1 fi diff --git a/.github/workflows/warden.yml b/.github/workflows/warden.yml index 378c8d3f..60ea138c 100644 --- a/.github/workflows/warden.yml +++ b/.github/workflows/warden.yml @@ -18,6 +18,8 @@ jobs: WARDEN_MODEL: anthropic/claude-sonnet-4-6 WARDEN_OPENAI_API_KEY: ${{ secrets.WARDEN_OPENAI_API_KEY }} WARDEN_ANTHROPIC_API_KEY: ${{ secrets.WARDEN_ANTHROPIC_API_KEY }} + WARDEN_OPENROUTER_API_KEY: ${{ secrets.WARDEN_OPENROUTER_API_KEY || secrets.OPENROUTER_API_KEY }} + OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY || secrets.WARDEN_OPENROUTER_API_KEY }} WARDEN_SENTRY_DSN: ${{ secrets.WARDEN_SENTRY_DSN }} steps: - uses: actions/checkout@v4 diff --git a/packages/evals/eval-bug-detection.yaml b/packages/evals/eval-bug-detection.yaml index 7726618d..fafadf98 100644 --- a/packages/evals/eval-bug-detection.yaml +++ b/packages/evals/eval-bug-detection.yaml @@ -1,6 +1,6 @@ skill: skills/bug-detection.md runtime: pi -model: anthropic/claude-sonnet-4-6 +model: openrouter/anthropic/claude-sonnet-4.6 evals: - name: null-property-access diff --git a/packages/evals/eval-precision.yaml b/packages/evals/eval-precision.yaml index 5099d08d..e780c9b4 100644 --- a/packages/evals/eval-precision.yaml +++ b/packages/evals/eval-precision.yaml @@ -1,6 +1,6 @@ skill: skills/precision.md runtime: pi -model: anthropic/claude-sonnet-4-6 +model: openrouter/anthropic/claude-sonnet-4.6 evals: - name: ignores-style-issues diff --git a/packages/evals/eval-security-scanning.yaml b/packages/evals/eval-security-scanning.yaml index 2f8df994..3fbe0d3a 100644 --- a/packages/evals/eval-security-scanning.yaml +++ b/packages/evals/eval-security-scanning.yaml @@ -1,6 +1,6 @@ skill: skills/security-scanning.md runtime: pi -model: anthropic/claude-sonnet-4-6 +model: openrouter/anthropic/claude-sonnet-4.6 evals: - name: sql-injection diff --git a/packages/evals/src/auth.ts b/packages/evals/src/auth.ts new file mode 100644 index 00000000..634138ab --- /dev/null +++ b/packages/evals/src/auth.ts @@ -0,0 +1,46 @@ +import { bridgeWardenProviderApiKeyEnv } from '../../warden/src/utils/index.js'; +export { DEFAULT_EVAL_MODEL, DEFAULT_EVAL_RUNTIME } from './types.js'; +import { DEFAULT_EVAL_MODEL } from './types.js'; + +function providerFromModel(model: string): string | undefined { + const slashIndex = model.indexOf('/'); + if (slashIndex <= 0) { + return undefined; + } + + return model.slice(0, slashIndex); +} + +function providerEnvPrefix(provider: string): string { + return provider.toUpperCase().replace(/-/g, '_'); +} + +/** + * Returns a provider API key from the env for eval skip checks. + */ +export function getEvalProviderApiKey(model = defaultEvalModel()): string { + bridgeWardenProviderApiKeyEnv(); + + const provider = providerFromModel(model); + if (!provider) { + return ''; + } + + const prefix = providerEnvPrefix(provider); + return process.env[`WARDEN_${prefix}_API_KEY`] ?? process.env[`${prefix}_API_KEY`] ?? ''; +} + +/** + * Returns the legacy runtime API key override only for direct Anthropic Pi models. + */ +export function getEvalRuntimeApiKey(model = defaultEvalModel()): string { + const provider = providerFromModel(model); + return provider === 'anthropic' ? getEvalProviderApiKey(model) : ''; +} + +/** + * Returns the eval model override or the repo's OpenRouter default. + */ +export function defaultEvalModel(): string { + return process.env['WARDEN_MODEL'] ?? DEFAULT_EVAL_MODEL; +} diff --git a/packages/evals/src/code-review.eval.ts b/packages/evals/src/code-review.eval.ts index 4f79fac0..bb1f3b58 100644 --- a/packages/evals/src/code-review.eval.ts +++ b/packages/evals/src/code-review.eval.ts @@ -7,13 +7,16 @@ import { } from './harness.js'; import { discoverEvalScenarios } from './index.js'; import { formatEvalId, formatEvalTestName } from './names.js'; +import { DEFAULT_EVAL_RUNTIME, defaultEvalModel, getEvalProviderApiKey, getEvalRuntimeApiKey } from './auth.js'; -const apiKey = process.env['ANTHROPIC_API_KEY'] ?? ''; +const model = defaultEvalModel(); +const apiKey = getEvalRuntimeApiKey(model); +const providerApiKey = getEvalProviderApiKey(model); const evals = discoverEvalScenarios({ category: 'code-review', skill: '../warden/src/builtin-skills/code-review/SKILL.md', - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', + runtime: DEFAULT_EVAL_RUNTIME, + model, }); const CODE_REVIEW_RUN_TIMEOUT_MS = 120_000; const CODE_REVIEW_EVAL_TIMEOUT_MS = CODE_REVIEW_RUN_TIMEOUT_MS + 60_000; @@ -30,7 +33,7 @@ describeEval( }), judges: [createWardenEvalJudge(apiKey)], judgeThreshold: 1, - skipIf: () => !apiKey, + skipIf: () => !providerApiKey, }, (it) => { for (const meta of evals) { diff --git a/packages/evals/src/e2e.eval.ts b/packages/evals/src/e2e.eval.ts index 6f43c7c4..6cae0662 100644 --- a/packages/evals/src/e2e.eval.ts +++ b/packages/evals/src/e2e.eval.ts @@ -7,8 +7,16 @@ import { } from './harness.js'; import { discoverEvals } from './index.js'; import { formatEvalId, formatEvalTestName } from './names.js'; +import { + DEFAULT_EVAL_RUNTIME, + defaultEvalModel, + getEvalProviderApiKey, + getEvalRuntimeApiKey, +} from './auth.js'; -const apiKey = process.env['ANTHROPIC_API_KEY'] ?? ''; +const model = defaultEvalModel(); +const apiKey = getEvalRuntimeApiKey(model); +const providerApiKey = getEvalProviderApiKey(model); const evals = discoverEvals(); describeEval( @@ -16,11 +24,13 @@ describeEval( { harness: createWardenEvalHarness({ apiKey, + runtime: DEFAULT_EVAL_RUNTIME, + model, verbose: true, }), judges: [createWardenEvalJudge(apiKey)], judgeThreshold: 1, - skipIf: () => !apiKey, + skipIf: () => !providerApiKey, }, (it) => { for (const meta of evals) { diff --git a/packages/evals/src/harness.ts b/packages/evals/src/harness.ts index a060f95d..9f6c7838 100644 --- a/packages/evals/src/harness.ts +++ b/packages/evals/src/harness.ts @@ -174,7 +174,13 @@ export function createWardenEvalJudge(apiKey: string) { const meta = input; const findings = output.data.findings; - const judgeResult = await runJudge(meta, findings, apiKey); + const judgeResult = await runJudge(meta, findings, { + apiKey, + runtime: output.data.runtime === 'claude' || output.data.runtime === 'pi' + ? output.data.runtime + : meta.runtime, + model: output.data.model ?? meta.model, + }); if (judgeResult.error) { return { score: 0, diff --git a/packages/evals/src/judge.ts b/packages/evals/src/judge.ts index 7c8a0fbd..a3bb381d 100644 --- a/packages/evals/src/judge.ts +++ b/packages/evals/src/judge.ts @@ -1,12 +1,16 @@ -import Anthropic from '@anthropic-ai/sdk'; -import { anthropicUsageToStats, parseJsonFromOutput, type Finding, type UsageStats } from '@sentry/warden'; +import { getRuntime, type Finding, type RuntimeName, type UsageStats } from '@sentry/warden'; import type { EvalMeta, JudgeResponse } from './types.js'; -import { DEFAULT_EVAL_MODEL, JudgeResponseSchema } from './types.js'; +import { DEFAULT_EVAL_MODEL, DEFAULT_EVAL_RUNTIME, JudgeResponseSchema } from './types.js'; -const JUDGE_MODEL = DEFAULT_EVAL_MODEL; const JUDGE_MAX_TOKENS = 4096; const JUDGE_TIMEOUT_MS = 30_000; +export interface RunJudgeOptions { + apiKey?: string; + runtime?: RuntimeName; + model?: string; +} + export interface JudgeResult { response: JudgeResponse; usage: UsageStats; @@ -107,22 +111,24 @@ Requirements: export async function runJudge( meta: EvalMeta, findings: Finding[], - apiKey: string + options: RunJudgeOptions = {} ): Promise { - const client = new Anthropic({ apiKey, timeout: JUDGE_TIMEOUT_MS, maxRetries: 0 }); - + const runtimeName = options.runtime ?? DEFAULT_EVAL_RUNTIME; + const model = options.model ?? DEFAULT_EVAL_MODEL; const prompt = buildJudgePrompt(meta, findings); - - const messages: Anthropic.MessageParam[] = [ - { role: 'user', content: prompt }, - ]; - - let response: Anthropic.Message; + const runtime = getRuntime(runtimeName); + let result: Awaited>>; try { - response = await client.messages.create({ - model: JUDGE_MODEL, - max_tokens: JUDGE_MAX_TOKENS, - messages, + result = await runtime.runAuxiliary({ + task: 'eval_judge', + agentName: 'warden-eval-judge', + apiKey: options.apiKey, + prompt, + schema: JudgeResponseSchema, + model, + maxTokens: JUDGE_MAX_TOKENS, + timeout: JUDGE_TIMEOUT_MS, + maxRetries: 0, }); } catch (error) { const reason = `Judge API call failed: ${error instanceof Error ? error.message : String(error)}`; @@ -144,58 +150,36 @@ export async function runJudge( }; } - const usage = anthropicUsageToStats(JUDGE_MODEL, response.usage); - - const textBlock = response.content.find( - (b): b is Anthropic.TextBlock => b.type === 'text' - ); - - if (!textBlock) { - return { - response: buildFallbackResponse(meta, 'No text in judge response'), - usage, - error: 'No text in judge response', - }; - } - - const parsed = await parseJsonFromOutput({ - output: textBlock.text, - schema: JudgeResponseSchema, - }); - - if (!parsed.success) { - const reason = `Judge response parse failed: ${parsed.error}`; + if (!result.success) { + const reason = `Judge response failed: ${result.error}`; return { response: buildFallbackResponse(meta, reason), - usage, + usage: result.usage, error: reason, }; } // Validate array lengths match assertions - const judgeResp = parsed.data; + const judgeResp = result.data; if (judgeResp.expectations.length !== meta.should_find.length) { return { response: buildFallbackResponse(meta, `Judge returned ${judgeResp.expectations.length} verdicts, expected ${meta.should_find.length}`), - usage, + usage: result.usage, error: `Judge returned ${judgeResp.expectations.length} verdicts, expected ${meta.should_find.length}`, }; } if (judgeResp.antiExpectations.length !== meta.should_not_find.length) { return { response: buildFallbackResponse(meta, `Judge returned ${judgeResp.antiExpectations.length} anti-verdicts, expected ${meta.should_not_find.length}`), - usage, + usage: result.usage, error: `Judge returned ${judgeResp.antiExpectations.length} anti-verdicts, expected ${meta.should_not_find.length}`, }; } - return { response: judgeResp, usage }; + return { response: judgeResp, usage: result.usage }; } -/** - * Build a fallback judge response when parsing fails. - * Marks all assertions as not met with the error reason. - */ +/** Build a failed judge response when the judge cannot produce a usable verdict. */ function buildFallbackResponse(meta: EvalMeta, reason: string): JudgeResponse { return { expectations: meta.should_find.map(() => ({ diff --git a/packages/evals/src/security-review.eval.ts b/packages/evals/src/security-review.eval.ts index 477a08e3..eec75e1e 100644 --- a/packages/evals/src/security-review.eval.ts +++ b/packages/evals/src/security-review.eval.ts @@ -7,13 +7,16 @@ import { } from './harness.js'; import { discoverEvalScenarios } from './index.js'; import { formatEvalId, formatEvalTestName } from './names.js'; +import { DEFAULT_EVAL_RUNTIME, defaultEvalModel, getEvalProviderApiKey, getEvalRuntimeApiKey } from './auth.js'; -const apiKey = process.env['ANTHROPIC_API_KEY'] ?? ''; +const model = defaultEvalModel(); +const apiKey = getEvalRuntimeApiKey(model); +const providerApiKey = getEvalProviderApiKey(model); const evals = discoverEvalScenarios({ category: 'security-review', skill: '../warden/src/builtin-skills/security-review/SKILL.md', - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', + runtime: DEFAULT_EVAL_RUNTIME, + model, }); describeEval( @@ -25,7 +28,7 @@ describeEval( }), judges: [createWardenEvalJudge(apiKey)], judgeThreshold: 1, - skipIf: () => !apiKey, + skipIf: () => !providerApiKey, }, (it) => { for (const meta of evals) { diff --git a/packages/evals/src/semantic-chunking.eval.ts b/packages/evals/src/semantic-chunking.eval.ts new file mode 100644 index 00000000..baa2094c --- /dev/null +++ b/packages/evals/src/semantic-chunking.eval.ts @@ -0,0 +1,123 @@ +import { describe, expect, it } from 'vitest'; +import type { EventContext } from '../../warden/src/types/index.js'; +import { prepareFiles } from '../../warden/src/sdk/prepare.js'; +import { planSemanticReviewChunks } from '../../warden/src/sdk/semantic-chunk-planner.js'; +import { + DEFAULT_EVAL_RUNTIME, + defaultEvalModel, + getEvalProviderApiKey, + getEvalRuntimeApiKey, +} from './auth.js'; + +const model = process.env['WARDEN_SEMANTIC_CHUNK_EVAL_MODEL'] ?? defaultEvalModel(); +const apiKey = getEvalRuntimeApiKey(model); +const providerApiKey = getEvalProviderApiKey(model); + +function makeSemanticChunkingContext(): EventContext { + return { + eventType: 'pull_request', + action: 'opened', + repository: { + owner: 'getsentry', + name: 'semantic-chunking-eval', + fullName: 'getsentry/semantic-chunking-eval', + defaultBranch: 'main', + }, + repoPath: '/tmp/warden-semantic-chunking-eval', + pullRequest: { + number: 313, + title: 'Update dashboard behavior', + body: '', + author: 'eval', + baseBranch: 'main', + headBranch: 'semantic-axis-range', + headSha: 'head', + baseSha: 'base', + files: [{ + filename: 'src/dashboard.ts', + status: 'modified', + additions: 3, + deletions: 3, + chunks: 3, + patch: [ + '@@ -10,1 +10,1 @@', + '-const range = getDefaultAxisRange(widget);', + '+const range = widget.axisRange ?? getDefaultAxisRange(widget);', + '@@ -80,1 +80,1 @@', + '-const chart = convertWidgetToChart(widget);', + '+const chart = convertWidgetToChart(widget, range);', + '@@ -140,1 +140,1 @@', + '-return renderChart(chart, series);', + '+return renderChart(chart, series, range);', + ].join('\n'), + }, { + filename: 'tests/dashboard.test.ts', + status: 'modified', + additions: 1, + deletions: 1, + chunks: 1, + patch: [ + '@@ -220,1 +220,1 @@', + '-expect(rendered.range).toEqual(defaultAxisRange);', + '+expect(rendered.range).toEqual(customAxisRange);', + ].join('\n'), + }], + }, + }; +} + +describe.skipIf(!providerApiKey)('semantic chunking eval', () => { + it('plans a behavior-level delta for many tiny related hunks', { timeout: 120_000 }, async () => { + const context = makeSemanticChunkingContext(); + const prepared = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + }); + + const atomicChunks = prepared.files.flatMap((file) => file.chunks); + expect(atomicChunks).toHaveLength(4); + + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + apiKey, + runtime: DEFAULT_EVAL_RUNTIME, + model, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + }); + + const chunks = planned.groups.flatMap((group) => group.chunks); + expect(chunks.length).toBeLessThan(atomicChunks.length); + const crossFileChunk = chunks.find((chunk) => chunk.files.length > 1); + expect(crossFileChunk).toBeDefined(); + expect(crossFileChunk?.changedLineMap).toEqual(expect.arrayContaining([ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'src/dashboard.ts', start: 80, end: 80 }, + { path: 'src/dashboard.ts', start: 140, end: 140 }, + { path: 'tests/dashboard.test.ts', start: 220, end: 220 }, + ])); + + for (const chunk of chunks) { + expect(chunk.summary).toBeTruthy(); + expect(chunk.summary).not.toMatch(/lines?\s+\d+/i); + expect(chunk.summary).not.toMatch(/\b10\b.*\b80\b.*\b140\b/); + } + + const summaryText = chunks.map((chunk) => chunk.summary ?? '').join(' '); + expect(summaryText).toMatch(/axisRange|axis range|range/i); + expect(summaryText).toMatch(/widget|convert|render|chart/i); + + const chunkText = crossFileChunk?.files.map((file) => file.content).join('\n') ?? ''; + expect(chunkText).toContain('convertWidgetToChart(widget, range)'); + expect(chunkText).toContain('renderChart(chart, series, range)'); + expect(chunkText).toContain('expect(rendered.range).toEqual(customAxisRange)'); + }); +}); diff --git a/packages/evals/src/types.ts b/packages/evals/src/types.ts index 62142707..2ad65ebe 100644 --- a/packages/evals/src/types.ts +++ b/packages/evals/src/types.ts @@ -2,10 +2,10 @@ import { z } from 'zod'; import { RuntimeNameSchema, SeveritySchema, type Finding, type RuntimeName } from '@sentry/warden'; /** Default model for eval skill execution and judging. */ -export const DEFAULT_EVAL_MODEL = 'claude-sonnet-4-6'; +export const DEFAULT_EVAL_MODEL = 'openrouter/anthropic/claude-sonnet-4.6'; /** Default runtime for eval skill execution. */ -export const DEFAULT_EVAL_RUNTIME: RuntimeName = 'claude'; +export const DEFAULT_EVAL_RUNTIME: RuntimeName = 'pi'; /** * A "should find" assertion in BDD style. diff --git a/packages/evals/src/verify.eval.ts b/packages/evals/src/verify.eval.ts index 2bcbe55b..30e3c644 100644 --- a/packages/evals/src/verify.eval.ts +++ b/packages/evals/src/verify.eval.ts @@ -6,13 +6,20 @@ import { VerificationEvalOutputSchema, } from './verify.js'; import { formatEvalTestName } from './names.js'; +import { + DEFAULT_EVAL_RUNTIME, + defaultEvalModel, + getEvalProviderApiKey, + getEvalRuntimeApiKey, +} from './auth.js'; -const apiKey = process.env['ANTHROPIC_API_KEY'] ?? ''; +const model = defaultEvalModel(); +const apiKey = getEvalRuntimeApiKey(model); const evals = discoverVerificationEvalScenarios({ category: 'verification', skill: '../warden/src/builtin-skills/security-review/SKILL.md', - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', + runtime: DEFAULT_EVAL_RUNTIME, + model, }); describeEval( @@ -24,7 +31,7 @@ describeEval( }), judges: [createVerificationEvalJudge()], judgeThreshold: 1, - skipIf: () => !apiKey, + skipIf: () => !getEvalRuntimeApiKey(model) && !getEvalProviderApiKey(model), }, (it) => { for (const meta of evals) { diff --git a/packages/warden/src/action/triggers/executor.test.ts b/packages/warden/src/action/triggers/executor.test.ts index 4aa23a6d..2fbd0da5 100644 --- a/packages/warden/src/action/triggers/executor.test.ts +++ b/packages/warden/src/action/triggers/executor.test.ts @@ -198,7 +198,11 @@ describe('executeTrigger', () => { maxContextFiles: 12, ignore: { paths: ['**/fixtures/**'] }, scan: { maxFiles: 5 }, - chunking: { maxContextFiles: 12, filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }] }, + chunking: { + maxContextFiles: 12, + filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], + semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, preferWholeFileBelowLines: 800 }, + }, auxiliaryMaxRetries: 9, }, { ...mockDeps, @@ -211,7 +215,10 @@ describe('executeTrigger', () => { maxContextFiles: 12, ignore: { paths: ['**/fixtures/**'] }, scan: { maxFiles: 5 }, - chunking: { filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }] }, + chunking: { + filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], + semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, preferWholeFileBelowLines: 800 }, + }, auxiliaryMaxRetries: 9, }), }), diff --git a/packages/warden/src/action/triggers/executor.ts b/packages/warden/src/action/triggers/executor.ts index 90affb87..d06e502b 100644 --- a/packages/warden/src/action/triggers/executor.ts +++ b/packages/warden/src/action/triggers/executor.ts @@ -43,8 +43,11 @@ function toAnalysisChunkingConfig( if (chunking.coalesce) { analysisChunking.coalesce = chunking.coalesce; } + if (chunking.semantic) { + analysisChunking.semantic = chunking.semantic; + } - return analysisChunking.filePatterns || analysisChunking.coalesce + return analysisChunking.filePatterns || analysisChunking.coalesce || analysisChunking.semantic ? analysisChunking : undefined; } @@ -105,6 +108,8 @@ export interface TriggerExecutorDeps { circuitBreaker?: ProviderFailureCircuitBreaker; /** Optional context-bound check writer. Omit for analyze mode. */ checks?: TriggerCheckReporter; + /** Prepared review chunks shared by the action workflow across matching triggers. */ + preparedFiles?: SkillTaskOptions['preparedFiles']; } /** @@ -131,6 +136,47 @@ export interface TriggerResult { // Executor // ----------------------------------------------------------------------------- +/** Build the skill task options used by action trigger execution. */ +export function createTriggerSkillTaskOptions( + trigger: ResolvedTrigger, + deps: TriggerExecutorDeps +): SkillTaskOptions { + const { context, anthropicApiKey, claudePath } = deps; + const failOn = trigger.failOn ?? deps.globalFailOn; + const skillRoot = trigger.useBuiltinSkill ? undefined : (trigger.skillRoot ?? context.repoPath); + + return { + name: trigger.name, + displayName: trigger.skill, + triggerName: trigger.name, + failOn, + resolveSkill: () => resolveSkillAsync(trigger.skill, skillRoot, { + remote: trigger.remote, + }), + context: filterContextByPaths(context, trigger.filters), + preparedFiles: deps.preparedFiles, + runnerOptions: { + apiKey: anthropicApiKey, + model: trigger.model, + runtime: trigger.runtime, + effort: trigger.effort, + auxiliaryModel: trigger.auxiliaryModel, + synthesisModel: trigger.synthesisModel, + maxTurns: trigger.maxTurns, + batchDelayMs: trigger.batchDelayMs, + maxContextFiles: trigger.maxContextFiles, + ignore: trigger.ignore, + scan: trigger.scan, + chunking: toAnalysisChunkingConfig(trigger.chunking), + pathToClaudeCodeExecutable: claudePath, + auxiliaryMaxRetries: trigger.auxiliaryMaxRetries, + verifyFindings: trigger.verifyFindings, + abortController: deps.abortController, + circuitBreaker: deps.circuitBreaker, + }, + }; +} + /** * Execute a single trigger and return results. * @@ -148,7 +194,7 @@ export async function executeTrigger( async (span) => { span.setAttribute('gen_ai.agent.name', trigger.skill); span.setAttribute('warden.trigger.name', trigger.name); - const { context, anthropicApiKey, claudePath } = deps; + const { context } = deps; logGroup(`Running trigger: ${trigger.name} (skill: ${trigger.skill})`); @@ -169,40 +215,11 @@ export async function executeTrigger( const minConfidence = trigger.minConfidence ?? 'medium'; const requestChanges = trigger.requestChanges ?? deps.globalRequestChanges; const failCheck = trigger.failCheck ?? deps.globalFailCheck; - const skillRoot = trigger.useBuiltinSkill ? undefined : (trigger.skillRoot ?? context.repoPath); try { assertValidPiModelSelectors([trigger]); - const taskOptions: SkillTaskOptions = { - name: trigger.name, - displayName: trigger.skill, - triggerName: trigger.name, - failOn, - resolveSkill: () => resolveSkillAsync(trigger.skill, skillRoot, { - remote: trigger.remote, - }), - context: filterContextByPaths(context, trigger.filters), - runnerOptions: { - apiKey: anthropicApiKey, - model: trigger.model, - runtime: trigger.runtime, - effort: trigger.effort, - auxiliaryModel: trigger.auxiliaryModel, - synthesisModel: trigger.synthesisModel, - maxTurns: trigger.maxTurns, - batchDelayMs: trigger.batchDelayMs, - maxContextFiles: trigger.maxContextFiles, - ignore: trigger.ignore, - scan: trigger.scan, - chunking: toAnalysisChunkingConfig(trigger.chunking), - pathToClaudeCodeExecutable: claudePath, - auxiliaryMaxRetries: trigger.auxiliaryMaxRetries, - verifyFindings: trigger.verifyFindings, - abortController: deps.abortController, - circuitBreaker: deps.circuitBreaker, - }, - }; + const taskOptions = createTriggerSkillTaskOptions(trigger, deps); const callbacks = createDefaultCallbacks([taskOptions], CI_OUTPUT_MODE, Verbosity.Normal); const fileConcurrency = deps.semaphore ? Number.MAX_SAFE_INTEGER : DEFAULT_FILE_CONCURRENCY; diff --git a/packages/warden/src/action/workflow/pr-workflow.test.ts b/packages/warden/src/action/workflow/pr-workflow.test.ts index bb7d2cf0..86f333d9 100644 --- a/packages/warden/src/action/workflow/pr-workflow.test.ts +++ b/packages/warden/src/action/workflow/pr-workflow.test.ts @@ -40,6 +40,7 @@ vi.mock('../../cli/output/tasks.js', async () => { return { ...actual, runSkillTask: vi.fn(), + prepareSemanticPlansForTasks: vi.fn(async (tasks) => tasks), }; }); @@ -115,7 +116,7 @@ vi.mock('./base.js', async () => { }); // Import after mocks -import { runSkillTask } from '../../cli/output/tasks.js'; +import { prepareSemanticPlansForTasks, runSkillTask } from '../../cli/output/tasks.js'; import { fetchExistingComments, deduplicateFindings, processDuplicateActions } from '../../output/dedup.js'; import { evaluateFixAttempts } from '../fix-evaluation/index.js'; import { setFailed, writeFindingsOutput } from './base.js'; @@ -126,6 +127,7 @@ import { buildFindingsOutput } from '../reporting/output.js'; // Type the mocks const mockRunSkillTask = vi.mocked(runSkillTask); +const mockPrepareSemanticPlansForTasks = vi.mocked(prepareSemanticPlansForTasks); const mockFetchExistingComments = vi.mocked(fetchExistingComments); const mockDeduplicateFindings = vi.mocked(deduplicateFindings); const mockProcessDuplicateActions = vi.mocked(processDuplicateActions); @@ -1251,6 +1253,12 @@ describe('runPRWorkflow', () => { name: 'test-skill', displayName: 'test-skill', })); + expect(mockPrepareSemanticPlansForTasks).toHaveBeenCalledWith([ + expect.objectContaining({ + name: 'test-skill', + displayName: 'test-skill', + }), + ]); // When a semaphore is provided, fileConcurrency is unlimited (semaphore is the gate) expect(fileConcurrency).toBe(Number.MAX_SAFE_INTEGER); expect(semaphore).toBeInstanceOf(Semaphore); diff --git a/packages/warden/src/action/workflow/pr-workflow.ts b/packages/warden/src/action/workflow/pr-workflow.ts index 7546b383..8b1e5e19 100644 --- a/packages/warden/src/action/workflow/pr-workflow.ts +++ b/packages/warden/src/action/workflow/pr-workflow.ts @@ -39,8 +39,9 @@ import { formatCost, formatTokens, formatDuration } from '../../cli/output/forma import { findBotReviewState } from '../review-state.js'; import type { BotReviewInfo } from '../review-state.js'; import type { ActionInputs } from '../inputs.js'; -import { executeTrigger } from '../triggers/executor.js'; +import { createTriggerSkillTaskOptions, executeTrigger } from '../triggers/executor.js'; import type { TriggerCheckReporter, TriggerResult } from '../triggers/executor.js'; +import { prepareSemanticPlansForTasks } from '../../cli/output/tasks.js'; import { postTriggerReview } from '../review/poster.js'; import { shouldResolveStaleComments } from '../review/coordination.js'; import type { FindingObservation } from '../reporting/outcomes.js'; @@ -485,24 +486,32 @@ async function executeAllTriggers( const semaphore = new Semaphore(concurrency); const abortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController }); + const baseTriggerDeps = { + context, + anthropicApiKey: inputs.anthropicApiKey, + claudePath: runtimeEnv.pathToClaudeCodeExecutable, + globalFailOn: inputs.failOn, + globalReportOn: inputs.reportOn, + globalMaxFindings: inputs.maxFindings, + globalRequestChanges: inputs.requestChanges, + globalFailCheck: inputs.failCheck, + semaphore, + abortController, + circuitBreaker, + }; + const plannedTaskOptions = await prepareSemanticPlansForTasks( + matchedTriggers.map((trigger) => createTriggerSkillTaskOptions(trigger, baseTriggerDeps)) + ); + const preparedFilesByTriggerIndex = plannedTaskOptions.map((task) => task.preparedFiles); // Limit trigger dispatch too; the semaphore only gates work after a trigger starts. return runPool( matchedTriggers, concurrency, - (trigger) => + (trigger, index) => executeTrigger(trigger, { - context, - anthropicApiKey: inputs.anthropicApiKey, - claudePath: runtimeEnv.pathToClaudeCodeExecutable, - globalFailOn: inputs.failOn, - globalReportOn: inputs.reportOn, - globalMaxFindings: inputs.maxFindings, - globalRequestChanges: inputs.requestChanges, - globalFailCheck: inputs.failCheck, - semaphore, - abortController, - circuitBreaker, + ...baseTriggerDeps, + preparedFiles: preparedFilesByTriggerIndex[index], checks: options.checks, }), { shouldAbort: () => abortController.signal.aborted }, diff --git a/packages/warden/src/cli/output/ink-runner.test.tsx b/packages/warden/src/cli/output/ink-runner.test.tsx index f6583ec5..3fa12ecd 100644 --- a/packages/warden/src/cli/output/ink-runner.test.tsx +++ b/packages/warden/src/cli/output/ink-runner.test.tsx @@ -36,6 +36,7 @@ vi.mock('ink', () => ({ vi.mock('./tasks.js', () => ({ composeTasksWithFailFast: vi.fn((tasks) => tasks), + prepareSemanticPlansForTasks: vi.fn(async (tasks) => tasks), runComposedSkillTasks: mockRunComposedSkillTasks, })); diff --git a/packages/warden/src/cli/output/ink-runner.tsx b/packages/warden/src/cli/output/ink-runner.tsx index eacae891..dd9181d1 100644 --- a/packages/warden/src/cli/output/ink-runner.tsx +++ b/packages/warden/src/cli/output/ink-runner.tsx @@ -18,6 +18,7 @@ import chalk from 'chalk'; import { composeTasksWithFailFast, runComposedSkillTasks, + prepareSemanticPlansForTasks, type SkillTaskOptions, type SkillTaskResult, type RunTasksOptions, @@ -280,8 +281,9 @@ export async function runSkillTasksWithInk( const semaphore = new Semaphore(concurrency); const circuitAbortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController: circuitAbortController }); + const plannedTasks = await prepareSemanticPlansForTasks(tasks); const composedTasks = composeTasksWithFailFast( - tasks, + plannedTasks, failFastController, circuitBreaker, circuitAbortController, @@ -480,8 +482,9 @@ export async function runSkillTasksWithInk( // Compose per-task abort controllers: fire on SIGINT, fail-fast, or provider circuit breaker. const circuitAbortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController: circuitAbortController }); + const plannedTasks = await prepareSemanticPlansForTasks(tasks); const composedTasks = composeTasksWithFailFast( - tasks, + plannedTasks, failFastController, circuitBreaker, circuitAbortController, diff --git a/packages/warden/src/cli/output/tasks.test.ts b/packages/warden/src/cli/output/tasks.test.ts index 3b7ea1b8..6269c83d 100644 --- a/packages/warden/src/cli/output/tasks.test.ts +++ b/packages/warden/src/cli/output/tasks.test.ts @@ -6,7 +6,7 @@ import type { OutputMode } from './tty.js'; import type { SkillReport, Finding, HunkFailure } from '../../types/index.js'; import type { SkillTaskOptions } from './tasks.js'; import type { FileAnalysisResult } from '../../sdk/types.js'; -import type { HunkWithContext } from '../../diff/index.js'; +import type { ReviewChunk } from '../../diff/index.js'; import type { SkillDefinition } from '../../config/schema.js'; import { Semaphore, runPool } from '../../utils/index.js'; import { SkillRunnerError, WardenAuthenticationError } from '../../sdk/errors.js'; @@ -43,6 +43,23 @@ function makeTask(name: string, displayName?: string): SkillTaskOptions { }; } +function makeReviewChunk(path = 'a.ts', start = 1, end = 10): ReviewChunk { + return { + id: `${path}:${start}-${end}`, + title: `${path}:${start}-${end}`, + summary: `Changes in ${path}`, + files: [{ + path, + language: 'typescript', + changedRanges: [{ path, start, end }], + content: '@@ -1,1 +1,1 @@\n-old\n+new', + contentMode: 'raw-hunks', + sourceLines: [{ line: start, content: 'new' }], + }], + changedLineMap: [{ path, start, end }], + }; +} + function logMode(): OutputMode { return { isTTY: false, supportsColor: false, columns: 80 }; } @@ -663,12 +680,10 @@ describe('runSkillTasks', () => { it('does not fail-fast on findings rejected by post-processing', async () => { const candidate = makeFinding(); const controller = new AbortController(); - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -711,15 +726,107 @@ describe('runSkillTasks', () => { postProcessFindings.mockRestore(); }); + it('shares semantic planning across skill tasks for the same changeset', async () => { + const fakeChunk = makeReviewChunk('a.ts', 1, 1); + const context = { + eventType: 'pull_request', + repository: { owner: 'o', name: 'n', fullName: 'o/n', defaultBranch: 'main' }, + repoPath: '/tmp', + pullRequest: { + number: 1, + title: 't', + body: '', + author: 'octocat', + baseBranch: 'main', + headBranch: 'feature', + headSha: 'abc', + baseSha: 'def', + files: [{ + filename: 'a.ts', + status: 'modified', + additions: 1, + deletions: 1, + patch: '@@ -1,1 +1,1 @@\n-old\n+new', + }], + }, + } as unknown as SkillTaskOptions['context']; + const copiedContext = { + ...context, + repository: { ...context.repository }, + pullRequest: { + ...context.pullRequest, + files: [...(context.pullRequest?.files ?? [])], + }, + } as SkillTaskOptions['context']; + const runnerOptions: SkillTaskOptions['runnerOptions'] = { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + postProcessFindings: false, + }; + + const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + skippedFiles: [], + }); + const planSemanticReviewChunks = vi.spyOn(sdkRunner, 'planSemanticReviewChunks').mockResolvedValue({ + groups: [{ displayName: 'a.ts', filenames: ['a.ts'], chunks: [fakeChunk] }], + usage: { inputTokens: 10, outputTokens: 5, costUSD: 0.001 }, + }); + const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ + filename: 'a.ts', + findings: [], + usage: { inputTokens: 1, outputTokens: 1, costUSD: 0.001 }, + failedHunks: 0, + failedExtractions: 0, + hunkFailures: [], + } satisfies FileAnalysisResult); + + const results = await runSkillTasks([ + { + name: 'skill-a', + resolveSkill: async () => ({ name: 'skill-a', description: '', prompt: '' }), + context, + runnerOptions, + }, + { + name: 'skill-b', + resolveSkill: async () => ({ name: 'skill-b', description: '', prompt: '' }), + context: copiedContext, + runnerOptions, + }, + ], { + mode: logMode(), + verbosity: Verbosity.Quiet, + concurrency: 2, + }); + + expect(prepareFiles).toHaveBeenCalledTimes(1); + expect(planSemanticReviewChunks).toHaveBeenCalledTimes(1); + expect(analyzeFile).toHaveBeenCalledTimes(2); + expect(results).toHaveLength(2); + expect(results.filter((result) => + Boolean(result.report?.auxiliaryUsage?.['semantic-chunk-planner']) + )).toHaveLength(1); + + prepareFiles.mockRestore(); + planSemanticReviewChunks.mockRestore(); + analyzeFile.mockRestore(); + }); + it('fail-fast aborts after final findings are post-processed', async () => { const finalFinding = makeFinding(); const controller = new AbortController(); - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -821,15 +928,13 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('synthesizes a report with error.code=auth_failed when every hunk fails with auth errors', async () => { - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const hunkFailures: HunkFailure[] = [ { type: 'analysis', filename: 'a.ts', lineRange: '1-10', code: 'auth_failed', message: 'bad key' }, ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); @@ -877,10 +982,10 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('preserves auth_failed when all analysis failures are auth alongside extraction failures', async () => { - const fakeHunks = [ - { hunk: { newStart: 1, newCount: 10 } }, - { hunk: { newStart: 20, newCount: 5 } }, - ] as unknown as HunkWithContext[]; + const fakeChunks = [ + makeReviewChunk('a.ts', 1, 10), + makeReviewChunk('a.ts', 20, 24), + ]; const hunkFailures: HunkFailure[] = [ { type: 'analysis', filename: 'a.ts', lineRange: '1-10', code: 'auth_failed', message: 'bad key' }, { @@ -893,7 +998,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: fakeHunks }], + files: [{ filename: 'a.ts', chunks: fakeChunks }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -926,9 +1031,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('synthesizes provider_unavailable when every hunk fails with provider errors', async () => { - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const hunkFailures: HunkFailure[] = [ { type: 'analysis', @@ -940,7 +1043,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -972,9 +1075,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('preserves invalid_model_selector when every hunk fails from Pi model validation', async () => { - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const hunkFailures: HunkFailure[] = [ { type: 'analysis', @@ -986,7 +1087,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1018,16 +1119,14 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('ignores unrelated circuit state when this skill completed without failures', async () => { - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const circuitBreaker = new ProviderFailureCircuitBreaker({ maxConsecutiveProviderFailures: 1, }); circuitBreaker.recordFailure('provider_unavailable', 'temporary outage'); vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1070,9 +1169,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { // If every hunk fails extraction (SDK call succeeds, parsing fails) // then failedHunks is 0 — naive `failedHunks === totalHunks` checks // would silently produce a "0 findings" run. Detection must sum both. - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const hunkFailures: HunkFailure[] = [ { type: 'extraction', @@ -1084,7 +1181,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); @@ -1119,16 +1216,14 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('does not convert user interruption into all_hunks_failed', async () => { - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const hunkFailures: HunkFailure[] = [ { type: 'analysis', filename: 'a.ts', lineRange: '1-10', code: 'aborted', message: 'Analysis aborted' }, ]; const controller = new AbortController(); vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); @@ -1237,13 +1332,11 @@ describe('runSkillTask model lanes', () => { }); it('passes model lanes to shared finding post-processing', async () => { - const fakeHunk = { - hunk: { newStart: 1, newCount: 10 }, - } as unknown as HunkWithContext; + const fakeChunk = makeReviewChunk(); const finding = makeFinding(); vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', hunks: [fakeHunk] }], + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ diff --git a/packages/warden/src/cli/output/tasks.ts b/packages/warden/src/cli/output/tasks.ts index 095d441a..34316402 100644 --- a/packages/warden/src/cli/output/tasks.ts +++ b/packages/warden/src/cli/output/tasks.ts @@ -5,7 +5,7 @@ * Reporter spec: specs/reporters.md */ -import type { SkillReport, SeverityThreshold, ConfidenceThreshold, Finding, UsageStats, EventContext, HunkFailure, AuxiliaryUsageMap, ErrorCode, HunkTrace } from '../../types/index.js'; +import type { SkillReport, SeverityThreshold, ConfidenceThreshold, Finding, UsageStats, EventContext, HunkFailure, AuxiliaryUsageMap, ErrorCode, HunkTrace, SkippedFile } from '../../types/index.js'; import type { SkillDefinition } from '../../config/schema.js'; import { Sentry, emitSkillMetrics, logger } from '../../sentry.js'; import { SkillRunnerError, WardenAuthenticationError, classifyError } from '../../sdk/errors.js'; @@ -15,11 +15,12 @@ import { aggregateUsage, aggregateAuxiliaryUsage, postProcessFindings, + planSemanticReviewChunks, generateSummary, type AuxiliaryUsageEntry, type SkillRunnerOptions, type FileAnalysisCallbacks, - type PreparedFile, + type ReviewChunkGroup, type PRPromptContext, type ChunkAnalysisResult, type FindingProcessingEvent, @@ -196,6 +197,26 @@ export interface SkillTaskOptions { context: EventContext; /** Options passed to the runner */ runnerOptions?: SkillRunnerOptions; + /** Prepared review chunks shared across skill tasks for the same changeset. */ + preparedFiles?: PreparedTaskFiles; +} + +export interface PreparedTaskFiles { + groups: ReviewChunkGroup[]; + skippedFiles: SkippedFile[]; + semanticUsage?: UsageStats; +} + +function fileReportInputsFromGroup(args: { + group: ReviewChunkGroup; + durationMs?: number; + usage?: UsageStats; +}) { + return args.group.filenames.map((filename, index) => ({ + filename, + durationMs: index === 0 ? args.durationMs : undefined, + usage: index === 0 ? args.usage : undefined, + })); } /** @@ -220,7 +241,7 @@ export interface SkillProgressCallbacks { onSkillStart: (skill: SkillState) => void; onSkillUpdate: (name: string, updates: Partial) => void; onFileUpdate: (skillName: string, filename: string, updates: Partial) => void; - /** Called when a hunk analysis starts (one SDK invocation per hunk) */ + /** Called when a chunk analysis starts (one SDK invocation per review chunk) */ onHunkStart?: (skillName: string, filename: string, hunkNum: number, totalHunks: number, lineRange: string) => void; onChunkComplete?: (skillName: string, chunk: ChunkAnalysisResult) => void; onSkillComplete: (name: string, report: SkillReport) => void; @@ -234,7 +255,7 @@ export interface SkillProgressCallbacks { onExtractionResult?: (skillName: string, filename: string, lineRange: string, findingsCount: number, method: 'regex' | 'llm' | 'none') => void; /** Called when findings are dropped, revised, merged, or stripped after analysis */ onFindingProcessing?: (skillName: string, event: FindingProcessingEvent) => void; - /** Called when hunk analysis fails (SDK error, API error, abort) */ + /** Called when chunk analysis fails (SDK error, API error, abort) */ onHunkFailed?: (skillName: string, filename: string, lineRange: string, error: string) => void; /** Called when findings extraction fails (both regex and LLM fallback failed) */ onExtractionFailure?: (skillName: string, filename: string, lineRange: string, error: string, preview: string) => void; @@ -242,6 +263,34 @@ export interface SkillProgressCallbacks { onRetry?: (skillName: string, filename: string, lineRange: string, attempt: number, maxRetries: number, error: string, delayMs: number) => void; } +async function prepareTaskFiles( + context: EventContext, + runnerOptions: SkillRunnerOptions, +): Promise { + const { files: initialPreparedFiles, skippedFiles } = prepareFiles(context, { + contextLines: runnerOptions.contextLines, + ignore: runnerOptions.ignore, + scan: runnerOptions.scan, + chunking: runnerOptions.chunking, + }); + const semanticPlan = await planSemanticReviewChunks(initialPreparedFiles, context, { + enabled: runnerOptions.chunking?.semantic?.enabled, + apiKey: runnerOptions.apiKey, + runtime: runnerOptions.runtime, + model: runnerOptions.auxiliaryModel ?? runnerOptions.model, + maxRetries: runnerOptions.auxiliaryMaxRetries, + maxChunks: runnerOptions.chunking?.semantic?.maxChunks, + maxChunkChars: runnerOptions.chunking?.semantic?.maxChunkChars, + maxHunksPerChunk: runnerOptions.chunking?.semantic?.maxHunksPerChunk, + }); + + return { + groups: semanticPlan.groups, + skippedFiles, + semanticUsage: semanticPlan.usage, + }; +} + /** * Run a single skill task. */ @@ -285,15 +334,12 @@ export async function runSkillTask( throw new SkillRunnerError(message, { cause: err, code: 'skill_resolution_failed' }); } - // Prepare files (parse patches into hunks) - const { files: preparedFiles, skippedFiles } = prepareFiles(context, { - contextLines: runnerOptions.contextLines, - ignore: runnerOptions.ignore, - scan: runnerOptions.scan, - chunking: runnerOptions.chunking, - }); + // Prepare files into review chunks before optional semantic planning. + const prepared = options.preparedFiles ?? await prepareTaskFiles(context, runnerOptions); + const chunkGroups = prepared.groups; + const skippedFiles = prepared.skippedFiles; - if (preparedFiles.length === 0) { + if (chunkGroups.length === 0) { // No files to analyze - skip const skippedReport: SkillReport = { skill: skill.name, @@ -318,11 +364,11 @@ export async function runSkillTask( } // Initialize file states - const fileStates: FileState[] = preparedFiles.map((file) => ({ - filename: file.filename, + const fileStates: FileState[] = chunkGroups.map((group) => ({ + filename: group.displayName, status: 'pending', currentHunk: 0, - totalHunks: file.hunks.length, + totalHunks: group.chunks.length, findings: [], })); @@ -338,7 +384,7 @@ export async function runSkillTask( // Build PR context for inclusion in prompts (if available) // For non-PR contexts (CLI file/diff mode), skip the "Other Files" list to avoid - // bloating every hunk prompt with thousands of filenames. + // bloating every chunk prompt with thousands of filenames. const isPullRequest = context.pullRequest ? context.pullRequest.number !== 0 : false; const prContext: PRPromptContext | undefined = context.pullRequest ? { @@ -350,8 +396,8 @@ export async function runSkillTask( : undefined; // Process files with concurrency - const processFile = async (prepared: PreparedFile, index: number): Promise => { - const filename = prepared.filename; + const processFile = async (group: ReviewChunkGroup, index: number): Promise => { + const filename = group.displayName; const fileStartTime = Date.now(); // Update file state to running (local + callback) @@ -437,7 +483,7 @@ export async function runSkillTask( const result = await analyzeFile( skill, - prepared, + group, context.repoPath, runnerOptions, fileCallbacks, @@ -474,7 +520,7 @@ export async function runSkillTask( const processSkippedFile = (index: number): FileProcessResult => { const localState = fileStates[index]; if (localState) localState.status = 'skipped'; - const filename = preparedFiles[index]?.filename ?? 'unknown'; + const filename = chunkGroups[index]?.displayName ?? 'unknown'; callbacks.onFileUpdate(name, filename, { status: 'skipped' }); return { findings: [], durationMs: 0, failedHunks: 0, failedExtractions: 0, hunkFailures: [] }; }; @@ -485,8 +531,8 @@ export async function runSkillTask( // The effective concurrency for batch delay: when a semaphore gates work, // use its permit count (the actual concurrency limit) rather than fileConcurrency. const effectiveConcurrency = semaphore ? semaphore.initialPermits : fileConcurrency; - const allResults = await runPool(preparedFiles, fileConcurrency, - async (file, index) => { + const allResults = await runPool(chunkGroups, fileConcurrency, + async (group, index) => { if (semaphore) await semaphore.acquire(); try { // Check abort after acquiring the semaphore -- the file may have @@ -496,7 +542,7 @@ export async function runSkillTask( if (index >= effectiveConcurrency && batchDelayMs > 0) { await new Promise((resolve) => setTimeout(resolve, batchDelayMs)); } - return await processFile(file, index); + return await processFile(group, index); } finally { if (semaphore) semaphore.release(); } @@ -516,15 +562,23 @@ export async function runSkillTask( const allFindings = allResults.flatMap((r) => r.findings); const allUsage = allResults.map((r) => r.usage).filter((u): u is UsageStats => u !== undefined); const allAuxEntries = allResults.flatMap((r) => r.auxiliaryUsage ?? []); + if (prepared.semanticUsage) { + allAuxEntries.push({ + agent: 'semantic-chunk-planner', + usage: prepared.semanticUsage, + model: runnerOptions.auxiliaryModel ?? runnerOptions.model, + runtime: runnerOptions.runtime, + }); + } const allTraces = allResults.flatMap((r) => r.traces ?? []); const totalFailedHunks = allResults.reduce((sum, r) => sum + r.failedHunks, 0); const totalFailedExtractions = allResults.reduce((sum, r) => sum + r.failedExtractions, 0); const allHunkFailures: HunkFailure[] = allResults.flatMap((r) => r.hunkFailures); - const totalHunks = preparedFiles.reduce((sum, f) => sum + f.hunks.length, 0); - // Each hunk contributes to at most one of failedHunks / failedExtractions + const totalHunks = chunkGroups.reduce((sum, group) => sum + group.chunks.length, 0); + // Each chunk contributes to at most one of failedHunks / failedExtractions // (mutually exclusive in analyzeFile), so summing them gives the total - // failed-hunk count. Counting only analysis failures would miss the - // scenario where every hunk's SDK call succeeded but every extraction + // failed chunk count. Counting only analysis failures would miss the + // scenario where every chunk's SDK call succeeded but every extraction // failed — a silent zero-findings run otherwise. const totalAttemptFailures = totalFailedHunks + totalFailedExtractions; @@ -556,19 +610,21 @@ export async function runSkillTask( durationMs: duration, model: runnerOptions?.model, runtime, - // Preserve per-file metadata (timing, partial usage, attempted + // Preserve per-group metadata (timing, partial usage, attempted // filenames) on failure runs too — `warden runs` and JSONL // consumers iterate this array to count attempted files. Without // it, a failed run shows totalFiles: 0. - files: preparedFiles.map((file, i) => { - const r = allResults[i]; - return { - filename: file.filename, - findings: r?.findings.length ?? 0, - durationMs: r?.durationMs, - usage: r?.usage, - }; - }), + files: buildFileReports( + chunkGroups.flatMap((group, i) => { + const r = allResults[i]; + return fileReportInputsFromGroup({ + group, + durationMs: r?.durationMs, + usage: r?.usage, + }); + }), + allFindings, + ), failedHunks: totalFailedHunks, hunkFailures: allHunkFailures, error: { @@ -630,13 +686,13 @@ export async function runSkillTask( model: runnerOptions?.model, runtime, files: buildFileReports( - preparedFiles.map((file, i) => { + chunkGroups.flatMap((group, i) => { const r = allResults[i]; - return { - filename: file.filename, + return fileReportInputsFromGroup({ + group, durationMs: r?.durationMs, usage: r?.usage, - }; + }); }), finalFindings, ), @@ -865,7 +921,7 @@ export function createDefaultCallbacks( debugLog(mode, formatFindingProcessingEvent(event)); } : undefined, - // Verbose mode: show per-hunk analysis failures (spec: event #16 hunk_failed) + // Verbose mode: show per-chunk analysis failures (spec: event #16 hunk_failed) onHunkFailed: verbosity >= Verbosity.Verbose ? (_skillName, filename, lineRange, error) => { const location = `${filename}:${lineRange}`; @@ -876,7 +932,7 @@ export function createDefaultCallbacks( } } : undefined, - // Verbose mode: show per-hunk extraction failures (spec: event #17 extraction_failure) + // Verbose mode: show per-chunk extraction failures (spec: event #17 extraction_failure) onExtractionFailure: verbosity >= Verbosity.Verbose ? (_skillName, filename, lineRange, error, preview) => { const location = `${filename}:${lineRange}`; @@ -971,6 +1027,85 @@ export function composeTasksWithFailFast( })); } +function semanticContextKey(context: EventContext): object { + const pullRequest = context.pullRequest + ? { + number: context.pullRequest.number, + title: context.pullRequest.title, + body: context.pullRequest.body, + headSha: context.pullRequest.headSha, + baseSha: context.pullRequest.baseSha, + files: [...context.pullRequest.files] + .map((file) => ({ + filename: file.filename, + status: file.status, + additions: file.additions, + deletions: file.deletions, + patch: file.patch, + chunks: file.chunks, + })) + .sort((a, b) => a.filename.localeCompare(b.filename)), + } + : undefined; + + return { + eventType: context.eventType, + action: context.action, + label: context.label, + repoPath: context.repoPath, + diffContextSource: context.diffContextSource, + repository: context.repository, + pullRequest, + }; +} + +function semanticPlanKey(task: SkillTaskOptions): string | undefined { + if (!task.runnerOptions?.chunking?.semantic?.enabled) return undefined; + return JSON.stringify({ + context: semanticContextKey(task.context), + contextLines: task.runnerOptions.contextLines, + ignore: task.runnerOptions.ignore, + scan: task.runnerOptions.scan, + chunking: task.runnerOptions.chunking, + apiKey: Boolean(task.runnerOptions.apiKey), + runtime: task.runnerOptions.runtime, + model: task.runnerOptions.auxiliaryModel ?? task.runnerOptions.model, + auxiliaryMaxRetries: task.runnerOptions.auxiliaryMaxRetries, + }); +} + +/** Share semantic chunk planning across skill tasks for the same changeset. */ +export async function prepareSemanticPlansForTasks(tasks: SkillTaskOptions[]): Promise { + const plansByKey = new Map>(); + const chargedPlans = new Set(); + + return Promise.all(tasks.map(async (task) => { + const key = semanticPlanKey(task); + if (!key || task.preparedFiles) { + return task; + } + + let plan = plansByKey.get(key); + if (!plan) { + plan = prepareTaskFiles(task.context, task.runnerOptions ?? {}); + plansByKey.set(key, plan); + } + + const chargeSemanticUsage = !chargedPlans.has(key); + chargedPlans.add(key); + + const preparedFiles = await plan; + + return { + ...task, + preparedFiles: { + ...preparedFiles, + semanticUsage: chargeSemanticUsage ? preparedFiles.semanticUsage : undefined, + }, + }; + })); +} + /** * Launch all skill tasks in parallel using a shared semaphore for concurrency. */ @@ -1035,8 +1170,9 @@ export async function runSkillTasks( const circuitAbortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController: circuitAbortController }); + const plannedTasks = await prepareSemanticPlansForTasks(tasks); const composedTasks = composeTasksWithFailFast( - tasks, + plannedTasks, failFastController, circuitBreaker, circuitAbortController, diff --git a/packages/warden/src/config/loader.test.ts b/packages/warden/src/config/loader.test.ts index b6a443f6..f72a831f 100644 --- a/packages/warden/src/config/loader.test.ts +++ b/packages/warden/src/config/loader.test.ts @@ -595,6 +595,7 @@ describe('mergeWardenConfigs', () => { chunking: { filePatterns: [{ pattern: '**/*.lock', mode: 'skip' }], coalesce: { enabled: true, maxGapLines: 20, maxChunkSize: 4000 }, + semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, preferWholeFileBelowLines: 800 }, maxContextFiles: 25, }, }, @@ -614,6 +615,7 @@ describe('mergeWardenConfigs', () => { chunking: { filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], coalesce: { maxGapLines: 5, maxChunkSize: 2000, enabled: true }, + semantic: { enabled: true, maxChunks: 10, maxChunkChars: 12000, maxHunksPerChunk: 25, preferWholeFileBelowLines: 500 }, maxContextFiles: 10, }, }, @@ -643,12 +645,46 @@ describe('mergeWardenConfigs', () => { { pattern: '**/*.snap', mode: 'skip' }, ], coalesce: { enabled: true, maxGapLines: 5, maxChunkSize: 2000 }, + semantic: { enabled: true, maxChunks: 10, maxChunkChars: 12000, maxHunksPerChunk: 25, preferWholeFileBelowLines: 500 }, maxContextFiles: 10, }, }); expect(merged.skills.map((skill) => skill.name)).toEqual(['org-skill', 'repo-skill']); }); + it('field-merges semantic chunking defaults across config layers', () => { + const baseConfig = WardenConfigSchema.parse({ + version: 1, + defaults: { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 50000, + }, + }, + }, + skills: [], + }); + const repoConfig = WardenConfigSchema.parse({ + version: 1, + defaults: { + chunking: { + semantic: { + maxChunks: 8, + }, + }, + }, + skills: [], + }); + + expect(mergeWardenConfigs(baseConfig, repoConfig).defaults?.chunking?.semantic).toEqual({ + enabled: true, + maxChunks: 8, + maxChunkChars: 50000, + }); + }); + it('deep-merges nested default model lanes across layers', () => { const baseConfig: WardenConfig = { version: 1, diff --git a/packages/warden/src/config/loader.ts b/packages/warden/src/config/loader.ts index 20e43d5a..b21a03b7 100644 --- a/packages/warden/src/config/loader.ts +++ b/packages/warden/src/config/loader.ts @@ -13,6 +13,7 @@ import { type Defaults, type ChunkingConfig, type CoalesceConfig, + type SemanticChunkingConfig, type IgnoreConfig, type ScanConfig, type RunnerConfig, @@ -97,6 +98,15 @@ function mergeCoalesceConfig( return { ...base, ...overlay }; } +function mergeSemanticChunkingConfig( + base?: SemanticChunkingConfig, + overlay?: SemanticChunkingConfig +): SemanticChunkingConfig | undefined { + if (!base) return overlay; + if (!overlay) return base; + return { ...base, ...overlay }; +} + function mergeChunkingConfig( base?: ChunkingConfig, overlay?: ChunkingConfig @@ -108,6 +118,7 @@ function mergeChunkingConfig( ...overlay, filePatterns: mergeArray(base.filePatterns, overlay.filePatterns), coalesce: mergeCoalesceConfig(base.coalesce, overlay.coalesce), + semantic: mergeSemanticChunkingConfig(base.semantic, overlay.semantic), }; } diff --git a/packages/warden/src/config/schema.ts b/packages/warden/src/config/schema.ts index 245c30df..65753f61 100644 --- a/packages/warden/src/config/schema.ts +++ b/packages/warden/src/config/schema.ts @@ -144,7 +144,7 @@ export const SkillConfigSchema = z.object({ failCheck: z.boolean().optional(), /** Model to use for this skill (e.g., 'openai/gpt-5.5'). Uses SDK default if not specified. */ model: z.string().optional(), - /** Maximum agentic turns (API round-trips) per hunk analysis. Overrides defaults.maxTurns. */ + /** Maximum agentic turns (API round-trips) per chunk analysis. Overrides defaults.maxTurns. */ maxTurns: z.number().int().positive().optional(), /** Minimum confidence level for findings. Findings below this are filtered from output. */ minConfidence: ConfidenceThresholdSchema.optional(), @@ -180,13 +180,29 @@ export const CoalesceConfigSchema = z.object({ }); export type CoalesceConfig = z.infer; +export const SemanticChunkingConfigSchema = z.object({ + /** Enable semantic review chunk materialization (default: false) */ + enabled: z.boolean().optional(), + /** Maximum number of review chunks to emit after semantic grouping */ + maxChunks: z.number().int().positive().optional(), + /** Target max size per semantic review chunk in characters */ + maxChunkChars: z.number().int().positive().optional(), + /** Maximum atomic hunks to group into one semantic review chunk */ + maxHunksPerChunk: z.number().int().positive().optional(), + /** Reserved for whole-file materialization below this line count */ + preferWholeFileBelowLines: z.number().int().positive().optional(), +}); +export type SemanticChunkingConfig = z.infer; + // Chunking configuration for controlling how files are processed export const ChunkingConfigSchema = z.object({ /** Patterns to control file processing mode */ filePatterns: z.array(FilePatternSchema).optional(), /** Coalescing options for merging nearby hunks */ coalesce: CoalesceConfigSchema.optional(), - /** Max number of "other files" to list in hunk prompts for PR context. 0 disables the section entirely. Default: 50 */ + /** Semantic review chunk options */ + semantic: SemanticChunkingConfigSchema.optional(), + /** Max number of "other files" to list in chunk prompts for PR context. 0 disables the section entirely. Default: 50 */ maxContextFiles: z.number().int().nonnegative().default(50), }); export type ChunkingConfig = z.infer; @@ -231,7 +247,7 @@ export const DefaultsSchema = z.object({ failCheck: z.boolean().optional(), /** Default model for all skills (e.g., 'openai/gpt-5.5') */ model: z.string().optional(), - /** Maximum agentic turns (API round-trips) per hunk analysis. Default: 50 */ + /** Maximum agentic turns (API round-trips) per chunk analysis. Default: 50 */ maxTurns: z.number().int().positive().optional(), /** Runtime backend for all model-backed execution. Default: pi */ runtime: RuntimeNameSchema.optional(), diff --git a/packages/warden/src/diff/index.ts b/packages/warden/src/diff/index.ts index 3dd9078e..019df42d 100644 --- a/packages/warden/src/diff/index.ts +++ b/packages/warden/src/diff/index.ts @@ -1,5 +1,6 @@ export * from './parser.js'; export * from './apply.js'; export * from './context.js'; +export * from './review-chunk.js'; export * from './classify.js'; export * from './coalesce.js'; diff --git a/packages/warden/src/diff/review-chunk.test.ts b/packages/warden/src/diff/review-chunk.test.ts new file mode 100644 index 00000000..7524a8b1 --- /dev/null +++ b/packages/warden/src/diff/review-chunk.test.ts @@ -0,0 +1,32 @@ +import { describe, expect, it } from 'vitest'; +import { reviewChunkFromHunk } from './review-chunk.js'; +import type { HunkWithContext } from './context.js'; + +function makeHunkContext(overrides: Partial = {}): HunkWithContext { + return { + filename: 'src/example.ts', + hunk: { + oldStart: 10, + oldCount: 1, + newStart: 10, + newCount: 0, + header: '@@ -10,1 +10,0 @@', + lines: ['-const removed = true;'], + content: '-const removed = true;', + }, + contextBefore: [], + contextAfter: [], + contextStartLine: 10, + language: 'typescript', + ...overrides, + }; +} + +describe('reviewChunkFromHunk', () => { + it('uses hunk coordinates and an anchor range for deletion-only chunks', () => { + const chunk = reviewChunkFromHunk(makeHunkContext()); + + expect(chunk.changedLineMap).toEqual([{ path: 'src/example.ts', start: 10, end: 10 }]); + expect(chunk.id).toBe('src/example.ts:old10-10:new10-9'); + }); +}); diff --git a/packages/warden/src/diff/review-chunk.ts b/packages/warden/src/diff/review-chunk.ts new file mode 100644 index 00000000..4b389db9 --- /dev/null +++ b/packages/warden/src/diff/review-chunk.ts @@ -0,0 +1,121 @@ +import type { SourceSnippetLine } from '../types/index.js'; +import type { HunkWithContext } from './context.js'; +import { formatHunkForAnalysis } from './context.js'; + +export type ReviewChunkContentMode = 'whole-file' | 'stitched-file' | 'raw-hunks'; + +export interface ChangedLineRange { + path: string; + start: number; + end: number; +} + +export interface ReviewChunkFile { + path: string; + language: string; + changedRanges: ChangedLineRange[]; + content: string; + contentMode: ReviewChunkContentMode; + sourceLines: SourceSnippetLine[]; +} + +export interface ReviewChunk { + id: string; + title: string; + summary?: string; + files: ReviewChunkFile[]; + changedLineMap: ChangedLineRange[]; +} + +function changedRangesFromHunk(path: string, hunkCtx: HunkWithContext): ChangedLineRange[] { + const ranges: ChangedLineRange[] = []; + let newLine = hunkCtx.hunk.newStart; + let currentStart: number | undefined; + let currentEnd: number | undefined; + + function flush(): void { + if (currentStart === undefined || currentEnd === undefined) return; + ranges.push({ path, start: currentStart, end: currentEnd }); + currentStart = undefined; + currentEnd = undefined; + } + + for (const diffLine of hunkCtx.hunk.lines) { + if (diffLine.startsWith('+')) { + if (currentStart === undefined) { + currentStart = newLine; + } + currentEnd = newLine; + newLine += 1; + continue; + } + + flush(); + if (diffLine.startsWith(' ') || !diffLine.startsWith('-')) { + newLine += 1; + } + } + + flush(); + if ( + ranges.length === 0 + && hunkCtx.hunk.lines.some((line) => line.startsWith('-')) + ) { + const start = hunkCtx.hunk.newStart; + const end = Math.max(start, hunkCtx.hunk.newStart + hunkCtx.hunk.newCount - 1); + return [{ path, start, end }]; + } + + return ranges; +} + +function hunkSourceLines(hunkCtx: HunkWithContext): SourceSnippetLine[] { + const lines: SourceSnippetLine[] = []; + for (const [index, content] of hunkCtx.contextBefore.entries()) { + lines.push({ line: hunkCtx.contextStartLine + index, content }); + } + + let newLine = hunkCtx.hunk.newStart; + for (const diffLine of hunkCtx.hunk.lines) { + if (diffLine.startsWith('-')) continue; + if (!diffLine.startsWith('+') && !diffLine.startsWith(' ')) continue; + const content = diffLine.slice(1); + lines.push({ line: newLine, content }); + newLine += 1; + } + + const afterStart = hunkCtx.hunk.newStart + hunkCtx.hunk.newCount; + for (const [index, content] of hunkCtx.contextAfter.entries()) { + lines.push({ line: afterStart + index, content }); + } + + return lines; +} + +/** Convert an expanded git hunk into Warden's scanner-facing review chunk. */ +export function reviewChunkFromHunk(hunkCtx: HunkWithContext): ReviewChunk { + const changedRanges = changedRangesFromHunk(hunkCtx.filename, hunkCtx); + const hasAddedLines = hunkCtx.hunk.lines.some((line) => line.startsWith('+')); + const lineRange = changedRanges + .map((range) => range.start === range.end ? `${range.start}` : `${range.start}-${range.end}`) + .join(','); + const hunkCoordinate = [ + `old${hunkCtx.hunk.oldStart}-${hunkCtx.hunk.oldStart + hunkCtx.hunk.oldCount - 1}`, + `new${hunkCtx.hunk.newStart}-${hunkCtx.hunk.newStart + hunkCtx.hunk.newCount - 1}`, + ].join(':'); + const chunkKey = hasAddedLines ? lineRange : hunkCoordinate; + + return { + id: `${hunkCtx.filename}:${chunkKey}`, + title: `${hunkCtx.filename}:${chunkKey}`, + files: [{ + path: hunkCtx.filename, + language: hunkCtx.language, + changedRanges, + content: formatHunkForAnalysis(hunkCtx), + contentMode: 'raw-hunks', + sourceLines: hunkSourceLines(hunkCtx), + }], + changedLineMap: changedRanges, + }; +} diff --git a/packages/warden/src/sdk/analyze.test.ts b/packages/warden/src/sdk/analyze.test.ts index 086ed47e..0060a097 100644 --- a/packages/warden/src/sdk/analyze.test.ts +++ b/packages/warden/src/sdk/analyze.test.ts @@ -1,7 +1,7 @@ import { describe, it, expect, vi, afterEach, beforeAll, afterAll } from 'vitest'; import { APIError } from '@anthropic-ai/sdk'; import type { SkillDefinition } from '../config/schema.js'; -import type { HunkWithContext } from '../diff/index.js'; +import { reviewChunkFromHunk, type HunkWithContext } from '../diff/index.js'; import type { EventContext, Finding, UsageStats } from '../types/index.js'; import { analyzeFile, buildSourceSnippet, filterOutOfRangeFindings, runSkill } from './analyze.js'; import type { PreparedFile } from './types.js'; @@ -30,14 +30,14 @@ afterAll(async () => { await Sentry.close(0); }); -function makeFinding(startLine: number, id = `f-${startLine}`): Finding { +function makeFinding(startLine: number, id = `f-${startLine}`, path = 'file.ts'): Finding { return { id, severity: 'medium', confidence: 'high', title: `Finding at line ${startLine}`, description: 'test', - location: { path: 'file.ts', startLine }, + location: { path, startLine }, }; } @@ -81,7 +81,7 @@ function makePreparedFile(hunkCount = 1): PreparedFile { }); return { filename: 'src/example.ts', - hunks, + chunks: hunks.map(reviewChunkFromHunk), }; } @@ -237,6 +237,27 @@ describe('filterOutOfRangeFindings', () => { expect(dropped).toEqual([belowRange, aboveRange]); }); + it('filters findings against multi-file changed line maps', () => { + const firstFileFinding = makeFinding(15, 'first-file', 'src/one.ts'); + const secondFileFinding = makeFinding(42, 'second-file', 'src/two.ts'); + const wrongFileFinding = makeFinding(15, 'wrong-file', 'src/three.ts'); + const partialRangeFinding: Finding = { + ...makeFinding(42, 'partial-range', 'src/two.ts'), + location: { path: 'src/two.ts', startLine: 42, endLine: 45 }, + }; + + const { filtered, dropped } = filterOutOfRangeFindings( + [firstFileFinding, secondFileFinding, wrongFileFinding, partialRangeFinding], + [ + { path: 'src/one.ts', start: 10, end: 20 }, + { path: 'src/two.ts', start: 40, end: 43 }, + ], + ); + + expect(filtered).toEqual([firstFileFinding, secondFileFinding]); + expect(dropped).toEqual([wrongFileFinding, partialRangeFinding]); + }); + it('returns empty arrays for empty input', () => { const { filtered, dropped } = filterOutOfRangeFindings([], hunkRange); expect(filtered).toEqual([]); @@ -268,10 +289,10 @@ describe('buildSourceSnippet', () => { language: 'typescript', }; - const snippet = buildSourceSnippet(makeFinding(11), hunk, 2); + const snippet = buildSourceSnippet(makeFinding(11, 'f-11', 'src/example.ts'), reviewChunkFromHunk(hunk), 2); expect(snippet).toEqual({ - path: 'file.ts', + path: 'src/example.ts', language: 'typescript', startLine: 9, endLine: 13, @@ -304,7 +325,7 @@ describe('buildSourceSnippet', () => { language: 'typescript', }; - const snippet = buildSourceSnippet(makeFinding(10), hunk, 1); + const snippet = buildSourceSnippet(makeFinding(10, 'f-10', 'src/example.ts'), reviewChunkFromHunk(hunk), 1); expect(snippet?.lines).toEqual([ { line: 10, content: 'newCall();', highlighted: true }, @@ -722,6 +743,73 @@ describe('runSkill', () => { expect(report.runtime).toBe('pi'); }); + it('sends planner-provided semantic chunks to the scanner', async () => { + const runSkillMock = vi.fn().mockResolvedValue({ + result: { + status: 'success', + text: JSON.stringify({ findings: [] }), + errors: [], + usage: makeUsage(), + }, + }); + const runAuxiliaryMock = vi.fn().mockResolvedValue({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart conversion, rendering, and tests.', + chunkIds: [ + 'src/example.ts:10', + 'src/example.ts:100', + 'src/example.ts:200', + ], + }], + }, + usage: makeUsage(), + }); + vi.mocked(getRuntime).mockReturnValue({ + name: 'pi', + runSkill: runSkillMock, + runAuxiliary: runAuxiliaryMock, + runSynthesis: vi.fn(), + } as unknown as Runtime); + + const report = await runSkill( + { + name: 'security-review', + description: 'Security review.', + prompt: 'Return findings as JSON.', + }, + makeContextWithThreeHunks(), + { + runtime: 'pi', + model: 'anthropic/claude-sonnet-4-6', + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + postProcessFindings: false, + }, + ); + + expect(runAuxiliaryMock).toHaveBeenCalledWith(expect.objectContaining({ + task: 'semantic_chunking', + agentName: 'semantic-chunk-planner', + })); + expect(runSkillMock).toHaveBeenCalledTimes(1); + const request = runSkillMock.mock.calls[0]?.[0]; + expect(request.userPrompt).toContain('## Semantic Summary: Dashboard charts now carry a widget-provided axis range through chart conversion, rendering, and tests.'); + expect(request.userPrompt).toContain('- src/example.ts:10-10'); + expect(request.userPrompt).toContain('- src/example.ts:100-100'); + expect(request.userPrompt).toContain('- src/example.ts:200-200'); + expect(report.auxiliaryUsageAttribution?.['semantic-chunk-planner']).toBeDefined(); + }); + it('preserves candidate findings when verification is interrupted', async () => { const controller = new AbortController(); const runSkillMock = vi.fn() @@ -729,7 +817,7 @@ describe('runSkill', () => { result: { status: 'success', text: JSON.stringify({ - findings: [makeFinding(10, 'candidate-finding')], + findings: [makeFinding(10, 'candidate-finding', 'src/example.ts')], }), errors: [], usage: makeUsage(), @@ -777,7 +865,7 @@ describe('runSkill', () => { result: { status: 'success', text: JSON.stringify({ - findings: [makeFinding(10, 'first-finding')], + findings: [makeFinding(10, 'first-finding', 'src/example.ts')], }), errors: [], usage: makeUsage(), diff --git a/packages/warden/src/sdk/analyze.ts b/packages/warden/src/sdk/analyze.ts index 14bcc5a1..558cda9b 100644 --- a/packages/warden/src/sdk/analyze.ts +++ b/packages/warden/src/sdk/analyze.ts @@ -1,13 +1,13 @@ import type { Span } from '@sentry/node'; import type { SkillDefinition } from '../config/schema.js'; import type { ErrorCode, Finding, RetryConfig } from '../types/index.js'; -import { getHunkLineRange, type HunkWithContext } from '../diff/index.js'; +import type { ChangedLineRange, ReviewChunk } from '../diff/index.js'; import { Sentry, emitExtractionMetrics, emitRetryMetric, emitSkillMetrics, ensureLocalTracing } from '../sentry.js'; import { SkillRunnerError, WardenAuthenticationError, isRetryableError, isAuthenticationError, isAuthenticationErrorMessage, isSubprocessError, classifyError, mapExtractionErrorCode, sanitizeErrorMessage } from './errors.js'; import type { CircuitBreakerReason } from './circuit-breaker.js'; import { DEFAULT_RETRY_CONFIG, calculateRetryDelay, sleep } from './retry.js'; import { aggregateUsage, emptyUsage, estimateTokens, aggregateAuxiliaryUsage, aggregateAuxiliaryUsageAttribution } from './usage.js'; -import { buildHunkSystemPrompt, buildHunkUserPrompt, type PRPromptContext } from './prompt.js'; +import { buildHunkSystemPrompt, buildReviewChunkUserPrompt, type PRPromptContext } from './prompt.js'; import { extractFindingsJson, extractFindingsWithLLM, validateFindings } from './extract.js'; import { postProcessFindings } from './post-process.js'; import { buildFileReports } from './report-files.js'; @@ -21,17 +21,19 @@ import { type HunkAnalysisCallbacks, type SkillRunnerOptions, type PreparedFile, + type ReviewChunkGroup, type FileAnalysisCallbacks, type FileAnalysisResult, type ChunkAnalysisResult, } from './types.js'; import { prepareFiles } from './prepare.js'; +import { planSemanticReviewChunks } from './semantic-chunk-planner.js'; import type { EventContext, SkillReport, UsageStats, HunkFailure, HunkTrace } from '../types/index.js'; -import type { SourceSnippet, SourceSnippetLine } from '../types/index.js'; +import type { SourceSnippet } from '../types/index.js'; import { runPool } from '../utils/index.js'; import { getSpanContext, startTraceRecorder, withTraceRecorder, type TraceRecorder } from '../sentry-trace.js'; -/** Result from parsing hunk output */ +/** Result from parsing review chunk output */ interface ParseHunkOutputResult { findings: Finding[]; /** Whether extraction failed (both regex and LLM fallback) */ @@ -102,6 +104,30 @@ function allHunksFailedGuidance(runtime: SkillRunnerOptions['runtime'] | undefin return "Verify WARDEN_ANTHROPIC_API_KEY is set correctly, or run 'claude login' when using the Claude runtime without an API key."; } +function normalizeReviewChunkGroup(input: PreparedFile | ReviewChunkGroup): ReviewChunkGroup { + if ('filenames' in input) { + return input; + } + + return { + displayName: input.filename, + filenames: [input.filename], + chunks: input.chunks, + }; +} + +function fileReportInputsFromGroup(args: { + group: ReviewChunkGroup; + durationMs?: number; + usage?: UsageStats; +}) { + return args.group.filenames.map((filename, index) => ({ + filename, + durationMs: index === 0 ? args.durationMs : undefined, + usage: index === 0 ? args.usage : undefined, + })); +} + function buildHunkTrace(args: { enabled: boolean | undefined; span: Span; @@ -137,14 +163,14 @@ function buildHunkTrace(args: { } /** - * Parse findings from a hunk analysis result. + * Parse findings from a review chunk analysis result. * Uses a two-tier extraction strategy: * 1. Regex-based extraction (fast, handles well-formed output) * 2. LLM fallback using haiku (handles malformed output gracefully) */ async function parseHunkOutput( result: SkillRunResult, - filename: string, + defaultFilename: string | undefined, skillName: string, options: SkillRunnerOptions ): Promise { @@ -157,7 +183,7 @@ async function parseHunkOutput( const extracted = extractFindingsJson(result.text); if (extracted.success) { - return { findings: validateFindings(extracted.findings, filename), extractionFailed: false, extractionMethod: 'regex' }; + return { findings: validateFindings(extracted.findings, defaultFilename), extractionFailed: false, extractionMethod: 'regex' }; } // Tier 2: Try LLM fallback for malformed output @@ -170,7 +196,7 @@ async function parseHunkOutput( }); if (fallback.success) { - return { findings: validateFindings(fallback.findings, filename), extractionFailed: false, extractionMethod: 'llm', extractionUsage: fallback.usage }; + return { findings: validateFindings(fallback.findings, defaultFilename), extractionFailed: false, extractionMethod: 'llm', extractionUsage: fallback.usage }; } // Both tiers failed - return extraction failure info @@ -185,20 +211,28 @@ async function parseHunkOutput( } /** - * Filter findings whose startLine falls outside the hunk line range. + * Filter findings whose location falls outside the changed line map. * Findings without a location are kept (general findings). */ export function filterOutOfRangeFindings( findings: Finding[], - hunkRange: { start: number; end: number } + changedLineMap: ChangedLineRange[] | { start: number; end: number } ): { filtered: Finding[]; dropped: Finding[] } { + const ranges: (ChangedLineRange | { path?: undefined; start: number; end: number })[] = Array.isArray(changedLineMap) + ? changedLineMap + : [{ start: changedLineMap.start, end: changedLineMap.end }]; const filtered: Finding[] = []; const dropped: Finding[] = []; function isWithinHunk(finding: Finding): boolean { if (!finding.location) return true; - const { startLine } = finding.location; - return startLine >= hunkRange.start && startLine <= hunkRange.end; + const { path, startLine } = finding.location; + const endLine = finding.location.endLine ?? startLine; + return ranges.some((range) => + (range.path === undefined || range.path === path) + && startLine >= range.start + && endLine <= range.end + ); } for (const finding of findings) { @@ -211,41 +245,21 @@ export function filterOutOfRangeFindings( return { filtered, dropped }; } -function hunkSourceLines(hunkCtx: HunkWithContext): SourceSnippetLine[] { - const lines: SourceSnippetLine[] = []; - for (const [index, content] of hunkCtx.contextBefore.entries()) { - lines.push({ line: hunkCtx.contextStartLine + index, content }); - } - - let newLine = hunkCtx.hunk.newStart; - for (const diffLine of hunkCtx.hunk.lines) { - if (diffLine.startsWith('-')) continue; - if (!diffLine.startsWith('+') && !diffLine.startsWith(' ')) continue; - const content = diffLine.slice(1); - lines.push({ line: newLine, content }); - newLine += 1; - } - - const afterStart = hunkCtx.hunk.newStart + hunkCtx.hunk.newCount; - for (const [index, content] of hunkCtx.contextAfter.entries()) { - lines.push({ line: afterStart + index, content }); - } - - return lines; -} - +/** Build a source snippet for a finding from the matching review chunk file. */ export function buildSourceSnippet( finding: Finding, - hunkCtx: HunkWithContext, + chunk: ReviewChunk, contextLines = 3 ): SourceSnippet | undefined { if (!finding.location) return undefined; + const chunkFile = chunk.files.find((file) => file.path === finding.location?.path); + if (!chunkFile) return undefined; const targetStartLine = finding.location.startLine; const targetEndLine = finding.location.endLine ?? targetStartLine; const startLine = Math.max(1, targetStartLine - contextLines); const endLine = targetEndLine + contextLines; - const lines = hunkSourceLines(hunkCtx) + const lines = chunkFile.sourceLines .filter((line) => line.line >= startLine && line.line <= endLine) .map((line) => ({ ...line, @@ -259,7 +273,7 @@ export function buildSourceSnippet( return { path: finding.location.path, - language: hunkCtx.language, + language: chunkFile.language, startLine: firstLine.line, endLine: lastLine.line, targetStartLine, @@ -268,20 +282,20 @@ export function buildSourceSnippet( }; } -function attachSourceSnippets(findings: Finding[], hunkCtx: HunkWithContext): Finding[] { +function attachSourceSnippets(findings: Finding[], chunk: ReviewChunk): Finding[] { return findings.map((finding) => { if (!finding.location) return finding; - const sourceSnippet = buildSourceSnippet(finding, hunkCtx); + const sourceSnippet = buildSourceSnippet(finding, chunk); return sourceSnippet ? { ...finding, sourceSnippet } : finding; }); } /** - * Analyze a single hunk with retry logic for transient failures. + * Analyze a single review chunk with retry logic for transient failures. */ -async function analyzeHunk( +async function analyzeReviewChunk( skill: SkillDefinition, - hunkCtx: HunkWithContext, + chunk: ReviewChunk, repoPath: string, options: SkillRunnerOptions, callbacks?: HunkAnalysisCallbacks, @@ -291,15 +305,16 @@ async function analyzeHunk( ensureLocalTracing(); } - const lineRange = callbacks?.lineRange ?? formatHunkLineRange(hunkCtx); + const lineRange = callbacks?.lineRange ?? formatChunkLineRange(chunk); + const primaryFile = chunk.files[0]?.path ?? 'unknown'; return Sentry.startSpan( { op: 'skill.analyze_hunk', - name: `analyze hunk ${hunkCtx.filename}:${lineRange}`, + name: `analyze chunk ${primaryFile}:${lineRange}`, attributes: { 'gen_ai.agent.name': skill.name, - 'code.file.path': hunkCtx.filename, + 'code.file.path': primaryFile, 'warden.hunk.line_range': lineRange, }, }, @@ -309,7 +324,7 @@ async function analyzeHunk( const traceRecorder = options.captureTraces ? startTraceRecorder(span) : undefined; const systemPrompt = buildHunkSystemPrompt(skill); - const userPrompt = buildHunkUserPrompt(skill, hunkCtx, prContext); + const userPrompt = buildReviewChunkUserPrompt(skill, chunk, prContext); // Report prompt size information const systemChars = systemPrompt.length; @@ -345,7 +360,7 @@ async function analyzeHunk( buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: circuitReason.code, @@ -368,7 +383,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: 'aborted', @@ -418,7 +433,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: 'missing_result', @@ -466,7 +481,7 @@ async function analyzeHunk( buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: resultMessage.status, @@ -486,7 +501,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: resultMessage.status, @@ -499,22 +514,26 @@ async function analyzeHunk( options.circuitBreaker?.recordSuccess(); const parseResult = await withTraceRecorder( traceRecorder, - () => parseHunkOutput(resultMessage, hunkCtx.filename, skill.name, options), + () => parseHunkOutput( + resultMessage, + chunk.files.length === 1 ? primaryFile : undefined, + skill.name, + options, + ), ); - // Filter findings outside hunk line range (defense-in-depth) - const hunkRange = getHunkLineRange(hunkCtx.hunk); - const { filtered, dropped } = filterOutOfRangeFindings(parseResult.findings, hunkRange); - const filteredFindings = attachSourceSnippets(filtered, hunkCtx); + // Filter findings outside changed line ranges (defense-in-depth) + const { filtered, dropped } = filterOutOfRangeFindings(parseResult.findings, chunk.changedLineMap); + const filteredFindings = attachSourceSnippets(filtered, chunk); if (dropped.length > 0) { Sentry.addBreadcrumb({ category: 'finding.out_of_range', - message: `Dropped ${dropped.length} finding(s) outside hunk range ${hunkRange.start}-${hunkRange.end}`, + message: `Dropped ${dropped.length} finding(s) outside changed line map`, level: 'warning', data: { skill: skill.name, - filename: hunkCtx.filename, - hunkRange, + filename: primaryFile, + changedLineMap: chunk.changedLineMap, droppedLines: dropped.map((f) => f.location?.startLine), }, }); @@ -560,7 +579,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: resultMessage.status, @@ -584,7 +603,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: 'aborted', @@ -631,7 +650,7 @@ async function analyzeHunk( Sentry.addBreadcrumb({ category: 'retry', - message: `Retrying hunk analysis`, + message: `Retrying review chunk analysis`, data: { attempt: attempt + 1, error: errorMessage, delayMs }, level: 'warning', }); @@ -662,7 +681,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: 'aborted', @@ -706,7 +725,7 @@ async function analyzeHunk( buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: retryCode, @@ -725,7 +744,7 @@ async function analyzeHunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: hunkCtx.filename, + filename: primaryFile, lineRange, runtime: runtimeName, status: retryCode, @@ -737,12 +756,15 @@ async function analyzeHunk( } /** - * Format a hunk's line range as a display string (e.g. "10-20" or "10"). + * Format a review chunk's changed ranges as a display string. */ -function formatHunkLineRange(hunk: HunkWithContext): string { - const start = hunk.hunk.newStart; - const end = start + hunk.hunk.newCount - 1; - return start === end ? `${start}` : `${start}-${end}`; +function formatChunkLineRange(chunk: ReviewChunk): string { + return chunk.changedLineMap + .map((range) => { + const lineRange = range.start === range.end ? `${range.start}` : `${range.start}-${range.end}`; + return chunk.files.length === 1 ? lineRange : `${range.path}:${lineRange}`; + }) + .join(', '); } /** @@ -757,24 +779,25 @@ function attachElapsedTime(findings: Finding[], skillStartTime: number | undefin } /** - * Analyze a single prepared file's hunks. + * Analyze a single prepared file's review chunks. */ export async function analyzeFile( skill: SkillDefinition, - file: PreparedFile, + file: PreparedFile | ReviewChunkGroup, repoPath: string, options: SkillRunnerOptions = {}, callbacks?: FileAnalysisCallbacks, prContext?: PRPromptContext ): Promise { + const group = normalizeReviewChunkGroup(file); return Sentry.startSpan( { op: 'skill.analyze_file', - name: `analyze file ${file.filename}`, + name: `analyze chunk group ${group.displayName}`, attributes: { 'gen_ai.agent.name': skill.name, - 'code.file.path': file.filename, - 'warden.hunk.count': file.hunks.length, + 'code.file.path': group.displayName, + 'warden.hunk.count': group.chunks.length, }, }, async (span) => { @@ -787,11 +810,11 @@ export async function analyzeFile( let failedHunks = 0; let failedExtractions = 0; - for (const [hunkIndex, hunk] of file.hunks.entries()) { + for (const [chunkIndex, chunk] of group.chunks.entries()) { if (abortController?.signal.aborted) break; - const lineRange = formatHunkLineRange(hunk); - callbacks?.onHunkStart?.(hunkIndex + 1, file.hunks.length, lineRange); + const lineRange = formatChunkLineRange(chunk); + callbacks?.onHunkStart?.(chunkIndex + 1, group.chunks.length, lineRange); const hunkCallbacks: HunkAnalysisCallbacks | undefined = callbacks ? { @@ -806,7 +829,7 @@ export async function analyzeFile( : undefined; const hunkStartTime = Date.now(); - const result = await analyzeHunk(skill, hunk, repoPath, options, hunkCallbacks, prContext); + const result = await analyzeReviewChunk(skill, chunk, repoPath, options, hunkCallbacks, prContext); const hunkDurationMs = Date.now() - hunkStartTime; // `failed` and `extractionFailed` are conceptually mutually exclusive: @@ -818,7 +841,7 @@ export async function analyzeFile( failedHunks++; hunkFailures.push({ type: 'analysis', - filename: file.filename, + filename: group.displayName, lineRange, code: result.failureCode ?? 'unknown', message: result.failureMessage ?? 'unknown error', @@ -828,7 +851,7 @@ export async function analyzeFile( failedExtractions++; hunkFailures.push({ type: 'extraction', - filename: file.filename, + filename: group.displayName, lineRange, code: mapExtractionErrorCode(result.extractionError), message: result.extractionError ?? 'unknown extraction error', @@ -837,15 +860,15 @@ export async function analyzeFile( } attachElapsedTime(result.findings, callbacks?.skillStartTime); - callbacks?.onHunkComplete?.(hunkIndex + 1, result.findings, result.usage); + callbacks?.onHunkComplete?.(chunkIndex + 1, result.findings, result.usage); if (result.trace) { hunkTraces.push(result.trace); } const chunkResult: ChunkAnalysisResult = { - filename: file.filename, + filename: group.displayName, model: options.model, - index: hunkIndex + 1, - total: file.hunks.length, + index: chunkIndex + 1, + total: group.chunks.length, lineRange, findings: result.findings, usage: result.usage, @@ -873,7 +896,8 @@ export async function analyzeFile( span.setAttribute('warden.extraction.failed_count', failedExtractions); return { - filename: file.filename, + filename: group.displayName, + filenames: group.filenames, findings: fileFindings, usage: aggregateUsage(fileUsage), failedHunks, @@ -908,7 +932,7 @@ export function generateSummary(skillName: string, findings: Finding[]): string } /** - * Run a skill on a PR, analyzing each hunk separately. + * Run a skill on a PR, analyzing each prepared review chunk separately. */ export async function runSkill( skill: SkillDefinition, @@ -951,14 +975,25 @@ async function runSkillAnalysis( throw new SkillRunnerError('Pull request context required for skill execution'); } - const { files: fileHunks, skippedFiles } = prepareFiles(context, { + const { files: initialPreparedFiles, skippedFiles } = prepareFiles(context, { contextLines: options.contextLines, ignore: options.ignore, scan: options.scan, chunking: options.chunking, }); + const semanticPlan = await planSemanticReviewChunks(initialPreparedFiles, context, { + enabled: options.chunking?.semantic?.enabled, + apiKey: options.apiKey, + runtime: options.runtime, + model: options.auxiliaryModel ?? options.model, + maxRetries: options.auxiliaryMaxRetries, + maxChunks: options.chunking?.semantic?.maxChunks, + maxChunkChars: options.chunking?.semantic?.maxChunkChars, + maxHunksPerChunk: options.chunking?.semantic?.maxHunksPerChunk, + }); + const chunkGroups = semanticPlan.groups; - if (fileHunks.length === 0) { + if (chunkGroups.length === 0) { const report: SkillReport = { skill: skill.name, summary: 'No code changes to analyze', @@ -974,13 +1009,21 @@ async function runSkillAnalysis( return report; } - const totalFiles = fileHunks.length; - const totalHunks = fileHunks.reduce((sum, file) => sum + file.hunks.length, 0); + const totalFiles = chunkGroups.length; + const totalHunks = chunkGroups.reduce((sum, group) => sum + group.chunks.length, 0); const allFindings: Finding[] = []; // Track all usage stats for aggregation const allUsage: UsageStats[] = []; const allAuxiliaryUsage: AuxiliaryUsageEntry[] = []; + if (semanticPlan.usage) { + allAuxiliaryUsage.push({ + agent: 'semantic-chunk-planner', + usage: semanticPlan.usage, + model: options.auxiliaryModel ?? options.model, + runtime: options.runtime, + }); + } const allTraces: HunkTrace[] = []; // Track failed hunks across all files @@ -989,7 +1032,7 @@ async function runSkillAnalysis( // Build PR context for inclusion in prompts (helps LLM understand the full scope of changes) // For non-PR contexts (CLI file/diff mode), skip the "Other Files" list to avoid - // bloating every hunk prompt with thousands of filenames. + // bloating every chunk prompt with thousands of filenames. const isPullRequest = context.pullRequest.number !== 0; const prContext: PRPromptContext = { repository: context.repository.fullName, @@ -1000,14 +1043,14 @@ async function runSkillAnalysis( }; /** - * Process all hunks for a single file sequentially. + * Process all review chunks for a single file sequentially. * Wraps analyzeFile with progress callbacks. */ async function processFile( - fileHunkEntry: PreparedFile, + group: ReviewChunkGroup, fileIndex: number ): Promise { - const { filename } = fileHunkEntry; + const filename = group.displayName; callbacks?.onFileStart?.(filename, fileIndex, totalFiles); @@ -1051,7 +1094,7 @@ async function runSkillAnalysis( : undefined, }; - const result = await analyzeFile(skill, fileHunkEntry, context.repoPath, options, fileCallbacks, prContext); + const result = await analyzeFile(skill, group, context.repoPath, options, fileCallbacks, prContext); callbacks?.onFileComplete?.(filename, fileIndex, totalFiles); @@ -1059,15 +1102,15 @@ async function runSkillAnalysis( } /** Process a file with timing, returning a self-contained result. */ - async function processFileWithTiming(fileHunkEntry: PreparedFile, fileIndex: number) { + async function processFileWithTiming(group: ReviewChunkGroup, fileIndex: number) { const fileStart = Date.now(); - const result = await processFile(fileHunkEntry, fileIndex); + const result = await processFile(group, fileIndex); const durationMs = Date.now() - fileStart; - return { filename: fileHunkEntry.filename, result, durationMs }; + return { group, result, durationMs }; } // Collect results in input order (Promise.all preserves order) - const fileResults: { filename: string; result: FileAnalysisResult; durationMs: number }[] = []; + const fileResults: { group: ReviewChunkGroup; result: FileAnalysisResult; durationMs: number }[] = []; // Process files - parallel or sequential based on options if (parallel) { @@ -1075,23 +1118,23 @@ async function runSkillAnalysis( const fileConcurrency = options.concurrency ?? DEFAULT_FILE_CONCURRENCY; const batchDelayMs = options.batchDelayMs ?? 0; - fileResults.push(...await runPool(fileHunks, fileConcurrency, - async (fileHunkEntry, index) => { + fileResults.push(...await runPool(chunkGroups, fileConcurrency, + async (group, index) => { // Rate-limit: delay items beyond the first concurrent wave if (index >= fileConcurrency && batchDelayMs > 0) { await new Promise((resolve) => setTimeout(resolve, batchDelayMs)); } - return processFileWithTiming(fileHunkEntry, index); + return processFileWithTiming(group, index); }, { shouldAbort: () => abortController?.signal.aborted ?? false } )); } else { // Process files sequentially - for (const [fileIndex, fileHunkEntry] of fileHunks.entries()) { + for (const [fileIndex, group] of chunkGroups.entries()) { // Check for abort before starting new file if (abortController?.signal.aborted) break; - fileResults.push(await processFileWithTiming(fileHunkEntry, fileIndex)); + fileResults.push(await processFileWithTiming(group, fileIndex)); } } @@ -1113,11 +1156,11 @@ async function runSkillAnalysis( } } - // All hunks failed — typically a systemic problem (auth, subprocess, etc). + // All chunks failed — typically a systemic problem (auth, subprocess, etc). // Throw so direct SDK consumers (evals, scheduled workflows) keep their // prior exception-based contract. The CLI path (tasks.ts) has its own // all-hunks-fail detection that emits a structured JSONL record instead. - // Count both analysis and extraction failures: each hunk contributes to + // Count both analysis and extraction failures: each chunk contributes to // at most one (analyzeFile makes them mutually exclusive), and an // extraction-only failure scenario would otherwise slip through silently. const totalAttemptFailures = totalFailedHunks + totalFailedExtractions; @@ -1177,7 +1220,7 @@ async function runSkillAnalysis( // Generate summary const summary = generateSummary(skill.name, finalFindings); - // Aggregate usage across all hunks + // Aggregate usage across all chunks const totalUsage = aggregateUsage(allUsage); const report: SkillReport = { @@ -1188,8 +1231,8 @@ async function runSkillAnalysis( durationMs: Date.now() - startTime, model: options.model, files: buildFileReports( - fileResults.map((fr) => ({ - filename: fr.filename, + fileResults.flatMap((fr) => fileReportInputsFromGroup({ + group: fr.group, durationMs: fr.durationMs, usage: fr.result.usage, })), diff --git a/packages/warden/src/sdk/extract.ts b/packages/warden/src/sdk/extract.ts index b7ef0229..3a3a9144 100644 --- a/packages/warden/src/sdk/extract.ts +++ b/packages/warden/src/sdk/extract.ts @@ -274,7 +274,7 @@ export function generateShortId(): string { * Validate and normalize findings from extracted JSON. * Replaces the LLM-provided ID with a short ID for cross-referencing. */ -export function validateFindings(findings: unknown[], filename: string): Finding[] { +export function validateFindings(findings: unknown[], defaultFilename?: string): Finding[] { const validated: Finding[] = []; for (const f of findings) { @@ -282,20 +282,30 @@ export function validateFindings(findings: unknown[], filename: string): Finding ? { ...(f as Record) } : f; - // Normalize location path before validation + // Normalize missing location paths before validation. Multi-file review + // chunks may provide explicit paths, so preserve them when present. if (typeof candidate === 'object' && candidate !== null && 'location' in candidate) { const loc = (candidate as Record)['location']; if (loc && typeof loc === 'object') { + const location = loc as Record; (candidate as Record)['location'] = { - ...(loc as Record), - path: filename, + ...location, + path: typeof location['path'] === 'string' && location['path'].length > 0 + ? location['path'] + : defaultFilename, }; } } const result = ExtractedFindingSchema.safeParse(candidate); if (result.success) { - const location = result.data.location ? { ...result.data.location, path: filename } : undefined; + const locationPath = result.data.location?.path || defaultFilename; + if (result.data.location && !locationPath) { + continue; + } + const location = result.data.location + ? { ...result.data.location, path: locationPath as string } + : undefined; validated.push({ ...result.data, id: generateShortId(), diff --git a/packages/warden/src/sdk/prepare.test.ts b/packages/warden/src/sdk/prepare.test.ts index 3bb58799..b8957315 100644 --- a/packages/warden/src/sdk/prepare.test.ts +++ b/packages/warden/src/sdk/prepare.test.ts @@ -208,6 +208,68 @@ describe('prepareFiles', () => { ]); }); + it('leaves distant same-file hunks atomic for semantic planning', () => { + const context = makeContext([ + { + filename: 'src/example.ts', + status: 'modified', + patch: [ + '@@ -10,1 +10,1 @@', + '-old10', + '+new10', + '@@ -100,1 +100,1 @@', + '-old100', + '+new100', + '@@ -200,1 +200,1 @@', + '-old200', + '+new200', + ].join('\n'), + }, + ]); + + const result = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + }); + + expect(result.files).toHaveLength(1); + expect(result.files[0]?.chunks).toHaveLength(3); + expect(result.files[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ + [{ path: 'src/example.ts', start: 10, end: 10 }], + [{ path: 'src/example.ts', start: 100, end: 100 }], + [{ path: 'src/example.ts', start: 200, end: 200 }], + ]); + }); + + it('maps only changed added lines, not unchanged hunk context', () => { + const context = makeContext([ + { + filename: 'src/example.ts', + status: 'modified', + patch: [ + '@@ -10,3 +10,3 @@', + ' const before = true;', + '-const value = oldValue;', + '+const value = newValue;', + ' const after = true;', + ].join('\n'), + }, + ]); + + const result = prepareFiles(context); + + expect(result.files[0]?.chunks[0]?.changedLineMap).toEqual([ + { path: 'src/example.ts', start: 11, end: 11 }, + ]); + }); + it('does not skip go.mod as a built-in ignored file', () => { const context = makeContext([ { filename: 'go.mod', patch: '@@ -0,0 +1,1 @@\n+module example.com/app' }, @@ -351,7 +413,10 @@ describe('prepareFiles', () => { const context = buildLocalEventContext({ base: 'HEAD^', head: 'HEAD', cwd: repoPath }); const result = prepareFiles(context, { contextLines: 1 }); - expect(result.files[0]?.hunks[0]?.contextAfter).toEqual(['clean context']); + expect(result.files[0]?.chunks[0]?.files[0]?.sourceLines).toContainEqual({ + line: 8, + content: 'clean context', + }); }, 30_000); it('applies git-ref scan limits to the analyzed commit, not the dirty working tree', () => { @@ -396,6 +461,9 @@ describe('prepareFiles', () => { const context = buildLocalEventContext({ cwd: repoPath, staged: true }); const result = prepareFiles(context, { contextLines: 1 }); - expect(result.files[0]?.hunks[0]?.contextAfter).toEqual(['index context']); + expect(result.files[0]?.chunks[0]?.files[0]?.sourceLines).toContainEqual({ + line: 8, + content: 'index context', + }); }, 30_000); }); diff --git a/packages/warden/src/sdk/prepare.ts b/packages/warden/src/sdk/prepare.ts index 84808b91..296141a8 100644 --- a/packages/warden/src/sdk/prepare.ts +++ b/packages/warden/src/sdk/prepare.ts @@ -5,7 +5,8 @@ import { classifyFile, coalesceHunks, splitLargeHunks, - type HunkWithContext, + reviewChunkFromHunk, + type ReviewChunk, } from '../diff/index.js'; import type { PreparedFile, PrepareFilesOptions, PrepareFilesResult } from './types.js'; import { applyScanPolicy } from './scan-policy.js'; @@ -18,25 +19,27 @@ function matchingChunkingSkipPattern( } /** - * Group hunks by filename into PreparedFile entries. + * Group review chunks by filename into PreparedFile entries. */ -export function groupHunksByFile(hunks: HunkWithContext[]): PreparedFile[] { - const fileMap = new Map(); +export function groupChunksByFile(chunks: ReviewChunk[]): PreparedFile[] { + const fileMap = new Map(); - for (const hunk of hunks) { - const existing = fileMap.get(hunk.filename); + for (const chunk of chunks) { + const filename = chunk.files[0]?.path; + if (!filename) continue; + const existing = fileMap.get(filename); if (existing) { - existing.push(hunk); + existing.push(chunk); } else { - fileMap.set(hunk.filename, [hunk]); + fileMap.set(filename, [chunk]); } } - return Array.from(fileMap, ([filename, fileHunks]) => ({ filename, hunks: fileHunks })); + return Array.from(fileMap, ([filename, chunks]) => ({ filename, chunks })); } /** - * Prepare files for analysis by parsing patches into hunks with context. + * Prepare files for analysis by parsing patches into review chunks with context. * Returns files that have changes to analyze and files that were skipped. */ export function prepareFiles( @@ -50,7 +53,7 @@ export function prepareFiles( } const pr = context.pullRequest; - const allHunks: HunkWithContext[] = []; + const allChunks: ReviewChunk[] = []; const skippedFiles: SkippedFile[] = []; const scanPolicy = applyScanPolicy(pr.files, { @@ -109,11 +112,12 @@ export function prepareFiles( contextLines, contentSource: context.diffContextSource, }); - allHunks.push(...hunksWithContext); + const rawChunks = hunksWithContext.map(reviewChunkFromHunk); + allChunks.push(...rawChunks); } return { - files: groupHunksByFile(allHunks), + files: groupChunksByFile(allChunks), skippedFiles, }; } diff --git a/packages/warden/src/sdk/prompt.ts b/packages/warden/src/sdk/prompt.ts index 8c0e92b4..258801e9 100644 --- a/packages/warden/src/sdk/prompt.ts +++ b/packages/warden/src/sdk/prompt.ts @@ -1,7 +1,7 @@ import { existsSync } from 'node:fs'; import { join } from 'node:path'; import type { SkillDefinition } from '../config/schema.js'; -import { formatHunkForAnalysis, type HunkWithContext } from '../diff/index.js'; +import type { ReviewChunk } from '../diff/index.js'; import { buildChangedFilesSection, buildJsonOutputSection, @@ -12,11 +12,43 @@ import { export type PRPromptContext = PromptPRContext; +function formatChangedLineMap(chunk: ReviewChunk): string { + return chunk.changedLineMap + .map((range) => `- ${range.path}:${range.start}-${range.end}`) + .join('\n'); +} + +/** + * Format a review chunk for LLM analysis. + */ +export function formatReviewChunkForAnalysis(chunk: ReviewChunk): string { + const lines: string[] = []; + + lines.push(`## Review Chunk: ${chunk.title}`); + if (chunk.summary) { + lines.push(`## Semantic Summary: ${chunk.summary}`); + } + lines.push(''); + lines.push('## Changed Line Map'); + lines.push(formatChangedLineMap(chunk)); + + for (const file of chunk.files) { + lines.push(''); + lines.push(`## File: ${file.path}`); + lines.push(`## Language: ${file.language}`); + lines.push(`## Content Mode: ${file.contentMode}`); + lines.push(''); + lines.push(file.content); + } + + return lines.join('\n'); +} + /** - * Builds the system prompt for hunk-based analysis. + * Builds the system prompt for review chunk analysis. * * Future enhancement: Could have the agent output a structured `contextAssessment` - * (applicationType, trustBoundaries, filesChecked) to cache across hunks, allow + * (applicationType, trustBoundaries, filesChecked) to cache across chunks, allow * user overrides, or build analytics. Not implemented since we don't consume it yet. */ export function buildHunkSystemPrompt(skill: SkillDefinition): string { @@ -30,7 +62,7 @@ Before reporting a finding: 1. Read the relevant source code to understand the full context 2. Trace through the code path — follow imports, base classes, and indirect references, not just the immediate file 3. Verify your assumptions — confirm the issue exists, don't infer from incomplete information -4. Ensure the finding references lines within the hunk being analyzed +4. Ensure each finding's location starts on a line listed in the review chunk's changed-line map 5. Document the evidence trace in the 'verification' field of each finding `, @@ -66,15 +98,15 @@ Full schema: Requirements: - Return valid JSON starting with {"findings": - "findings" array can be empty if no issues found -- "location.path" is auto-filled from context - just provide startLine (and optionally endLine). Omit location entirely for general findings not about a specific line. -- "location.startLine" MUST be within the hunk line range (shown in the "## Hunk" header). If the issue originates in surrounding code, anchor to the nearest changed line in the hunk and note the actual location in the description. +- "location.path" must be one of the files in the changed line map. For single-file chunks, use that file path. +- "location.startLine" MUST be within one of the changed line map ranges. If the issue originates in surrounding code, anchor to the nearest changed line in the changed line map and note the actual location in the description. - "confidence" reflects how certain you are this is a real issue given the codebase context - "description" is rendered directly in GitHub inline comments. Keep it brief and actionable, usually one sentence. - Put the concrete evidence trace in "verification", not "description". - Write "verification" as evidence, not reasoning: facts from the code path, guards, conditions, and observed behavior that make the finding believable. - Do not format "verification" as any labeled checklist or template. - Do not include severity, confidence, finding ID, skill name, or generic review framing in "description". -- Focus your analysis on the code changes in the hunk. Surrounding context and tool results are for understanding only -- all findings must reference lines within the hunk range. +- Focus your analysis on the code changes in the review chunk. Surrounding context and tool results are for understanding only -- all findings must reference lines within the changed line map. `), ]; @@ -96,22 +128,26 @@ You can read files from ${dirList} subdirectories using the Read tool with the f } /** - * Builds the user prompt for a single hunk. + * Builds the user prompt for a semantic review chunk. */ -export function buildHunkUserPrompt( +export function buildReviewChunkUserPrompt( skill: SkillDefinition, - hunkCtx: HunkWithContext, + chunk: ReviewChunk, prContext?: PRPromptContext ): string { + const currentFile = chunk.files.length === 1 ? chunk.files[0]?.path : undefined; return joinPromptSections([ ` -Analyze this code change according to the "${skill.name}" skill criteria. +Analyze this semantic review chunk according to the "${skill.name}" skill criteria. `, buildPullRequestContextSection(prContext), - buildChangedFilesSection(prContext, hunkCtx.filename), - formatHunkForAnalysis(hunkCtx), + buildChangedFilesSection(prContext, currentFile), + formatReviewChunkForAnalysis(chunk), ` -Only report findings that are explicitly covered by the skill instructions. Do not report general code quality issues, bugs, or improvements unless the skill specifically asks for them. Return an empty findings array if no issues match the skill's criteria. +Only report findings that are explicitly covered by the skill instructions. Do not report general code quality issues, bugs, or improvements unless the skill specifically asks for them. Locations must be inside the changed line map. Return an empty findings array if no issues match the skill's criteria. `, ]); } + +/** Legacy alias for callers that still use the old hunk prompt name. */ +export const buildHunkUserPrompt = buildReviewChunkUserPrompt; diff --git a/packages/warden/src/sdk/runner.test.ts b/packages/warden/src/sdk/runner.test.ts index 17de9dd2..9af27ea6 100644 --- a/packages/warden/src/sdk/runner.test.ts +++ b/packages/warden/src/sdk/runner.test.ts @@ -944,7 +944,7 @@ describe('validateFindings', () => { expect(validated[0]!.id).not.toBe(validated[1]!.id); }); - it('normalizes location path to the provided filename', () => { + it('preserves explicit location paths for multi-file chunks', () => { const rawFindings = [ { id: 'id-1', @@ -955,6 +955,21 @@ describe('validateFindings', () => { }, ]; + const validated = validateFindings(rawFindings, 'correct-path.ts'); + expect(validated[0]!.location!.path).toBe('wrong-path.ts'); + }); + + it('defaults missing location paths to the provided filename', () => { + const rawFindings = [ + { + id: 'id-1', + severity: 'medium', + title: 'Issue', + description: 'Details', + location: { startLine: 5 }, + }, + ]; + const validated = validateFindings(rawFindings, 'correct-path.ts'); expect(validated[0]!.location!.path).toBe('correct-path.ts'); }); diff --git a/packages/warden/src/sdk/runner.ts b/packages/warden/src/sdk/runner.ts index 22b0dac0..f1d4ba5a 100644 --- a/packages/warden/src/sdk/runner.ts +++ b/packages/warden/src/sdk/runner.ts @@ -29,7 +29,7 @@ export { anthropicUsageToStats, apiUsageToStats } from './pricing.js'; export type { AnthropicApiUsage } from './pricing.js'; // Re-export prompt building (with legacy alias) -export { buildHunkSystemPrompt, buildHunkUserPrompt } from './prompt.js'; +export { buildHunkSystemPrompt, buildHunkUserPrompt, buildReviewChunkUserPrompt, formatReviewChunkForAnalysis } from './prompt.js'; export type { PRPromptContext } from './prompt.js'; // Legacy export for backwards compatibility export { buildHunkSystemPrompt as buildSystemPrompt } from './prompt.js'; @@ -57,6 +57,7 @@ export type { // Re-export file preparation export { prepareFiles } from './prepare.js'; +export { planSemanticReviewChunks } from './semantic-chunk-planner.js'; // Re-export verification utilities export { verifyFindings } from './verify.js'; @@ -105,6 +106,7 @@ export type { SkillRunnerCallbacks, SkillRunnerOptions, PreparedFile, + ReviewChunkGroup, PrepareFilesOptions, PrepareFilesResult, FileAnalysisCallbacks, diff --git a/packages/warden/src/sdk/runtimes/types.ts b/packages/warden/src/sdk/runtimes/types.ts index e1a16eb5..0bbedad8 100644 --- a/packages/warden/src/sdk/runtimes/types.ts +++ b/packages/warden/src/sdk/runtimes/types.ts @@ -89,7 +89,9 @@ export interface AuxiliaryTool { export type AuxiliaryTask = | 'extraction' | 'deduplication' - | 'fix_evaluation'; + | 'fix_evaluation' + | 'semantic_chunking' + | 'eval_judge'; export type SynthesisTask = | 'consolidation' diff --git a/packages/warden/src/sdk/semantic-chunk-planner.test.ts b/packages/warden/src/sdk/semantic-chunk-planner.test.ts new file mode 100644 index 00000000..cb1da3d0 --- /dev/null +++ b/packages/warden/src/sdk/semantic-chunk-planner.test.ts @@ -0,0 +1,246 @@ +import { describe, expect, it, vi } from 'vitest'; +import type { EventContext } from '../types/index.js'; +import { prepareFiles } from './prepare.js'; +import { planSemanticReviewChunks } from './semantic-chunk-planner.js'; +import type { Runtime } from './runtimes/index.js'; +import type * as RuntimeModule from './runtimes/index.js'; + +const runAuxiliary = vi.fn(); + +vi.mock('./runtimes/index.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + getRuntime: vi.fn(() => ({ + name: 'pi', + runSkill: vi.fn(), + runAuxiliary, + runSynthesis: vi.fn(), + } satisfies Runtime)), + }; +}); + +function makeContext(): EventContext { + return { + eventType: 'pull_request', + action: 'opened', + repository: { + owner: 'qa', + name: 'repo', + fullName: 'qa/repo', + defaultBranch: 'main', + }, + repoPath: '/tmp/warden-semantic-planner-test', + pullRequest: { + number: 1, + title: 'Preserve user-selected dashboard axis range', + body: '', + author: 'qa', + baseBranch: 'main', + headBranch: 'feature', + headSha: 'head', + baseSha: 'base', + files: [{ + filename: 'src/dashboard.ts', + status: 'modified', + additions: 3, + deletions: 3, + chunks: 3, + patch: [ + '@@ -10,1 +10,1 @@', + '-const range = getDefaultRange(widget);', + '+const range = widget.axisRange ?? getDefaultRange(widget);', + '@@ -100,1 +100,1 @@', + '-return buildChart(series);', + '+return buildChart(series, range);', + '@@ -200,1 +200,1 @@', + '-expect(chart.range).toEqual(defaultRange);', + '+expect(chart.range).toEqual(customAxisRange);', + ].join('\n'), + }], + }, + }; +} + +describe('planSemanticReviewChunks', () => { + it('groups atomic chunks using a model-provided semantic delta', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: [ + 'src/dashboard.ts:10', + 'src/dashboard.ts:100', + 'src/dashboard.ts:200', + ], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + }); + expect(prepared.files[0]?.chunks).toHaveLength(3); + + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + model: 'anthropic/claude-sonnet-4-6', + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + }); + + expect(planned.groups).toHaveLength(1); + expect(planned.groups[0]?.chunks).toHaveLength(1); + expect(planned.groups[0]?.chunks[0]).toMatchObject({ + id: 'semantic:1', + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + changedLineMap: [ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'src/dashboard.ts', start: 100, end: 100 }, + { path: 'src/dashboard.ts', start: 200, end: 200 }, + ], + }); + expect(runAuxiliary).toHaveBeenCalledWith(expect.objectContaining({ + task: 'semantic_chunking', + agentName: 'semantic-chunk-planner', + })); + const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; + expect(prompt).toContain('Group atomic git chunks into semantic review chunks.'); + expect(prompt).toContain('Do not restate filenames or line ranges.'); + }); + + it('rejects planned groups that exceed maxChunkChars after materialization', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100'], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context); + + await expect(planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxChunkChars: 10, + })).rejects.toThrow('exceeding maxChunkChars 10'); + }); + + it('materializes semantic groups across multiple files', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry widget axis ranges through rendering and assert that behavior in tests.', + chunkIds: [ + 'src/dashboard.ts:10', + 'tests/dashboard.test.ts:20', + ], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + context.pullRequest!.files = [ + { + filename: 'src/dashboard.ts', + status: 'modified', + additions: 1, + deletions: 1, + chunks: 1, + patch: [ + '@@ -10,1 +10,1 @@', + '-const range = getDefaultRange(widget);', + '+const range = widget.axisRange ?? getDefaultRange(widget);', + ].join('\n'), + }, + { + filename: 'tests/dashboard.test.ts', + status: 'modified', + additions: 1, + deletions: 1, + chunks: 1, + patch: [ + '@@ -20,1 +20,1 @@', + '-expect(chart.range).toEqual(defaultRange);', + '+expect(chart.range).toEqual(customAxisRange);', + ].join('\n'), + }, + ]; + + const prepared = prepareFiles(context); + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + }); + + const chunk = planned.groups[0]?.chunks[0]; + expect(planned.groups).toHaveLength(1); + expect(chunk).toMatchObject({ + id: 'semantic:1', + title: 'Preserve dashboard axis range', + changedLineMap: [ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'tests/dashboard.test.ts', start: 20, end: 20 }, + ], + }); + expect(chunk?.files).toHaveLength(2); + expect(chunk?.files.map((file) => file.path)).toEqual([ + 'src/dashboard.ts', + 'tests/dashboard.test.ts', + ]); + }); +}); diff --git a/packages/warden/src/sdk/semantic-chunk-planner.ts b/packages/warden/src/sdk/semantic-chunk-planner.ts new file mode 100644 index 00000000..57ad7c48 --- /dev/null +++ b/packages/warden/src/sdk/semantic-chunk-planner.ts @@ -0,0 +1,253 @@ +import { z } from 'zod'; +import type { EventContext, UsageStats } from '../types/index.js'; +import type { ReviewChunk } from '../diff/index.js'; +import { SkillRunnerError } from './errors.js'; +import { getRuntime } from './runtimes/index.js'; +import type { RuntimeName } from './runtimes/index.js'; +import type { PreparedFile, ReviewChunkGroup } from './types.js'; + +const SemanticChunkPlanSchema = z.object({ + groups: z.array(z.object({ + title: z.string().min(5), + summary: z.string().min(20), + chunkIds: z.array(z.string().min(1)).min(1), + })), +}); + +export interface SemanticChunkPlanningOptions { + enabled?: boolean; + apiKey?: string; + runtime?: RuntimeName; + model?: string; + maxRetries?: number; + maxChunks?: number; + maxChunkChars?: number; + maxHunksPerChunk?: number; +} + +export interface SemanticChunkPlanningResult { + groups: ReviewChunkGroup[]; + usage?: UsageStats; +} + +function formatChangedRanges(chunk: ReviewChunk): string { + return chunk.changedLineMap + .map((range) => `${range.path}:${range.start}-${range.end}`) + .join(', '); +} + +function formatChunkForPlanning(chunk: ReviewChunk): string { + const fileSummaries = chunk.files.map((file) => [ + `File: ${file.path}`, + `Language: ${file.language}`, + `Content mode: ${file.contentMode}`, + 'Content:', + file.content, + ].join('\n')).join('\n\n'); + + return [ + `Chunk ID: ${chunk.id}`, + `Title: ${chunk.title}`, + `Changed ranges: ${formatChangedRanges(chunk)}`, + fileSummaries, + ].join('\n'); +} + +function buildSemanticChunkPlanningPrompt( + context: EventContext, + chunks: ReviewChunk[], + options: SemanticChunkPlanningOptions +): string { + const pr = context.pullRequest; + const maxChunks = options.maxChunks ?? 20; + const maxHunksPerChunk = options.maxHunksPerChunk ?? 50; + const maxChunkChars = options.maxChunkChars ?? 30000; + const header = [ + 'You are planning code review chunks for Warden.', + '', + 'Group atomic git chunks into semantic review chunks.', + 'A semantic chunk should contain changes that a reviewer should understand together: one behavior change, API contract, data flow, migration, validation rule, or test expectation.', + 'For each planned group, write a semantic delta summary: what behavior, contract, data flow, API, or test expectation changed.', + 'Do not restate filenames or line ranges. Do not say "lines 10, 100, 200".', + 'Keep each summary one sentence. Be concrete enough that a scanner knows why the grouped changes belong together.', + `Target at most ${maxChunks} planned groups.`, + `Use at most ${maxHunksPerChunk} atomic chunks per group.`, + `Keep each planned group under roughly ${maxChunkChars} characters of provided content.`, + 'Every input Chunk ID must appear in exactly one group. Do not invent Chunk IDs.', + '', + `Repository: ${context.repository.fullName}`, + pr ? `Pull request title: ${pr.title}` : undefined, + pr?.body ? `Pull request body: ${pr.body}` : undefined, + ].filter((line): line is string => Boolean(line)); + + return [ + ...header, + '', + 'Return JSON with this exact shape:', + '{"groups":[{"title":"semantic title","summary":"semantic delta summary","chunkIds":["chunk id"]}]}', + '', + 'Atomic chunks:', + chunks.map(formatChunkForPlanning).join('\n\n---\n\n'), + ].join('\n'); +} + +function mergeSourceLines(chunks: ReviewChunk[], path: string) { + const byLine = new Map(); + for (const chunk of chunks) { + for (const file of chunk.files.filter((file) => file.path === path)) { + for (const line of file.sourceLines) { + byLine.set(line.line, line.content); + } + } + } + + return Array.from(byLine, ([line, content]) => ({ line, content })) + .sort((a, b) => a.line - b.line); +} + +function materializePlannedChunk( + index: number, + title: string, + summary: string, + chunks: ReviewChunk[] +): ReviewChunk { + const changedLineMap = chunks.flatMap((chunk) => chunk.changedLineMap); + const paths = [...new Set(chunks.flatMap((chunk) => chunk.files.map((file) => file.path)))]; + const files = paths.map((path) => { + const chunkFiles = chunks.flatMap((chunk) => chunk.files.filter((file) => file.path === path)); + const first = chunkFiles[0]; + if (!first) { + throw new Error(`Cannot materialize planned chunk without file content for ${path}`); + } + + return { + path, + language: first.language, + changedRanges: changedLineMap.filter((range) => range.path === path), + content: chunkFiles.map((file) => file.content).join('\n\n---\n\n'), + contentMode: 'raw-hunks' as const, + sourceLines: mergeSourceLines(chunks, path), + }; + }); + + return { + id: `semantic:${index + 1}`, + title, + summary, + files, + changedLineMap, + }; +} + +function groupFromPreparedFile(file: PreparedFile): ReviewChunkGroup { + const filenames = [...new Set(file.chunks.flatMap((chunk) => chunk.files.map((chunkFile) => chunkFile.path)))]; + return { + displayName: file.filename, + filenames: filenames.length > 0 ? filenames : [file.filename], + chunks: file.chunks, + }; +} + +function groupFromPlannedChunk(chunk: ReviewChunk): ReviewChunkGroup { + const filenames = [...new Set(chunk.files.map((file) => file.path))]; + const [firstFilename] = filenames; + return { + displayName: filenames.length <= 1 + ? firstFilename ?? chunk.title + : `${firstFilename ?? chunk.title} + ${filenames.length - 1} file${filenames.length === 2 ? '' : 's'}`, + filenames, + chunks: [chunk], + }; +} + +/** + * Populate ReviewChunk semantic summaries with a model-backed planning pass. + */ +export async function planSemanticReviewChunks( + files: PreparedFile[], + context: EventContext, + options: SemanticChunkPlanningOptions = {} +): Promise { + const chunks = files.flatMap((file) => file.chunks); + if (!options.enabled || chunks.length === 0) { + return { groups: files.map(groupFromPreparedFile) }; + } + + const runtimeName = options.runtime ?? 'pi'; + const result = await getRuntime(runtimeName).runAuxiliary({ + task: 'semantic_chunking', + agentName: 'semantic-chunk-planner', + apiKey: options.apiKey, + prompt: buildSemanticChunkPlanningPrompt(context, chunks, options), + schema: SemanticChunkPlanSchema, + model: options.model, + maxRetries: options.maxRetries, + }); + + if (!result.success) { + throw new SkillRunnerError(`Semantic chunk planning failed: ${result.error}`, { + code: 'sdk_error', + }); + } + + const chunksById = new Map(chunks.map((chunk) => [chunk.id, chunk])); + const seen = new Set(); + const plannedChunks: ReviewChunk[] = []; + const maxChunks = options.maxChunks ?? 20; + const maxHunksPerChunk = options.maxHunksPerChunk ?? 50; + const maxChunkChars = options.maxChunkChars ?? 30000; + + if (result.data.groups.length > maxChunks) { + throw new SkillRunnerError( + `Semantic chunk planning returned ${result.data.groups.length} groups, exceeding maxChunks ${maxChunks}`, + { code: 'sdk_error' }, + ); + } + + for (const [index, group] of result.data.groups.entries()) { + if (group.chunkIds.length > maxHunksPerChunk) { + throw new SkillRunnerError( + `Semantic chunk planning returned ${group.chunkIds.length} chunks in one group, exceeding maxHunksPerChunk ${maxHunksPerChunk}`, + { code: 'sdk_error' }, + ); + } + const groupChunks: ReviewChunk[] = []; + for (const chunkId of group.chunkIds) { + if (seen.has(chunkId)) { + throw new SkillRunnerError(`Semantic chunk planning assigned chunk ${chunkId} more than once`, { + code: 'sdk_error', + }); + } + const chunk = chunksById.get(chunkId); + if (!chunk) { + throw new SkillRunnerError(`Semantic chunk planning returned unknown chunk ${chunkId}`, { + code: 'sdk_error', + }); + } + seen.add(chunkId); + groupChunks.push(chunk); + } + const plannedChunk = materializePlannedChunk(index, group.title, group.summary, groupChunks); + const plannedChunkChars = plannedChunk.files.reduce((sum, file) => sum + file.content.length, 0); + if (plannedChunkChars > maxChunkChars) { + throw new SkillRunnerError( + `Semantic chunk planning returned a ${plannedChunkChars} character group, exceeding maxChunkChars ${maxChunkChars}`, + { code: 'sdk_error' }, + ); + } + plannedChunks.push(plannedChunk); + } + + const missing = chunks.filter((chunk) => !seen.has(chunk.id)); + if (missing.length > 0) { + throw new SkillRunnerError( + `Semantic chunk planning omitted ${missing.length} chunk${missing.length === 1 ? '' : 's'}`, + { code: 'sdk_error' }, + ); + } + + return { + groups: plannedChunks.map(groupFromPlannedChunk), + usage: result.usage, + }; +} diff --git a/packages/warden/src/sdk/types.ts b/packages/warden/src/sdk/types.ts index 7c773562..4d7a2dee 100644 --- a/packages/warden/src/sdk/types.ts +++ b/packages/warden/src/sdk/types.ts @@ -1,5 +1,5 @@ import type { Finding, UsageStats, SkippedFile, RetryConfig, ErrorCode, HunkFailure, HunkTrace } from '../types/index.js'; -import type { HunkWithContext } from '../diff/index.js'; +import type { ReviewChunk } from '../diff/index.js'; import type { ChunkingConfig, Effort, IgnoreConfig, ScanConfig } from '../config/schema.js'; import type { RuntimeName } from './runtimes/index.js'; import type { ProviderFailureCircuitBreaker } from './circuit-breaker.js'; @@ -146,18 +146,27 @@ export interface SkillRunnerOptions { postProcessFindings?: boolean; /** Trigger name to attach to skill-level telemetry when the caller has one. */ telemetryTriggerName?: string; - /** Capture per-hunk runtime traces in structured run output. Defaults to false. */ + /** Capture per-chunk runtime traces in structured run output. Defaults to false. */ captureTraces?: boolean; } -export type AnalysisChunkingConfig = Pick; +export type AnalysisChunkingConfig = Pick; /** - * A file prepared for analysis with its hunks. + * A file prepared for analysis with its review chunks. */ export interface PreparedFile { filename: string; - hunks: HunkWithContext[]; + chunks: ReviewChunk[]; +} + +/** + * One scanner-facing group of review chunks. + */ +export interface ReviewChunkGroup { + displayName: string; + filenames: string[]; + chunks: ReviewChunk[]; } /** @@ -211,6 +220,7 @@ export interface FileAnalysisCallbacks { */ export interface FileAnalysisResult { filename: string; + filenames?: string[]; findings: Finding[]; usage: UsageStats; /** Number of hunks that failed to analyze */ diff --git a/skills/warden/references/config-schema.md b/skills/warden/references/config-schema.md index dfb829c7..d45fdc73 100644 --- a/skills/warden/references/config-schema.md +++ b/skills/warden/references/config-schema.md @@ -56,14 +56,18 @@ maxRetries = 5 # Retries for auxiliary structured calls [defaults.synthesis] model = "anthropic/claude-opus-4-5" # Consolidation and generated-skill build model -[defaults.chunking] -enabled = true # Enable hunk-based chunking - [defaults.chunking.coalesce] enabled = true # Merge nearby hunks maxGapLines = 30 # Lines between hunks to merge maxChunkSize = 8000 # Max chars per chunk +[defaults.chunking.semantic] +enabled = true # Group small related review chunks +maxChunks = 20 # Target max review chunks after grouping +maxChunkChars = 30000 # Max chars per grouped review chunk +maxHunksPerChunk = 50 # Max raw hunks per grouped review chunk +preferWholeFileBelowLines = 800 # Reserved for whole-file materialization + [[defaults.chunking.filePatterns]] pattern = "*.config.*" # Glob pattern mode = "whole-file" # per-hunk | whole-file | skip diff --git a/skills/warden/references/configuration.md b/skills/warden/references/configuration.md index 2ddb81b6..8ec1d829 100644 --- a/skills/warden/references/configuration.md +++ b/skills/warden/references/configuration.md @@ -84,6 +84,16 @@ pattern = "*.config.*" mode = "whole-file" ``` +**Group small review chunks:** +```toml +[defaults.chunking.semantic] +enabled = true +maxChunks = 20 +maxChunkChars = 30000 +maxHunksPerChunk = 50 +preferWholeFileBelowLines = 800 +``` + ## Model Lanes Warden uses different model lanes for different kinds of work: diff --git a/specs/semantic-review-chunks.md b/specs/semantic-review-chunks.md new file mode 100644 index 00000000..653204ac --- /dev/null +++ b/specs/semantic-review-chunks.md @@ -0,0 +1,283 @@ +# Semantic Review Chunks + +Warden currently prepares code for review from git hunks. Git hunks are useful +for changed-line anchoring, but they are a poor unit of review. A single logical +change can produce dozens of tiny hunks, especially in tests, generated catalogs, +or repeated call-site updates. Reviewing each tiny hunk independently repeats +prompt setup, repeats codebase exploration, increases cost, and can hide the +shape of the actual change. + +Semantic review chunks make the scanner read a coherent change unit while +Warden still uses git hunks as the source of truth for where inline comments may +land. + +## Current Behavior + +The current pipeline is: + +```text +git patch + -> parse file diffs into hunks + -> split large hunks + -> coalesce nearby hunks by line distance and size + -> expand context around each hunk + -> run one scanner call per hunk-like chunk +``` + +This is simple and safe, but it only fixes nearby fragmentation. If a file has +50 small hunks spread across distant test cases, Warden still makes many scanner +calls for one logical change. + +## Desired Outcome + +The scanner should receive a larger review packet when that better matches how a +human would review the change. + +Examples: + +- one test file with many small related changes becomes one review chunk with + the whole file or a stitched excerpt +- one implementation change plus related tests becomes one semantic chunk when + both sides are needed to understand the behavior +- one behavior change touching several files remains one semantic chunk with + multiple file payloads under the same summary +- unrelated edits in the same file stay separate chunks +- very large or generated files stay governed by scan policy and existing skip + behavior + +The target pipeline is: + +```text +git patch + -> atomic hunk inventory + -> semantic chunk planner + -> ReviewChunk materialization + -> run one scanner call per ReviewChunk + -> validate findings against changedLineMap +``` + +Git hunks remain the evidence and anchoring primitive. Review chunks become the +scanner-facing primitive. + +## ReviewChunk Contract + +```ts +export interface ReviewChunk { + id: string; + title: string; + summary?: string; + files: ReviewChunkFile[]; + changedLineMap: ChangedLineRange[]; +} + +export interface ReviewChunkFile { + path: string; + changedRanges: ChangedLineRange[]; + content: string; + contentMode: 'whole-file' | 'stitched-file' | 'raw-hunks'; +} + +export interface ChangedLineRange { + path: string; + start: number; + end: number; +} +``` + +`title` is a stable label for progress, logging, and trace output. Deterministic +chunking may use filenames and changed ranges. + +`summary` is optional and planner-owned. It must only be set when a semantic +planner has described the logical change. Deterministic grouping must not +populate `summary` with filenames or changed ranges and call that semantic. + +`files[].content` is the readable review packet. It can be larger than a hunk +and may include unchanged surrounding code. This content is for understanding. +One `ReviewChunk` may include multiple files when the same logical change spans +implementation, tests, config, or call sites. The chunk title and summary +describe the shared change; each `files[]` entry carries the file-local content +needed to review that change. + +`changedLineMap` is the hard validation boundary. A scanner finding may only +anchor to a line inside this map. Surrounding content can explain a finding but +cannot be used as the comment location. + +## Atomic Hunk Inventory + +Before planning, Warden should normalize parsed hunks into stable atomic units: + +```ts +export interface AtomicHunk { + id: string; + path: string; + oldStart: number; + oldEnd: number; + newStart: number; + newEnd: number; + header?: string; + changedLines: string[]; + excerpt: string; +} +``` + +The planner operates on this inventory. It does not invent changed lines. Each +atomic hunk is either assigned to exactly one review chunk or excluded by an +existing scan/ignore policy before planning. + +## Planner + +The semantic chunk planner groups atomic hunks into review chunks and selects a +content mode for each file in each chunk. + +Planner input: + +- repository and PR metadata +- PR title and body +- changed file list +- atomic hunk inventory with compact excerpts +- file sizes and line counts where available +- hard limits for chunk size and count + +Planner output: + +```ts +export interface PlannedReviewChunk { + id: string; + title: string; + summary: string; + hunkIds: string[]; + files: Array<{ + path: string; + contentMode: 'whole-file' | 'stitched-file' | 'raw-hunks'; + }>; +} +``` + +The planner should optimize for scanner usefulness, not for diff prettiness. +Good chunks are cohesive enough that the scanner can understand the change in +one pass and small enough that the scanner can stay precise. +Cross-file groups are allowed and expected when the files are part of the same +semantic delta. A source change and its test assertion should usually stay +together if reviewing them independently would hide the behavior change. + +## Materialization + +Materialization turns a planned chunk into the scanner-facing `ReviewChunk`. +For cross-file groups, materialization keeps one shared `ReviewChunk` and +creates one `ReviewChunkFile` per path. It must not flatten file contents into a +single synthetic file, because extraction and changed-line validation depend on +real paths. + +Content modes: + +| Mode | Use When | Content | +|------|----------|---------| +| `whole-file` | File is below configured line/byte limits and many small hunks are spread across it | Current file contents | +| `stitched-file` | File is too large for whole-file but related hunks need broad structure | Ordered excerpts around changed ranges, with omitted sections marked | +| `raw-hunks` | Planning is unnecessary or the change is already compact | Existing formatted hunks with context | + +For test files with many tiny hunks, `whole-file` should usually be preferred +when file limits allow it. Tests often need the surrounding `describe`/`it` +structure to make the change reviewable. + +## Finding Validation + +Scanner prompts must say that findings can only anchor to changed lines in the +review chunk's changed-line map. + +Warden must also enforce that rule after extraction: + +- a finding with no location may remain a general finding +- a finding with a location is accepted only if `location.path` and + `location.startLine` fall inside `changedLineMap` +- multi-line findings must be fully contained by a changed range +- out-of-range findings are dropped and recorded in telemetry + +This replaces the current single hunk-range check with a multi-range check. + +## Configuration + +Semantic review chunks should be configured with a small public surface: + +```toml +[defaults.chunking.semantic] +enabled = true +maxChunks = 20 +maxChunkChars = 30000 +maxHunksPerChunk = 50 +preferWholeFileBelowLines = 800 +``` + +Do not expose fallback behavior as config. If semantic planning fails mechanical +validation or cannot run, any recovery behavior should remain internal. Users +should not need to choose recovery strategy. + +## Validation Rules + +Planner output must pass deterministic validation before scanner execution: + +- every planned `hunkId` exists +- no hunk is assigned to more than one chunk +- every included hunk is assigned to a chunk +- every planned path exists in the changed file set +- each chunk respects hard size and hunk-count limits after materialization +- each `changedLineMap` range comes from assigned atomic hunks +- chunk ids are stable within the run and unique + +Invalid plans are not partially trusted. + +## Prompt Changes + +The scanner task should move from hunk-specific language to chunk-specific +language: + +```text +Analyze this review chunk according to the skill criteria. + +If a semantic summary is present, use it as planner context for why these +changed ranges are being reviewed together. File content may include unchanged +surrounding code for context. Only report findings covered by the skill +instructions, and only anchor locations to lines listed in the changed-line map. +``` + +The JSON output schema can stay mostly unchanged. The location rules need to +reference changed-line maps instead of a single hunk range. + +## Telemetry + +Warden should record enough data to prove whether semantic chunking helps: + +- original atomic hunk count +- planned review chunk count +- materialized chunk count +- content mode counts +- planner duration and usage +- scanner duration and usage +- internal recovery reason when semantic chunking is not used +- number of dropped out-of-range findings + +This should make cost and latency changes visible without reading logs line by +line. + +## Rollout + +1. Add `AtomicHunk`, `ReviewChunk`, and multi-range finding validation. +2. Adapt the existing hunk/coalescing flow to emit `ReviewChunk` values using + `raw-hunks`. +3. Update scanner prompt construction to accept `ReviewChunk`. +4. Add the model-backed semantic planner behind `chunking.semantic.enabled`. + Planner output is the first place `ReviewChunk.summary` should be populated. +5. Add materializers for `whole-file`, `stitched-file`, and `raw-hunks`. +6. Add telemetry for chunk counts, cost, duration, and planner failures. +7. Add regression fixtures for tiny-hunk pathological changes, including the + `getsentry/sentry-mcp` style case from Warden issue 313. +8. Enable selectively, compare eval recall and cost, then consider making it + default. + +## Non-Goals + +- replacing git diff parsing +- letting the planner decide where comments may land +- exposing planner recovery strategy as user config +- building a full AST differ +- reviewing unchanged lines as primary finding locations diff --git a/warden.toml b/warden.toml index 3e40e43c..51d3fafc 100644 --- a/warden.toml +++ b/warden.toml @@ -2,7 +2,7 @@ version = 1 [defaults] runtime = "pi" -model = "anthropic/claude-sonnet-4-6" +model = "openrouter/anthropic/claude-sonnet-4.6" # Fail check on high+ severity findings (critical, high) failOn = "high" # Show annotations for medium+ severity findings @@ -10,6 +10,13 @@ reportOn = "medium" # Exclude build output and internal eval fixtures from all skills ignorePaths = ["dist/**", "packages/evals/**"] +[defaults.chunking.semantic] +enabled = true +maxChunks = 20 +maxChunkChars = 30000 +maxHunksPerChunk = 50 +preferWholeFileBelowLines = 800 + [[skills]] name = "security-review" paths = [ From 64feb2ad5e2822c9ef9b81b48c8eca1b2c3052c9 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Mon, 29 Jun 2026 20:32:15 -0700 Subject: [PATCH 02/18] fix(warden): Split oversized semantic review groups Semantic planning could abort a review when the model grouped related chunks beyond maxChunkChars. Split oversized planned groups into valid scanner chunks while preserving semantic titles and summaries. Co-Authored-By: GPT-5 Codex --- .../src/sdk/semantic-chunk-planner.test.ts | 22 ++++-- .../warden/src/sdk/semantic-chunk-planner.ts | 75 +++++++++++++++---- 2 files changed, 76 insertions(+), 21 deletions(-) diff --git a/packages/warden/src/sdk/semantic-chunk-planner.test.ts b/packages/warden/src/sdk/semantic-chunk-planner.test.ts index cb1da3d0..e7fbb855 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.test.ts +++ b/packages/warden/src/sdk/semantic-chunk-planner.test.ts @@ -133,14 +133,14 @@ describe('planSemanticReviewChunks', () => { expect(prompt).toContain('Do not restate filenames or line ranges.'); }); - it('rejects planned groups that exceed maxChunkChars after materialization', async () => { + it('splits planned groups that exceed maxChunkChars after materialization', async () => { runAuxiliary.mockResolvedValueOnce({ success: true, data: { groups: [{ title: 'Preserve dashboard axis range', summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100'], + chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], }], }, usage: { @@ -158,11 +158,23 @@ describe('planSemanticReviewChunks', () => { const context = makeContext(); const prepared = prepareFiles(context); - await expect(planSemanticReviewChunks(prepared.files, context, { + const planned = await planSemanticReviewChunks(prepared.files, context, { enabled: true, runtime: 'pi', - maxChunkChars: 10, - })).rejects.toThrow('exceeding maxChunkChars 10'); + maxChunkChars: 200, + }); + + expect(planned.groups).toHaveLength(3); + expect(planned.groups.map((group) => group.chunks[0]?.title)).toEqual([ + 'Preserve dashboard axis range (1/3)', + 'Preserve dashboard axis range (2/3)', + 'Preserve dashboard axis range (3/3)', + ]); + expect(planned.groups.map((group) => group.chunks[0]?.changedLineMap)).toEqual([ + [{ path: 'src/dashboard.ts', start: 10, end: 10 }], + [{ path: 'src/dashboard.ts', start: 100, end: 100 }], + [{ path: 'src/dashboard.ts', start: 200, end: 200 }], + ]); }); it('materializes semantic groups across multiple files', async () => { diff --git a/packages/warden/src/sdk/semantic-chunk-planner.ts b/packages/warden/src/sdk/semantic-chunk-planner.ts index 57ad7c48..6f0261f8 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.ts +++ b/packages/warden/src/sdk/semantic-chunk-planner.ts @@ -139,6 +139,60 @@ function materializePlannedChunk( }; } +function reviewChunkContentChars(chunk: ReviewChunk): number { + return chunk.files.reduce((sum, file) => sum + file.content.length, 0); +} + +function plannedChunkTitle(title: string, partIndex: number, totalParts: number): string { + return totalParts <= 1 ? title : `${title} (${partIndex}/${totalParts})`; +} + +function splitPlannedChunks( + startIndex: number, + title: string, + summary: string, + chunks: ReviewChunk[], + limits: { maxChunkChars: number; maxHunksPerChunk: number } +): ReviewChunk[] { + const batches: ReviewChunk[][] = []; + let current: ReviewChunk[] = []; + + const pushCurrent = () => { + if (current.length > 0) { + batches.push(current); + current = []; + } + }; + + for (const chunk of chunks) { + if (current.length === 0) { + current = [chunk]; + continue; + } + + const candidate = [...current, chunk]; + const candidateChunk = materializePlannedChunk(startIndex + batches.length, title, summary, candidate); + const exceedsHunks = candidate.length > limits.maxHunksPerChunk; + const exceedsChars = reviewChunkContentChars(candidateChunk) > limits.maxChunkChars; + + if (exceedsHunks || exceedsChars) { + pushCurrent(); + current = [chunk]; + } else { + current = candidate; + } + } + + pushCurrent(); + + return batches.map((batch, batchIndex) => materializePlannedChunk( + startIndex + batchIndex, + plannedChunkTitle(title, batchIndex + 1, batches.length), + summary, + batch, + )); +} + function groupFromPreparedFile(file: PreparedFile): ReviewChunkGroup { const filenames = [...new Set(file.chunks.flatMap((chunk) => chunk.files.map((chunkFile) => chunkFile.path)))]; return { @@ -204,13 +258,7 @@ export async function planSemanticReviewChunks( ); } - for (const [index, group] of result.data.groups.entries()) { - if (group.chunkIds.length > maxHunksPerChunk) { - throw new SkillRunnerError( - `Semantic chunk planning returned ${group.chunkIds.length} chunks in one group, exceeding maxHunksPerChunk ${maxHunksPerChunk}`, - { code: 'sdk_error' }, - ); - } + for (const group of result.data.groups) { const groupChunks: ReviewChunk[] = []; for (const chunkId of group.chunkIds) { if (seen.has(chunkId)) { @@ -227,15 +275,10 @@ export async function planSemanticReviewChunks( seen.add(chunkId); groupChunks.push(chunk); } - const plannedChunk = materializePlannedChunk(index, group.title, group.summary, groupChunks); - const plannedChunkChars = plannedChunk.files.reduce((sum, file) => sum + file.content.length, 0); - if (plannedChunkChars > maxChunkChars) { - throw new SkillRunnerError( - `Semantic chunk planning returned a ${plannedChunkChars} character group, exceeding maxChunkChars ${maxChunkChars}`, - { code: 'sdk_error' }, - ); - } - plannedChunks.push(plannedChunk); + plannedChunks.push(...splitPlannedChunks(plannedChunks.length, group.title, group.summary, groupChunks, { + maxChunkChars, + maxHunksPerChunk, + })); } const missing = chunks.filter((chunk) => !seen.has(chunk.id)); From 1e036d785670ea1d0c7b5beed571daffdf25155d Mon Sep 17 00:00:00 2001 From: David Cramer Date: Mon, 29 Jun 2026 20:38:15 -0700 Subject: [PATCH 03/18] fix(warden): Enforce semantic chunk limits Validate materialized semantic chunks after size splitting so maxChunks remains a real scanner cap. Reject oversized atomic chunks with a specific planner error instead of silently bypassing maxChunkChars. Co-Authored-By: GPT-5 Codex --- .../src/sdk/semantic-chunk-planner.test.ts | 86 +++++++++++++++++-- .../warden/src/sdk/semantic-chunk-planner.ts | 15 ++++ 2 files changed, 93 insertions(+), 8 deletions(-) diff --git a/packages/warden/src/sdk/semantic-chunk-planner.test.ts b/packages/warden/src/sdk/semantic-chunk-planner.test.ts index e7fbb855..5d49f334 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.test.ts +++ b/packages/warden/src/sdk/semantic-chunk-planner.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it, vi } from 'vitest'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; import type { EventContext } from '../types/index.js'; import { prepareFiles } from './prepare.js'; import { planSemanticReviewChunks } from './semantic-chunk-planner.js'; @@ -20,6 +20,10 @@ vi.mock('./runtimes/index.js', async (importOriginal) => { }; }); +beforeEach(() => { + runAuxiliary.mockReset(); +}); + function makeContext(): EventContext { return { eventType: 'pull_request', @@ -161,22 +165,88 @@ describe('planSemanticReviewChunks', () => { const planned = await planSemanticReviewChunks(prepared.files, context, { enabled: true, runtime: 'pi', - maxChunkChars: 200, + maxChunkChars: 500, }); - expect(planned.groups).toHaveLength(3); + expect(planned.groups).toHaveLength(2); expect(planned.groups.map((group) => group.chunks[0]?.title)).toEqual([ - 'Preserve dashboard axis range (1/3)', - 'Preserve dashboard axis range (2/3)', - 'Preserve dashboard axis range (3/3)', + 'Preserve dashboard axis range (1/2)', + 'Preserve dashboard axis range (2/2)', ]); expect(planned.groups.map((group) => group.chunks[0]?.changedLineMap)).toEqual([ - [{ path: 'src/dashboard.ts', start: 10, end: 10 }], - [{ path: 'src/dashboard.ts', start: 100, end: 100 }], + [ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'src/dashboard.ts', start: 100, end: 100 }, + ], [{ path: 'src/dashboard.ts', start: 200, end: 200 }], ]); }); + it('rejects materialized groups that exceed maxChunks after splitting', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context); + + await expect(planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxChunks: 1, + maxChunkChars: 500, + })).rejects.toThrow('materialized 2 groups, exceeding maxChunks 1'); + }); + + it('rejects atomic chunks that exceed maxChunkChars', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: ['src/dashboard.ts:10'], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context); + + await expect(planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxChunkChars: 10, + })).rejects.toThrow('cannot split atomic chunk src/dashboard.ts:10 under maxChunkChars 10'); + }); + it('materializes semantic groups across multiple files', async () => { runAuxiliary.mockResolvedValueOnce({ success: true, diff --git a/packages/warden/src/sdk/semantic-chunk-planner.ts b/packages/warden/src/sdk/semantic-chunk-planner.ts index 6f0261f8..57217533 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.ts +++ b/packages/warden/src/sdk/semantic-chunk-planner.ts @@ -165,6 +165,14 @@ function splitPlannedChunks( }; for (const chunk of chunks) { + const singleChunk = materializePlannedChunk(startIndex + batches.length, title, summary, [chunk]); + if (reviewChunkContentChars(singleChunk) > limits.maxChunkChars) { + throw new SkillRunnerError( + `Semantic chunk planning cannot split atomic chunk ${chunk.id} under maxChunkChars ${limits.maxChunkChars}`, + { code: 'sdk_error' }, + ); + } + if (current.length === 0) { current = [chunk]; continue; @@ -279,6 +287,13 @@ export async function planSemanticReviewChunks( maxChunkChars, maxHunksPerChunk, })); + + if (plannedChunks.length > maxChunks) { + throw new SkillRunnerError( + `Semantic chunk planning materialized ${plannedChunks.length} groups, exceeding maxChunks ${maxChunks}`, + { code: 'sdk_error' }, + ); + } } const missing = chunks.filter((chunk) => !seen.has(chunk.id)); From 72c6a2e59be72769cbaaf405c8e1f6bba3edc9ce Mon Sep 17 00:00:00 2001 From: David Cramer Date: Mon, 29 Jun 2026 20:43:51 -0700 Subject: [PATCH 04/18] fix(warden): Infer paths in semantic findings Use the changed-line map to infer missing finding paths for semantic chunks when a line range maps to exactly one file. Keep ambiguous multi-file findings strict so invalid locations are still dropped. Co-Authored-By: GPT-5 Codex --- packages/warden/src/sdk/analyze.ts | 50 ++++++++++++++++++++++++-- packages/warden/src/sdk/extract.ts | 12 ++++++- packages/warden/src/sdk/runner.test.ts | 37 +++++++++++++++++++ 3 files changed, 96 insertions(+), 3 deletions(-) diff --git a/packages/warden/src/sdk/analyze.ts b/packages/warden/src/sdk/analyze.ts index 558cda9b..d274a0d6 100644 --- a/packages/warden/src/sdk/analyze.ts +++ b/packages/warden/src/sdk/analyze.ts @@ -9,6 +9,7 @@ import { DEFAULT_RETRY_CONFIG, calculateRetryDelay, sleep } from './retry.js'; import { aggregateUsage, emptyUsage, estimateTokens, aggregateAuxiliaryUsage, aggregateAuxiliaryUsageAttribution } from './usage.js'; import { buildHunkSystemPrompt, buildReviewChunkUserPrompt, type PRPromptContext } from './prompt.js'; import { extractFindingsJson, extractFindingsWithLLM, validateFindings } from './extract.js'; +import type { FindingPathResolver } from './extract.js'; import { postProcessFindings } from './post-process.js'; import { buildFileReports } from './report-files.js'; import { getRuntime, getRuntimeProviderOptions } from './runtimes/index.js'; @@ -170,7 +171,7 @@ function buildHunkTrace(args: { */ async function parseHunkOutput( result: SkillRunResult, - defaultFilename: string | undefined, + defaultFilename: string | FindingPathResolver | undefined, skillName: string, options: SkillRunnerOptions ): Promise { @@ -210,6 +211,47 @@ async function parseHunkOutput( }; } +function numberFromRecord(record: Record, key: string): number | undefined { + const value = record[key]; + return typeof value === 'number' && Number.isInteger(value) && value > 0 ? value : undefined; +} + +function resolveFindingPathFromChangedLines( + changedLineMap: ChangedLineRange[], + fallbackFilename: string | undefined, + finding: Record +): string | undefined { + const location = finding['location']; + if (!location || typeof location !== 'object') { + return fallbackFilename; + } + + const locationRecord = location as Record; + const explicitPath = locationRecord['path']; + if (typeof explicitPath === 'string' && explicitPath.length > 0) { + return explicitPath; + } + + if (fallbackFilename) { + return fallbackFilename; + } + + const startLine = numberFromRecord(locationRecord, 'startLine'); + if (!startLine) { + return undefined; + } + + const endLine = numberFromRecord(locationRecord, 'endLine') ?? startLine; + const matchingPaths = new Set(); + for (const range of changedLineMap) { + if (startLine >= range.start && endLine <= range.end) { + matchingPaths.add(range.path); + } + } + + return matchingPaths.size === 1 ? [...matchingPaths][0] : undefined; +} + /** * Filter findings whose location falls outside the changed line map. * Findings without a location are kept (general findings). @@ -516,7 +558,11 @@ async function analyzeReviewChunk( traceRecorder, () => parseHunkOutput( resultMessage, - chunk.files.length === 1 ? primaryFile : undefined, + (finding) => resolveFindingPathFromChangedLines( + chunk.changedLineMap, + chunk.files.length === 1 ? primaryFile : undefined, + finding, + ), skill.name, options, ), diff --git a/packages/warden/src/sdk/extract.ts b/packages/warden/src/sdk/extract.ts index 3a3a9144..f1a02a72 100644 --- a/packages/warden/src/sdk/extract.ts +++ b/packages/warden/src/sdk/extract.ts @@ -16,6 +16,8 @@ import { const ExtractedFindingSchema = FindingSchema.omit({ sourceSnippet: true }); +export type FindingPathResolver = (finding: Record) => string | undefined; + /** Pattern to match the start of findings JSON (allows whitespace after brace) */ export const FINDINGS_JSON_START = /\{\s*"findings"/; @@ -274,13 +276,21 @@ export function generateShortId(): string { * Validate and normalize findings from extracted JSON. * Replaces the LLM-provided ID with a short ID for cross-referencing. */ -export function validateFindings(findings: unknown[], defaultFilename?: string): Finding[] { +export function validateFindings( + findings: unknown[], + defaultFilenameOrResolver?: string | FindingPathResolver +): Finding[] { const validated: Finding[] = []; for (const f of findings) { const candidate = typeof f === 'object' && f !== null ? { ...(f as Record) } : f; + const defaultFilename = typeof defaultFilenameOrResolver === 'function' + && typeof candidate === 'object' + && candidate !== null + ? defaultFilenameOrResolver(candidate as Record) + : defaultFilenameOrResolver; // Normalize missing location paths before validation. Multi-file review // chunks may provide explicit paths, so preserve them when present. diff --git a/packages/warden/src/sdk/runner.test.ts b/packages/warden/src/sdk/runner.test.ts index 9af27ea6..b304fc27 100644 --- a/packages/warden/src/sdk/runner.test.ts +++ b/packages/warden/src/sdk/runner.test.ts @@ -974,6 +974,43 @@ describe('validateFindings', () => { expect(validated[0]!.location!.path).toBe('correct-path.ts'); }); + it('defaults missing location paths with a resolver', () => { + const rawFindings = [ + { + id: 'id-1', + severity: 'medium', + title: 'Issue', + description: 'Details', + location: { startLine: 42 }, + }, + ]; + + const validated = validateFindings(rawFindings, (finding) => { + const location = finding['location']; + if (!location || typeof location !== 'object') return undefined; + return (location as Record)['startLine'] === 42 + ? 'src/cross-file.ts' + : undefined; + }); + + expect(validated[0]!.location!.path).toBe('src/cross-file.ts'); + }); + + it('drops findings when the resolver cannot infer a missing path', () => { + const rawFindings = [ + { + id: 'id-1', + severity: 'medium', + title: 'Issue', + description: 'Details', + location: { startLine: 42 }, + }, + ]; + + const validated = validateFindings(rawFindings, () => undefined); + expect(validated).toHaveLength(0); + }); + it('strips model-provided source snippets during extraction', () => { const rawFindings = [ { From 19a693673e6abdef014c8c1def19fa24d4710ca7 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Mon, 29 Jun 2026 21:12:38 -0700 Subject: [PATCH 05/18] fix(warden): Tighten semantic chunk validation Handle coalesced hunk boundaries when building changed-line maps, tighten review chunk and finding validation types, and make the semantic chunking eval participate in scored eval status checks. Co-Authored-By: GPT-5 Codex --- packages/evals/src/semantic-chunking.eval.ts | 205 ++++++++++++++---- .../warden/src/action/triggers/executor.ts | 2 +- packages/warden/src/diff/review-chunk.test.ts | 52 +++++ packages/warden/src/diff/review-chunk.ts | 67 ++++-- packages/warden/src/sdk/analyze.test.ts | 124 ++++++++++- packages/warden/src/sdk/analyze.ts | 12 +- packages/warden/src/sdk/extract.ts | 2 +- packages/warden/src/sdk/prepare.ts | 4 +- packages/warden/src/sdk/prompt.ts | 10 +- .../warden/src/sdk/semantic-chunk-planner.ts | 6 +- 10 files changed, 402 insertions(+), 82 deletions(-) diff --git a/packages/evals/src/semantic-chunking.eval.ts b/packages/evals/src/semantic-chunking.eval.ts index baa2094c..8520f827 100644 --- a/packages/evals/src/semantic-chunking.eval.ts +++ b/packages/evals/src/semantic-chunking.eval.ts @@ -1,4 +1,8 @@ -import { describe, expect, it } from 'vitest'; +import { expect } from 'vitest'; +import { createJudge, describeEval } from 'vitest-evals'; +import type { JudgeContext } from 'vitest-evals'; +import { normalizeContent, toJsonValue, type Harness, type JsonValue } from 'vitest-evals/harness'; +import { z } from 'zod'; import type { EventContext } from '../../warden/src/types/index.js'; import { prepareFiles } from '../../warden/src/sdk/prepare.js'; import { planSemanticReviewChunks } from '../../warden/src/sdk/semantic-chunk-planner.js'; @@ -13,6 +17,25 @@ const model = process.env['WARDEN_SEMANTIC_CHUNK_EVAL_MODEL'] ?? defaultEvalMode const apiKey = getEvalRuntimeApiKey(model); const providerApiKey = getEvalProviderApiKey(model); +const SemanticChunkingEvalInputSchema = z.object({ + name: z.string(), +}); +type SemanticChunkingEvalInput = z.infer; + +const SemanticChunkingEvalOutputSchema = z.object({ + atomicChunkCount: z.number().int().nonnegative(), + chunks: z.array(z.object({ + summary: z.string().optional(), + files: z.array(z.string()), + changedLineMap: z.array(z.object({ + path: z.string(), + start: z.number().int(), + end: z.number().int(), + })), + content: z.string(), + })), +}); + function makeSemanticChunkingContext(): EventContext { return { eventType: 'pull_request', @@ -66,58 +89,144 @@ function makeSemanticChunkingContext(): EventContext { }; } -describe.skipIf(!providerApiKey)('semantic chunking eval', () => { - it('plans a behavior-level delta for many tiny related hunks', { timeout: 120_000 }, async () => { - const context = makeSemanticChunkingContext(); - const prepared = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, +function createSemanticChunkingHarness(): Harness { + return { + name: 'semantic-chunking', + run: async (input) => { + SemanticChunkingEvalInputSchema.parse(input); + const startTime = Date.now(); + const context = makeSemanticChunkingContext(); + const prepared = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, }, - }, - }); - - const atomicChunks = prepared.files.flatMap((file) => file.chunks); - expect(atomicChunks).toHaveLength(4); + }); + const atomicChunks = prepared.files.flatMap((file) => file.chunks); + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + apiKey, + runtime: DEFAULT_EVAL_RUNTIME, + model, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + }); + const chunks = planned.groups.flatMap((group) => group.chunks); + const output = { + atomicChunkCount: atomicChunks.length, + chunks: chunks.map((chunk) => ({ + summary: chunk.summary, + files: chunk.files.map((file) => file.path), + changedLineMap: chunk.changedLineMap, + content: chunk.files.map((file) => file.content).join('\n'), + })), + }; - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - apiKey, - runtime: DEFAULT_EVAL_RUNTIME, - model, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }); - - const chunks = planned.groups.flatMap((group) => group.chunks); - expect(chunks.length).toBeLessThan(atomicChunks.length); - const crossFileChunk = chunks.find((chunk) => chunk.files.length > 1); - expect(crossFileChunk).toBeDefined(); - expect(crossFileChunk?.changedLineMap).toEqual(expect.arrayContaining([ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'src/dashboard.ts', start: 80, end: 80 }, - { path: 'src/dashboard.ts', start: 140, end: 140 }, - { path: 'tests/dashboard.test.ts', start: 220, end: 220 }, - ])); + return { + output: toJsonValue(output) as JsonValue, + session: { + messages: [ + { + role: 'user', + content: normalizeContent({ + name: input.name, + goal: 'Group related dashboard range changes semantically across implementation and test files.', + }), + }, + { + role: 'assistant', + content: normalizeContent(output), + }, + ], + provider: DEFAULT_EVAL_RUNTIME, + model, + }, + usage: {}, + timings: { totalMs: Date.now() - startTime }, + artifacts: {}, + errors: [], + }; + }, + }; +} - for (const chunk of chunks) { - expect(chunk.summary).toBeTruthy(); - expect(chunk.summary).not.toMatch(/lines?\s+\d+/i); - expect(chunk.summary).not.toMatch(/\b10\b.*\b80\b.*\b140\b/); +function createSemanticChunkingJudge() { + return createJudge>('SemanticChunkingJudge', async ({ run }) => { + const output = SemanticChunkingEvalOutputSchema.safeParse(run.output); + if (!output.success) { + return { + score: 0, + metadata: { rationale: `Invalid semantic chunking output: ${output.error.message}` }, + }; } + const chunks = output.data.chunks; + const crossFileChunk = chunks.find((chunk) => chunk.files.length > 1); const summaryText = chunks.map((chunk) => chunk.summary ?? '').join(' '); - expect(summaryText).toMatch(/axisRange|axis range|range/i); - expect(summaryText).toMatch(/widget|convert|render|chart/i); + const hasSemanticSummary = /axisRange|axis range|range/i.test(summaryText) + && /widget|convert|render|chart/i.test(summaryText) + && !/lines?\s+\d+/i.test(summaryText) + && !/\b10\b.*\b80\b.*\b140\b/.test(summaryText); + const hasExpectedLineMap = Boolean(crossFileChunk) + && [ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'src/dashboard.ts', start: 80, end: 80 }, + { path: 'src/dashboard.ts', start: 140, end: 140 }, + { path: 'tests/dashboard.test.ts', start: 220, end: 220 }, + ].every((range) => + crossFileChunk?.changedLineMap.some((actual) => + actual.path === range.path && actual.start === range.start && actual.end === range.end + ) + ); + const crossFileContent = crossFileChunk?.content ?? ''; + const hasExpectedContent = Boolean(crossFileChunk) + && crossFileContent.includes('convertWidgetToChart(widget, range)') + && crossFileContent.includes('renderChart(chart, series, range)') + && crossFileContent.includes('expect(rendered.range).toEqual(customAxisRange)'); + const reducedChunks = chunks.length < output.data.atomicChunkCount; + const passed = reducedChunks && Boolean(crossFileChunk) && hasExpectedLineMap + && hasSemanticSummary && hasExpectedContent; - const chunkText = crossFileChunk?.files.map((file) => file.content).join('\n') ?? ''; - expect(chunkText).toContain('convertWidgetToChart(widget, range)'); - expect(chunkText).toContain('renderChart(chart, series, range)'); - expect(chunkText).toContain('expect(rendered.range).toEqual(customAxisRange)'); + return { + score: passed ? 1 : 0, + metadata: { + rationale: passed + ? 'Planner produced a semantic cross-file chunk with behavior-level summary.' + : 'Planner did not produce the expected semantic cross-file chunk.', + atomicChunkCount: output.data.atomicChunkCount, + chunkCount: chunks.length, + hasCrossFileChunk: Boolean(crossFileChunk), + hasExpectedLineMap, + hasSemanticSummary, + hasExpectedContent, + }, + }; }); -}); +} + +describeEval( + 'semantic-chunking', + { + harness: createSemanticChunkingHarness(), + judges: [createSemanticChunkingJudge()], + judgeThreshold: 1, + skipIf: () => !providerApiKey, + }, + (it) => { + it('plans a behavior-level delta for many tiny related hunks', { timeout: 120_000 }, async ({ run }) => { + const result = await run({ name: 'dashboard-axis-range' }); + const output = SemanticChunkingEvalOutputSchema.parse(result.output); + const chunks = output.chunks; + + expect(output.atomicChunkCount).toBe(4); + expect(chunks.length).toBeLessThan(output.atomicChunkCount); + expect(chunks.some((chunk) => chunk.files.length > 1)).toBe(true); + }); + }, +); diff --git a/packages/warden/src/action/triggers/executor.ts b/packages/warden/src/action/triggers/executor.ts index d06e502b..93be020a 100644 --- a/packages/warden/src/action/triggers/executor.ts +++ b/packages/warden/src/action/triggers/executor.ts @@ -136,7 +136,7 @@ export interface TriggerResult { // Executor // ----------------------------------------------------------------------------- -/** Build the skill task options used by action trigger execution. */ +/** Translate a resolved action trigger into the shared skill-task boundary. */ export function createTriggerSkillTaskOptions( trigger: ResolvedTrigger, deps: TriggerExecutorDeps diff --git a/packages/warden/src/diff/review-chunk.test.ts b/packages/warden/src/diff/review-chunk.test.ts index 7524a8b1..5037cc46 100644 --- a/packages/warden/src/diff/review-chunk.test.ts +++ b/packages/warden/src/diff/review-chunk.test.ts @@ -1,4 +1,6 @@ import { describe, expect, it } from 'vitest'; +import { coalesceHunks } from './coalesce.js'; +import { parsePatch } from './parser.js'; import { reviewChunkFromHunk } from './review-chunk.js'; import type { HunkWithContext } from './context.js'; @@ -29,4 +31,54 @@ describe('reviewChunkFromHunk', () => { expect(chunk.changedLineMap).toEqual([{ path: 'src/example.ts', start: 10, end: 10 }]); expect(chunk.id).toBe('src/example.ts:old10-10:new10-9'); }); + + it('resets changed-line ranges at embedded hunk headers from coalesced content', () => { + const parsedHunks = parsePatch([ + '@@ -10,1 +10,1 @@', + '-const first = oldValue;', + '+const first = newValue;', + '@@ -100,1 +100,1 @@', + '-const second = oldValue;', + '+const second = newValue;', + ].join('\n')); + const [hunk] = coalesceHunks(parsedHunks, { maxGapLines: 100 }); + expect(hunk?.lines).toEqual([ + '-const first = oldValue;', + '+const first = newValue;', + '-const second = oldValue;', + '+const second = newValue;', + ]); + + const chunk = reviewChunkFromHunk(makeHunkContext({ + hunk, + })); + + expect(chunk.changedLineMap).toEqual([ + { path: 'src/example.ts', start: 10, end: 10 }, + { path: 'src/example.ts', start: 100, end: 100 }, + ]); + expect(chunk.files[0].sourceLines).toEqual([ + { line: 10, content: 'const first = newValue;' }, + { line: 100, content: 'const second = newValue;' }, + ]); + }); + + it('anchors deletion-only embedded hunk sections independently', () => { + const parsedHunks = parsePatch([ + '@@ -10,1 +10,0 @@', + '-const first = removed;', + '@@ -100,1 +100,0 @@', + '-const second = removed;', + ].join('\n')); + const [hunk] = coalesceHunks(parsedHunks, { maxGapLines: 100 }); + + const chunk = reviewChunkFromHunk(makeHunkContext({ + hunk, + })); + + expect(chunk.changedLineMap).toEqual([ + { path: 'src/example.ts', start: 10, end: 10 }, + { path: 'src/example.ts', start: 100, end: 100 }, + ]); + }); }); diff --git a/packages/warden/src/diff/review-chunk.ts b/packages/warden/src/diff/review-chunk.ts index 4b389db9..5ac2a85a 100644 --- a/packages/warden/src/diff/review-chunk.ts +++ b/packages/warden/src/diff/review-chunk.ts @@ -19,19 +19,40 @@ export interface ReviewChunkFile { sourceLines: SourceSnippetLine[]; } +export type ReviewChunkFiles = [ReviewChunkFile, ...ReviewChunkFile[]]; + export interface ReviewChunk { id: string; title: string; summary?: string; - files: ReviewChunkFile[]; + files: ReviewChunkFiles; changedLineMap: ChangedLineRange[]; } +function parseEmbeddedHunkHeader(line: string): { newStart: number; newCount: number } | undefined { + const match = line.match(/^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/); + if (!match?.[1]) return undefined; + return { + newStart: parseInt(match[1], 10), + newCount: parseInt(match[2] ?? '1', 10), + }; +} + +function linesWithHunkBoundaries(hunkCtx: HunkWithContext): string[] { + const contentLines = hunkCtx.hunk.content.split('\n'); + const hunkHeaderCount = contentLines.filter((line) => parseEmbeddedHunkHeader(line)).length; + return hunkHeaderCount > 1 ? contentLines : hunkCtx.hunk.lines; +} + function changedRangesFromHunk(path: string, hunkCtx: HunkWithContext): ChangedLineRange[] { const ranges: ChangedLineRange[] = []; let newLine = hunkCtx.hunk.newStart; let currentStart: number | undefined; let currentEnd: number | undefined; + let segmentStart = hunkCtx.hunk.newStart; + let segmentEnd = Math.max(segmentStart, segmentStart + hunkCtx.hunk.newCount - 1); + let segmentHasAddedLine = false; + let segmentHasDeletedLine = false; function flush(): void { if (currentStart === undefined || currentEnd === undefined) return; @@ -40,8 +61,27 @@ function changedRangesFromHunk(path: string, hunkCtx: HunkWithContext): ChangedL currentEnd = undefined; } - for (const diffLine of hunkCtx.hunk.lines) { + function flushDeletionOnlySegment(): void { + flush(); + if (!segmentHasAddedLine && segmentHasDeletedLine) { + ranges.push({ path, start: segmentStart, end: segmentEnd }); + } + segmentHasAddedLine = false; + segmentHasDeletedLine = false; + } + + for (const diffLine of linesWithHunkBoundaries(hunkCtx)) { + const embeddedHeader = parseEmbeddedHunkHeader(diffLine); + if (embeddedHeader) { + flushDeletionOnlySegment(); + newLine = embeddedHeader.newStart; + segmentStart = embeddedHeader.newStart; + segmentEnd = Math.max(segmentStart, segmentStart + embeddedHeader.newCount - 1); + continue; + } + if (diffLine.startsWith('+')) { + segmentHasAddedLine = true; if (currentStart === undefined) { currentStart = newLine; } @@ -51,21 +91,14 @@ function changedRangesFromHunk(path: string, hunkCtx: HunkWithContext): ChangedL } flush(); - if (diffLine.startsWith(' ') || !diffLine.startsWith('-')) { + if (diffLine.startsWith('-')) { + segmentHasDeletedLine = true; + } else if (diffLine.startsWith(' ')) { newLine += 1; } } - flush(); - if ( - ranges.length === 0 - && hunkCtx.hunk.lines.some((line) => line.startsWith('-')) - ) { - const start = hunkCtx.hunk.newStart; - const end = Math.max(start, hunkCtx.hunk.newStart + hunkCtx.hunk.newCount - 1); - return [{ path, start, end }]; - } - + flushDeletionOnlySegment(); return ranges; } @@ -76,7 +109,13 @@ function hunkSourceLines(hunkCtx: HunkWithContext): SourceSnippetLine[] { } let newLine = hunkCtx.hunk.newStart; - for (const diffLine of hunkCtx.hunk.lines) { + for (const diffLine of linesWithHunkBoundaries(hunkCtx)) { + const embeddedHeader = parseEmbeddedHunkHeader(diffLine); + if (embeddedHeader) { + newLine = embeddedHeader.newStart; + continue; + } + if (diffLine.startsWith('-')) continue; if (!diffLine.startsWith('+') && !diffLine.startsWith(' ')) continue; const content = diffLine.slice(1); diff --git a/packages/warden/src/sdk/analyze.test.ts b/packages/warden/src/sdk/analyze.test.ts index 0060a097..eae2f18d 100644 --- a/packages/warden/src/sdk/analyze.test.ts +++ b/packages/warden/src/sdk/analyze.test.ts @@ -187,6 +187,48 @@ function makeContextWithOneHunk(): EventContext { }; } +function makeContextWithCrossFileHunks(): EventContext { + return { + eventType: 'pull_request', + action: 'opened', + repository: { owner: 'o', name: 'r', fullName: 'o/r', defaultBranch: 'main' }, + repoPath: '/tmp/repo', + pullRequest: { + number: 1, + title: 'Test PR', + body: '', + author: 'test', + baseBranch: 'main', + headBranch: 'feature', + headSha: 'head', + baseSha: 'base', + files: [{ + filename: 'src/example.ts', + status: 'modified', + additions: 1, + deletions: 1, + patch: [ + '@@ -10,1 +10,1 @@', + '-old10', + '+new10', + ].join('\n'), + chunks: 1, + }, { + filename: 'tests/example.test.ts', + status: 'modified', + additions: 1, + deletions: 1, + patch: [ + '@@ -50,1 +50,1 @@', + '-old50', + '+new50', + ].join('\n'), + chunks: 1, + }], + }, + }; +} + describe('filterOutOfRangeFindings', () => { const hunkRange = { start: 10, end: 20 }; @@ -245,16 +287,21 @@ describe('filterOutOfRangeFindings', () => { ...makeFinding(42, 'partial-range', 'src/two.ts'), location: { path: 'src/two.ts', startLine: 42, endLine: 45 }, }; + const crossRangeFinding: Finding = { + ...makeFinding(42, 'cross-range', 'src/two.ts'), + location: { path: 'src/two.ts', startLine: 42, endLine: 50 }, + }; const { filtered, dropped } = filterOutOfRangeFindings( - [firstFileFinding, secondFileFinding, wrongFileFinding, partialRangeFinding], + [firstFileFinding, secondFileFinding, wrongFileFinding, partialRangeFinding, crossRangeFinding], [ { path: 'src/one.ts', start: 10, end: 20 }, { path: 'src/two.ts', start: 40, end: 43 }, + { path: 'src/two.ts', start: 50, end: 50 }, ], ); - expect(filtered).toEqual([firstFileFinding, secondFileFinding]); + expect(filtered).toEqual([firstFileFinding, secondFileFinding, crossRangeFinding]); expect(dropped).toEqual([wrongFileFinding, partialRangeFinding]); }); @@ -810,6 +857,79 @@ describe('runSkill', () => { expect(report.auxiliaryUsageAttribution?.['semantic-chunk-planner']).toBeDefined(); }); + it('infers missing finding paths from cross-file semantic chunk line maps', async () => { + const runSkillMock = vi.fn().mockResolvedValue({ + result: { + status: 'success', + text: JSON.stringify({ + findings: [{ + id: 'pathless-finding', + severity: 'medium', + confidence: 'high', + title: 'Pathless finding', + description: 'test', + location: { startLine: 50 }, + }], + }), + errors: [], + usage: makeUsage(), + }, + }); + const runAuxiliaryMock = vi.fn().mockResolvedValue({ + success: true, + data: { + groups: [{ + title: 'Preserve behavior and test expectation', + summary: 'The implementation and test update the same behavior contract.', + chunkIds: [ + 'src/example.ts:10', + 'tests/example.test.ts:50', + ], + }], + }, + usage: makeUsage(), + }); + vi.mocked(getRuntime).mockReturnValue({ + name: 'pi', + runSkill: runSkillMock, + runAuxiliary: runAuxiliaryMock, + runSynthesis: vi.fn(), + } as unknown as Runtime); + + const report = await runSkill( + { + name: 'security-review', + description: 'Security review.', + prompt: 'Return findings as JSON.', + }, + makeContextWithCrossFileHunks(), + { + runtime: 'pi', + model: 'anthropic/claude-sonnet-4-6', + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + postProcessFindings: false, + }, + ); + + expect(report.findings).toEqual([ + expect.objectContaining({ + title: 'Pathless finding', + location: { + path: 'tests/example.test.ts', + startLine: 50, + }, + }), + ]); + }); + it('preserves candidate findings when verification is interrupted', async () => { const controller = new AbortController(); const runSkillMock = vi.fn() diff --git a/packages/warden/src/sdk/analyze.ts b/packages/warden/src/sdk/analyze.ts index d274a0d6..4da6a1b2 100644 --- a/packages/warden/src/sdk/analyze.ts +++ b/packages/warden/src/sdk/analyze.ts @@ -182,9 +182,10 @@ async function parseHunkOutput( // Tier 1: Try regex-based extraction first (fast) const extracted = extractFindingsJson(result.text); + const filenameOrResolver = defaultFilename ?? (() => undefined); if (extracted.success) { - return { findings: validateFindings(extracted.findings, defaultFilename), extractionFailed: false, extractionMethod: 'regex' }; + return { findings: validateFindings(extracted.findings, filenameOrResolver), extractionFailed: false, extractionMethod: 'regex' }; } // Tier 2: Try LLM fallback for malformed output @@ -197,7 +198,7 @@ async function parseHunkOutput( }); if (fallback.success) { - return { findings: validateFindings(fallback.findings, defaultFilename), extractionFailed: false, extractionMethod: 'llm', extractionUsage: fallback.usage }; + return { findings: validateFindings(fallback.findings, filenameOrResolver), extractionFailed: false, extractionMethod: 'llm', extractionUsage: fallback.usage }; } // Both tiers failed - return extraction failure info @@ -270,11 +271,12 @@ export function filterOutOfRangeFindings( if (!finding.location) return true; const { path, startLine } = finding.location; const endLine = finding.location.endLine ?? startLine; - return ranges.some((range) => + const lineInRange = (line: number): boolean => ranges.some((range) => (range.path === undefined || range.path === path) - && startLine >= range.start - && endLine <= range.end + && line >= range.start + && line <= range.end ); + return lineInRange(startLine) && lineInRange(endLine); } for (const finding of findings) { diff --git a/packages/warden/src/sdk/extract.ts b/packages/warden/src/sdk/extract.ts index f1a02a72..a2da1ac6 100644 --- a/packages/warden/src/sdk/extract.ts +++ b/packages/warden/src/sdk/extract.ts @@ -278,7 +278,7 @@ export function generateShortId(): string { */ export function validateFindings( findings: unknown[], - defaultFilenameOrResolver?: string | FindingPathResolver + defaultFilenameOrResolver: string | FindingPathResolver ): Finding[] { const validated: Finding[] = []; diff --git a/packages/warden/src/sdk/prepare.ts b/packages/warden/src/sdk/prepare.ts index 296141a8..abe0632e 100644 --- a/packages/warden/src/sdk/prepare.ts +++ b/packages/warden/src/sdk/prepare.ts @@ -18,9 +18,7 @@ function matchingChunkingSkipPattern( return patterns?.find((pattern) => classifyFile(filename, [pattern]) === 'skip')?.pattern; } -/** - * Group review chunks by filename into PreparedFile entries. - */ +/** Adapt atomic review chunks into the per-file shape used before semantic planning. */ export function groupChunksByFile(chunks: ReviewChunk[]): PreparedFile[] { const fileMap = new Map(); diff --git a/packages/warden/src/sdk/prompt.ts b/packages/warden/src/sdk/prompt.ts index 258801e9..c80f9618 100644 --- a/packages/warden/src/sdk/prompt.ts +++ b/packages/warden/src/sdk/prompt.ts @@ -18,9 +18,7 @@ function formatChangedLineMap(chunk: ReviewChunk): string { .join('\n'); } -/** - * Format a review chunk for LLM analysis. - */ +/** Format one semantic review chunk with its summary, line map, and file content. */ export function formatReviewChunkForAnalysis(chunk: ReviewChunk): string { const lines: string[] = []; @@ -99,7 +97,7 @@ Requirements: - Return valid JSON starting with {"findings": - "findings" array can be empty if no issues found - "location.path" must be one of the files in the changed line map. For single-file chunks, use that file path. -- "location.startLine" MUST be within one of the changed line map ranges. If the issue originates in surrounding code, anchor to the nearest changed line in the changed line map and note the actual location in the description. +- "location.startLine" MUST be within one of the changed line map ranges. If "location.endLine" is present, it must also be within one of the changed line map ranges for the same file. If the issue originates in surrounding code, anchor to the nearest changed line in the changed line map and note the actual location in the description. - "confidence" reflects how certain you are this is a real issue given the codebase context - "description" is rendered directly in GitHub inline comments. Keep it brief and actionable, usually one sentence. - Put the concrete evidence trace in "verification", not "description". @@ -127,9 +125,7 @@ You can read files from ${dirList} subdirectories using the Read tool with the f return sections.join('\n\n'); } -/** - * Builds the user prompt for a semantic review chunk. - */ +/** Build the scanner prompt around one semantic chunk while preserving location constraints. */ export function buildReviewChunkUserPrompt( skill: SkillDefinition, chunk: ReviewChunk, diff --git a/packages/warden/src/sdk/semantic-chunk-planner.ts b/packages/warden/src/sdk/semantic-chunk-planner.ts index 57217533..3b07f69a 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.ts +++ b/packages/warden/src/sdk/semantic-chunk-planner.ts @@ -129,12 +129,16 @@ function materializePlannedChunk( sourceLines: mergeSourceLines(chunks, path), }; }); + const firstFile = files[0]; + if (!firstFile) { + throw new Error('Cannot materialize planned chunk without files'); + } return { id: `semantic:${index + 1}`, title, summary, - files, + files: [firstFile, ...files.slice(1)], changedLineMap, }; } From 129e70e8f762b9631be3cf0b793ade30a68dd5c9 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 08:27:45 -0700 Subject: [PATCH 06/18] ci: Retrigger hosted Warden checks Refresh the PR head so hosted Warden checks run after the OpenRouter credential update. Co-Authored-By: GPT-5 Codex From 9412b833c60b53217c228a962e0cedff671bbf41 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 10:33:09 -0700 Subject: [PATCH 07/18] feat(chunking): Add semantic review chunk planning Move semantic grouping into its own module and run the scanner over planned review chunks. Add planner tooling, config knobs, eval coverage, and the benchmark plan with the first captured smoke baseline. Co-Authored-By: GPT-5 Codex --- packages/evals/src/semantic-chunking.eval.ts | 142 +++++--- packages/warden/src/cli/output/tasks.test.ts | 82 ++++- packages/warden/src/cli/output/tasks.ts | 11 +- packages/warden/src/config/loader.test.ts | 26 +- packages/warden/src/config/schema.ts | 6 + packages/warden/src/sdk/analyze.ts | 10 +- packages/warden/src/sdk/runner.ts | 1 - packages/warden/src/semantic/index.ts | 8 + .../planner.test.ts} | 142 +++++++- .../planner.ts} | 217 +++++++++-- packages/warden/src/semantic/tools.test.ts | 93 +++++ packages/warden/src/semantic/tools.ts | 156 ++++++++ specs/semantic-chunk-benchmark.md | 195 ++++++++++ specs/semantic-review-chunks.md | 337 ++++++++++++++++-- warden.toml | 3 + 15 files changed, 1292 insertions(+), 137 deletions(-) create mode 100644 packages/warden/src/semantic/index.ts rename packages/warden/src/{sdk/semantic-chunk-planner.test.ts => semantic/planner.test.ts} (67%) rename packages/warden/src/{sdk/semantic-chunk-planner.ts => semantic/planner.ts} (55%) create mode 100644 packages/warden/src/semantic/tools.test.ts create mode 100644 packages/warden/src/semantic/tools.ts create mode 100644 specs/semantic-chunk-benchmark.md diff --git a/packages/evals/src/semantic-chunking.eval.ts b/packages/evals/src/semantic-chunking.eval.ts index 8520f827..80808a0c 100644 --- a/packages/evals/src/semantic-chunking.eval.ts +++ b/packages/evals/src/semantic-chunking.eval.ts @@ -1,3 +1,6 @@ +import { mkdtemp, mkdir, rm, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; import { expect } from 'vitest'; import { createJudge, describeEval } from 'vitest-evals'; import type { JudgeContext } from 'vitest-evals'; @@ -5,7 +8,7 @@ import { normalizeContent, toJsonValue, type Harness, type JsonValue } from 'vit import { z } from 'zod'; import type { EventContext } from '../../warden/src/types/index.js'; import { prepareFiles } from '../../warden/src/sdk/prepare.js'; -import { planSemanticReviewChunks } from '../../warden/src/sdk/semantic-chunk-planner.js'; +import { planSemanticReviewChunks } from '../../warden/src/semantic/index.js'; import { DEFAULT_EVAL_RUNTIME, defaultEvalModel, @@ -36,7 +39,22 @@ const SemanticChunkingEvalOutputSchema = z.object({ })), }); -function makeSemanticChunkingContext(): EventContext { +async function createSemanticChunkingRepo(): Promise { + const repoPath = await mkdtemp(join(tmpdir(), 'warden-semantic-chunking-eval-')); + await mkdir(join(repoPath, 'src'), { recursive: true }); + await mkdir(join(repoPath, 'tests'), { recursive: true }); + await writeFile(join(repoPath, 'src/dashboard.ts'), [ + 'const range = widget.axisRange ?? getDefaultAxisRange(widget);', + 'const chart = convertWidgetToChart(widget, range);', + 'return renderChart(chart, series, range);', + ].join('\n')); + await writeFile(join(repoPath, 'tests/dashboard.test.ts'), [ + 'expect(rendered.range).toEqual(customAxisRange);', + ].join('\n')); + return repoPath; +} + +function makeSemanticChunkingContext(repoPath: string): EventContext { return { eventType: 'pull_request', action: 'opened', @@ -46,7 +64,7 @@ function makeSemanticChunkingContext(): EventContext { fullName: 'getsentry/semantic-chunking-eval', defaultBranch: 'main', }, - repoPath: '/tmp/warden-semantic-chunking-eval', + repoPath, pullRequest: { number: 313, title: 'Update dashboard behavior', @@ -95,63 +113,71 @@ function createSemanticChunkingHarness(): Harness { SemanticChunkingEvalInputSchema.parse(input); const startTime = Date.now(); - const context = makeSemanticChunkingContext(); - const prepared = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, - }, - }, - }); - const atomicChunks = prepared.files.flatMap((file) => file.chunks); - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - apiKey, - runtime: DEFAULT_EVAL_RUNTIME, - model, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }); - const chunks = planned.groups.flatMap((group) => group.chunks); - const output = { - atomicChunkCount: atomicChunks.length, - chunks: chunks.map((chunk) => ({ - summary: chunk.summary, - files: chunk.files.map((file) => file.path), - changedLineMap: chunk.changedLineMap, - content: chunk.files.map((file) => file.content).join('\n'), - })), - }; - - return { - output: toJsonValue(output) as JsonValue, - session: { - messages: [ - { - role: 'user', - content: normalizeContent({ - name: input.name, - goal: 'Group related dashboard range changes semantically across implementation and test files.', - }), + const repoPath = await createSemanticChunkingRepo(); + try { + const context = makeSemanticChunkingContext(repoPath); + const prepared = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, }, - { - role: 'assistant', - content: normalizeContent(output), - }, - ], - provider: DEFAULT_EVAL_RUNTIME, + }, + }); + const atomicChunks = prepared.files.flatMap((file) => file.chunks); + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + apiKey, + runtime: DEFAULT_EVAL_RUNTIME, model, - }, - usage: {}, - timings: { totalMs: Date.now() - startTime }, - artifacts: {}, - errors: [], - }; + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + maxEmbeddedDiffChars: 0, + maxEmbeddedDiffChunks: 0, + maxEmbeddedDiffRanges: 0, + }); + const chunks = planned.groups.flatMap((group) => group.chunks); + const output = { + atomicChunkCount: atomicChunks.length, + chunks: chunks.map((chunk) => ({ + summary: chunk.summary, + files: chunk.files.map((file) => file.path), + changedLineMap: chunk.changedLineMap, + content: chunk.files.map((file) => file.content).join('\n'), + })), + }; + + return { + output: toJsonValue(output) as JsonValue, + session: { + messages: [ + { + role: 'user', + content: normalizeContent({ + name: input.name, + goal: 'Group related dashboard range changes semantically across implementation and test files.', + }), + }, + { + role: 'assistant', + content: normalizeContent(output), + }, + ], + provider: DEFAULT_EVAL_RUNTIME, + model, + }, + usage: {}, + timings: { totalMs: Date.now() - startTime }, + artifacts: {}, + errors: [], + }; + } finally { + await rm(repoPath, { recursive: true, force: true }); + } }, }; } diff --git a/packages/warden/src/cli/output/tasks.test.ts b/packages/warden/src/cli/output/tasks.test.ts index 6269c83d..27e9b94b 100644 --- a/packages/warden/src/cli/output/tasks.test.ts +++ b/packages/warden/src/cli/output/tasks.test.ts @@ -12,6 +12,7 @@ import { Semaphore, runPool } from '../../utils/index.js'; import { SkillRunnerError, WardenAuthenticationError } from '../../sdk/errors.js'; import { ProviderFailureCircuitBreaker } from '../../sdk/circuit-breaker.js'; import * as sdkRunner from '../../sdk/runner.js'; +import * as semanticPlanner from '../../semantic/index.js'; function makeFinding(overrides: Partial = {}): Finding { return { @@ -775,7 +776,7 @@ describe('runSkillTasks', () => { files: [{ filename: 'a.ts', chunks: [fakeChunk] }], skippedFiles: [], }); - const planSemanticReviewChunks = vi.spyOn(sdkRunner, 'planSemanticReviewChunks').mockResolvedValue({ + const planSemanticReviewChunks = vi.spyOn(semanticPlanner, 'planSemanticReviewChunks').mockResolvedValue({ groups: [{ displayName: 'a.ts', filenames: ['a.ts'], chunks: [fakeChunk] }], usage: { inputTokens: 10, outputTokens: 5, costUSD: 0.001 }, }); @@ -820,6 +821,85 @@ describe('runSkillTasks', () => { analyzeFile.mockRestore(); }); + it('keys shared semantic planning by primary model and not auxiliary model', async () => { + const fakeChunk = makeReviewChunk('a.ts', 1, 1); + const context = { + eventType: 'pull_request', + repository: { owner: 'o', name: 'n', fullName: 'o/n', defaultBranch: 'main' }, + repoPath: '/tmp', + pullRequest: { + number: 1, + title: 't', + body: '', + author: 'octocat', + baseBranch: 'main', + headBranch: 'feature', + headSha: 'abc', + baseSha: 'def', + files: [{ + filename: 'a.ts', + status: 'modified', + additions: 1, + deletions: 1, + patch: '@@ -1,1 +1,1 @@\n-old\n+new', + }], + }, + } as unknown as SkillTaskOptions['context']; + const baseRunnerOptions: SkillTaskOptions['runnerOptions'] = { + model: 'primary-a', + chunking: { semantic: { enabled: true } }, + postProcessFindings: false, + }; + const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ + files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + skippedFiles: [], + }); + const planSemanticReviewChunks = vi.spyOn(semanticPlanner, 'planSemanticReviewChunks').mockResolvedValue({ + groups: [{ displayName: 'a.ts', filenames: ['a.ts'], chunks: [fakeChunk] }], + usage: { inputTokens: 10, outputTokens: 5, costUSD: 0.001 }, + }); + const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ + filename: 'a.ts', + findings: [], + usage: { inputTokens: 1, outputTokens: 1, costUSD: 0.001 }, + failedHunks: 0, + failedExtractions: 0, + hunkFailures: [], + } satisfies FileAnalysisResult); + + await runSkillTasks([ + { + name: 'skill-a', + resolveSkill: async () => ({ name: 'skill-a', description: '', prompt: '' }), + context, + runnerOptions: { ...baseRunnerOptions, auxiliaryModel: 'aux-a' }, + }, + { + name: 'skill-b', + resolveSkill: async () => ({ name: 'skill-b', description: '', prompt: '' }), + context, + runnerOptions: { ...baseRunnerOptions, auxiliaryModel: 'aux-b', auxiliaryMaxRetries: 99 }, + }, + { + name: 'skill-c', + resolveSkill: async () => ({ name: 'skill-c', description: '', prompt: '' }), + context, + runnerOptions: { ...baseRunnerOptions, model: 'primary-b', auxiliaryModel: 'aux-a' }, + }, + ], { + mode: logMode(), + verbosity: Verbosity.Quiet, + concurrency: 3, + }); + + expect(prepareFiles).toHaveBeenCalledTimes(2); + expect(planSemanticReviewChunks).toHaveBeenCalledTimes(2); + + prepareFiles.mockRestore(); + planSemanticReviewChunks.mockRestore(); + analyzeFile.mockRestore(); + }); + it('fail-fast aborts after final findings are post-processed', async () => { const finalFinding = makeFinding(); const controller = new AbortController(); diff --git a/packages/warden/src/cli/output/tasks.ts b/packages/warden/src/cli/output/tasks.ts index 34316402..f29011ed 100644 --- a/packages/warden/src/cli/output/tasks.ts +++ b/packages/warden/src/cli/output/tasks.ts @@ -15,7 +15,6 @@ import { aggregateUsage, aggregateAuxiliaryUsage, postProcessFindings, - planSemanticReviewChunks, generateSummary, type AuxiliaryUsageEntry, type SkillRunnerOptions, @@ -25,6 +24,7 @@ import { type ChunkAnalysisResult, type FindingProcessingEvent, } from '../../sdk/runner.js'; +import { planSemanticReviewChunks } from '../../semantic/index.js'; import { ProviderFailureCircuitBreaker } from '../../sdk/circuit-breaker.js'; import { buildFileReports } from '../../sdk/report-files.js'; import chalk from 'chalk'; @@ -277,11 +277,13 @@ async function prepareTaskFiles( enabled: runnerOptions.chunking?.semantic?.enabled, apiKey: runnerOptions.apiKey, runtime: runnerOptions.runtime, - model: runnerOptions.auxiliaryModel ?? runnerOptions.model, - maxRetries: runnerOptions.auxiliaryMaxRetries, + model: runnerOptions.model, maxChunks: runnerOptions.chunking?.semantic?.maxChunks, maxChunkChars: runnerOptions.chunking?.semantic?.maxChunkChars, maxHunksPerChunk: runnerOptions.chunking?.semantic?.maxHunksPerChunk, + maxEmbeddedDiffChars: runnerOptions.chunking?.semantic?.maxEmbeddedDiffChars, + maxEmbeddedDiffChunks: runnerOptions.chunking?.semantic?.maxEmbeddedDiffChunks, + maxEmbeddedDiffRanges: runnerOptions.chunking?.semantic?.maxEmbeddedDiffRanges, }); return { @@ -1069,8 +1071,7 @@ function semanticPlanKey(task: SkillTaskOptions): string | undefined { chunking: task.runnerOptions.chunking, apiKey: Boolean(task.runnerOptions.apiKey), runtime: task.runnerOptions.runtime, - model: task.runnerOptions.auxiliaryModel ?? task.runnerOptions.model, - auxiliaryMaxRetries: task.runnerOptions.auxiliaryMaxRetries, + model: task.runnerOptions.model, }); } diff --git a/packages/warden/src/config/loader.test.ts b/packages/warden/src/config/loader.test.ts index f72a831f..5a795944 100644 --- a/packages/warden/src/config/loader.test.ts +++ b/packages/warden/src/config/loader.test.ts @@ -615,7 +615,16 @@ describe('mergeWardenConfigs', () => { chunking: { filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], coalesce: { maxGapLines: 5, maxChunkSize: 2000, enabled: true }, - semantic: { enabled: true, maxChunks: 10, maxChunkChars: 12000, maxHunksPerChunk: 25, preferWholeFileBelowLines: 500 }, + semantic: { + enabled: true, + maxChunks: 10, + maxChunkChars: 12000, + maxHunksPerChunk: 25, + maxEmbeddedDiffChars: 4000, + maxEmbeddedDiffChunks: 8, + maxEmbeddedDiffRanges: 8, + preferWholeFileBelowLines: 500, + }, maxContextFiles: 10, }, }, @@ -645,7 +654,16 @@ describe('mergeWardenConfigs', () => { { pattern: '**/*.snap', mode: 'skip' }, ], coalesce: { enabled: true, maxGapLines: 5, maxChunkSize: 2000 }, - semantic: { enabled: true, maxChunks: 10, maxChunkChars: 12000, maxHunksPerChunk: 25, preferWholeFileBelowLines: 500 }, + semantic: { + enabled: true, + maxChunks: 10, + maxChunkChars: 12000, + maxHunksPerChunk: 25, + maxEmbeddedDiffChars: 4000, + maxEmbeddedDiffChunks: 8, + maxEmbeddedDiffRanges: 8, + preferWholeFileBelowLines: 500, + }, maxContextFiles: 10, }, }); @@ -661,6 +679,7 @@ describe('mergeWardenConfigs', () => { enabled: true, maxChunks: 20, maxChunkChars: 50000, + maxEmbeddedDiffChars: 8000, }, }, }, @@ -672,6 +691,7 @@ describe('mergeWardenConfigs', () => { chunking: { semantic: { maxChunks: 8, + maxEmbeddedDiffRanges: 6, }, }, }, @@ -682,6 +702,8 @@ describe('mergeWardenConfigs', () => { enabled: true, maxChunks: 8, maxChunkChars: 50000, + maxEmbeddedDiffChars: 8000, + maxEmbeddedDiffRanges: 6, }); }); diff --git a/packages/warden/src/config/schema.ts b/packages/warden/src/config/schema.ts index 65753f61..b1b41d13 100644 --- a/packages/warden/src/config/schema.ts +++ b/packages/warden/src/config/schema.ts @@ -189,6 +189,12 @@ export const SemanticChunkingConfigSchema = z.object({ maxChunkChars: z.number().int().positive().optional(), /** Maximum atomic hunks to group into one semantic review chunk */ maxHunksPerChunk: z.number().int().positive().optional(), + /** Maximum total hunk content characters to embed in the semantic planner prompt */ + maxEmbeddedDiffChars: z.number().int().nonnegative().optional(), + /** Maximum prepared chunks allowed before the semantic planner stops embedding full hunk content */ + maxEmbeddedDiffChunks: z.number().int().nonnegative().optional(), + /** Maximum changed ranges allowed before the semantic planner stops embedding full hunk content */ + maxEmbeddedDiffRanges: z.number().int().nonnegative().optional(), /** Reserved for whole-file materialization below this line count */ preferWholeFileBelowLines: z.number().int().positive().optional(), }); diff --git a/packages/warden/src/sdk/analyze.ts b/packages/warden/src/sdk/analyze.ts index 4da6a1b2..78cbf964 100644 --- a/packages/warden/src/sdk/analyze.ts +++ b/packages/warden/src/sdk/analyze.ts @@ -28,7 +28,7 @@ import { type ChunkAnalysisResult, } from './types.js'; import { prepareFiles } from './prepare.js'; -import { planSemanticReviewChunks } from './semantic-chunk-planner.js'; +import { planSemanticReviewChunks } from '../semantic/index.js'; import type { EventContext, SkillReport, UsageStats, HunkFailure, HunkTrace } from '../types/index.js'; import type { SourceSnippet } from '../types/index.js'; import { runPool } from '../utils/index.js'; @@ -1033,11 +1033,13 @@ async function runSkillAnalysis( enabled: options.chunking?.semantic?.enabled, apiKey: options.apiKey, runtime: options.runtime, - model: options.auxiliaryModel ?? options.model, - maxRetries: options.auxiliaryMaxRetries, + model: options.model, maxChunks: options.chunking?.semantic?.maxChunks, maxChunkChars: options.chunking?.semantic?.maxChunkChars, maxHunksPerChunk: options.chunking?.semantic?.maxHunksPerChunk, + maxEmbeddedDiffChars: options.chunking?.semantic?.maxEmbeddedDiffChars, + maxEmbeddedDiffChunks: options.chunking?.semantic?.maxEmbeddedDiffChunks, + maxEmbeddedDiffRanges: options.chunking?.semantic?.maxEmbeddedDiffRanges, }); const chunkGroups = semanticPlan.groups; @@ -1068,7 +1070,7 @@ async function runSkillAnalysis( allAuxiliaryUsage.push({ agent: 'semantic-chunk-planner', usage: semanticPlan.usage, - model: options.auxiliaryModel ?? options.model, + model: options.model, runtime: options.runtime, }); } diff --git a/packages/warden/src/sdk/runner.ts b/packages/warden/src/sdk/runner.ts index f1d4ba5a..e2aafd44 100644 --- a/packages/warden/src/sdk/runner.ts +++ b/packages/warden/src/sdk/runner.ts @@ -57,7 +57,6 @@ export type { // Re-export file preparation export { prepareFiles } from './prepare.js'; -export { planSemanticReviewChunks } from './semantic-chunk-planner.js'; // Re-export verification utilities export { verifyFindings } from './verify.js'; diff --git a/packages/warden/src/semantic/index.ts b/packages/warden/src/semantic/index.ts new file mode 100644 index 00000000..8772b598 --- /dev/null +++ b/packages/warden/src/semantic/index.ts @@ -0,0 +1,8 @@ +export { + planSemanticReviewChunks, +} from './planner.js'; + +export type { + SemanticChunkPlanningOptions, + SemanticChunkPlanningResult, +} from './planner.js'; diff --git a/packages/warden/src/sdk/semantic-chunk-planner.test.ts b/packages/warden/src/semantic/planner.test.ts similarity index 67% rename from packages/warden/src/sdk/semantic-chunk-planner.test.ts rename to packages/warden/src/semantic/planner.test.ts index 5d49f334..cfe68cbc 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.test.ts +++ b/packages/warden/src/semantic/planner.test.ts @@ -1,13 +1,13 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import type { EventContext } from '../types/index.js'; -import { prepareFiles } from './prepare.js'; -import { planSemanticReviewChunks } from './semantic-chunk-planner.js'; -import type { Runtime } from './runtimes/index.js'; -import type * as RuntimeModule from './runtimes/index.js'; +import { prepareFiles } from '../sdk/prepare.js'; +import { planSemanticReviewChunks } from './planner.js'; +import type { Runtime } from '../sdk/runtimes/index.js'; +import type * as RuntimeModule from '../sdk/runtimes/index.js'; const runAuxiliary = vi.fn(); -vi.mock('./runtimes/index.js', async (importOriginal) => { +vi.mock('../sdk/runtimes/index.js', async (importOriginal) => { const actual = await importOriginal(); return { ...actual, @@ -131,10 +131,138 @@ describe('planSemanticReviewChunks', () => { expect(runAuxiliary).toHaveBeenCalledWith(expect.objectContaining({ task: 'semantic_chunking', agentName: 'semantic-chunk-planner', + model: 'anthropic/claude-sonnet-4-6', + tools: expect.arrayContaining([ + expect.objectContaining({ name: 'read_review_chunk' }), + expect.objectContaining({ name: 'read_changed_file' }), + expect.objectContaining({ name: 'search_repo' }), + ]), + executeTool: expect.any(Function), + maxIterations: 5, })); const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; - expect(prompt).toContain('Group atomic git chunks into semantic review chunks.'); - expect(prompt).toContain('Do not restate filenames or line ranges.'); + expect(prompt).toContain('Changed-line preview:'); + expect(prompt).toContain('Embedded small diff:'); + expect(prompt).toContain('@@ -10,1 +10,1 @@'); + }); + + it('keeps many-hunk planner prompts metadata-first and tool-backed', async () => { + const context = makeContext(); + context.pullRequest!.files = [{ + filename: 'src/dashboard.ts', + status: 'modified', + additions: 13, + deletions: 13, + chunks: 13, + patch: Array.from({ length: 13 }, (_, index) => { + const line = index + 1; + return [ + `@@ -${line},1 +${line},1 @@`, + `-const value${line} = getDefaultRange(widget);`, + `+const value${line} = widget.axisRange ?? getDefaultRange(widget);`, + ].join('\n'); + }).join('\n'), + }]; + const prepared = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + }); + const chunkIds = prepared.files.flatMap((file) => file.chunks.map((chunk) => chunk.id)); + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard range values now prefer widget-provided axis ranges across repeated dashboard call sites.', + chunkIds, + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + model: 'anthropic/claude-sonnet-4-6', + }); + + const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; + expect(prompt).toContain('Changed-line preview:'); + expect(prompt).not.toContain('Embedded small diff:'); + expect(prompt).not.toContain('@@ -1,1 +1,1 @@'); + const executeTool = runAuxiliary.mock.calls[0]?.[0].executeTool as ( + name: string, + input: Record, + ) => Promise; + await expect(executeTool('read_review_chunk', { chunkId: chunkIds[0] })) + .resolves.toContain('@@ -1,1 +1,1 @@'); + }); + + it('uses configured embedded diff limits for smaller-context models', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context, { + chunking: { + semantic: { + enabled: true, + maxChunks: 20, + maxChunkChars: 30000, + maxHunksPerChunk: 50, + preferWholeFileBelowLines: 800, + }, + }, + }); + + await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + model: 'small-context-model', + maxEmbeddedDiffChars: 0, + maxEmbeddedDiffChunks: 0, + maxEmbeddedDiffRanges: 0, + }); + + const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; + expect(prompt).toContain('0 chars, 0 chunks, 0 changed ranges'); + expect(prompt).toContain('Changed-line preview:'); + expect(prompt).not.toContain('Embedded small diff:'); + expect(prompt).not.toContain('@@ -10,1 +10,1 @@'); }); it('splits planned groups that exceed maxChunkChars after materialization', async () => { diff --git a/packages/warden/src/sdk/semantic-chunk-planner.ts b/packages/warden/src/semantic/planner.ts similarity index 55% rename from packages/warden/src/sdk/semantic-chunk-planner.ts rename to packages/warden/src/semantic/planner.ts index 3b07f69a..f1c5e18f 100644 --- a/packages/warden/src/sdk/semantic-chunk-planner.ts +++ b/packages/warden/src/semantic/planner.ts @@ -1,10 +1,11 @@ import { z } from 'zod'; import type { EventContext, UsageStats } from '../types/index.js'; -import type { ReviewChunk } from '../diff/index.js'; -import { SkillRunnerError } from './errors.js'; -import { getRuntime } from './runtimes/index.js'; -import type { RuntimeName } from './runtimes/index.js'; -import type { PreparedFile, ReviewChunkGroup } from './types.js'; +import type { ChangedLineRange, ReviewChunk } from '../diff/index.js'; +import { SkillRunnerError } from '../sdk/errors.js'; +import { getRuntime } from '../sdk/runtimes/index.js'; +import type { RuntimeName } from '../sdk/runtimes/index.js'; +import type { PreparedFile, ReviewChunkGroup } from '../sdk/types.js'; +import { createSemanticPlannerToolExecutor, SEMANTIC_PLANNER_TOOLS } from './tools.js'; const SemanticChunkPlanSchema = z.object({ groups: z.array(z.object({ @@ -14,15 +15,49 @@ const SemanticChunkPlanSchema = z.object({ })), }); +const MAX_EMBEDDED_DIFF_CHARS = 8000; +const MAX_EMBEDDED_DIFF_CHUNKS = 12; +const MAX_EMBEDDED_DIFF_RANGES = 12; + export interface SemanticChunkPlanningOptions { enabled?: boolean; apiKey?: string; runtime?: RuntimeName; model?: string; - maxRetries?: number; maxChunks?: number; maxChunkChars?: number; maxHunksPerChunk?: number; + maxEmbeddedDiffChars?: number; + maxEmbeddedDiffChunks?: number; + maxEmbeddedDiffRanges?: number; +} + +interface AtomicHunkSummary { + id: string; + path: string; + language?: string; + changedRanges: ChangedLineRange[]; + title: string; + contentMode: ReviewChunk['files'][number]['contentMode']; + additions: number; + deletions: number; + changedLinePreview: string[]; + embeddedDiff?: string; +} + +interface SemanticChunkLimits { + maxChunks: number; + maxChunkChars: number; + maxHunksPerChunk: number; + maxEmbeddedDiffChars: number; + maxEmbeddedDiffChunks: number; + maxEmbeddedDiffRanges: number; +} + +interface SemanticChunkPlannerInput { + context: EventContext; + chunks: AtomicHunkSummary[]; + limits: SemanticChunkLimits; } export interface SemanticChunkPlanningResult { @@ -30,27 +65,133 @@ export interface SemanticChunkPlanningResult { usage?: UsageStats; } -function formatChangedRanges(chunk: ReviewChunk): string { - return chunk.changedLineMap - .map((range) => `${range.path}:${range.start}-${range.end}`) - .join(', '); +function countChangedLines(chunk: ReviewChunk, prefix: '+' | '-'): number { + return chunk.files.reduce((sum, file) => { + const matches = file.content.match(new RegExp(`^\\${prefix}(?!\\${prefix}|\\+\\+)`, 'gm')); + return sum + (matches?.length ?? 0); + }, 0); } -function formatChunkForPlanning(chunk: ReviewChunk): string { - const fileSummaries = chunk.files.map((file) => [ - `File: ${file.path}`, - `Language: ${file.language}`, - `Content mode: ${file.contentMode}`, - 'Content:', - file.content, - ].join('\n')).join('\n\n'); +function compactChangedLinePreview(chunk: ReviewChunk, maxLines = 8): string[] { + const preview: string[] = []; + for (const file of chunk.files) { + for (const line of file.content.split('\n')) { + if ((!line.startsWith('+') && !line.startsWith('-')) || line.startsWith('+++') || line.startsWith('---')) { + continue; + } + preview.push(line.length > 180 ? `${line.slice(0, 177)}...` : line); + if (preview.length >= maxLines) return preview; + } + } + return preview; +} - return [ - `Chunk ID: ${chunk.id}`, - `Title: ${chunk.title}`, - `Changed ranges: ${formatChangedRanges(chunk)}`, - fileSummaries, - ].join('\n'); +function atomicHunkSummaryFromChunk(chunk: ReviewChunk): AtomicHunkSummary { + const firstFile = chunk.files[0]; + return { + id: chunk.id, + path: firstFile.path, + language: firstFile.language, + changedRanges: chunk.changedLineMap, + title: chunk.title, + contentMode: firstFile.contentMode, + additions: countChangedLines(chunk, '+'), + deletions: countChangedLines(chunk, '-'), + changedLinePreview: compactChangedLinePreview(chunk), + }; +} + +function shouldEmbedDiffs(chunks: ReviewChunk[], limits: SemanticChunkLimits): boolean { + if (chunks.length > limits.maxEmbeddedDiffChunks) return false; + const changedRanges = chunks.reduce((sum, chunk) => sum + chunk.changedLineMap.length, 0); + if (changedRanges > limits.maxEmbeddedDiffRanges) return false; + const totalChars = chunks.reduce( + (sum, chunk) => sum + chunk.files.reduce((fileSum, file) => fileSum + file.content.length, 0), + 0, + ); + return totalChars <= limits.maxEmbeddedDiffChars; +} + +function atomicHunkSummariesFromChunks(chunks: ReviewChunk[], limits: SemanticChunkLimits): AtomicHunkSummary[] { + const embedDiffs = shouldEmbedDiffs(chunks, limits); + return chunks.map((chunk) => { + const summary = atomicHunkSummaryFromChunk(chunk); + if (!embedDiffs) return summary; + + return { + ...summary, + embeddedDiff: chunk.files.map((file) => [ + `File: ${file.path}`, + file.content, + ].join('\n')).join('\n\n'), + }; + }); +} + +function formatFileInventory(context: EventContext): string { + return (context.pullRequest?.files ?? []).map((file) => [ + `${file.filename} (${file.status})`, + ` additions: ${file.additions}`, + ` deletions: ${file.deletions}`, + ` hunks: ${file.chunks}`, + ].join('\n')).join('\n'); +} + +function semanticChunkLimits(options: SemanticChunkPlanningOptions): SemanticChunkLimits { + return { + maxChunks: options.maxChunks ?? 20, + maxChunkChars: options.maxChunkChars ?? 30000, + maxHunksPerChunk: options.maxHunksPerChunk ?? 50, + maxEmbeddedDiffChars: options.maxEmbeddedDiffChars ?? MAX_EMBEDDED_DIFF_CHARS, + maxEmbeddedDiffChunks: options.maxEmbeddedDiffChunks ?? MAX_EMBEDDED_DIFF_CHUNKS, + maxEmbeddedDiffRanges: options.maxEmbeddedDiffRanges ?? MAX_EMBEDDED_DIFF_RANGES, + }; +} + +function buildSemanticChunkPlannerInput( + context: EventContext, + chunks: ReviewChunk[], + options: SemanticChunkPlanningOptions, +): SemanticChunkPlannerInput { + const limits = semanticChunkLimits(options); + return { + context, + chunks: atomicHunkSummariesFromChunks(chunks, limits), + limits, + }; +} + +function formatPlannerInput(input: SemanticChunkPlannerInput): string { + const pr = input.context.pullRequest; + const sections = [ + `Repository: ${input.context.repository.fullName}`, + pr ? `Pull request title: ${pr.title}` : undefined, + pr?.body ? `Pull request body: ${pr.body}` : undefined, + formatFileInventory(input.context) + ? ['Changed files:', formatFileInventory(input.context)].join('\n') + : undefined, + [ + 'Atomic chunks:', + input.chunks.map((chunk) => [ + `Chunk ID: ${chunk.id}`, + `Path: ${chunk.path}`, + chunk.language ? `Language: ${chunk.language}` : undefined, + `Title: ${chunk.title}`, + `Changed ranges: ${chunk.changedRanges.map((range) => `${range.path}:${range.start}-${range.end}`).join(', ')}`, + `Content mode: ${chunk.contentMode}`, + `Additions: ${chunk.additions}`, + `Deletions: ${chunk.deletions}`, + chunk.changedLinePreview.length > 0 + ? ['Changed-line preview:', ...chunk.changedLinePreview].join('\n') + : undefined, + chunk.embeddedDiff + ? ['Embedded small diff:', chunk.embeddedDiff].join('\n') + : undefined, + ].filter((line): line is string => Boolean(line)).join('\n')).join('\n\n---\n\n'), + ].join('\n'), + ]; + + return sections.filter((section): section is string => Boolean(section)).join('\n\n'); } function buildSemanticChunkPlanningPrompt( @@ -58,10 +199,8 @@ function buildSemanticChunkPlanningPrompt( chunks: ReviewChunk[], options: SemanticChunkPlanningOptions ): string { - const pr = context.pullRequest; - const maxChunks = options.maxChunks ?? 20; - const maxHunksPerChunk = options.maxHunksPerChunk ?? 50; - const maxChunkChars = options.maxChunkChars ?? 30000; + const plannerInput = buildSemanticChunkPlannerInput(context, chunks, options); + const { maxChunks, maxHunksPerChunk, maxChunkChars } = plannerInput.limits; const header = [ 'You are planning code review chunks for Warden.', '', @@ -69,15 +208,16 @@ function buildSemanticChunkPlanningPrompt( 'A semantic chunk should contain changes that a reviewer should understand together: one behavior change, API contract, data flow, migration, validation rule, or test expectation.', 'For each planned group, write a semantic delta summary: what behavior, contract, data flow, API, or test expectation changed.', 'Do not restate filenames or line ranges. Do not say "lines 10, 100, 200".', + 'Do not summarize the input shape. Explain the product, security, data-flow, API, or test behavior being changed.', 'Keep each summary one sentence. Be concrete enough that a scanner knows why the grouped changes belong together.', + 'Inspect the changed code with tools before finalizing the plan, especially when grouping across files or distant hunks.', + 'Use read_review_chunk to inspect exact hunk content for non-embedded chunks; read_changed_file only shows current head content.', + 'If embedded small diffs are present, use them as initial evidence and use tools only for missing relationships or context.', `Target at most ${maxChunks} planned groups.`, `Use at most ${maxHunksPerChunk} atomic chunks per group.`, - `Keep each planned group under roughly ${maxChunkChars} characters of provided content.`, + `Keep each planned group under roughly ${maxChunkChars} characters once materialized for scanning.`, + `Embedded small diffs are only present when the changeset fits these planner limits: ${plannerInput.limits.maxEmbeddedDiffChars} chars, ${plannerInput.limits.maxEmbeddedDiffChunks} chunks, ${plannerInput.limits.maxEmbeddedDiffRanges} changed ranges.`, 'Every input Chunk ID must appear in exactly one group. Do not invent Chunk IDs.', - '', - `Repository: ${context.repository.fullName}`, - pr ? `Pull request title: ${pr.title}` : undefined, - pr?.body ? `Pull request body: ${pr.body}` : undefined, ].filter((line): line is string => Boolean(line)); return [ @@ -86,8 +226,7 @@ function buildSemanticChunkPlanningPrompt( 'Return JSON with this exact shape:', '{"groups":[{"title":"semantic title","summary":"semantic delta summary","chunkIds":["chunk id"]}]}', '', - 'Atomic chunks:', - chunks.map(formatChunkForPlanning).join('\n\n---\n\n'), + formatPlannerInput(plannerInput), ].join('\n'); } @@ -226,9 +365,7 @@ function groupFromPlannedChunk(chunk: ReviewChunk): ReviewChunkGroup { }; } -/** - * Populate ReviewChunk semantic summaries with a model-backed planning pass. - */ +/** Plan and materialize semantic review chunk groups with a model-backed planning pass. */ export async function planSemanticReviewChunks( files: PreparedFile[], context: EventContext, @@ -246,8 +383,10 @@ export async function planSemanticReviewChunks( apiKey: options.apiKey, prompt: buildSemanticChunkPlanningPrompt(context, chunks, options), schema: SemanticChunkPlanSchema, + tools: SEMANTIC_PLANNER_TOOLS, + executeTool: createSemanticPlannerToolExecutor(context, chunks), + maxIterations: 5, model: options.model, - maxRetries: options.maxRetries, }); if (!result.success) { diff --git a/packages/warden/src/semantic/tools.test.ts b/packages/warden/src/semantic/tools.test.ts new file mode 100644 index 00000000..8c8fa193 --- /dev/null +++ b/packages/warden/src/semantic/tools.test.ts @@ -0,0 +1,93 @@ +import { mkdtemp, rm, writeFile, mkdir } from 'node:fs/promises'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import type { EventContext } from '../types/index.js'; +import type { ReviewChunk } from '../diff/index.js'; +import { createSemanticPlannerToolExecutor } from './tools.js'; + +describe('createSemanticPlannerToolExecutor', () => { + let tempDir: string; + + beforeEach(async () => { + tempDir = await mkdtemp(join(tmpdir(), 'warden-semantic-tools-')); + await mkdir(join(tempDir, 'src'), { recursive: true }); + await writeFile(join(tempDir, 'src/dashboard.ts'), 'export const axisRange = widget.axisRange;\n'); + await writeFile(join(tempDir, 'src/secret.ts'), 'export const unrelated = true;\n'); + }); + + afterEach(async () => { + await rm(tempDir, { recursive: true, force: true }); + }); + + function makeContext(): EventContext { + return { + eventType: 'pull_request', + action: 'opened', + repository: { + owner: 'qa', + name: 'repo', + fullName: 'qa/repo', + defaultBranch: 'main', + }, + repoPath: tempDir, + pullRequest: { + number: 1, + title: 'Preserve dashboard range', + body: '', + author: 'qa', + baseBranch: 'main', + headBranch: 'feature', + headSha: 'head', + baseSha: 'base', + files: [{ + filename: 'src/dashboard.ts', + status: 'modified', + additions: 1, + deletions: 1, + patch: '@@ -1,1 +1,1 @@\n-old\n+new', + }], + }, + }; + } + + it('only reads changed files inside the repository', async () => { + const executeTool = createSemanticPlannerToolExecutor(makeContext(), []); + + await expect(executeTool('read_changed_file', { path: 'src/dashboard.ts' })) + .resolves.toContain('axisRange'); + await expect(executeTool('read_changed_file', { path: 'src/secret.ts' })) + .resolves.toBe('Refusing to read a file that is not in the changed file list'); + await expect(executeTool('read_changed_file', { path: '../outside.ts' })) + .resolves.toBe('Refusing to read a file that is not in the changed file list'); + }); + + it('searches the repository with capped output', async () => { + const executeTool = createSemanticPlannerToolExecutor(makeContext(), []); + + await expect(executeTool('search_repo', { query: 'axisRange' })) + .resolves.toContain('src/dashboard.ts:1'); + }); + + it('reads exact review chunk diffs by chunk id', async () => { + const chunk: ReviewChunk = { + id: 'src/dashboard.ts:1', + title: 'src/dashboard.ts:1', + changedLineMap: [{ path: 'src/dashboard.ts', start: 1, end: 1 }], + files: [{ + path: 'src/dashboard.ts', + language: 'typescript', + changedRanges: [{ path: 'src/dashboard.ts', start: 1, end: 1 }], + content: '@@ -1,1 +1,1 @@\n-old\n+new', + contentMode: 'raw-hunks', + sourceLines: [{ line: 1, content: 'new' }], + }], + }; + const executeTool = createSemanticPlannerToolExecutor(makeContext(), [chunk]); + + await expect(executeTool('read_review_chunk', { chunkId: 'src/dashboard.ts:1' })) + .resolves.toContain('-old'); + await expect(executeTool('read_review_chunk', { chunkId: 'missing' })) + .resolves.toBe('Unknown review chunk ID'); + }); +}); diff --git a/packages/warden/src/semantic/tools.ts b/packages/warden/src/semantic/tools.ts new file mode 100644 index 00000000..ae8bfb80 --- /dev/null +++ b/packages/warden/src/semantic/tools.ts @@ -0,0 +1,156 @@ +import { execFile } from 'node:child_process'; +import { readFile } from 'node:fs/promises'; +import { resolve, relative, isAbsolute } from 'node:path'; +import { promisify } from 'node:util'; +import { z } from 'zod'; +import type { EventContext } from '../types/index.js'; +import type { ReviewChunk } from '../diff/index.js'; +import type { AuxiliaryTool } from '../sdk/runtimes/index.js'; + +const execFileAsync = promisify(execFile); +const MAX_FILE_CHARS = 20000; +const MAX_SEARCH_CHARS = 12000; +const MAX_CHUNK_DIFF_CHARS = 12000; + +export const SEMANTIC_PLANNER_TOOLS: AuxiliaryTool[] = [ + { + name: 'read_review_chunk', + description: 'Read exact diff content for one atomic review chunk by Chunk ID. Use this to inspect the actual delta without embedding the whole patch.', + inputSchema: { + type: 'object' as const, + properties: { + chunkId: { type: 'string', description: 'Atomic review chunk ID from the planner input' }, + }, + required: ['chunkId'], + }, + }, + { + name: 'read_changed_file', + description: 'Read the current head content for a changed file in this pull request. Output is capped.', + inputSchema: { + type: 'object' as const, + properties: { + path: { type: 'string', description: 'Changed file path to read' }, + }, + required: ['path'], + }, + }, + { + name: 'search_repo', + description: 'Search the repository with ripgrep for imports, symbols, and usage relationships. Output is capped.', + inputSchema: { + type: 'object' as const, + properties: { + query: { type: 'string', description: 'Literal or regex query to search for' }, + path: { type: 'string', description: 'Optional path or directory to limit the search' }, + }, + required: ['query'], + }, + }, +]; + +const ReadChangedFileInput = z.object({ + path: z.string().min(1), +}); + +const ReadReviewChunkInput = z.object({ + chunkId: z.string().min(1), +}); + +const SearchRepoInput = z.object({ + query: z.string().min(1).max(200), + path: z.string().min(1).optional(), +}); + +function safeRepoPath(repoPath: string, inputPath: string): string | undefined { + if (isAbsolute(inputPath)) return undefined; + const absolute = resolve(repoPath, inputPath); + const repoRelative = relative(repoPath, absolute); + if (repoRelative.startsWith('..') || isAbsolute(repoRelative)) return undefined; + return absolute; +} + +function capText(text: string, maxChars: number): string { + if (text.length <= maxChars) return text; + return `${text.slice(0, maxChars)}\n[truncated at ${maxChars} chars]`; +} + +function errorStdout(error: unknown): string { + if ( + typeof error === 'object' + && error !== null + && 'stdout' in error + && ((typeof error.stdout === 'string') || Buffer.isBuffer(error.stdout)) + ) { + return String(error.stdout); + } + return ''; +} + +/** Create the read-only tool executor used by the semantic chunk planner. */ +export function createSemanticPlannerToolExecutor( + context: EventContext, + chunks: ReviewChunk[], +): (name: string, input: Record) => Promise { + const changedFiles = new Set((context.pullRequest?.files ?? []).map((file) => file.filename)); + const chunksById = new Map(chunks.map((chunk) => [chunk.id, chunk])); + + return async (name: string, input: Record): Promise => { + if (name === 'read_review_chunk') { + const parsed = ReadReviewChunkInput.safeParse(input); + if (!parsed.success) return `Invalid input: ${parsed.error.message}`; + + const chunk = chunksById.get(parsed.data.chunkId); + if (!chunk) return 'Unknown review chunk ID'; + + return capText(chunk.files.map((file) => [ + `File: ${file.path}`, + `Changed ranges: ${file.changedRanges.map((range) => `${range.path}:${range.start}-${range.end}`).join(', ')}`, + file.content, + ].join('\n')).join('\n\n---\n\n'), MAX_CHUNK_DIFF_CHARS); + } + + if (name === 'read_changed_file') { + const parsed = ReadChangedFileInput.safeParse(input); + if (!parsed.success) return `Invalid input: ${parsed.error.message}`; + if (!changedFiles.has(parsed.data.path)) return 'Refusing to read a file that is not in the changed file list'; + + const absolutePath = safeRepoPath(context.repoPath, parsed.data.path); + if (!absolutePath) return 'Invalid path'; + + try { + return capText(await readFile(absolutePath, 'utf8'), MAX_FILE_CHARS); + } catch (error) { + return `Unable to read file: ${error instanceof Error ? error.message : String(error)}`; + } + } + + if (name === 'search_repo') { + const parsed = SearchRepoInput.safeParse(input); + if (!parsed.success) return `Invalid input: ${parsed.error.message}`; + + const searchPath = parsed.data.path ? safeRepoPath(context.repoPath, parsed.data.path) : context.repoPath; + if (!searchPath) return 'Invalid path'; + + try { + const { stdout } = await execFileAsync('rg', [ + '--line-number', + '--max-count', + '40', + parsed.data.query, + searchPath, + ], { + cwd: context.repoPath, + maxBuffer: MAX_SEARCH_CHARS * 2, + }); + return capText(stdout, MAX_SEARCH_CHARS); + } catch (error) { + const maybeStdout = errorStdout(error); + if (maybeStdout) return capText(maybeStdout, MAX_SEARCH_CHARS); + return 'No matches found'; + } + } + + return `Unknown tool: ${name}`; + }; +} diff --git a/specs/semantic-chunk-benchmark.md b/specs/semantic-chunk-benchmark.md new file mode 100644 index 00000000..39a3a1c9 --- /dev/null +++ b/specs/semantic-chunk-benchmark.md @@ -0,0 +1,195 @@ +# Semantic Chunk Benchmark Plan + +Semantic chunking should be measured against real changesets where the scanner +needs to understand a logical change that git split across many small hunks. +The benchmark should compare Warden with semantic chunking disabled and enabled +on the same bug-introducing diffs. + +This is a maintainer benchmark plan, not public product documentation. Public +benchmark readouts belong under `packages/docs/src/content/docs/benchmarking/` +after the runner and result format are stable. + +## Goal + +Answer two questions: + +- Does semantic chunking preserve or improve known-finding recall? +- Does it reduce scanner calls, runtime, tokens, or cost on fragmented changes? + +The benchmark should not only prove that the planner can produce summaries. It +must prove that scanner behavior improves or stays correct when the scanner +receives semantic review chunks instead of atomic git-hunk chunks. + +## Dataset Shape + +Use existing Sentry eval cases that already identify: + +- a vulnerable parent commit +- a fixing commit +- the expected finding +- source files tied to the bug + +Run the inverse diff as a bug-introducing change: + +```text +base = fixing commit +head = vulnerable parent/source_ref +``` + +That lets Warden review a realistic pull-request-shaped delta that reintroduces +the known bug. + +Apply the whole fixing commit or PR diff in reverse. Do not limit the synthetic +patch to `notes.source_files`. Those files identify the expected finding +location and provenance, but trimming the patch can remove related tests and +call-site edits. The point of this benchmark is to preserve the real +fragmentation shape. + +The benchmark should run against a real `getsentry/sentry` checkout, not the +current fixture-only eval repository. The semantic planner has read-only tools, +and those tools are only meaningful when surrounding repository context exists. + +## Initial Cases + +Start with these cases from `packages/evals/code-review/`. + +| Case | Vulnerable commit | Fix commit | Shape | Expected finding | +| --- | --- | --- | --- | --- | +| `sentry-dashboard-axis-range-existing-widget` | `1b2028bb7455b62fba59a1eb3b9d00bdcff27d51` | `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` | 6 files, 14 hunks | Existing dashboard widgets lose their saved axis range because builder defaults override it. | +| `sentry-cursor-service-account-api-key` | `dd2e7671013b38550048c3c427b5152997e91ab9` | `652324b48217e89f4267bfc1bb6e5e390818c277` | 3 files, 36 hunks | Cursor service-account API keys fail validation because `/v0/me` is insufficient. | +| `sentry-fixability-missing-issue-summary` | `4199c6aeed84c7c359aa7ad6863534174769d436` | `62125c6514958cd89aa3cf7374be32f984adb683` | 4 files, 20 hunks | Fixability calculation misses the cached issue summary needed by Seer. | +| `sentry-workflow-status-missing-foreign-key` | `e36f46a85cf4a6c9a6ae0e5e545a7c13b789d478` | `f4cc09c52e73c2ab60a3b14291c60dd0db5458a7` | 2 files, 17 hunks | Workflow status processing fails hard on missing or deleted foreign keys. | + +These four cover the core semantic chunking risks: + +- many tiny hunks that describe one behavior change +- cross-file implementation and test updates +- changes that need nearby code inspection +- enough fragmentation to make atomic scanner calls expensive + +## Expansion Cases + +Add these after the initial four work. + +| Case | Vulnerable commit | Fix commit | Shape | Expected finding | +| --- | --- | --- | --- | --- | +| `sentry-dashboard-delete-side-nav-stale` | `14170bd6ae28abe4d4f0b5807185b116f777a1a0` | `86872f4f1491c4392a25f4d3fcba31a12726fa63` | 3 files, 14 hunks | Deleting a dashboard leaves stale side-nav state. | +| `sentry-action-dedup-by-workflow` | `1730e13b97865d3ee8943c6f860964388c4987a8` | `165be911ba388d402993b58f34dc8ad683827e32` | 4 files, 8 hunks | Action dedup keys include workflow id, causing duplicate notifications. | + +## Security Controls + +Use security-review cases as correctness controls, not the main fragmentation +stress set. These are useful because they exercise Warden's security skill, but +most are smaller diffs. + +| Case | Source | Fix commit(s) | Expected finding | +| --- | --- | --- | --- | +| `sentry-preprod-snapshot-project-access` | `https://github.com/getsentry/sentry/pull/114169` | `8fac324d82c903c8022b99dcd4329f3944e57196` | Snapshot detail GET/DELETE bypasses project access. | +| `sentry-slack-options-load-unscoped-group` | `https://github.com/getsentry/sentry/pull/114185` | `0f09491755f71a95343285cbe17c93bf272a0d62`, `b718661dd8560a20a826d90ee6755f153957969c` | Slack options-load resolves a group without verifying the integration belongs to the group's org. | +| `sentry-release-threshold-empty-project-filter` | `https://github.com/getsentry/sentry/pull/114049` | `8a93913509441a0c8e7d035f9c4bc24dabed2d86` | Empty accessible-project filters remove project scoping from release threshold queries. | + +Security cases whose provenance is only a blob SHA should stay out of the first +benchmark until the fixing commit is identified. + +## Run Method + +For each benchmark case: + +1. Create or reuse a temporary Sentry worktree. +2. Check out the fixing commit. +3. Create a benchmark branch. +4. Apply the full inverse fixing diff back to the vulnerable commit. +5. Run Warden with semantic chunking disabled. +6. Reset to the same benchmark branch. +7. Run Warden with semantic chunking enabled. +8. Judge both reports against the existing `should_find` assertion. + +The first implementation can be a one-off script under `packages/evals/scripts/` +or a local maintainer command. Do not add a root `pnpm` script until the output +format and workflow are worth preserving. + +## Metrics + +Record these for each semantic-off and semantic-on run: + +- pass/fail against expected finding +- total findings +- duplicate or near-duplicate findings +- failed chunks +- failed extractions +- wall time +- scanner chunk count +- semantic planner group count +- semantic planner summaries +- input tokens +- output tokens +- recorded cost + +The semantic-on run is successful when it keeps the expected finding and reduces +operational load. A lower chunk count is not enough if recall regresses. + +## Output + +Store raw run output separately from checked-in docs until the format stabilizes. +The final durable artifact should be a small JSON result set that can power a +public docs readout later, similar to the existing Sentry security benchmark +data under `packages/docs/src/data/benchmarking/`. + +At minimum, each result record should include: + +```json +{ + "case": "sentry-dashboard-axis-range-existing-widget", + "repository": "getsentry/sentry", + "base": "01217a6efb90cd26f0f8ac8b33c4f255379fe21d", + "head": "1b2028bb7455b62fba59a1eb3b9d00bdcff27d51", + "semantic": true, + "passed": true, + "findings": 1, + "scannerChunks": 3, + "semanticGroups": 1, + "durationMs": 0, + "usage": { + "inputTokens": 0, + "outputTokens": 0, + "costUSD": 0 + } +} +``` + +## Captured Baseline + +Initial smoke run: + +- date: 2026-06-30 +- case: `sentry-dashboard-axis-range-existing-widget` +- repository: `getsentry/sentry` +- synthetic base: `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` +- synthetic head: `4d022227929c4d923cfb9bb2b45721407570cf8c` +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- skill: `code-review` +- verification: disabled + +This smoke run intentionally used a trimmed two-file reverse patch. It proves +that the benchmark mechanics and semantic grouping path work, but it should not +be used as the dataset quality baseline. + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| semantic off | 2 | 1 | 11m54s | 445,997 | 39,484 | $1.2213 | +| semantic on | 1 | 1 | 3m03s | 160,096 | 10,079 | $0.4609 | + +The semantic-on run grouped both changed ranges into one scanner chunk and kept +the expected medium-confidence axis-range regression finding. + +## Acceptance + +Before using this benchmark to make decisions: + +- every initial case must run both modes from the same synthetic branch +- the runner must preserve enough logs to debug planner grouping decisions +- expected finding judgments must reuse the existing eval judge or equivalent + semantic matching +- semantic-on must expose planner summaries in the artifact +- interrupted or timed-out runs must be marked incomplete, not compared diff --git a/specs/semantic-review-chunks.md b/specs/semantic-review-chunks.md index 653204ac..b11e77b9 100644 --- a/specs/semantic-review-chunks.md +++ b/specs/semantic-review-chunks.md @@ -50,7 +50,7 @@ The target pipeline is: ```text git patch -> atomic hunk inventory - -> semantic chunk planner + -> semantic planning module -> ReviewChunk materialization -> run one scanner call per ReviewChunk -> validate findings against changedLineMap @@ -59,6 +59,53 @@ git patch Git hunks remain the evidence and anchoring primitive. Review chunks become the scanner-facing primitive. +Benchmarking for this work is tracked in +[`semantic-chunk-benchmark.md`](semantic-chunk-benchmark.md). + +## Module Boundary + +Semantic planning should be cleanly encapsulated in its own module because it is +an input/output planning pass, not scanner logic. The rest of Warden should not +know about planner prompts, tool definitions, model selection details, or +planner validation internals. + +Current layout: + +```text +packages/warden/src/semantic/ + index.ts + planner.ts + tools.ts +``` + +Split `inventory.ts`, `materialize.ts`, or `types.ts` out later only if the +module grows enough that the separation removes real complexity. + +Public surface: + +```ts +export interface SemanticChunkPlannerInput { + context: EventContext; + chunks: AtomicHunkSummary[]; + limits: SemanticChunkLimits; +} + +``` + +The integration point remains thin: + +```text +prepareFiles() + -> atomic ReviewChunks +semantic.planSemanticReviewChunks(...) + -> semantic ReviewChunkGroups +run scanner on groups +``` + +`prepareFiles` remains responsible for parsing diffs and creating deterministic +atomic chunks. The semantic module owns only the decision of which atomic chunks +belong together and why. + ## ReviewChunk Contract ```ts @@ -107,16 +154,19 @@ cannot be used as the comment location. Before planning, Warden should normalize parsed hunks into stable atomic units: ```ts -export interface AtomicHunk { +export interface AtomicHunkSummary { id: string; path: string; + language?: string; oldStart: number; oldEnd: number; newStart: number; newEnd: number; header?: string; - changedLines: string[]; - excerpt: string; + symbolHint?: string; + additions: number; + deletions: number; + changedLinePreview?: string[]; } ``` @@ -124,17 +174,76 @@ The planner operates on this inventory. It does not invent changed lines. Each atomic hunk is either assigned to exactly one review chunk or excluded by an existing scan/ignore policy before planning. +The planner should be tool-backed on every semantic run. It should have enough +compact metadata to decide where to inspect, then use read-only tools to inspect +the changed code before finalizing the grouping. + +Full content is still primarily for scanner execution after semantic grouping. +Planning input should stay small enough that the planner can reason over the +whole changeset without reproducing the same token cost that semantic chunking +is meant to avoid. + +Small diffs can be embedded directly as an optimization. The cutoff should be +tunable because model context windows and price/performance profiles vary. A +conservative default is: + +- embed full atomic hunk content only when the total planner diff payload is at + most 8k characters +- embed only when there are at most 12 atomic chunks +- embed only when there are at most 12 changed ranges after deterministic + preparation +- otherwise send compact metadata plus capped changed-line previews and require + tool inspection + +This keeps tiny PRs cheap while preserving the main goal for pathological PRs: +avoid passing a huge fragmented patch to the planner. + +The knobs belong under semantic chunking config, for example: + +```toml +[defaults.chunking.semantic] +maxEmbeddedDiffChars = 8000 +maxEmbeddedDiffChunks = 12 +maxEmbeddedDiffRanges = 12 +``` + +Smaller-context or expensive models can set these lower, including `0` to force +metadata-plus-tools planning. + +Allowed compact inputs include: + +- repository metadata +- PR title and body +- commit messages, when available +- changed file list and file statuses +- atomic chunk ids +- file paths and languages +- changed ranges +- hunk headers and symbol hints +- additions, deletions, and hunk counts +- small capped changed-line previews when useful +- full atomic hunk content only when the whole semantic planning input is below + the small-diff threshold + +Disallowed by default: + +- full rendered hunk content for medium or large changesets +- whole changed files +- scanner skill prompts +- full repository context + ## Planner -The semantic chunk planner groups atomic hunks into review chunks and selects a -content mode for each file in each chunk. +The semantic chunk planner groups atomic hunks into review chunks and writes a +semantic summary for each planned group. Planner input: - repository and PR metadata - PR title and body +- commit messages, when available - changed file list -- atomic hunk inventory with compact excerpts +- atomic hunk inventory with compact metadata - file sizes and line counts where available - hard limits for chunk size and count @@ -142,17 +251,15 @@ Planner output: ```ts export interface PlannedReviewChunk { - id: string; title: string; summary: string; - hunkIds: string[]; - files: Array<{ - path: string; - contentMode: 'whole-file' | 'stitched-file' | 'raw-hunks'; - }>; + chunkIds: string[]; } ``` +Current materialization keeps planned groups as `raw-hunks` review chunks. Whole +file and stitched-file materializers remain future work. + The planner should optimize for scanner usefulness, not for diff prettiness. Good chunks are cohesive enough that the scanner can understand the change in one pass and small enough that the scanner can stay precise. @@ -160,6 +267,35 @@ Cross-file groups are allowed and expected when the files are part of the same semantic delta. A source change and its test assertion should usually stay together if reviewing them independently would hide the behavior change. +The planner should use the primary scanner model by default, not the auxiliary +model. Semantic grouping is a hard reasoning task: it must infer relationships +between files, tests, call sites, and behavior. Auxiliary models remain +appropriate for extraction repair and other bounded post-processing work. + +### Tool-Using Planning Agent + +The planner should be a tool-using read-only agent, not only a single schema +call over a rendered prompt. It should always have tools available and should +inspect code before finalizing a plan. Embedded small diffs can reduce tool +turns, but tools remain the primary way to resolve cross-file relationships, +symbol usage, and behavior. + +Read-only tools should include: + +- read changed file content at head +- read base content or changed-range context +- inspect nearby symbols/functions +- search usages/imports with bounded `rg` +- inspect PR metadata and commit messages + +The planner must not have write tools. It should also not receive an unbounded +repo dump. Tool calls should be driven by grouping uncertainty: inspect enough +code to decide which chunks belong together, then stop. + +The scanner remains responsible for finding issues. The planner is responsible +only for grouping and writing semantic summaries that explain why the grouped +changes should be reviewed together. + ## Materialization Materialization turns a planned chunk into the scanner-facing `ReviewChunk`. @@ -205,6 +341,9 @@ enabled = true maxChunks = 20 maxChunkChars = 30000 maxHunksPerChunk = 50 +maxEmbeddedDiffChars = 8000 +maxEmbeddedDiffChunks = 12 +maxEmbeddedDiffRanges = 12 preferWholeFileBelowLines = 800 ``` @@ -259,19 +398,175 @@ Warden should record enough data to prove whether semantic chunking helps: This should make cost and latency changes visible without reading logs line by line. +## User-Facing Visibility + +Semantic segments may be useful beyond internal execution. Warden should be able +to expose them in debug or verbose output, and possibly in pull request reports, +without making them noisy by default. + +Useful fields to expose: + +- segment title +- semantic summary +- files included +- changed ranges included +- atomic chunk count +- content mode per file +- scanner result count for the segment + +CLI presentation can show a compact plan before scanning when verbose output is +enabled: + +```text +Semantic plan: 7 review chunks from 42 atomic hunks + 1. Enforce project access in token lookup + api/tokens.ts, auth/permissions.ts, api/tokens.test.ts + 2. Update dashboard range validation + dashboard/range.ts, dashboard/range.test.ts +``` + +Pull request presentation should be more conservative. A collapsed report +section can help reviewers understand why findings came from a cross-file +chunk, but inline comments should remain focused on findings. The plan should +not become another review surface unless it clearly helps explain Warden's +coverage and cost. + +## Benchmarking and Evals + +Semantic chunking needs more than a chunk-count benchmark. We need fixtures +where semantic grouping should materially improve or preserve scanner quality. + +Good fixtures: + +- real commit or PR SHA +- known security regression or confirmed product bug +- objective expected finding +- issue spans multiple files or many tiny hunks +- baseline hunk mode is worse in recall, duplicates, cost, or model calls + +Preferred security examples: + +- removed authorization check +- unsafe command execution +- path traversal +- SSRF +- SQL/query injection +- secret exposure +- unsafe deserialization + +Each fixture should run the same scanner twice: + +```text +same fixture + same skill + semantic off +same fixture + same skill + semantic on +``` + +Compare: + +- expected finding found or missed +- severity and confidence +- location correctness +- duplicate findings +- total findings +- chunk count +- model calls +- input and output tokens +- cost +- wall time + +The eval should also inspect planner output directly: + +- at least one expected semantic group exists +- group summary describes behavior, not files or line numbers +- cross-file changes can be grouped when they are one logical change +- unrelated edits remain separate + +The benchmark corpus already supports known Sentry vulnerabilities. We should +reuse that machinery where possible, but semantic chunking needs fixtures +selected for cross-file or many-hunk behavior, not just historical security +coverage. + +## Prior Art Notes + +This space has enough prior art to shape the implementation, but not enough +public detail to copy an architecture directly. + +- CodeRabbit documents context-aware PR review, path-specific instructions, + linked issue context, MCP context, multi-repo analysis, walkthrough comments, + and a review interface that reorganizes a PR into a logical walkthrough + rather than a flat file list: + +- CodeRabbit's path instructions and filters are a useful reminder that semantic + grouping should respect review scope and file-specific guidance. Generated + files, binaries, and lockfiles should be filtered before semantic planning, + and path-scoped review instructions should remain scanner inputs rather than + planner inputs: + +- Qodo describes its current review product as multi-agent PR analysis with + diff plus repository-aware reasoning, ticket-aware context, and a separate + semantic code intelligence layer for repository retrieval and multi-step + reasoning: + +- SWE-PRBench is the strongest warning against "just add more context." Its + results show AI review quality degrading as context expands, even with + structured semantic layers such as AST function context and import graph + resolution. It also found that a compact diff-with-summary context beat a + richer full-context prompt in their setup: + +- MutaGReP supports the same direction from code-use tasks: instead of putting + the whole repo into context, it builds grounded natural-language plans using + retrieval over relevant symbols. The reported plans use a small fraction of a + large context window while preserving task performance: + +- Semantic-aware AST diff work, especially RefactoringMiner-based approaches, + suggests a future deterministic enhancement: detect moved/renamed/refactored + code so the semantic planner starts with better symbol and relationship hints: + +- Empirical AI review studies suggest concise, actionable comments matter. + One study of GitHub Actions review tools found concise comments with code + snippets and hunk-level review tools were more likely to lead to code changes: + +- Industrial Qodo PR Agent studies found automated review can help bug + detection and awareness, but can also increase review time and produce faulty + or irrelevant comments: + + +Implementation lessons for Warden: + +- Keep planner context compact. The planner should operate over an inventory and + retrieve extra context through bounded tools only when needed. +- Treat semantic summaries as routing/context, not evidence. Findings still + need changed-line anchors and scanner evidence. +- Separate planning from scanning. A planning agent can inspect the repo to + group work; skill scanners still own domain-specific finding generation. +- Prefer explicit semantic segments over invisible behavior. Showing the plan in + verbose CLI output, JSONL, or a collapsed PR section can make Warden's work + auditable without overwhelming inline review. +- Evaluate against expected findings, not only cost. A cheaper run that misses a + known bug is worse. A more expensive planner may be justified only when it + improves or preserves recall while reducing scanner fragmentation. +- Avoid unbounded full-context prompts. Prior work suggests attention dilution + is a real failure mode; semantic chunking should reduce cognitive load, not + move the entire PR into an earlier prompt. + ## Rollout -1. Add `AtomicHunk`, `ReviewChunk`, and multi-range finding validation. +1. Add `AtomicHunkSummary`, `ReviewChunk`, and multi-range finding validation. 2. Adapt the existing hunk/coalescing flow to emit `ReviewChunk` values using `raw-hunks`. 3. Update scanner prompt construction to accept `ReviewChunk`. -4. Add the model-backed semantic planner behind `chunking.semantic.enabled`. - Planner output is the first place `ReviewChunk.summary` should be populated. -5. Add materializers for `whole-file`, `stitched-file`, and `raw-hunks`. -6. Add telemetry for chunk counts, cost, duration, and planner failures. -7. Add regression fixtures for tiny-hunk pathological changes, including the +4. Move semantic planning into `packages/warden/src/semantic/` with a narrow + public API. +5. Replace the simple auxiliary schema call with a primary-model, read-only + tool-using planning agent. +6. Keep planner input compact and make code inspection explicit through tools. +7. Add materializers for `whole-file`, `stitched-file`, and `raw-hunks`. +8. Add telemetry for chunk counts, cost, duration, and planner failures. +9. Add regression fixtures for tiny-hunk pathological changes, including the `getsentry/sentry-mcp` style case from Warden issue 313. -8. Enable selectively, compare eval recall and cost, then consider making it +10. Add semantic-vs-hunk eval fixtures with known expected findings. +11. Add optional verbose CLI and collapsed PR visibility for semantic segments. +12. Enable selectively, compare eval recall and cost, then consider making it default. ## Non-Goals @@ -281,3 +576,5 @@ line. - exposing planner recovery strategy as user config - building a full AST differ - reviewing unchanged lines as primary finding locations +- passing full diffs or full repository context to the planner by default +- making semantic segment display a required PR review surface diff --git a/warden.toml b/warden.toml index 51d3fafc..22665c42 100644 --- a/warden.toml +++ b/warden.toml @@ -15,6 +15,9 @@ enabled = true maxChunks = 20 maxChunkChars = 30000 maxHunksPerChunk = 50 +maxEmbeddedDiffChars = 8000 +maxEmbeddedDiffChunks = 12 +maxEmbeddedDiffRanges = 12 preferWholeFileBelowLines = 800 [[skills]] From df393bbca7e15e16483009b7e5b356d451939639 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 11:19:51 -0700 Subject: [PATCH 08/18] perf(chunking): Bound semantic scanner chunks Keep semantic changes grouped while splitting scanner payloads by hunk, changed range, and character limits. Treat semantic summaries as routing hints instead of correctness evidence and record the benchmark experiments. Co-Authored-By: GPT-5 Codex --- packages/warden/src/cli/output/tasks.ts | 1 + packages/warden/src/config/schema.ts | 8 +- packages/warden/src/sdk/analyze.ts | 1 + packages/warden/src/sdk/prompt.ts | 1 + packages/warden/src/semantic/planner.test.ts | 97 ++++++++++++++++++-- packages/warden/src/semantic/planner.ts | 59 +++++++----- specs/semantic-chunk-benchmark.md | 33 +++++++ specs/semantic-review-chunks.md | 18 ++-- warden.toml | 5 +- 9 files changed, 182 insertions(+), 41 deletions(-) diff --git a/packages/warden/src/cli/output/tasks.ts b/packages/warden/src/cli/output/tasks.ts index f29011ed..ef6337ea 100644 --- a/packages/warden/src/cli/output/tasks.ts +++ b/packages/warden/src/cli/output/tasks.ts @@ -281,6 +281,7 @@ async function prepareTaskFiles( maxChunks: runnerOptions.chunking?.semantic?.maxChunks, maxChunkChars: runnerOptions.chunking?.semantic?.maxChunkChars, maxHunksPerChunk: runnerOptions.chunking?.semantic?.maxHunksPerChunk, + maxChangedRangesPerChunk: runnerOptions.chunking?.semantic?.maxChangedRangesPerChunk, maxEmbeddedDiffChars: runnerOptions.chunking?.semantic?.maxEmbeddedDiffChars, maxEmbeddedDiffChunks: runnerOptions.chunking?.semantic?.maxEmbeddedDiffChunks, maxEmbeddedDiffRanges: runnerOptions.chunking?.semantic?.maxEmbeddedDiffRanges, diff --git a/packages/warden/src/config/schema.ts b/packages/warden/src/config/schema.ts index b1b41d13..a2fad82d 100644 --- a/packages/warden/src/config/schema.ts +++ b/packages/warden/src/config/schema.ts @@ -183,12 +183,14 @@ export type CoalesceConfig = z.infer; export const SemanticChunkingConfigSchema = z.object({ /** Enable semantic review chunk materialization (default: false) */ enabled: z.boolean().optional(), - /** Maximum number of review chunks to emit after semantic grouping */ + /** Maximum number of scanner review chunks to emit after semantic grouping */ maxChunks: z.number().int().positive().optional(), - /** Target max size per semantic review chunk in characters */ + /** Target max size per scanner review chunk in characters */ maxChunkChars: z.number().int().positive().optional(), - /** Maximum atomic hunks to group into one semantic review chunk */ + /** Maximum atomic hunks to place in one scanner review chunk inside a semantic change */ maxHunksPerChunk: z.number().int().positive().optional(), + /** Maximum changed line ranges to place in one scanner review chunk inside a semantic change */ + maxChangedRangesPerChunk: z.number().int().positive().optional(), /** Maximum total hunk content characters to embed in the semantic planner prompt */ maxEmbeddedDiffChars: z.number().int().nonnegative().optional(), /** Maximum prepared chunks allowed before the semantic planner stops embedding full hunk content */ diff --git a/packages/warden/src/sdk/analyze.ts b/packages/warden/src/sdk/analyze.ts index 78cbf964..735b8f83 100644 --- a/packages/warden/src/sdk/analyze.ts +++ b/packages/warden/src/sdk/analyze.ts @@ -1037,6 +1037,7 @@ async function runSkillAnalysis( maxChunks: options.chunking?.semantic?.maxChunks, maxChunkChars: options.chunking?.semantic?.maxChunkChars, maxHunksPerChunk: options.chunking?.semantic?.maxHunksPerChunk, + maxChangedRangesPerChunk: options.chunking?.semantic?.maxChangedRangesPerChunk, maxEmbeddedDiffChars: options.chunking?.semantic?.maxEmbeddedDiffChars, maxEmbeddedDiffChunks: options.chunking?.semantic?.maxEmbeddedDiffChunks, maxEmbeddedDiffRanges: options.chunking?.semantic?.maxEmbeddedDiffRanges, diff --git a/packages/warden/src/sdk/prompt.ts b/packages/warden/src/sdk/prompt.ts index c80f9618..2de2e273 100644 --- a/packages/warden/src/sdk/prompt.ts +++ b/packages/warden/src/sdk/prompt.ts @@ -25,6 +25,7 @@ export function formatReviewChunkForAnalysis(chunk: ReviewChunk): string { lines.push(`## Review Chunk: ${chunk.title}`); if (chunk.summary) { lines.push(`## Semantic Summary: ${chunk.summary}`); + lines.push('The semantic summary is only a grouping hint. Do not treat it as evidence that the changed behavior is correct or intended.'); } lines.push(''); lines.push('## Changed Line Map'); diff --git a/packages/warden/src/semantic/planner.test.ts b/packages/warden/src/semantic/planner.test.ts index cfe68cbc..f15854c2 100644 --- a/packages/warden/src/semantic/planner.test.ts +++ b/packages/warden/src/semantic/planner.test.ts @@ -265,7 +265,7 @@ describe('planSemanticReviewChunks', () => { expect(prompt).not.toContain('@@ -10,1 +10,1 @@'); }); - it('splits planned groups that exceed maxChunkChars after materialization', async () => { + it('splits planned semantic changes into bounded scanner chunks', async () => { runAuxiliary.mockResolvedValueOnce({ success: true, data: { @@ -296,12 +296,13 @@ describe('planSemanticReviewChunks', () => { maxChunkChars: 500, }); - expect(planned.groups).toHaveLength(2); - expect(planned.groups.map((group) => group.chunks[0]?.title)).toEqual([ + expect(planned.groups).toHaveLength(1); + expect(planned.groups[0]?.displayName).toBe('Preserve dashboard axis range'); + expect(planned.groups[0]?.chunks.map((chunk) => chunk.title)).toEqual([ 'Preserve dashboard axis range (1/2)', 'Preserve dashboard axis range (2/2)', ]); - expect(planned.groups.map((group) => group.chunks[0]?.changedLineMap)).toEqual([ + expect(planned.groups[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ [ { path: 'src/dashboard.ts', start: 10, end: 10 }, { path: 'src/dashboard.ts', start: 100, end: 100 }, @@ -310,7 +311,7 @@ describe('planSemanticReviewChunks', () => { ]); }); - it('rejects materialized groups that exceed maxChunks after splitting', async () => { + it('rejects materialized scanner chunks that exceed maxChunks after splitting', async () => { runAuxiliary.mockResolvedValueOnce({ success: true, data: { @@ -340,7 +341,91 @@ describe('planSemanticReviewChunks', () => { runtime: 'pi', maxChunks: 1, maxChunkChars: 500, - })).rejects.toThrow('materialized 2 groups, exceeding maxChunks 1'); + })).rejects.toThrow('materialized 2 chunks, exceeding maxChunks 1'); + }); + + it('preserves one semantic group when maxHunksPerChunk splits scanner chunks', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context); + + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxHunksPerChunk: 2, + }); + + expect(planned.groups).toHaveLength(1); + expect(planned.groups[0]?.chunks).toHaveLength(2); + expect(planned.groups[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ + [ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'src/dashboard.ts', start: 100, end: 100 }, + ], + [{ path: 'src/dashboard.ts', start: 200, end: 200 }], + ]); + }); + + it('preserves one semantic group when changed ranges split scanner chunks', async () => { + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Preserve dashboard axis range', + summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', + chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const context = makeContext(); + const prepared = prepareFiles(context); + + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxChangedRangesPerChunk: 2, + }); + + expect(planned.groups).toHaveLength(1); + expect(planned.groups[0]?.chunks).toHaveLength(2); + expect(planned.groups[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ + [ + { path: 'src/dashboard.ts', start: 10, end: 10 }, + { path: 'src/dashboard.ts', start: 100, end: 100 }, + ], + [{ path: 'src/dashboard.ts', start: 200, end: 200 }], + ]); }); it('rejects atomic chunks that exceed maxChunkChars', async () => { diff --git a/packages/warden/src/semantic/planner.ts b/packages/warden/src/semantic/planner.ts index f1c5e18f..f23c4fab 100644 --- a/packages/warden/src/semantic/planner.ts +++ b/packages/warden/src/semantic/planner.ts @@ -18,6 +18,9 @@ const SemanticChunkPlanSchema = z.object({ const MAX_EMBEDDED_DIFF_CHARS = 8000; const MAX_EMBEDDED_DIFF_CHUNKS = 12; const MAX_EMBEDDED_DIFF_RANGES = 12; +const MAX_CHUNK_CHARS = 20000; +const MAX_HUNKS_PER_CHUNK = 4; +const MAX_CHANGED_RANGES_PER_CHUNK = 4; export interface SemanticChunkPlanningOptions { enabled?: boolean; @@ -27,6 +30,7 @@ export interface SemanticChunkPlanningOptions { maxChunks?: number; maxChunkChars?: number; maxHunksPerChunk?: number; + maxChangedRangesPerChunk?: number; maxEmbeddedDiffChars?: number; maxEmbeddedDiffChunks?: number; maxEmbeddedDiffRanges?: number; @@ -49,6 +53,7 @@ interface SemanticChunkLimits { maxChunks: number; maxChunkChars: number; maxHunksPerChunk: number; + maxChangedRangesPerChunk: number; maxEmbeddedDiffChars: number; maxEmbeddedDiffChunks: number; maxEmbeddedDiffRanges: number; @@ -140,8 +145,9 @@ function formatFileInventory(context: EventContext): string { function semanticChunkLimits(options: SemanticChunkPlanningOptions): SemanticChunkLimits { return { maxChunks: options.maxChunks ?? 20, - maxChunkChars: options.maxChunkChars ?? 30000, - maxHunksPerChunk: options.maxHunksPerChunk ?? 50, + maxChunkChars: options.maxChunkChars ?? MAX_CHUNK_CHARS, + maxHunksPerChunk: options.maxHunksPerChunk ?? MAX_HUNKS_PER_CHUNK, + maxChangedRangesPerChunk: options.maxChangedRangesPerChunk ?? MAX_CHANGED_RANGES_PER_CHUNK, maxEmbeddedDiffChars: options.maxEmbeddedDiffChars ?? MAX_EMBEDDED_DIFF_CHARS, maxEmbeddedDiffChunks: options.maxEmbeddedDiffChunks ?? MAX_EMBEDDED_DIFF_CHUNKS, maxEmbeddedDiffRanges: options.maxEmbeddedDiffRanges ?? MAX_EMBEDDED_DIFF_RANGES, @@ -200,12 +206,12 @@ function buildSemanticChunkPlanningPrompt( options: SemanticChunkPlanningOptions ): string { const plannerInput = buildSemanticChunkPlannerInput(context, chunks, options); - const { maxChunks, maxHunksPerChunk, maxChunkChars } = plannerInput.limits; + const { maxChunks, maxHunksPerChunk, maxChangedRangesPerChunk, maxChunkChars } = plannerInput.limits; const header = [ 'You are planning code review chunks for Warden.', '', - 'Group atomic git chunks into semantic review chunks.', - 'A semantic chunk should contain changes that a reviewer should understand together: one behavior change, API contract, data flow, migration, validation rule, or test expectation.', + 'Group atomic git chunks into semantic changes.', + 'A semantic change should contain chunks that a reviewer should understand together: one behavior change, API contract, data flow, migration, validation rule, or test expectation.', 'For each planned group, write a semantic delta summary: what behavior, contract, data flow, API, or test expectation changed.', 'Do not restate filenames or line ranges. Do not say "lines 10, 100, 200".', 'Do not summarize the input shape. Explain the product, security, data-flow, API, or test behavior being changed.', @@ -214,8 +220,10 @@ function buildSemanticChunkPlanningPrompt( 'Use read_review_chunk to inspect exact hunk content for non-embedded chunks; read_changed_file only shows current head content.', 'If embedded small diffs are present, use them as initial evidence and use tools only for missing relationships or context.', `Target at most ${maxChunks} planned groups.`, - `Use at most ${maxHunksPerChunk} atomic chunks per group.`, - `Keep each planned group under roughly ${maxChunkChars} characters once materialized for scanning.`, + `Warden may split a planned semantic change into bounded scanner chunks after planning.`, + `Each scanner chunk will use at most ${maxHunksPerChunk} atomic chunks.`, + `Each scanner chunk will use at most ${maxChangedRangesPerChunk} changed line ranges.`, + `Each scanner chunk will stay under roughly ${maxChunkChars} characters once materialized.`, `Embedded small diffs are only present when the changeset fits these planner limits: ${plannerInput.limits.maxEmbeddedDiffChars} chars, ${plannerInput.limits.maxEmbeddedDiffChunks} chunks, ${plannerInput.limits.maxEmbeddedDiffRanges} changed ranges.`, 'Every input Chunk ID must appear in exactly one group. Do not invent Chunk IDs.', ].filter((line): line is string => Boolean(line)); @@ -295,7 +303,7 @@ function splitPlannedChunks( title: string, summary: string, chunks: ReviewChunk[], - limits: { maxChunkChars: number; maxHunksPerChunk: number } + limits: { maxChunkChars: number; maxHunksPerChunk: number; maxChangedRangesPerChunk: number } ): ReviewChunk[] { const batches: ReviewChunk[][] = []; let current: ReviewChunk[] = []; @@ -324,9 +332,10 @@ function splitPlannedChunks( const candidate = [...current, chunk]; const candidateChunk = materializePlannedChunk(startIndex + batches.length, title, summary, candidate); const exceedsHunks = candidate.length > limits.maxHunksPerChunk; + const exceedsChangedRanges = candidateChunk.changedLineMap.length > limits.maxChangedRangesPerChunk; const exceedsChars = reviewChunkContentChars(candidateChunk) > limits.maxChunkChars; - if (exceedsHunks || exceedsChars) { + if (exceedsHunks || exceedsChangedRanges || exceedsChars) { pushCurrent(); current = [chunk]; } else { @@ -353,15 +362,12 @@ function groupFromPreparedFile(file: PreparedFile): ReviewChunkGroup { }; } -function groupFromPlannedChunk(chunk: ReviewChunk): ReviewChunkGroup { - const filenames = [...new Set(chunk.files.map((file) => file.path))]; - const [firstFilename] = filenames; +function groupFromPlannedChunks(title: string, chunks: ReviewChunk[]): ReviewChunkGroup { + const filenames = [...new Set(chunks.flatMap((chunk) => chunk.files.map((file) => file.path)))]; return { - displayName: filenames.length <= 1 - ? firstFilename ?? chunk.title - : `${firstFilename ?? chunk.title} + ${filenames.length - 1} file${filenames.length === 2 ? '' : 's'}`, + displayName: title, filenames, - chunks: [chunk], + chunks, }; } @@ -397,10 +403,12 @@ export async function planSemanticReviewChunks( const chunksById = new Map(chunks.map((chunk) => [chunk.id, chunk])); const seen = new Set(); - const plannedChunks: ReviewChunk[] = []; + const plannedGroups: ReviewChunkGroup[] = []; + let plannedChunkCount = 0; const maxChunks = options.maxChunks ?? 20; - const maxHunksPerChunk = options.maxHunksPerChunk ?? 50; - const maxChunkChars = options.maxChunkChars ?? 30000; + const maxHunksPerChunk = options.maxHunksPerChunk ?? MAX_HUNKS_PER_CHUNK; + const maxChangedRangesPerChunk = options.maxChangedRangesPerChunk ?? MAX_CHANGED_RANGES_PER_CHUNK; + const maxChunkChars = options.maxChunkChars ?? MAX_CHUNK_CHARS; if (result.data.groups.length > maxChunks) { throw new SkillRunnerError( @@ -426,14 +434,17 @@ export async function planSemanticReviewChunks( seen.add(chunkId); groupChunks.push(chunk); } - plannedChunks.push(...splitPlannedChunks(plannedChunks.length, group.title, group.summary, groupChunks, { + const chunksForGroup = splitPlannedChunks(plannedChunkCount, group.title, group.summary, groupChunks, { maxChunkChars, maxHunksPerChunk, - })); + maxChangedRangesPerChunk, + }); + plannedChunkCount += chunksForGroup.length; + plannedGroups.push(groupFromPlannedChunks(group.title, chunksForGroup)); - if (plannedChunks.length > maxChunks) { + if (plannedChunkCount > maxChunks) { throw new SkillRunnerError( - `Semantic chunk planning materialized ${plannedChunks.length} groups, exceeding maxChunks ${maxChunks}`, + `Semantic chunk planning materialized ${plannedChunkCount} chunks, exceeding maxChunks ${maxChunks}`, { code: 'sdk_error' }, ); } @@ -448,7 +459,7 @@ export async function planSemanticReviewChunks( } return { - groups: plannedChunks.map(groupFromPlannedChunk), + groups: plannedGroups, usage: result.usage, }; } diff --git a/specs/semantic-chunk-benchmark.md b/specs/semantic-chunk-benchmark.md index 39a3a1c9..f1273d55 100644 --- a/specs/semantic-chunk-benchmark.md +++ b/specs/semantic-chunk-benchmark.md @@ -120,6 +120,7 @@ Record these for each semantic-off and semantic-on run: - wall time - scanner chunk count - semantic planner group count +- scanner chunks per semantic group - semantic planner summaries - input tokens - output tokens @@ -127,6 +128,9 @@ Record these for each semantic-off and semantic-on run: The semantic-on run is successful when it keeps the expected finding and reduces operational load. A lower chunk count is not enough if recall regresses. +Semantic groups are allowed to contain multiple scanner chunks; the target is +fewer, better bounded scanner calls than raw git chunks, not one giant scanner +prompt per logical change. ## Output @@ -183,6 +187,35 @@ be used as the dataset quality baseline. The semantic-on run grouped both changed ranges into one scanner chunk and kept the expected medium-confidence axis-range regression finding. +Full reverse-patch run: + +- date: 2026-06-30 +- case: `sentry-dashboard-axis-range-existing-widget` +- repository: `getsentry/sentry` +- synthetic base: `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` +- synthetic head: `a4778dc5fcf` +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- skill: `code-review` +- verification: disabled + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| semantic on, bounded groups | 5 | 0 | 10m45s | 481,391 | 34,884 | $1.2564 | +| semantic on, bounded groups plus summary anti-bias prompt | 4 | 0 | 8m19s | 472,104 | 26,978 | $1.1046 | +| semantic on, bounded groups plus changed-range cap | 5 | 0 | 7m24s | 432,183 | 23,708 | $0.9862 | + +The full reverse-patch run preserved the real patch shape but did not find the +expected axis-range regression. This makes the case useful as a recall warning: +semantic grouping can make a bad behavior look like a coherent migration when +tests are changed to match it. The scanner prompt now states that semantic +summaries are grouping hints, not evidence of correctness. + +The changed-range cap run is incomplete for score comparison because one +test-heavy scanner chunk hit `turn_limit`. It is still useful operationally: it +split one broad semantic group into two scanner chunks and reduced recorded +cost, but the run cannot be treated as a clean recall result. + ## Acceptance Before using this benchmark to make decisions: diff --git a/specs/semantic-review-chunks.md b/specs/semantic-review-chunks.md index b11e77b9..b0068e96 100644 --- a/specs/semantic-review-chunks.md +++ b/specs/semantic-review-chunks.md @@ -137,13 +137,17 @@ chunking may use filenames and changed ranges. `summary` is optional and planner-owned. It must only be set when a semantic planner has described the logical change. Deterministic grouping must not populate `summary` with filenames or changed ranges and call that semantic. +Scanner prompts must treat summaries as grouping hints, not evidence that the +change is intended or correct. `files[].content` is the readable review packet. It can be larger than a hunk and may include unchanged surrounding code. This content is for understanding. -One `ReviewChunk` may include multiple files when the same logical change spans -implementation, tests, config, or call sites. The chunk title and summary -describe the shared change; each `files[]` entry carries the file-local content -needed to review that change. +One semantic change may include multiple files when the same logical change +spans implementation, tests, config, or call sites. That semantic change may +still materialize as multiple bounded `ReviewChunk` scanner calls when the +group would otherwise be too large. The chunk title and summary describe the +shared change; each `files[]` entry carries the file-local content needed to +review that scanner slice. `changedLineMap` is the hard validation boundary. A scanner finding may only anchor to a line inside this map. Surrounding content can explain a finding but @@ -339,8 +343,9 @@ Semantic review chunks should be configured with a small public surface: [defaults.chunking.semantic] enabled = true maxChunks = 20 -maxChunkChars = 30000 -maxHunksPerChunk = 50 +maxChunkChars = 20000 +maxHunksPerChunk = 4 +maxChangedRangesPerChunk = 4 maxEmbeddedDiffChars = 8000 maxEmbeddedDiffChunks = 12 maxEmbeddedDiffRanges = 12 @@ -411,6 +416,7 @@ Useful fields to expose: - files included - changed ranges included - atomic chunk count +- scanner chunk count - content mode per file - scanner result count for the segment diff --git a/warden.toml b/warden.toml index 22665c42..e164954c 100644 --- a/warden.toml +++ b/warden.toml @@ -13,8 +13,9 @@ ignorePaths = ["dist/**", "packages/evals/**"] [defaults.chunking.semantic] enabled = true maxChunks = 20 -maxChunkChars = 30000 -maxHunksPerChunk = 50 +maxChunkChars = 20000 +maxHunksPerChunk = 4 +maxChangedRangesPerChunk = 4 maxEmbeddedDiffChars = 8000 maxEmbeddedDiffChunks = 12 maxEmbeddedDiffRanges = 12 From 1cfb29ad3237a239dba96eb0bd6e4a6390445b8b Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 11:28:44 -0700 Subject: [PATCH 09/18] docs(chunking): Snapshot semantic benchmark results Record compact benchmark metrics for the trimmed and full axis-range semantic chunking runs. Note that the full non-semantic run was partial but produced the expected-style finding before interruption. Co-Authored-By: GPT-5 Codex --- specs/semantic-chunk-benchmark-results.json | 171 ++++++++++++++++++++ specs/semantic-chunk-benchmark.md | 14 +- 2 files changed, 180 insertions(+), 5 deletions(-) create mode 100644 specs/semantic-chunk-benchmark-results.json diff --git a/specs/semantic-chunk-benchmark-results.json b/specs/semantic-chunk-benchmark-results.json new file mode 100644 index 00000000..60875d0f --- /dev/null +++ b/specs/semantic-chunk-benchmark-results.json @@ -0,0 +1,171 @@ +{ + "schemaVersion": 1, + "capturedAt": "2026-06-30", + "case": "sentry-dashboard-axis-range-existing-widget", + "repository": "getsentry/sentry", + "model": "openrouter/anthropic/claude-sonnet-4.6", + "runtime": "pi", + "skill": "code-review", + "verification": false, + "runs": [ + { + "name": "trimmed-off", + "patch": "trimmed two-file reverse patch", + "semantic": false, + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-off.jsonl", + "sha256": "0bfacc26a78f572b0d31e06d75c9a3dda55077b2d61a09fcca6092227e884895" + }, + "scannerChunks": 2, + "completedScannerChunks": 2, + "failedScannerChunks": 0, + "findings": [ + { + "title": "`convertWidgetToBuilderStateParams` no longer defaults `axisRange` to `'auto'` for null/invalid widget values, breaking existing contracts and tests", + "severity": "medium", + "confidence": "high", + "location": { + "path": "static/app/views/dashboards/widgetBuilder/utils/convertWidgetToBuilderStateParams.ts", + "startLine": 79 + } + } + ], + "durationMs": 714272, + "usage": { + "inputTokens": 445997, + "outputTokens": 39484, + "costUSD": 1.2213379500000001 + } + }, + { + "name": "trimmed-on", + "patch": "trimmed two-file reverse patch", + "semantic": true, + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-on.jsonl", + "sha256": "9ad2d4b7565cab4780285e0e653bdd90124d74537c39c21d570fb0656ff9778e" + }, + "scannerChunks": 1, + "completedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Removing `?? 'auto'` fallback from `getAxisRange` causes test failures and propagates `undefined` axisRange downstream", + "severity": "medium", + "confidence": "high", + "location": { + "path": "static/app/views/dashboards/widgetBuilder/utils/convertWidgetToBuilderStateParams.ts", + "startLine": 79 + } + } + ], + "durationMs": 182759, + "usage": { + "inputTokens": 160096, + "outputTokens": 10079, + "costUSD": 0.46092255 + } + }, + { + "name": "full-off-partial", + "patch": "full reverse patch", + "semantic": false, + "complete": false, + "reason": "interrupted after first completed raw git chunk", + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-full-off.jsonl", + "sha256": "e975e8cf1346d66726d1fd4b8846969e695883817d963996287c009179c9aafa" + }, + "scannerChunks": 1, + "completedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Removal of `axisRange: 'auto'` assertion masks regression where axis range is not forwarded to widget builder", + "severity": "medium", + "confidence": "high", + "location": { + "path": "static/app/components/modals/widgetBuilder/addToDashboardModal.spec.tsx", + "startLine": 340, + "endLine": 345 + } + } + ], + "durationMs": 173444, + "usage": { + "inputTokens": 160637, + "outputTokens": 9024, + "costUSD": 0.3190759499999999 + } + }, + { + "name": "full-semantic-bounded", + "patch": "full reverse patch", + "semantic": true, + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-full-bounded.jsonl", + "sha256": "cdb283b9469e05b8f9e2b7c6601849b8959668dd7eba5526e0b966105763dd65" + }, + "scannerChunks": 5, + "completedScannerChunks": 5, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 644575, + "usage": { + "inputTokens": 481391, + "outputTokens": 34884, + "costUSD": 1.2563874 + } + }, + { + "name": "full-semantic-antibias", + "patch": "full reverse patch", + "semantic": true, + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-full-bounded-v2.jsonl", + "sha256": "10334451cffeb231e5b6ff05190bcaf39ed6233cf521c5864c7f5858ef4b6c29" + }, + "scannerChunks": 4, + "completedScannerChunks": 4, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 499429, + "usage": { + "inputTokens": 472104, + "outputTokens": 26978, + "costUSD": 1.10464215 + } + }, + { + "name": "full-semantic-range-cap", + "patch": "full reverse patch", + "semantic": true, + "complete": false, + "reason": "one scanner chunk hit turn_limit", + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-full-bounded-ranges.jsonl", + "sha256": "d188e68a3af7753c83626b2c666ac675ae51eea3d678b3c940308223b738891d" + }, + "scannerChunks": 5, + "completedScannerChunks": 4, + "failedScannerChunks": 1, + "findings": [], + "durationMs": 443935, + "usage": { + "inputTokens": 432183, + "outputTokens": 23708, + "costUSD": 0.9861766500000001 + } + } + ], + "notes": [ + "The trimmed smoke case is useful for mechanics but not dataset quality.", + "The full non-semantic run is incomplete but produced the expected-style test-regression finding before interruption.", + "Full semantic runs completed faster after tuning but missed the expected finding.", + "The current evidence points to recall loss from semantic framing of implementation and test updates, not only oversized scanner chunks." + ] +} diff --git a/specs/semantic-chunk-benchmark.md b/specs/semantic-chunk-benchmark.md index f1273d55..8508adb5 100644 --- a/specs/semantic-chunk-benchmark.md +++ b/specs/semantic-chunk-benchmark.md @@ -201,15 +201,19 @@ Full reverse-patch run: | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| non-semantic, partial | 1/13 completed | 1 | 2m53s | 160,637 | 9,024 | $0.3191 | | semantic on, bounded groups | 5 | 0 | 10m45s | 481,391 | 34,884 | $1.2564 | | semantic on, bounded groups plus summary anti-bias prompt | 4 | 0 | 8m19s | 472,104 | 26,978 | $1.1046 | | semantic on, bounded groups plus changed-range cap | 5 | 0 | 7m24s | 432,183 | 23,708 | $0.9862 | -The full reverse-patch run preserved the real patch shape but did not find the -expected axis-range regression. This makes the case useful as a recall warning: -semantic grouping can make a bad behavior look like a coherent migration when -tests are changed to match it. The scanner prompt now states that semantic -summaries are grouping hints, not evidence of correctness. +The full non-semantic run was interrupted after one completed scanner chunk, so +it is not a clean cost comparison. That completed chunk did find the expected +style of regression: removing `axisRange: 'auto'` assertions masked a behavior +break. The semantic full-patch runs preserved the real patch shape but did not +find the expected axis-range regression. This makes the case useful as a recall +warning: semantic grouping can make a bad behavior look like a coherent +migration when tests are changed to match it. The scanner prompt now states +that semantic summaries are grouping hints, not evidence of correctness. The changed-range cap run is incomplete for score comparison because one test-heavy scanner chunk hit `turn_limit`. It is still useful operationally: it From ccc75514a86005ef32fdf8ff9ed347ac4ceadd2c Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 11:56:10 -0700 Subject: [PATCH 10/18] perf(chunking): Keep semantic scanner prompts neutral Stop passing semantic summaries into scanner prompts so planner output cannot be treated as evidence of correctness. Record the neutral-prompt and stricter range-cap benchmark attempts. Co-Authored-By: GPT-5 Codex --- packages/warden/src/sdk/analyze.test.ts | 4 +- packages/warden/src/sdk/prompt.ts | 6 +-- specs/semantic-chunk-benchmark-results.json | 46 ++++++++++++++++++++- specs/semantic-chunk-benchmark.md | 10 +++++ 4 files changed, 59 insertions(+), 7 deletions(-) diff --git a/packages/warden/src/sdk/analyze.test.ts b/packages/warden/src/sdk/analyze.test.ts index eae2f18d..1aa1d9a6 100644 --- a/packages/warden/src/sdk/analyze.test.ts +++ b/packages/warden/src/sdk/analyze.test.ts @@ -850,7 +850,9 @@ describe('runSkill', () => { })); expect(runSkillMock).toHaveBeenCalledTimes(1); const request = runSkillMock.mock.calls[0]?.[0]; - expect(request.userPrompt).toContain('## Semantic Summary: Dashboard charts now carry a widget-provided axis range through chart conversion, rendering, and tests.'); + expect(request.userPrompt).toContain('## Review Chunk: semantic scanner slice'); + expect(request.userPrompt).not.toContain('## Semantic Summary:'); + expect(request.userPrompt).not.toContain('Dashboard charts now carry a widget-provided axis range through chart conversion, rendering, and tests.'); expect(request.userPrompt).toContain('- src/example.ts:10-10'); expect(request.userPrompt).toContain('- src/example.ts:100-100'); expect(request.userPrompt).toContain('- src/example.ts:200-200'); diff --git a/packages/warden/src/sdk/prompt.ts b/packages/warden/src/sdk/prompt.ts index 2de2e273..d043d353 100644 --- a/packages/warden/src/sdk/prompt.ts +++ b/packages/warden/src/sdk/prompt.ts @@ -22,11 +22,7 @@ function formatChangedLineMap(chunk: ReviewChunk): string { export function formatReviewChunkForAnalysis(chunk: ReviewChunk): string { const lines: string[] = []; - lines.push(`## Review Chunk: ${chunk.title}`); - if (chunk.summary) { - lines.push(`## Semantic Summary: ${chunk.summary}`); - lines.push('The semantic summary is only a grouping hint. Do not treat it as evidence that the changed behavior is correct or intended.'); - } + lines.push(`## Review Chunk: ${chunk.summary ? 'semantic scanner slice' : chunk.title}`); lines.push(''); lines.push('## Changed Line Map'); lines.push(formatChangedLineMap(chunk)); diff --git a/specs/semantic-chunk-benchmark-results.json b/specs/semantic-chunk-benchmark-results.json index 60875d0f..6025a16b 100644 --- a/specs/semantic-chunk-benchmark-results.json +++ b/specs/semantic-chunk-benchmark-results.json @@ -160,12 +160,56 @@ "outputTokens": 23708, "costUSD": 0.9861766500000001 } + }, + { + "name": "full-semantic-neutral-prompt", + "patch": "full reverse patch", + "semantic": true, + "complete": false, + "reason": "one scanner chunk hit turn_limit", + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-full-neutral-prompt.jsonl", + "sha256": "9d8de49aefee67a7fc7fa06adfccecc6ce64c3a7f6edd29a9e91ea0798e7d5bb" + }, + "scannerChunks": 4, + "completedScannerChunks": 3, + "failedScannerChunks": 1, + "findings": [], + "durationMs": 314311, + "usage": { + "inputTokens": 374416, + "outputTokens": 15714, + "costUSD": 0.7503958500000002 + } + }, + { + "name": "full-semantic-range-cap-2", + "patch": "full reverse patch", + "semantic": true, + "complete": false, + "reason": "one scanner chunk hit turn_limit", + "rawArtifact": { + "path": "/tmp/warden-semantic-axis-full-range2.jsonl", + "sha256": "27a9fc77a4522219dbd6bfb06b17c54c23a276c9514421d96425d58221ac6744" + }, + "scannerChunks": 7, + "completedScannerChunks": 6, + "failedScannerChunks": 1, + "findings": [], + "durationMs": 820311, + "usage": { + "inputTokens": 807047, + "outputTokens": 45554, + "costUSD": 2.050875 + } } ], "notes": [ "The trimmed smoke case is useful for mechanics but not dataset quality.", "The full non-semantic run is incomplete but produced the expected-style test-regression finding before interruption.", "Full semantic runs completed faster after tuning but missed the expected finding.", - "The current evidence points to recall loss from semantic framing of implementation and test updates, not only oversized scanner chunks." + "Removing semantic summaries from scanner prompts reduced cost but did not restore recall.", + "A stricter changed-range cap increased scanner calls and cost without restoring recall.", + "The current evidence points to recall loss from semantic framing and test/update grouping, not only oversized scanner chunks." ] } diff --git a/specs/semantic-chunk-benchmark.md b/specs/semantic-chunk-benchmark.md index 8508adb5..1b62904b 100644 --- a/specs/semantic-chunk-benchmark.md +++ b/specs/semantic-chunk-benchmark.md @@ -205,6 +205,8 @@ Full reverse-patch run: | semantic on, bounded groups | 5 | 0 | 10m45s | 481,391 | 34,884 | $1.2564 | | semantic on, bounded groups plus summary anti-bias prompt | 4 | 0 | 8m19s | 472,104 | 26,978 | $1.1046 | | semantic on, bounded groups plus changed-range cap | 5 | 0 | 7m24s | 432,183 | 23,708 | $0.9862 | +| semantic on, neutral scanner prompt | 4 | 0 | 5m14s | 374,416 | 15,714 | $0.7504 | +| semantic on, neutral prompt plus changed-range cap 2 | 7 | 0 | 13m40s | 807,047 | 45,554 | $2.0509 | The full non-semantic run was interrupted after one completed scanner chunk, so it is not a clean cost comparison. That completed chunk did find the expected @@ -220,6 +222,14 @@ test-heavy scanner chunk hit `turn_limit`. It is still useful operationally: it split one broad semantic group into two scanner chunks and reduced recorded cost, but the run cannot be treated as a clean recall result. +The neutral scanner prompt removed semantic summaries from scanner prompts. It +reduced cost again but still missed recall, and a stricter changed-range cap of +2 increased scanner calls and cost without recovering the finding. This points +away from size alone. The stronger hypothesis is that semantically grouping +implementation and test-update chunks makes the scanner evaluate a coherent +intentional migration, while raw hunk mode caught an isolated assertion removal +as masking a regression. + ## Acceptance Before using this benchmark to make decisions: From 53bfc88a95ff9b05ba20455d6b58d439c855cde4 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 13:10:30 -0700 Subject: [PATCH 11/18] test(chunking): Capture semantic group eval shape Record the current semantic chunking direction before trying a different batching strategy. The eval now checks semantic groups separately from scanner chunks, and the benchmark notes call out the rejected syntax-specific split experiment. Co-Authored-By: GPT-5 Codex --- packages/evals/src/semantic-chunking.eval.ts | 43 ++++++++++++++------ packages/warden/src/semantic/planner.ts | 1 + specs/semantic-chunk-benchmark.md | 5 +++ 3 files changed, 37 insertions(+), 12 deletions(-) diff --git a/packages/evals/src/semantic-chunking.eval.ts b/packages/evals/src/semantic-chunking.eval.ts index 80808a0c..ec56615d 100644 --- a/packages/evals/src/semantic-chunking.eval.ts +++ b/packages/evals/src/semantic-chunking.eval.ts @@ -27,6 +27,17 @@ type SemanticChunkingEvalInput = z.infer const SemanticChunkingEvalOutputSchema = z.object({ atomicChunkCount: z.number().int().nonnegative(), + groups: z.array(z.object({ + displayName: z.string(), + files: z.array(z.string()), + changedLineMap: z.array(z.object({ + path: z.string(), + start: z.number().int(), + end: z.number().int(), + })), + summary: z.string().optional(), + scannerChunkCount: z.number().int().positive(), + })), chunks: z.array(z.object({ summary: z.string().optional(), files: z.array(z.string()), @@ -143,6 +154,13 @@ function createSemanticChunkingHarness(): Harness group.chunks); const output = { atomicChunkCount: atomicChunks.length, + groups: planned.groups.map((group) => ({ + displayName: group.displayName, + files: group.filenames, + changedLineMap: group.chunks.flatMap((chunk) => chunk.changedLineMap), + summary: group.chunks.find((chunk) => chunk.summary)?.summary, + scannerChunkCount: group.chunks.length, + })), chunks: chunks.map((chunk) => ({ summary: chunk.summary, files: chunk.files.map((file) => file.path), @@ -193,30 +211,31 @@ function createSemanticChunkingJudge() { } const chunks = output.data.chunks; - const crossFileChunk = chunks.find((chunk) => chunk.files.length > 1); - const summaryText = chunks.map((chunk) => chunk.summary ?? '').join(' '); + const groups = output.data.groups; + const crossFileGroup = groups.find((group) => group.files.length > 1); + const summaryText = groups.map((group) => group.summary ?? '').join(' '); const hasSemanticSummary = /axisRange|axis range|range/i.test(summaryText) && /widget|convert|render|chart/i.test(summaryText) && !/lines?\s+\d+/i.test(summaryText) && !/\b10\b.*\b80\b.*\b140\b/.test(summaryText); - const hasExpectedLineMap = Boolean(crossFileChunk) + const hasExpectedLineMap = Boolean(crossFileGroup) && [ { path: 'src/dashboard.ts', start: 10, end: 10 }, { path: 'src/dashboard.ts', start: 80, end: 80 }, { path: 'src/dashboard.ts', start: 140, end: 140 }, { path: 'tests/dashboard.test.ts', start: 220, end: 220 }, ].every((range) => - crossFileChunk?.changedLineMap.some((actual) => + crossFileGroup?.changedLineMap.some((actual) => actual.path === range.path && actual.start === range.start && actual.end === range.end ) ); - const crossFileContent = crossFileChunk?.content ?? ''; - const hasExpectedContent = Boolean(crossFileChunk) - && crossFileContent.includes('convertWidgetToChart(widget, range)') - && crossFileContent.includes('renderChart(chart, series, range)') - && crossFileContent.includes('expect(rendered.range).toEqual(customAxisRange)'); + const allContent = chunks.map((chunk) => chunk.content).join('\n'); + const hasExpectedContent = Boolean(crossFileGroup) + && allContent.includes('convertWidgetToChart(widget, range)') + && allContent.includes('renderChart(chart, series, range)') + && allContent.includes('expect(rendered.range).toEqual(customAxisRange)'); const reducedChunks = chunks.length < output.data.atomicChunkCount; - const passed = reducedChunks && Boolean(crossFileChunk) && hasExpectedLineMap + const passed = reducedChunks && Boolean(crossFileGroup) && hasExpectedLineMap && hasSemanticSummary && hasExpectedContent; return { @@ -227,7 +246,7 @@ function createSemanticChunkingJudge() { : 'Planner did not produce the expected semantic cross-file chunk.', atomicChunkCount: output.data.atomicChunkCount, chunkCount: chunks.length, - hasCrossFileChunk: Boolean(crossFileChunk), + hasCrossFileGroup: Boolean(crossFileGroup), hasExpectedLineMap, hasSemanticSummary, hasExpectedContent, @@ -252,7 +271,7 @@ describeEval( expect(output.atomicChunkCount).toBe(4); expect(chunks.length).toBeLessThan(output.atomicChunkCount); - expect(chunks.some((chunk) => chunk.files.length > 1)).toBe(true); + expect(output.groups.some((group) => group.files.length > 1)).toBe(true); }); }, ); diff --git a/packages/warden/src/semantic/planner.ts b/packages/warden/src/semantic/planner.ts index f23c4fab..26d61160 100644 --- a/packages/warden/src/semantic/planner.ts +++ b/packages/warden/src/semantic/planner.ts @@ -212,6 +212,7 @@ function buildSemanticChunkPlanningPrompt( '', 'Group atomic git chunks into semantic changes.', 'A semantic change should contain chunks that a reviewer should understand together: one behavior change, API contract, data flow, migration, validation rule, or test expectation.', + 'Collapse repeated small similar hunks into one semantic change when they apply the same transformation, update the same contract, or repeat the same expectation across call sites.', 'For each planned group, write a semantic delta summary: what behavior, contract, data flow, API, or test expectation changed.', 'Do not restate filenames or line ranges. Do not say "lines 10, 100, 200".', 'Do not summarize the input shape. Explain the product, security, data-flow, API, or test behavior being changed.', diff --git a/specs/semantic-chunk-benchmark.md b/specs/semantic-chunk-benchmark.md index 1b62904b..1b8c55bf 100644 --- a/specs/semantic-chunk-benchmark.md +++ b/specs/semantic-chunk-benchmark.md @@ -230,6 +230,11 @@ implementation and test-update chunks makes the scanner evaluate a coherent intentional migration, while raw hunk mode caught an isolated assertion removal as masking a regression. +Rejected follow-up experiments isolated changed test assertions as scanner +chunks. Those runs restored findings, but the rule was too benchmark-specific +and worked against the broader goal: semantic grouping should collapse repeated +small similar hunks into precise scanner slices, not special-case test syntax. + ## Acceptance Before using this benchmark to make decisions: From 27c7151a06f7c67f84f421702cf0273ac9a12f6a Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 16:53:56 -0700 Subject: [PATCH 12/18] perf(chunking): Add benchmark runner Move chunking benchmark work out of evals and into top-level benchmark infrastructure. Record the current baseline and keep semantic grouping notes tied to benchmark strategy instead of eval semantics. Also preserve the compact repeated tiny hunk behavior as the current comparison point for future benchmark runs. Refs GH-313 Co-Authored-By: GPT-5 Codex --- benchmarks/chunking/runner.ts | 551 ++++++++++++++++++ package.json | 2 +- packages/warden/src/semantic/planner.test.ts | 55 ++ packages/warden/src/semantic/planner.ts | 126 +++- .../chunking-strategy-benchmark-results.json | 492 ++++++++++++++++ specs/chunking-strategy-benchmark.md | 470 +++++++++++++++ specs/semantic-chunk-benchmark-results.json | 215 ------- specs/semantic-chunk-benchmark.md | 247 -------- specs/semantic-review-chunks.md | 2 +- 9 files changed, 1684 insertions(+), 476 deletions(-) create mode 100644 benchmarks/chunking/runner.ts create mode 100644 specs/chunking-strategy-benchmark-results.json create mode 100644 specs/chunking-strategy-benchmark.md delete mode 100644 specs/semantic-chunk-benchmark-results.json delete mode 100644 specs/semantic-chunk-benchmark.md diff --git a/benchmarks/chunking/runner.ts b/benchmarks/chunking/runner.ts new file mode 100644 index 00000000..adb569f3 --- /dev/null +++ b/benchmarks/chunking/runner.ts @@ -0,0 +1,551 @@ +#!/usr/bin/env tsx +import { execFileSync, spawnSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + writeFileSync, +} from 'node:fs'; +import { join, resolve } from 'node:path'; +import { tmpdir } from 'node:os'; +import { z } from 'zod'; + +const DEFAULT_CASES = [ + 'sentry-dashboard-axis-range-existing-widget', + 'sentry-cursor-service-account-api-key', + 'sentry-fixability-missing-issue-summary', + 'sentry-workflow-status-missing-foreign-key', +]; + +const PERFORMANCE_CASES = [ + { + name: 'sentry-mcp-search-issues-period-30d', + repository: 'getsentry/sentry-mcp', + base: 'df680f28fa705c447679bb8e0afa3f24e72387e0', + head: 'd4dbb32e7e05cd61024ec2adf06e53c358e77599', + }, + { + name: 'sentry-mcp-openrouter-provider', + repository: 'getsentry/sentry-mcp', + base: '032e7f6f28cd513699755920748bd10e9f429df5', + head: 'c42ef54a1de9e3640fc74f3e209c15704d094329', + }, + { + name: 'sentry-mcp-ai-conversation-search', + repository: 'getsentry/sentry-mcp', + base: '3bcaf5fd1db4ab1b13270606cc8808c8fa3fffea', + head: 'c11f3b9386ab33c4c85a90e8b9604bffafa14939', + }, + { + name: 'sentry-mcp-node-pnpm-baseline', + repository: 'getsentry/sentry-mcp', + base: '6e63abecbc732f61536f6df88a47a1fcde9d4c3e', + head: '7567694fcd28c36a0ffd7312eb0fd746cabb97fe', + }, + { + name: 'sentry-mcp-issue-search-project-period-endpoint', + repository: 'getsentry/sentry-mcp', + base: 'df680f28fa705c447679bb8e0afa3f24e72387e0', + head: '609a52120e0356da333280f301e9a41fcb55256e', + }, + { + name: 'sentry-mcp-ai-conversation-period-schema-default', + repository: 'getsentry/sentry-mcp', + base: 'df680f28fa705c447679bb8e0afa3f24e72387e0', + head: 'd4c8ec34cb1643d9a569ad7dd85a3ab653353511', + }, + { + name: 'sentry-mcp-ai-conversation-absolute-range-period-conflict', + repository: 'getsentry/sentry-mcp', + base: 'df680f28fa705c447679bb8e0afa3f24e72387e0', + head: 'e5df63d64bef3140deb88c8c80701f0583348afc', + }, + { + name: 'warden-split-pr-workflow', + repository: 'getsentry/warden', + base: '86699d45ec2ba2743f0a0c13dba46628ddaeeeb9', + head: 'df091dd43664d40ab9cf55c4407d5749c1ecc295', + }, + { + name: 'warden-global-scan-policy-limits', + repository: 'getsentry/warden', + base: '60bb7855a42922d5e36fbc30509ed5787caa9861', + head: 'ecf3162593bf019328b400c40500070c5f6af933', + }, + { + name: 'warden-remove-suggested-fix', + repository: 'getsentry/warden', + base: '876e1689996a6f599e5c64d81cb039fa4fbdf726', + head: 'bd8a4134197734cd843432ca9a349d16565467cc', + }, + { + name: 'warden-hoist-skills', + repository: 'getsentry/warden', + base: '7849f77b36c2ec3e024702007f39a7676a879d6d', + head: '36fcd770e9f9200961614043fcd0c76c62869ed4', + }, +] as const; + +const DEFAULT_MODEL = 'openrouter/anthropic/claude-sonnet-4.6'; +const DEFAULT_RUNTIME = 'pi'; + +const BenchmarkFindingSchema = z.object({ + title: z.string().optional(), + severity: z.string().optional(), + confidence: z.string().optional(), + location: z.object({ + path: z.string(), + startLine: z.number().optional(), + endLine: z.number().optional(), + }).optional(), +}).passthrough(); + +const ChunkRecordSchema = z.object({ + run: z.object({ + durationMs: z.number().optional(), + }).optional(), + chunk: z.object({ + lineRange: z.string().optional(), + }).optional(), + status: z.string().optional(), + findings: z.array(BenchmarkFindingSchema).optional(), + usageBreakdown: z.object({ + scan: z.object({ + usage: z.object({ + inputTokens: z.number().optional(), + outputTokens: z.number().optional(), + costUSD: z.number().optional(), + }).optional(), + }).optional(), + }).optional(), +}).passthrough(); + +const HistoricalScenarioSchema = z.object({ + notes: z.object({ + repository: z.string().optional(), + source: z.string().optional(), + source_ref: z.string().optional(), + }).optional(), + should_find: z.array(z.object({ + finding: z.string(), + })), +}).passthrough(); + +interface Args { + suite: 'historical' | 'performance'; + mode: 'nonsemantic' | 'semantic' | 'both'; + sentryRepo?: string; + sentryMcpRepo: string; + wardenRepo: string; + cases: string[]; + output: string; + artifactsDir: string; + model: string; + runtime: string; + keepWorktrees: boolean; +} + +interface BenchmarkCase { + name: string; + repository: string; + fixCommit: string; + vulnerableCommit: string; + expectedFindings: string[]; +} + +interface PerformanceCase { + name: string; + repository: string; + base: string; + head: string; +} + +interface RunSummary { + semantic: boolean; + complete: boolean; + outputPath: string; + exitCode: number; + expectedFindingMatched: boolean | null; + scannerChunks: number; + completedScannerChunks: number; + skippedScannerChunks: number; + failedScannerChunks: number; + findings: z.infer[]; + durationMs: number; + usage: { + inputTokens: number; + outputTokens: number; + costUSD: number; + }; +} + +function usage(exitCode = 2): never { + const text = [ + 'usage: pnpm exec tsx benchmarks/chunking/runner.ts [options]', + '', + 'Options:', + ' --suite historical or performance (default: historical)', + ' --mode nonsemantic, semantic, or both (default: both)', + ' --sentry-repo Existing getsentry/sentry checkout for historical cases', + ' --sentry-mcp-repo Existing getsentry/sentry-mcp checkout for performance cases', + ' --warden-repo Existing getsentry/warden checkout for performance cases', + ' --case Run one case; repeatable (default: initial four cases)', + ' --output Summary JSON path (default: /tmp/warden-chunking-benchmark.json)', + ' --artifacts-dir Raw JSONL artifact directory (default: /tmp/warden-chunking-benchmark-artifacts)', + ' --model Model override', + ' --runtime Runtime override', + ' --keep-worktrees Keep temporary synthetic worktrees', + ' -h, --help Show this help', + ].join('\n'); + if (exitCode === 0) console.log(text); + else console.error(text); + process.exit(exitCode); +} + +function parseArgs(argv: string[]): Args { + const args: Args = { + suite: 'historical', + mode: 'both', + cases: [], + sentryMcpRepo: '/home/dcramer/src/sentry-mcp', + wardenRepo: resolve(import.meta.dirname, '../..'), + output: '/tmp/warden-chunking-benchmark.json', + artifactsDir: '/tmp/warden-chunking-benchmark-artifacts', + model: process.env['WARDEN_BENCHMARK_MODEL'] ?? DEFAULT_MODEL, + runtime: process.env['WARDEN_BENCHMARK_RUNTIME'] ?? DEFAULT_RUNTIME, + keepWorktrees: false, + }; + + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (!arg) continue; + + if (arg === '-h' || arg === '--help') usage(0); + else if (arg === '--suite') { + const suite = argv[++i]; + if (suite !== 'historical' && suite !== 'performance') usage(); + args.suite = suite; + } else if (arg === '--mode') { + const mode = argv[++i]; + if (mode !== 'nonsemantic' && mode !== 'semantic' && mode !== 'both') usage(); + args.mode = mode; + } else if (arg === '--sentry-repo') args.sentryRepo = resolve(argv[++i] ?? usage()); + else if (arg === '--sentry-mcp-repo') args.sentryMcpRepo = resolve(argv[++i] ?? usage()); + else if (arg === '--warden-repo') args.wardenRepo = resolve(argv[++i] ?? usage()); + else if (arg === '--case') args.cases.push(argv[++i] ?? usage()); + else if (arg === '--output') args.output = resolve(argv[++i] ?? usage()); + else if (arg === '--artifacts-dir') args.artifactsDir = resolve(argv[++i] ?? usage()); + else if (arg === '--model') args.model = argv[++i] ?? usage(); + else if (arg === '--runtime') args.runtime = argv[++i] ?? usage(); + else if (arg === '--keep-worktrees') args.keepWorktrees = true; + else usage(); + } + + if (args.suite === 'historical' && !args.sentryRepo) usage(); + if (args.cases.length === 0) { + args.cases = args.suite === 'historical' + ? [...DEFAULT_CASES] + : PERFORMANCE_CASES.map((benchmarkCase) => benchmarkCase.name); + } + return args; +} + +function execGit(repo: string, gitArgs: string[]): string { + return execFileSync('git', gitArgs, { + cwd: repo, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim(); +} + +function fixCommitFromSource(source?: string): string | undefined { + return source?.match(/[0-9a-f]{40}/i)?.[0]; +} + +function loadBenchmarkCase(name: string): BenchmarkCase { + const scenarioDir = resolve(import.meta.dirname, '../../packages/evals/code-review'); + const filePath = readdirSync(scenarioDir) + .map((file) => join(scenarioDir, file)) + .find((file) => file.endsWith(`/${name}.json`)); + if (!filePath) { + throw new Error(`Unknown code-review eval case: ${name}`); + } + + const scenario = HistoricalScenarioSchema.parse(JSON.parse(readFileSync(filePath, 'utf8'))); + const fixCommit = fixCommitFromSource(scenario.notes?.source); + const vulnerableCommit = scenario.notes?.source_ref; + const repository = scenario.notes?.repository; + + if (!fixCommit || !vulnerableCommit || repository !== 'getsentry/sentry') { + throw new Error(`Case ${name} does not have getsentry/sentry fix/source refs`); + } + + return { + name, + repository, + fixCommit, + vulnerableCommit, + expectedFindings: scenario.should_find.map((item) => item.finding), + }; +} + +function createBenchmarkWorktree(sourceRepo: string, benchmarkCase: BenchmarkCase): string { + const worktree = mkdtempSync(join(tmpdir(), `warden-chunking-${benchmarkCase.name}-`)); + execGit(sourceRepo, ['worktree', 'add', '--detach', worktree, benchmarkCase.fixCommit]); + execGit(worktree, ['config', 'commit.gpgsign', 'false']); + execGit(worktree, ['config', 'tag.gpgsign', 'false']); + execGit(worktree, ['config', 'user.name', 'Warden Benchmark']); + execGit(worktree, ['config', 'user.email', 'warden-benchmark@example.com']); + const reverse = spawnSync('git', ['diff', `${benchmarkCase.vulnerableCommit}..${benchmarkCase.fixCommit}`, '--binary'], { + cwd: worktree, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }); + if (reverse.status !== 0) { + throw new Error(`Failed to create reverse diff for ${benchmarkCase.name}: ${reverse.stderr}`); + } + const apply = spawnSync('git', ['apply', '--reverse', '--index'], { + cwd: worktree, + input: reverse.stdout, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + if (apply.status !== 0) { + throw new Error(`Failed to apply reverse diff for ${benchmarkCase.name}: ${apply.stderr}`); + } + execGit(worktree, ['commit', '--no-verify', '-m', `benchmark: reintroduce ${benchmarkCase.name}`]); + return worktree; +} + +function writeBenchmarkConfig(path: string, semantic: boolean, model: string, runtime: string): string { + const configPath = join(path, semantic ? 'warden.semantic.toml' : 'warden.nonsemantic.toml'); + writeFileSync(configPath, [ + 'version = 1', + '', + '[defaults]', + `runtime = "${runtime}"`, + `model = "${model}"`, + 'failOn = "off"', + 'reportOn = "medium"', + 'minConfidence = "medium"', + '', + '[defaults.verification]', + 'enabled = false', + '', + '[defaults.chunking.semantic]', + `enabled = ${semantic ? 'true' : 'false'}`, + 'maxChunks = 20', + 'maxChunkChars = 20000', + 'maxHunksPerChunk = 4', + 'maxChangedRangesPerChunk = 4', + 'maxEmbeddedDiffChars = 8000', + 'maxEmbeddedDiffChunks = 12', + 'maxEmbeddedDiffRanges = 12', + 'preferWholeFileBelowLines = 800', + '', + '[[skills]]', + 'name = "code-review"', + 'paths = ["**/*"]', + '', + ].join('\n')); + return configPath; +} + +function summarizeJsonl(path: string, exitCode: number): RunSummary { + const findings: z.infer[] = []; + const usage = { inputTokens: 0, outputTokens: 0, costUSD: 0 }; + let scannerChunks = 0; + let completedScannerChunks = 0; + let skippedScannerChunks = 0; + let failedScannerChunks = 0; + let durationMs = 0; + + const content = existsSync(path) ? readFileSync(path, 'utf8').trim() : ''; + if (!content) { + return { + semantic: false, + complete: exitCode === 0, + outputPath: path, + exitCode, + expectedFindingMatched: null, + scannerChunks, + completedScannerChunks, + skippedScannerChunks, + failedScannerChunks, + findings, + durationMs, + usage, + }; + } + + for (const line of content.split('\n')) { + const parsed = ChunkRecordSchema.safeParse(JSON.parse(line)); + if (!parsed.success) continue; + const record = parsed.data; + if (record.run?.durationMs) durationMs = Math.max(durationMs, record.run.durationMs); + if (!record.chunk || record.chunk.lineRange === 'post-processing') continue; + + scannerChunks += 1; + if (record.status === 'ok') completedScannerChunks += 1; + else if (record.status === 'skipped') skippedScannerChunks += 1; + else failedScannerChunks += 1; + findings.push(...(record.findings ?? [])); + + const scanUsage = record.usageBreakdown?.scan?.usage; + usage.inputTokens += scanUsage?.inputTokens ?? 0; + usage.outputTokens += scanUsage?.outputTokens ?? 0; + usage.costUSD += scanUsage?.costUSD ?? 0; + } + + return { + semantic: false, + complete: exitCode === 0 && failedScannerChunks === 0, + outputPath: path, + exitCode, + expectedFindingMatched: null, + scannerChunks, + completedScannerChunks, + skippedScannerChunks, + failedScannerChunks, + findings, + durationMs, + usage, + }; +} + +function runWarden(args: Args, worktree: string, benchmarkCase: { name: string; base: string }, semantic: boolean): RunSummary { + const config = writeBenchmarkConfig(worktree, semantic, args.model, args.runtime); + const outputPath = join(args.artifactsDir, `${benchmarkCase.name}.${semantic ? 'semantic' : 'nonsemantic'}.jsonl`); + const cliArgs = [ + 'cli', + '--', + 'run', + `${benchmarkCase.base}..HEAD`, + '-C', + worktree, + '--skill', + 'code-review', + '-c', + config, + '--runtime', + args.runtime, + '--model', + args.model, + '--output', + outputPath, + '--log', + '--no-color', + ]; + const result = spawnSync('pnpm', cliArgs, { + cwd: resolve(import.meta.dirname, '../..'), + encoding: 'utf8', + stdio: 'inherit', + }); + + const summary = summarizeJsonl(outputPath, result.status ?? 1); + summary.semantic = semantic; + return summary; +} + +function selectedModes(args: Args): boolean[] { + if (args.mode === 'nonsemantic') return [false]; + if (args.mode === 'semantic') return [true]; + return [false, true]; +} + +function sourceRepoForPerformanceCase(args: Args, benchmarkCase: PerformanceCase): string { + if (benchmarkCase.repository === 'getsentry/sentry-mcp') return args.sentryMcpRepo; + if (benchmarkCase.repository === 'getsentry/warden') return args.wardenRepo; + throw new Error(`No source repo configured for ${benchmarkCase.repository}`); +} + +function loadPerformanceCase(name: string): PerformanceCase { + const benchmarkCase = PERFORMANCE_CASES.find((item) => item.name === name); + if (!benchmarkCase) throw new Error(`Unknown performance case: ${name}`); + return benchmarkCase; +} + +function createPerformanceWorktree(sourceRepo: string, benchmarkCase: PerformanceCase): string { + const worktree = mkdtempSync(join(tmpdir(), `warden-performance-${benchmarkCase.name}-`)); + execGit(sourceRepo, ['worktree', 'add', '--detach', worktree, benchmarkCase.head]); + return worktree; +} + +const args = parseArgs(process.argv.slice(2)); +mkdirSync(args.artifactsDir, { recursive: true }); + +const results = []; +if (args.suite === 'historical') { + const sentryRepo = args.sentryRepo; + if (!sentryRepo) usage(); + const cases = args.cases.map(loadBenchmarkCase); + for (const benchmarkCase of cases) { + console.log(`\n=== ${benchmarkCase.name} ===`); + const worktree = createBenchmarkWorktree(sentryRepo, benchmarkCase); + console.log(`Worktree: ${worktree}`); + const runs: Record = {}; + for (const semantic of selectedModes(args)) { + runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( + args, + worktree, + { name: benchmarkCase.name, base: benchmarkCase.fixCommit }, + semantic, + ); + } + results.push({ + case: benchmarkCase.name, + repository: benchmarkCase.repository, + base: benchmarkCase.fixCommit, + head: benchmarkCase.vulnerableCommit, + expectedFindings: benchmarkCase.expectedFindings, + runs, + worktree: args.keepWorktrees ? worktree : undefined, + }); + if (!args.keepWorktrees) { + execGit(sentryRepo, ['worktree', 'remove', '--force', worktree]); + } + } +} else { + const cases = args.cases.map(loadPerformanceCase); + for (const benchmarkCase of cases) { + console.log(`\n=== ${benchmarkCase.name} ===`); + const sourceRepo = sourceRepoForPerformanceCase(args, benchmarkCase); + const worktree = createPerformanceWorktree(sourceRepo, benchmarkCase); + console.log(`Worktree: ${worktree}`); + const runs: Record = {}; + for (const semantic of selectedModes(args)) { + runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( + args, + worktree, + { name: benchmarkCase.name, base: benchmarkCase.base }, + semantic, + ); + } + results.push({ + case: benchmarkCase.name, + repository: benchmarkCase.repository, + base: benchmarkCase.base, + head: benchmarkCase.head, + expectedFindings: [], + runs, + worktree: args.keepWorktrees ? worktree : undefined, + }); + if (!args.keepWorktrees) { + execGit(sourceRepo, ['worktree', 'remove', '--force', worktree]); + } + } +} + +writeFileSync(args.output, JSON.stringify({ + schemaVersion: 1, + capturedAt: new Date().toISOString(), + suite: args.suite, + mode: args.mode, + model: args.model, + runtime: args.runtime, + cases: results, +}, null, 2)); + +console.log(`\nWrote ${args.output}`); diff --git a/package.json b/package.json index e0be0b86..290fa6c0 100644 --- a/package.json +++ b/package.json @@ -36,7 +36,7 @@ "packages/evals/src/**/*.ts": [ "oxlint --fix" ], - "packages/evals/scripts/**/*.ts": [ + "benchmarks/**/*.ts": [ "oxlint --fix" ], "packages/evals/*.ts": [ diff --git a/packages/warden/src/semantic/planner.test.ts b/packages/warden/src/semantic/planner.test.ts index f15854c2..9ed413b9 100644 --- a/packages/warden/src/semantic/planner.test.ts +++ b/packages/warden/src/semantic/planner.test.ts @@ -428,6 +428,61 @@ describe('planSemanticReviewChunks', () => { ]); }); + it('collapses repeated tiny similar hunks into one compact scanner chunk', async () => { + const context = makeContext(); + context.pullRequest!.files = [{ + filename: 'src/repeated.ts', + status: 'modified', + additions: 5, + deletions: 5, + chunks: 5, + patch: [10, 60, 110, 160, 210].map((line) => [ + `@@ -${line},1 +${line},1 @@`, + "-params.push({axisRange: 'auto'});", + '+params.push({axisRange: undefined});', + ].join('\n')).join('\n'), + }]; + + const prepared = prepareFiles(context); + const chunkIds = prepared.files.flatMap((file) => file.chunks.map((chunk) => chunk.id)); + expect(chunkIds).toHaveLength(5); + + runAuxiliary.mockResolvedValueOnce({ + success: true, + data: { + groups: [{ + title: 'Update repeated axisRange defaults', + summary: 'Repeated call sites now pass an undefined axis range instead of the auto default.', + chunkIds, + }], + }, + usage: { + inputTokens: 100, + outputTokens: 25, + cacheReadInputTokens: 0, + cacheCreationInputTokens: 0, + cacheCreation5mInputTokens: 0, + cacheCreation1hInputTokens: 0, + webSearchRequests: 0, + costUSD: 0.001, + }, + }); + + const planned = await planSemanticReviewChunks(prepared.files, context, { + enabled: true, + runtime: 'pi', + maxHunksPerChunk: 2, + maxChangedRangesPerChunk: 2, + }); + + const chunk = planned.groups[0]?.chunks[0]; + expect(planned.groups).toHaveLength(1); + expect(planned.groups[0]?.chunks).toHaveLength(1); + expect(chunk?.changedLineMap).toHaveLength(5); + expect(chunk?.files[0]?.content).toContain("params.push({axisRange: 'auto'});"); + expect(chunk?.files[0]?.content).not.toContain('### Context Before'); + }); + it('rejects atomic chunks that exceed maxChunkChars', async () => { runAuxiliary.mockResolvedValueOnce({ success: true, diff --git a/packages/warden/src/semantic/planner.ts b/packages/warden/src/semantic/planner.ts index 26d61160..a9661d9f 100644 --- a/packages/warden/src/semantic/planner.ts +++ b/packages/warden/src/semantic/planner.ts @@ -21,6 +21,8 @@ const MAX_EMBEDDED_DIFF_RANGES = 12; const MAX_CHUNK_CHARS = 20000; const MAX_HUNKS_PER_CHUNK = 4; const MAX_CHANGED_RANGES_PER_CHUNK = 4; +const SIMILAR_TINY_RANGE_MULTIPLIER = 3; +const MAX_TINY_CHANGED_LINES = 6; export interface SemanticChunkPlanningOptions { enabled?: boolean; @@ -253,11 +255,86 @@ function mergeSourceLines(chunks: ReviewChunk[], path: string) { .sort((a, b) => a.line - b.line); } +function changedDiffLines(content: string): string[] { + return content.split('\n').filter((line) => + (line.startsWith('+') || line.startsWith('-')) && !line.startsWith('+++') && !line.startsWith('---') + ); +} + +function countChangedDiffLines(chunk: ReviewChunk): number { + return chunk.files.reduce((sum, file) => sum + changedDiffLines(file.content).length, 0); +} + +function normalizeChangedLine(line: string): string { + const sign = line[0]; + return `${sign}${line + .slice(1) + .replace(/(['"`])(?:\\.|(?!\1).)*\1/g, '') + .replace(/\b\d+(?:\.\d+)?\b/g, '') + .replace(/\s+/g, ' ') + .trim()}`; +} + +function similarityKey(chunk: ReviewChunk): string | undefined { + if (countChangedDiffLines(chunk) > MAX_TINY_CHANGED_LINES) return undefined; + + const firstFile = chunk.files[0]; + if (!firstFile) return undefined; + + const normalized = chunk.files + .flatMap((file) => changedDiffLines(file.content).map(normalizeChangedLine)) + .filter(Boolean); + if (normalized.length === 0) return undefined; + + return [ + firstFile.language, + normalized.join('\n'), + ].join(':'); +} + +function orderedSimilarityBatches(chunks: ReviewChunk[]): ReviewChunk[][] { + const batches: ReviewChunk[][] = []; + const indexByKey = new Map(); + + for (const chunk of chunks) { + const key = similarityKey(chunk); + if (!key) { + batches.push([chunk]); + continue; + } + + const existingIndex = indexByKey.get(key); + if (existingIndex !== undefined) { + batches[existingIndex]?.push(chunk); + continue; + } + + indexByKey.set(key, batches.length); + batches.push([chunk]); + } + + return batches; +} + +function compactRepeatedHunkContent(files: ReviewChunk['files'][number][]): string { + return files.map((file) => { + const lines = file.content.split('\n'); + const compactLines = lines.filter((line) => + line.startsWith('## Hunk:') || + line.startsWith('## Scope:') || + line.startsWith('@@') || + ((line.startsWith('+') || line.startsWith('-')) && !line.startsWith('+++') && !line.startsWith('---')) + ); + return compactLines.join('\n'); + }).join('\n\n---\n\n'); +} + function materializePlannedChunk( index: number, title: string, summary: string, - chunks: ReviewChunk[] + chunks: ReviewChunk[], + options: { compact?: boolean } = {} ): ReviewChunk { const changedLineMap = chunks.flatMap((chunk) => chunk.changedLineMap); const paths = [...new Set(chunks.flatMap((chunk) => chunk.files.map((file) => file.path)))]; @@ -272,7 +349,9 @@ function materializePlannedChunk( path, language: first.language, changedRanges: changedLineMap.filter((range) => range.path === path), - content: chunkFiles.map((file) => file.content).join('\n\n---\n\n'), + content: options.compact + ? compactRepeatedHunkContent(chunkFiles) + : chunkFiles.map((file) => file.content).join('\n\n---\n\n'), contentMode: 'raw-hunks' as const, sourceLines: mergeSourceLines(chunks, path), }; @@ -308,6 +387,7 @@ function splitPlannedChunks( ): ReviewChunk[] { const batches: ReviewChunk[][] = []; let current: ReviewChunk[] = []; + const plannedBatches = orderedSimilarityBatches(chunks); const pushCurrent = () => { if (current.length > 0) { @@ -316,7 +396,7 @@ function splitPlannedChunks( } }; - for (const chunk of chunks) { + const addChunk = (chunk: ReviewChunk, maxHunks: number, maxChangedRanges: number) => { const singleChunk = materializePlannedChunk(startIndex + batches.length, title, summary, [chunk]); if (reviewChunkContentChars(singleChunk) > limits.maxChunkChars) { throw new SkillRunnerError( @@ -327,13 +407,13 @@ function splitPlannedChunks( if (current.length === 0) { current = [chunk]; - continue; + return; } const candidate = [...current, chunk]; const candidateChunk = materializePlannedChunk(startIndex + batches.length, title, summary, candidate); - const exceedsHunks = candidate.length > limits.maxHunksPerChunk; - const exceedsChangedRanges = candidateChunk.changedLineMap.length > limits.maxChangedRangesPerChunk; + const exceedsHunks = candidate.length > maxHunks; + const exceedsChangedRanges = candidateChunk.changedLineMap.length > maxChangedRanges; const exceedsChars = reviewChunkContentChars(candidateChunk) > limits.maxChunkChars; if (exceedsHunks || exceedsChangedRanges || exceedsChars) { @@ -342,16 +422,38 @@ function splitPlannedChunks( } else { current = candidate; } + }; + + for (const plannedBatch of plannedBatches) { + const compactSimilar = plannedBatch.length > 1; + if (compactSimilar) pushCurrent(); + + const maxHunks = compactSimilar + ? limits.maxHunksPerChunk * SIMILAR_TINY_RANGE_MULTIPLIER + : limits.maxHunksPerChunk; + const maxChangedRanges = compactSimilar + ? limits.maxChangedRangesPerChunk * SIMILAR_TINY_RANGE_MULTIPLIER + : limits.maxChangedRangesPerChunk; + + for (const chunk of plannedBatch) { + addChunk(chunk, maxHunks, maxChangedRanges); + } + + if (compactSimilar) pushCurrent(); } pushCurrent(); - return batches.map((batch, batchIndex) => materializePlannedChunk( - startIndex + batchIndex, - plannedChunkTitle(title, batchIndex + 1, batches.length), - summary, - batch, - )); + return batches.map((batch, batchIndex) => { + const compact = batch.length > 1 && new Set(batch.map(similarityKey)).size === 1; + return materializePlannedChunk( + startIndex + batchIndex, + plannedChunkTitle(title, batchIndex + 1, batches.length), + summary, + batch, + { compact }, + ); + }); } function groupFromPreparedFile(file: PreparedFile): ReviewChunkGroup { diff --git a/specs/chunking-strategy-benchmark-results.json b/specs/chunking-strategy-benchmark-results.json new file mode 100644 index 00000000..09065e92 --- /dev/null +++ b/specs/chunking-strategy-benchmark-results.json @@ -0,0 +1,492 @@ +{ + "schemaVersion": 2, + "capturedAt": "2026-06-30", + "repository": "getsentry/sentry", + "model": "openrouter/anthropic/claude-sonnet-4.6", + "runtime": "pi", + "skill": "code-review", + "verification": false, + "notes": [ + "Each paired case uses the same synthetic bug-introducing head for semantic and non-semantic runs.", + "expectedFindingMatched is manual until the benchmark reuses the eval judge.", + "Axis-range was rerun with the maintainer runner after earlier one-off experiments; neither mode found the expected issue in that paired rerun.", + "Cursor and workflow-status preserve expected recall in both modes.", + "Fixability did not recover the expected issue in either mode.", + "Semantic mode reduced scanner chunks on two cases but increased token cost on every completed paired run." + ], + "cases": [ + { + "case": "sentry-dashboard-axis-range-existing-widget", + "base": "01217a6efb90cd26f0f8ac8b33c4f255379fe21d", + "head": "1b2028bb7455b62fba59a1eb3b9d00bdcff27d51", + "expectedFindingMatched": { + "nonsemantic": false, + "semantic": false + }, + "runs": { + "nonsemantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-dashboard-axis-range-existing-widget.nonsemantic.jsonl", + "sha256": "82ed06512dad89ac261b47d49a05634bbd191a131ba136082cad284a01e367d0" + }, + "complete": true, + "scannerChunks": 9, + "completedScannerChunks": 9, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 126324, + "usage": { + "inputTokens": 119788, + "outputTokens": 8589, + "costUSD": 0.3427662 + } + }, + "semantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-dashboard-axis-range-existing-widget.semantic.jsonl", + "sha256": "688d5510b204f11e4d4f0665afc768ccb902d59ecdcac4478edc2992215319b1" + }, + "complete": true, + "scannerChunks": 5, + "completedScannerChunks": 5, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 122745, + "usage": { + "inputTokens": 207575, + "outputTokens": 7501, + "costUSD": 0.3904299 + } + } + } + }, + { + "case": "sentry-cursor-service-account-api-key", + "base": "652324b48217e89f4267bfc1bb6e5e390818c277", + "head": "dd2e7671013b38550048c3c427b5152997e91ab9", + "expectedFindingMatched": { + "nonsemantic": true, + "semantic": true + }, + "runs": { + "nonsemantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-cursor-service-account-api-key.nonsemantic.jsonl", + "sha256": "19d7c300003680761d1f1e26da8eb3237398e840ad42d00cd716b3efe15ffe37" + }, + "complete": true, + "scannerChunks": 7, + "completedScannerChunks": 7, + "failedScannerChunks": 0, + "findings": [ + { + "title": "No tests for changed `build_integration` logic in Cursor integration", + "severity": "medium", + "location": { + "path": "src/sentry/integrations/cursor/integration.py", + "startLine": 133 + } + }, + { + "title": "Service account API keys broken by removal of /v0/models fallback", + "severity": "high", + "location": { + "path": "src/sentry/integrations/cursor/client.py", + "startLine": 92 + } + }, + { + "title": "`validation_error` may be unbound if `parse_obj({})` doesn't raise", + "severity": "low", + "location": { + "path": "tests/sentry/integrations/cursor/test_integration.py", + "startLine": 132 + } + } + ], + "durationMs": 289218, + "usage": { + "inputTokens": 217727, + "outputTokens": 16902, + "costUSD": 0.5859831 + } + }, + "semantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-cursor-service-account-api-key.semantic.jsonl", + "sha256": "601f1ec71529d5763fceb00453d1e69bdcaf71f29121c7e853a9698b555cf337" + }, + "complete": true, + "scannerChunks": 7, + "completedScannerChunks": 7, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Service account API keys can no longer be configured - /v0/me fallback removed", + "severity": "high", + "location": { + "path": "src/sentry/integrations/cursor/client.py", + "startLine": 85 + } + }, + { + "title": "`validation_error` may be unbound if `CursorApiKeyMetadata.parse_obj({})` does not raise", + "severity": "low", + "location": { + "path": "tests/sentry/integrations/cursor/test_integration.py", + "startLine": 136 + } + } + ], + "durationMs": 262986, + "usage": { + "inputTokens": 384304, + "outputTokens": 20268, + "costUSD": 0.71739075 + } + } + } + }, + { + "case": "sentry-fixability-missing-issue-summary", + "base": "62125c6514958cd89aa3cf7374be32f984adb683", + "head": "4199c6aeed84c7c359aa7ad6863534174769d436", + "expectedFindingMatched": { + "nonsemantic": false, + "semantic": false + }, + "runs": { + "nonsemantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-fixability-missing-issue-summary.nonsemantic.jsonl", + "sha256": "c8e1c874acab80a463c80c75ec99f26aadaa1eba24e116216c209f4e7e2ce15a" + }, + "complete": true, + "scannerChunks": 5, + "completedScannerChunks": 5, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Removed test leaves the `summary=None` -> no payload key path untested", + "severity": "low", + "location": { + "path": "tests/sentry/seer/autofix/test_issue_summary.py", + "startLine": 1219 + } + } + ], + "durationMs": 72248, + "usage": { + "inputTokens": 127987, + "outputTokens": 11044, + "costUSD": 0.35859315 + } + }, + "semantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-fixability-missing-issue-summary.semantic.jsonl", + "sha256": "86ebcd8341d3e649231d6be13743c5c1f13a9b7885aedc603d6ec8ab53028bfc" + }, + "complete": true, + "scannerChunks": 4, + "completedScannerChunks": 4, + "failedScannerChunks": 0, + "findings": [ + { + "title": "No test coverage for the new summary-passing logic in `generate_issue_summary_only`", + "severity": "low", + "location": { + "path": "src/sentry/tasks/autofix.py", + "startLine": 127 + } + } + ], + "durationMs": 189002, + "usage": { + "inputTokens": 602384, + "outputTokens": 23791, + "costUSD": 1.0902606 + } + } + } + }, + { + "case": "sentry-workflow-status-missing-foreign-key", + "base": "f4cc09c52e73c2ab60a3b14291c60dd0db5458a7", + "head": "e36f46a85cf4a6c9a6ae0e5e545a7c13b789d478", + "expectedFindingMatched": { + "nonsemantic": true, + "semantic": true + }, + "runs": { + "nonsemantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-workflow-status-missing-foreign-key.nonsemantic.jsonl", + "sha256": "86079775d10a098d5eefb5aac2a5d51e7cac5e4f2fd1f7e193c27bbad4e4ba5b" + }, + "complete": true, + "scannerChunks": 5, + "completedScannerChunks": 5, + "failedScannerChunks": 0, + "findings": [ + { + "title": "`connection.cursor()` bypasses Django's database router for a region-silo model", + "severity": "medium", + "location": { + "path": "src/sentry/workflow_engine/processors/action.py", + "startLine": 4 + } + }, + { + "title": "Unhandled IntegrityError when FK targets are deleted between read and INSERT", + "severity": "high", + "location": { + "path": "src/sentry/workflow_engine/processors/action.py", + "startLine": 147 + } + } + ], + "durationMs": 444294, + "usage": { + "inputTokens": 535681, + "outputTokens": 23518, + "costUSD": 1.1353497 + } + }, + "semantic": { + "rawArtifact": { + "path": "/tmp/warden-semantic-benchmark-artifacts/sentry-workflow-status-missing-foreign-key.semantic.jsonl", + "sha256": "6271846aa3cbb5a995ea2c8173a639bc867ebff5499c62ca2b8cbb1434203b45" + }, + "complete": true, + "scannerChunks": 5, + "completedScannerChunks": 5, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Raw SQL batch insert bypasses silo router, always targets `default` database", + "severity": "medium", + "location": { + "path": "src/sentry/workflow_engine/processors/action.py", + "startLine": 4 + } + }, + { + "title": "Unhandled IntegrityError on FK violation during batch insert", + "severity": "high", + "location": { + "path": "src/sentry/workflow_engine/processors/action.py", + "startLine": 146 + } + } + ], + "durationMs": 466334, + "usage": { + "inputTokens": 665740, + "outputTokens": 38759, + "costUSD": 1.45697175 + } + } + } + } + ], + "performanceBaselines": [ + { + "name": "nonsemantic-2026-06-30", + "suite": "performance", + "semantic": false, + "model": "openrouter/anthropic/claude-sonnet-4.6", + "runtime": "pi", + "skill": "code-review", + "verification": false, + "artifactsDir": "/tmp/warden-performance-baseline-artifacts", + "runs": [ + { + "case": "sentry-mcp-search-issues-period-30d", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/sentry-mcp-search-issues-period-30d.nonsemantic.jsonl", + "sha256": "6403c1fbd5663c3e94639adc6e72ac93c869e1e2609c6e90676cefcc3690d76c" + }, + "scannerChunks": 95, + "completedScannerChunks": 94, + "skippedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 420663, + "usage": { + "inputTokens": 2064917, + "outputTokens": 54596, + "costUSD": 3.87904965 + } + }, + { + "case": "sentry-mcp-openrouter-provider", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/sentry-mcp-openrouter-provider.nonsemantic.jsonl", + "sha256": "5523bc4b5f586a2cf54f01115a5f0f28a4267386116caf384064c881f04bca69" + }, + "scannerChunks": 56, + "completedScannerChunks": 56, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 278231, + "usage": { + "inputTokens": 970997, + "outputTokens": 46760, + "costUSD": 2.1067284 + } + }, + { + "case": "sentry-mcp-ai-conversation-search", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/sentry-mcp-ai-conversation-search.nonsemantic.jsonl", + "sha256": "1722ede696662d240aaf443d2cd01d4710f0e6b16b3e7ca4cb0df29343e774cd" + }, + "scannerChunks": 55, + "completedScannerChunks": 54, + "skippedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 1579419, + "usage": { + "inputTokens": 4902487, + "outputTokens": 231254, + "costUSD": 11.03394615 + } + }, + { + "case": "sentry-mcp-node-pnpm-baseline", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/sentry-mcp-node-pnpm-baseline.nonsemantic.jsonl", + "sha256": "ee51880c389de9a8393ecb44751a481733c8df55b9ef053644bfdc1435bba52e" + }, + "scannerChunks": 19, + "completedScannerChunks": 19, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [ + { + "title": "`@sentry/cli: false` in `allowBuilds` blocks binary download, breaking `@sentry/vite-plugin` at build time", + "severity": "high", + "confidence": "high", + "location": { + "path": "pnpm-workspace.yaml", + "startLine": 7 + } + } + ], + "durationMs": 214380, + "usage": { + "inputTokens": 559623, + "outputTokens": 10929, + "costUSD": 1.2961956 + } + }, + { + "case": "warden-split-pr-workflow", + "complete": false, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/warden-split-pr-workflow.nonsemantic.jsonl", + "sha256": "c2d24dc390726f92070b96bbccab1d16a270ba1b8b594ce08881cb52093aa229" + }, + "scannerChunks": 67, + "completedScannerChunks": 66, + "skippedScannerChunks": 0, + "failedScannerChunks": 1, + "findings": [], + "durationMs": 1436460, + "usage": { + "inputTokens": 3774012, + "outputTokens": 143749, + "costUSD": 6.92558805 + } + }, + { + "case": "warden-global-scan-policy-limits", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/warden-global-scan-policy-limits.nonsemantic.jsonl", + "sha256": "6016b62aff79a2f12bc381b1b2c14a67b596fef58b1e4ac3828979e4ab6aa6ee" + }, + "scannerChunks": 52, + "completedScannerChunks": 51, + "skippedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 708162, + "usage": { + "inputTokens": 2365164, + "outputTokens": 119140, + "costUSD": 4.787478 + } + }, + { + "case": "warden-remove-suggested-fix", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/warden-remove-suggested-fix.nonsemantic.jsonl", + "sha256": "536848cde519a240de05e4cec00231f3bc089fe7088926ff80be431ff78b9a3c" + }, + "scannerChunks": 58, + "completedScannerChunks": 57, + "skippedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 80827, + "usage": { + "inputTokens": 509490, + "outputTokens": 8588, + "costUSD": 0.7938351 + } + }, + { + "case": "warden-hoist-skills", + "complete": true, + "rawArtifact": { + "path": "/tmp/warden-performance-baseline-artifacts/warden-hoist-skills.nonsemantic.jsonl", + "sha256": "595d7b2c06a72e9ab3929930f3e73e7efbc19a7134e07007315c8ebb1827ebaf" + }, + "scannerChunks": 9, + "completedScannerChunks": 8, + "skippedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 69413, + "usage": { + "inputTokens": 108185, + "outputTokens": 10179, + "costUSD": 0.29617575 + } + } + ] + } + ], + "priorOneOffRuns": [ + { + "name": "axis-range-trimmed-semantic-off", + "note": "Trimmed two-file smoke test; useful for mechanics but not dataset quality.", + "scannerChunks": 2, + "findings": 1, + "costUSD": 1.22133795 + }, + { + "name": "axis-range-trimmed-semantic-on", + "note": "Trimmed two-file smoke test; useful for mechanics but not dataset quality.", + "scannerChunks": 1, + "findings": 1, + "costUSD": 0.46092255 + }, + { + "name": "axis-range-full-semantic-compact-similar", + "note": "Earlier one-off semantic experiment recovered the expected add-to-dashboard finding, but was not paired with a clean full non-semantic run.", + "scannerChunks": 6, + "findings": 1, + "costUSD": 1.95 + } + ] +} diff --git a/specs/chunking-strategy-benchmark.md b/specs/chunking-strategy-benchmark.md new file mode 100644 index 00000000..84e67383 --- /dev/null +++ b/specs/chunking-strategy-benchmark.md @@ -0,0 +1,470 @@ +# Chunking Strategy Benchmark + +Warden's scanner can become slow or expensive when a pull request has many git +hunks, very large generated/test chunks, or repeated tiny edits spread across +files. The scanner also loses precision when unrelated hunks are combined into +one broad prompt. This benchmark records both sides of that tradeoff. + +This is a maintainer benchmark plan, not public product documentation. Public +benchmark readouts belong under `packages/docs/src/content/docs/benchmarking/` +after the runner and result format are stable. + +## Current Takeaway + +The evidence so far does not support semantic grouping as the default +performance fix. Semantic grouping preserved recall on some historical cases, +but it did not reduce cost in the paired baseline and did not recover cases that +non-semantic chunking missed. + +The better next bet is a precision-preserving chunk optimizer: + +- collapse repeated tiny hunks when the edit shape is nearly identical +- merge sequential same-file hunks under strict size/range caps +- keep risky production chunks bounded +- treat generated/schema/test churn carefully because it dominates cost +- use semantic labels only as neutral metadata, not as authority to make giant + scanner prompts + +The benchmark suite now has three roles: + +- historical recall cases with known expected findings +- branch-evolution recall cases where later branch commits fix earlier branch + bugs +- real performance-shape PRs that measure cost, chunk count, failures, and + finding quality + +## Goal + +Answer two questions: + +- Does a chunking strategy preserve or improve known-finding recall? +- Does it reduce scanner calls, runtime, tokens, or cost on fragmented changes? + +The benchmark should not only prove that a chunking strategy produces fewer +chunks. It must prove that scanner behavior improves or stays correct when the +scanner receives optimized chunks instead of atomic git-hunk chunks. + +## Dataset Shape + +Use existing Sentry eval cases that already identify: + +- a vulnerable parent commit +- a fixing commit +- the expected finding +- source files tied to the bug + +Run the inverse diff as a bug-introducing change: + +```text +base = fixing commit +head = vulnerable parent/source_ref +``` + +That lets Warden review a realistic pull-request-shaped delta that reintroduces +the known bug. + +Apply the whole fixing commit or PR diff in reverse. Do not limit the synthetic +patch to `notes.source_files`. Those files identify the expected finding +location and provenance, but trimming the patch can remove related tests and +call-site edits. The point of this benchmark is to preserve the real +fragmentation shape. + +The benchmark should run against a real `getsentry/sentry` checkout, not the +current fixture-only eval repository. The semantic planner has read-only tools, +and those tools are only meaningful when surrounding repository context exists. + +## Initial Cases + +Start with these cases from `packages/evals/code-review/`. + +| Case | Vulnerable commit | Fix commit | Shape | Expected finding | +| --- | --- | --- | --- | --- | +| `sentry-dashboard-axis-range-existing-widget` | `1b2028bb7455b62fba59a1eb3b9d00bdcff27d51` | `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` | 6 files, 14 hunks | Existing dashboard widgets lose their saved axis range because builder defaults override it. | +| `sentry-cursor-service-account-api-key` | `dd2e7671013b38550048c3c427b5152997e91ab9` | `652324b48217e89f4267bfc1bb6e5e390818c277` | 3 files, 36 hunks | Cursor service-account API keys fail validation because `/v0/me` is insufficient. | +| `sentry-fixability-missing-issue-summary` | `4199c6aeed84c7c359aa7ad6863534174769d436` | `62125c6514958cd89aa3cf7374be32f984adb683` | 4 files, 20 hunks | Fixability calculation misses the cached issue summary needed by Seer. | +| `sentry-workflow-status-missing-foreign-key` | `e36f46a85cf4a6c9a6ae0e5e545a7c13b789d478` | `f4cc09c52e73c2ab60a3b14291c60dd0db5458a7` | 2 files, 17 hunks | Workflow status processing fails hard on missing or deleted foreign keys. | + +These four cover the core semantic chunking risks: + +- many tiny hunks that describe one behavior change +- cross-file implementation and test updates +- changes that need nearby code inspection +- enough fragmentation to make atomic scanner calls expensive + +Treat these as the historical recall suite. They answer whether a chunking +strategy still finds known bugs. They are not enough to answer whether a +chunking strategy handles slow, high-fragmentation real pull requests. + +## Performance Shape Cases + +Add a separate performance suite made of real pull requests that were slow or +pathologically fragmented. These cases do not need a known expected finding. +They measure scanner-call count, runtime, token cost, failed chunks, duplicate +findings, and whether reported findings are valid. + +| Case | Repository | Base | Head | Shape | Why it matters | +| --- | --- | --- | --- | --- | --- | +| `sentry-mcp-search-issues-period-30d` | `getsentry/sentry-mcp` | `df680f28fa705c447679bb8e0afa3f24e72387e0` | `d4dbb32e7e05cd61024ec2adf06e53c358e77599` | 21 files, 162 hunks, +324/-219 | Real slow PR. Repeated parameter rename/default/schema/test updates create many tiny hunks across related tool files. | +| `sentry-mcp-openrouter-provider` | `getsentry/sentry-mcp` | `032e7f6f28cd513699755920748bd10e9f429df5` | `c42ef54a1de9e3640fc74f3e209c15704d094329` | 34 files, 100 hunks, +650/-84 | Broad provider support change across config, docs, generated schemas, tests, and implementation. | +| `sentry-mcp-ai-conversation-search` | `getsentry/sentry-mcp` | `3bcaf5fd1db4ab1b13270606cc8808c8fa3fffea` | `c11f3b9386ab33c4c85a90e8b9604bffafa14939` | 27 files, 76 hunks, +2,608/-491 | Large feature PR where chunking must keep behavior, schemas, docs, and tests reviewable without making huge scanner prompts. | +| `sentry-mcp-node-pnpm-baseline` | `getsentry/sentry-mcp` | `6e63abecbc732f61536f6df88a47a1fcde9d4c3e` | `7567694fcd28c36a0ffd7312eb0fd746cabb97fe` | 17 files, 20 hunks, +42/-44 | Mechanical build/runtime baseline update with many small low-risk file edits. | +| `warden-split-pr-workflow` | `getsentry/warden` | `86699d45ec2ba2743f0a0c13dba46628ddaeeeb9` | `df091dd43664d40ab9cf55c4407d5749c1ecc295` | 29 files, 119 hunks, +2,578/-137 | Large workflow/action split with generated config, docs, and production behavior changes. | +| `warden-global-scan-policy-limits` | `getsentry/warden` | `60bb7855a42922d5e36fbc30509ed5787caa9861` | `ecf3162593bf019328b400c40500070c5f6af933` | 34 files, 80 hunks, +1,229/-257 | Broad product behavior change across policy loading, CLI/action paths, tests, and docs. | +| `warden-remove-suggested-fix` | `getsentry/warden` | `876e1689996a6f599e5c64d81cb039fa4fbdf726` | `bd8a4134197734cd843432ca9a349d16565467cc` | 41 files, 74 hunks, +17/-1,752 | Deletion-heavy cleanup where most hunks are removals and the scanner should avoid over-reviewing removed surfaces. | +| `warden-hoist-skills` | `getsentry/warden` | `7849f77b36c2ec3e024702007f39a7676a879d6d` | `36fcd770e9f9200961614043fcd0c76c62869ed4` | 34 files, 9 hunks, +20/-32 | Many-file move/rename-style change with low hunk count; useful control for avoiding unnecessary optimizer overhead. | + +The `sentry-mcp-search-issues-period-30d` final compare diff is the important +shape. Commit-patch output has 7 commits and double-counts files touched across +commits; Warden should benchmark the final PR diff from base to head. + +Run the full performance suite before claiming a strategy win. During local +iteration, use a smaller smoke subset: + +- `sentry-mcp-search-issues-period-30d` for repeated tiny hunks +- `warden-split-pr-workflow` for large workflow/config churn +- `warden-hoist-skills` as a low-risk many-file control + +Performance cases should be judged differently from historical recall cases: + +- valid findings per run, adjudicated manually until a judge exists +- scanner chunks and failed/interrupted chunks +- wall time and model usage +- duplicate or near-duplicate findings +- whether chunk grouping made findings more vague or less actionable +- whether mechanical repeated edits were reviewed once instead of many times + +Captured non-semantic baseline: + +- date: 2026-06-30 +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- skill: `code-review` +- verification: disabled +- raw artifacts: `/tmp/warden-performance-baseline-artifacts` + +| Case | Complete | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | yes | 95 | 0 | 7m01s | 2,064,917 | 54,596 | $3.8790 | +| `sentry-mcp-openrouter-provider` | yes | 56 | 0 | 4m38s | 970,997 | 46,760 | $2.1067 | +| `sentry-mcp-ai-conversation-search` | yes | 55 | 0 | 26m19s | 4,902,487 | 231,254 | $11.0339 | +| `sentry-mcp-node-pnpm-baseline` | yes | 19 | 1 | 3m34s | 559,623 | 10,929 | $1.2962 | +| `warden-split-pr-workflow` | no, 1 failed chunk | 67 | 0 | 23m56s | 3,774,012 | 143,749 | $6.9256 | +| `warden-global-scan-policy-limits` | yes | 52 | 0 | 11m48s | 2,365,164 | 119,140 | $4.7875 | +| `warden-remove-suggested-fix` | yes | 58 | 0 | 1m21s | 509,490 | 8,588 | $0.7938 | +| `warden-hoist-skills` | yes | 9 | 0 | 1m09s | 108,185 | 10,179 | $0.2962 | + +This baseline shows at least three different performance shapes: + +- repeated tiny schema/test hunks: `sentry-mcp-search-issues-period-30d` +- very large generated/schema/test chunks: `sentry-mcp-ai-conversation-search` + and `warden-split-pr-workflow` +- deletion-heavy or move/control cases that are already cheap enough: + `warden-remove-suggested-fix` and `warden-hoist-skills` + +This performance baseline is not a strong precision/recall benchmark yet. Only +`sentry-mcp-node-pnpm-baseline` produced a finding, so most of the suite can +only tell us whether a strategy is faster or cheaper. It cannot prove that a new +chunking strategy preserves bug-finding quality across large patches. + +Before using this suite as a quality gate, add at least: + +- one small or moderate PR-shaped case with a known valid finding +- one large or high-fragmentation PR-shaped case with a known valid finding + +Branch-evolution recall cases are the best near-term source: find PRs where a +later branch commit fixed a bug introduced by an earlier branch prefix, then run +Warden against the prefix before the fix. + +## Branch-Evolution Recall Cases + +Some slow PRs can also produce recall cases when a later branch commit fixes a +bug introduced by an earlier branch prefix. Use the later fix commit message and +diff as the expected finding, then run Warden against the earlier buggy prefix. + +For a linear branch: + +```text +base = PR base +buggy head = parent of fix commit +expected fix = fix commit +``` + +This gives us real-world recall without needing a known production incident. +The expected finding should be phrased from the later fix, but the scanner must +find it when reviewing `base..buggy head`. + +Candidate cases from `getsentry/sentry-mcp#1130`: + +| Case | Base | Buggy head | Fix commit | Expected finding | +| --- | --- | --- | --- | --- | +| `sentry-mcp-issue-search-project-period-endpoint` | `df680f28fa705c447679bb8e0afa3f24e72387e0` | `609a52120e0356da333280f301e9a41fcb55256e` | `d4c8ec34cb1643d9a569ad7dd85a3ab653353511` | Project-scoped longer-period issue searches still use the project issues endpoint instead of the organization issues endpoint with a numeric project filter. | +| `sentry-mcp-ai-conversation-period-schema-default` | `df680f28fa705c447679bb8e0afa3f24e72387e0` | `d4c8ec34cb1643d9a569ad7dd85a3ab653353511` | `601163d7c2259cf2d4da63a68698bbe67906fbc9` | `search_ai_conversations` exposes a runtime 30d default but the generated schema does not expose that default to clients. | +| `sentry-mcp-ai-conversation-absolute-range-period-conflict` | `df680f28fa705c447679bb8e0afa3f24e72387e0` | `e5df63d64bef3140deb88c8c80701f0583348afc` | `82890445a1f95d794a3cf7b99cb7c33dc7134c18` | Absolute `start`/`end` AI conversation searches conflict with injected/default `period` instead of letting absolute bounds win. | + +These are not replacements for the historical Sentry eval cases. They are a +middle ground: real branch context, known later fix, and high-fragmentation PR +shape. + +## Expansion Cases + +Add these after the initial four work. + +| Case | Vulnerable commit | Fix commit | Shape | Expected finding | +| --- | --- | --- | --- | --- | +| `sentry-dashboard-delete-side-nav-stale` | `14170bd6ae28abe4d4f0b5807185b116f777a1a0` | `86872f4f1491c4392a25f4d3fcba31a12726fa63` | 3 files, 14 hunks | Deleting a dashboard leaves stale side-nav state. | +| `sentry-action-dedup-by-workflow` | `1730e13b97865d3ee8943c6f860964388c4987a8` | `165be911ba388d402993b58f34dc8ad683827e32` | 4 files, 8 hunks | Action dedup keys include workflow id, causing duplicate notifications. | + +## Security Controls + +Use security-review cases as correctness controls, not the main fragmentation +stress set. These are useful because they exercise Warden's security skill, but +most are smaller diffs. + +| Case | Source | Fix commit(s) | Expected finding | +| --- | --- | --- | --- | +| `sentry-preprod-snapshot-project-access` | `https://github.com/getsentry/sentry/pull/114169` | `8fac324d82c903c8022b99dcd4329f3944e57196` | Snapshot detail GET/DELETE bypasses project access. | +| `sentry-slack-options-load-unscoped-group` | `https://github.com/getsentry/sentry/pull/114185` | `0f09491755f71a95343285cbe17c93bf272a0d62`, `b718661dd8560a20a826d90ee6755f153957969c` | Slack options-load resolves a group without verifying the integration belongs to the group's org. | +| `sentry-release-threshold-empty-project-filter` | `https://github.com/getsentry/sentry/pull/114049` | `8a93913509441a0c8e7d035f9c4bc24dabed2d86` | Empty accessible-project filters remove project scoping from release threshold queries. | + +Security cases whose provenance is only a blob SHA should stay out of the first +benchmark until the fixing commit is identified. + +## Run Method + +For each historical recall case: + +1. Create or reuse a temporary Sentry worktree. +2. Check out the fixing commit. +3. Create a benchmark branch. +4. Apply the full inverse fixing diff back to the vulnerable commit. +5. Run Warden with semantic chunking disabled. +6. Reset to the same benchmark branch. +7. Run Warden with semantic chunking enabled. +8. Judge both reports against the existing `should_find` assertion. + +For each performance shape case: + +1. Create or reuse a temporary worktree for the source repository. +2. Check out the PR base SHA. +3. Apply or check out the PR head SHA. +4. Run each chunking strategy against `base..head`. +5. Record scanner calls, runtime, tokens, cost, failed chunks, findings, and + manual finding validity. + +The runner lives under `benchmarks/chunking/` because this is maintainer +benchmark infrastructure, not the eval framework. It may read historical eval +fixture JSON for commit provenance, but it does not run through `pnpm evals` and +should not live under `packages/evals`. Do not add a root `pnpm` script until +the output format and workflow are worth preserving. + +Current maintainer runner: + +```bash +pnpm exec tsx benchmarks/chunking/runner.ts \ + --sentry-repo ~/src/sentry \ + --output /tmp/warden-chunking-benchmark.json \ + --artifacts-dir /tmp/warden-chunking-benchmark-artifacts +``` + +By default this runs the initial four cases. Use `--case ` one or more +times to run a smaller slice while iterating. Historical cases create synthetic +bug-introducing worktrees. Performance cases use explicit base/head SHAs from +real PR-shaped diffs. Both modes write paired summaries plus raw JSONL +artifacts. + +## Strategies To Compare + +Use the same cases across each strategy: + +- current non-semantic chunking +- current semantic grouping +- deterministic precision-preserving optimizer +- optimizer plus optional neutral semantic labels, if implemented + +The optimizer is not a replacement for all chunking. It should activate only +for pathological patch shapes: high raw chunk count, high hunk count in one +file, many tiny hunks, or large total diff size. + +## Metrics + +Record these for each semantic-off and semantic-on run: + +- pass/fail against expected finding +- total findings +- duplicate or near-duplicate findings +- failed chunks +- failed extractions +- wall time +- scanner chunk count +- semantic planner group count +- scanner chunks per semantic group +- semantic planner summaries +- input tokens +- output tokens +- recorded cost + +The semantic-on run is successful when it keeps the expected finding and reduces +operational load. A lower chunk count is not enough if recall regresses. +Semantic groups are allowed to contain multiple scanner chunks; the target is +fewer, better bounded scanner calls than raw git chunks, not one giant scanner +prompt per logical change. + +## Output + +Store raw run output separately from checked-in docs until the format stabilizes. +The final durable artifact should be a small JSON result set that can power a +public docs readout later, similar to the existing Sentry security benchmark +data under `packages/docs/src/data/benchmarking/`. + +At minimum, each result record should include: + +```json +{ + "case": "sentry-dashboard-axis-range-existing-widget", + "repository": "getsentry/sentry", + "base": "01217a6efb90cd26f0f8ac8b33c4f255379fe21d", + "head": "1b2028bb7455b62fba59a1eb3b9d00bdcff27d51", + "semantic": true, + "passed": true, + "findings": 1, + "scannerChunks": 3, + "semanticGroups": 1, + "durationMs": 0, + "usage": { + "inputTokens": 0, + "outputTokens": 0, + "costUSD": 0 + }, + "expectedFindingMatched": null +} +``` + +`expectedFindingMatched` stays `null` until the benchmark reuses the existing +eval judge for semantic finding matching. Do not use exact title matching as a +substitute; these reports often describe the same bug with different wording. + +## Captured Baseline + +Paired maintainer-runner baseline: + +- date: 2026-06-30 +- cases: initial four Sentry code-review cases +- repository: `getsentry/sentry` +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- skill: `code-review` +- verification: disabled +- results: `specs/chunking-strategy-benchmark-results.json` + +| Case | Mode | Expected found | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| axis range | semantic off | no | 9 | 0 | 2m06s | 119,788 | 8,589 | $0.3428 | +| axis range | semantic on | no | 5 | 0 | 2m03s | 207,575 | 7,501 | $0.3904 | +| cursor service account | semantic off | yes | 7 | 3 | 4m49s | 217,727 | 16,902 | $0.5860 | +| cursor service account | semantic on | yes | 7 | 2 | 4m23s | 384,304 | 20,268 | $0.7174 | +| fixability summary | semantic off | no | 5 | 1 | 1m12s | 127,987 | 11,044 | $0.3586 | +| fixability summary | semantic on | no | 4 | 1 | 3m09s | 602,384 | 23,791 | $1.0903 | +| workflow FK | semantic off | yes | 5 | 2 | 7m24s | 535,681 | 23,518 | $1.1353 | +| workflow FK | semantic on | yes | 5 | 2 | 7m46s | 665,740 | 38,759 | $1.4570 | + +This baseline is more useful than the earlier single-case experiments: + +- Recall was preserved on the two cases where either mode found the known issue. +- Semantic mode did not recover cases that non-semantic missed. +- Semantic mode reduced scanner chunks on axis-range and fixability, but did + not reduce cost on any completed paired run. +- Cursor and workflow-status show that semantic grouping can preserve high + signal findings, but the planner overhead currently dominates the cost model. + +Initial smoke run: + +- date: 2026-06-30 +- case: `sentry-dashboard-axis-range-existing-widget` +- repository: `getsentry/sentry` +- synthetic base: `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` +- synthetic head: `4d022227929c4d923cfb9bb2b45721407570cf8c` +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- skill: `code-review` +- verification: disabled + +This smoke run intentionally used a trimmed two-file reverse patch. It proves +that the benchmark mechanics and semantic grouping path work, but it should not +be used as the dataset quality baseline. + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| semantic off | 2 | 1 | 11m54s | 445,997 | 39,484 | $1.2213 | +| semantic on | 1 | 1 | 3m03s | 160,096 | 10,079 | $0.4609 | + +The semantic-on run grouped both changed ranges into one scanner chunk and kept +the expected medium-confidence axis-range regression finding. + +Full reverse-patch run: + +- date: 2026-06-30 +- case: `sentry-dashboard-axis-range-existing-widget` +- repository: `getsentry/sentry` +- synthetic base: `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` +- synthetic head: `a4778dc5fcf` +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- skill: `code-review` +- verification: disabled + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| non-semantic, partial | 1/13 completed | 1 | 2m53s | 160,637 | 9,024 | $0.3191 | +| semantic on, bounded groups | 5 | 0 | 10m45s | 481,391 | 34,884 | $1.2564 | +| semantic on, bounded groups plus summary anti-bias prompt | 4 | 0 | 8m19s | 472,104 | 26,978 | $1.1046 | +| semantic on, bounded groups plus changed-range cap | 5 | 0 | 7m24s | 432,183 | 23,708 | $0.9862 | +| semantic on, neutral scanner prompt | 4 | 0 | 5m14s | 374,416 | 15,714 | $0.7504 | +| semantic on, neutral prompt plus changed-range cap 2 | 7 | 0 | 13m40s | 807,047 | 45,554 | $2.0509 | +| semantic on, compact similar tiny hunks | 6 | 1 | 7m23s | 758,900 | 50,400 | $1.9500 | + +The full non-semantic run was interrupted after one completed scanner chunk, so +it is not a clean cost comparison. That completed chunk did find the expected +style of regression: removing `axisRange: 'auto'` assertions masked a behavior +break. The semantic full-patch runs preserved the real patch shape but did not +find the expected axis-range regression. This makes the case useful as a recall +warning: semantic grouping can make a bad behavior look like a coherent +migration when tests are changed to match it. The scanner prompt now states +that semantic summaries are grouping hints, not evidence of correctness. + +The changed-range cap run is incomplete for score comparison because one +test-heavy scanner chunk hit `turn_limit`. It is still useful operationally: it +split one broad semantic group into two scanner chunks and reduced recorded +cost, but the run cannot be treated as a clean recall result. + +The neutral scanner prompt removed semantic summaries from scanner prompts. It +reduced cost again but still missed recall, and a stricter changed-range cap of +2 increased scanner calls and cost without recovering the finding. This points +away from size alone. The stronger hypothesis is that semantically grouping +implementation and test-update chunks makes the scanner evaluate a coherent +intentional migration, while raw hunk mode caught an isolated assertion removal +as masking a regression. + +Rejected follow-up experiments isolated changed test assertions as scanner +chunks. Those runs restored findings, but the rule was too benchmark-specific +and worked against the broader goal: semantic grouping should collapse repeated +small similar hunks into precise scanner slices, not special-case test syntax. + +The compact-similar run restored the add-to-dashboard finding without a +test-specific rule by materializing repeated tiny hunks in a compact form. It is +not yet a clear cost win over earlier semantic runs, but it is the first generic +semantic variant that recovered the expected finding on the full patch. + +## Acceptance + +Before using this benchmark to make decisions: + +- every initial case must run both modes from the same synthetic branch +- performance cases must run every strategy against the same base/head SHA +- the runner must preserve enough logs to debug planner grouping decisions +- expected finding judgments must reuse the existing eval judge or equivalent + semantic matching +- semantic-on must expose planner summaries in the artifact +- interrupted or timed-out runs must be marked incomplete, not compared +- performance wins must preserve finding quality, not only reduce chunk count +- optimized chunking must beat semantic mode on runtime or cost before it becomes + the default for pathological patches diff --git a/specs/semantic-chunk-benchmark-results.json b/specs/semantic-chunk-benchmark-results.json deleted file mode 100644 index 6025a16b..00000000 --- a/specs/semantic-chunk-benchmark-results.json +++ /dev/null @@ -1,215 +0,0 @@ -{ - "schemaVersion": 1, - "capturedAt": "2026-06-30", - "case": "sentry-dashboard-axis-range-existing-widget", - "repository": "getsentry/sentry", - "model": "openrouter/anthropic/claude-sonnet-4.6", - "runtime": "pi", - "skill": "code-review", - "verification": false, - "runs": [ - { - "name": "trimmed-off", - "patch": "trimmed two-file reverse patch", - "semantic": false, - "complete": true, - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-off.jsonl", - "sha256": "0bfacc26a78f572b0d31e06d75c9a3dda55077b2d61a09fcca6092227e884895" - }, - "scannerChunks": 2, - "completedScannerChunks": 2, - "failedScannerChunks": 0, - "findings": [ - { - "title": "`convertWidgetToBuilderStateParams` no longer defaults `axisRange` to `'auto'` for null/invalid widget values, breaking existing contracts and tests", - "severity": "medium", - "confidence": "high", - "location": { - "path": "static/app/views/dashboards/widgetBuilder/utils/convertWidgetToBuilderStateParams.ts", - "startLine": 79 - } - } - ], - "durationMs": 714272, - "usage": { - "inputTokens": 445997, - "outputTokens": 39484, - "costUSD": 1.2213379500000001 - } - }, - { - "name": "trimmed-on", - "patch": "trimmed two-file reverse patch", - "semantic": true, - "complete": true, - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-on.jsonl", - "sha256": "9ad2d4b7565cab4780285e0e653bdd90124d74537c39c21d570fb0656ff9778e" - }, - "scannerChunks": 1, - "completedScannerChunks": 1, - "failedScannerChunks": 0, - "findings": [ - { - "title": "Removing `?? 'auto'` fallback from `getAxisRange` causes test failures and propagates `undefined` axisRange downstream", - "severity": "medium", - "confidence": "high", - "location": { - "path": "static/app/views/dashboards/widgetBuilder/utils/convertWidgetToBuilderStateParams.ts", - "startLine": 79 - } - } - ], - "durationMs": 182759, - "usage": { - "inputTokens": 160096, - "outputTokens": 10079, - "costUSD": 0.46092255 - } - }, - { - "name": "full-off-partial", - "patch": "full reverse patch", - "semantic": false, - "complete": false, - "reason": "interrupted after first completed raw git chunk", - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-full-off.jsonl", - "sha256": "e975e8cf1346d66726d1fd4b8846969e695883817d963996287c009179c9aafa" - }, - "scannerChunks": 1, - "completedScannerChunks": 1, - "failedScannerChunks": 0, - "findings": [ - { - "title": "Removal of `axisRange: 'auto'` assertion masks regression where axis range is not forwarded to widget builder", - "severity": "medium", - "confidence": "high", - "location": { - "path": "static/app/components/modals/widgetBuilder/addToDashboardModal.spec.tsx", - "startLine": 340, - "endLine": 345 - } - } - ], - "durationMs": 173444, - "usage": { - "inputTokens": 160637, - "outputTokens": 9024, - "costUSD": 0.3190759499999999 - } - }, - { - "name": "full-semantic-bounded", - "patch": "full reverse patch", - "semantic": true, - "complete": true, - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-full-bounded.jsonl", - "sha256": "cdb283b9469e05b8f9e2b7c6601849b8959668dd7eba5526e0b966105763dd65" - }, - "scannerChunks": 5, - "completedScannerChunks": 5, - "failedScannerChunks": 0, - "findings": [], - "durationMs": 644575, - "usage": { - "inputTokens": 481391, - "outputTokens": 34884, - "costUSD": 1.2563874 - } - }, - { - "name": "full-semantic-antibias", - "patch": "full reverse patch", - "semantic": true, - "complete": true, - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-full-bounded-v2.jsonl", - "sha256": "10334451cffeb231e5b6ff05190bcaf39ed6233cf521c5864c7f5858ef4b6c29" - }, - "scannerChunks": 4, - "completedScannerChunks": 4, - "failedScannerChunks": 0, - "findings": [], - "durationMs": 499429, - "usage": { - "inputTokens": 472104, - "outputTokens": 26978, - "costUSD": 1.10464215 - } - }, - { - "name": "full-semantic-range-cap", - "patch": "full reverse patch", - "semantic": true, - "complete": false, - "reason": "one scanner chunk hit turn_limit", - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-full-bounded-ranges.jsonl", - "sha256": "d188e68a3af7753c83626b2c666ac675ae51eea3d678b3c940308223b738891d" - }, - "scannerChunks": 5, - "completedScannerChunks": 4, - "failedScannerChunks": 1, - "findings": [], - "durationMs": 443935, - "usage": { - "inputTokens": 432183, - "outputTokens": 23708, - "costUSD": 0.9861766500000001 - } - }, - { - "name": "full-semantic-neutral-prompt", - "patch": "full reverse patch", - "semantic": true, - "complete": false, - "reason": "one scanner chunk hit turn_limit", - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-full-neutral-prompt.jsonl", - "sha256": "9d8de49aefee67a7fc7fa06adfccecc6ce64c3a7f6edd29a9e91ea0798e7d5bb" - }, - "scannerChunks": 4, - "completedScannerChunks": 3, - "failedScannerChunks": 1, - "findings": [], - "durationMs": 314311, - "usage": { - "inputTokens": 374416, - "outputTokens": 15714, - "costUSD": 0.7503958500000002 - } - }, - { - "name": "full-semantic-range-cap-2", - "patch": "full reverse patch", - "semantic": true, - "complete": false, - "reason": "one scanner chunk hit turn_limit", - "rawArtifact": { - "path": "/tmp/warden-semantic-axis-full-range2.jsonl", - "sha256": "27a9fc77a4522219dbd6bfb06b17c54c23a276c9514421d96425d58221ac6744" - }, - "scannerChunks": 7, - "completedScannerChunks": 6, - "failedScannerChunks": 1, - "findings": [], - "durationMs": 820311, - "usage": { - "inputTokens": 807047, - "outputTokens": 45554, - "costUSD": 2.050875 - } - } - ], - "notes": [ - "The trimmed smoke case is useful for mechanics but not dataset quality.", - "The full non-semantic run is incomplete but produced the expected-style test-regression finding before interruption.", - "Full semantic runs completed faster after tuning but missed the expected finding.", - "Removing semantic summaries from scanner prompts reduced cost but did not restore recall.", - "A stricter changed-range cap increased scanner calls and cost without restoring recall.", - "The current evidence points to recall loss from semantic framing and test/update grouping, not only oversized scanner chunks." - ] -} diff --git a/specs/semantic-chunk-benchmark.md b/specs/semantic-chunk-benchmark.md deleted file mode 100644 index 1b8c55bf..00000000 --- a/specs/semantic-chunk-benchmark.md +++ /dev/null @@ -1,247 +0,0 @@ -# Semantic Chunk Benchmark Plan - -Semantic chunking should be measured against real changesets where the scanner -needs to understand a logical change that git split across many small hunks. -The benchmark should compare Warden with semantic chunking disabled and enabled -on the same bug-introducing diffs. - -This is a maintainer benchmark plan, not public product documentation. Public -benchmark readouts belong under `packages/docs/src/content/docs/benchmarking/` -after the runner and result format are stable. - -## Goal - -Answer two questions: - -- Does semantic chunking preserve or improve known-finding recall? -- Does it reduce scanner calls, runtime, tokens, or cost on fragmented changes? - -The benchmark should not only prove that the planner can produce summaries. It -must prove that scanner behavior improves or stays correct when the scanner -receives semantic review chunks instead of atomic git-hunk chunks. - -## Dataset Shape - -Use existing Sentry eval cases that already identify: - -- a vulnerable parent commit -- a fixing commit -- the expected finding -- source files tied to the bug - -Run the inverse diff as a bug-introducing change: - -```text -base = fixing commit -head = vulnerable parent/source_ref -``` - -That lets Warden review a realistic pull-request-shaped delta that reintroduces -the known bug. - -Apply the whole fixing commit or PR diff in reverse. Do not limit the synthetic -patch to `notes.source_files`. Those files identify the expected finding -location and provenance, but trimming the patch can remove related tests and -call-site edits. The point of this benchmark is to preserve the real -fragmentation shape. - -The benchmark should run against a real `getsentry/sentry` checkout, not the -current fixture-only eval repository. The semantic planner has read-only tools, -and those tools are only meaningful when surrounding repository context exists. - -## Initial Cases - -Start with these cases from `packages/evals/code-review/`. - -| Case | Vulnerable commit | Fix commit | Shape | Expected finding | -| --- | --- | --- | --- | --- | -| `sentry-dashboard-axis-range-existing-widget` | `1b2028bb7455b62fba59a1eb3b9d00bdcff27d51` | `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` | 6 files, 14 hunks | Existing dashboard widgets lose their saved axis range because builder defaults override it. | -| `sentry-cursor-service-account-api-key` | `dd2e7671013b38550048c3c427b5152997e91ab9` | `652324b48217e89f4267bfc1bb6e5e390818c277` | 3 files, 36 hunks | Cursor service-account API keys fail validation because `/v0/me` is insufficient. | -| `sentry-fixability-missing-issue-summary` | `4199c6aeed84c7c359aa7ad6863534174769d436` | `62125c6514958cd89aa3cf7374be32f984adb683` | 4 files, 20 hunks | Fixability calculation misses the cached issue summary needed by Seer. | -| `sentry-workflow-status-missing-foreign-key` | `e36f46a85cf4a6c9a6ae0e5e545a7c13b789d478` | `f4cc09c52e73c2ab60a3b14291c60dd0db5458a7` | 2 files, 17 hunks | Workflow status processing fails hard on missing or deleted foreign keys. | - -These four cover the core semantic chunking risks: - -- many tiny hunks that describe one behavior change -- cross-file implementation and test updates -- changes that need nearby code inspection -- enough fragmentation to make atomic scanner calls expensive - -## Expansion Cases - -Add these after the initial four work. - -| Case | Vulnerable commit | Fix commit | Shape | Expected finding | -| --- | --- | --- | --- | --- | -| `sentry-dashboard-delete-side-nav-stale` | `14170bd6ae28abe4d4f0b5807185b116f777a1a0` | `86872f4f1491c4392a25f4d3fcba31a12726fa63` | 3 files, 14 hunks | Deleting a dashboard leaves stale side-nav state. | -| `sentry-action-dedup-by-workflow` | `1730e13b97865d3ee8943c6f860964388c4987a8` | `165be911ba388d402993b58f34dc8ad683827e32` | 4 files, 8 hunks | Action dedup keys include workflow id, causing duplicate notifications. | - -## Security Controls - -Use security-review cases as correctness controls, not the main fragmentation -stress set. These are useful because they exercise Warden's security skill, but -most are smaller diffs. - -| Case | Source | Fix commit(s) | Expected finding | -| --- | --- | --- | --- | -| `sentry-preprod-snapshot-project-access` | `https://github.com/getsentry/sentry/pull/114169` | `8fac324d82c903c8022b99dcd4329f3944e57196` | Snapshot detail GET/DELETE bypasses project access. | -| `sentry-slack-options-load-unscoped-group` | `https://github.com/getsentry/sentry/pull/114185` | `0f09491755f71a95343285cbe17c93bf272a0d62`, `b718661dd8560a20a826d90ee6755f153957969c` | Slack options-load resolves a group without verifying the integration belongs to the group's org. | -| `sentry-release-threshold-empty-project-filter` | `https://github.com/getsentry/sentry/pull/114049` | `8a93913509441a0c8e7d035f9c4bc24dabed2d86` | Empty accessible-project filters remove project scoping from release threshold queries. | - -Security cases whose provenance is only a blob SHA should stay out of the first -benchmark until the fixing commit is identified. - -## Run Method - -For each benchmark case: - -1. Create or reuse a temporary Sentry worktree. -2. Check out the fixing commit. -3. Create a benchmark branch. -4. Apply the full inverse fixing diff back to the vulnerable commit. -5. Run Warden with semantic chunking disabled. -6. Reset to the same benchmark branch. -7. Run Warden with semantic chunking enabled. -8. Judge both reports against the existing `should_find` assertion. - -The first implementation can be a one-off script under `packages/evals/scripts/` -or a local maintainer command. Do not add a root `pnpm` script until the output -format and workflow are worth preserving. - -## Metrics - -Record these for each semantic-off and semantic-on run: - -- pass/fail against expected finding -- total findings -- duplicate or near-duplicate findings -- failed chunks -- failed extractions -- wall time -- scanner chunk count -- semantic planner group count -- scanner chunks per semantic group -- semantic planner summaries -- input tokens -- output tokens -- recorded cost - -The semantic-on run is successful when it keeps the expected finding and reduces -operational load. A lower chunk count is not enough if recall regresses. -Semantic groups are allowed to contain multiple scanner chunks; the target is -fewer, better bounded scanner calls than raw git chunks, not one giant scanner -prompt per logical change. - -## Output - -Store raw run output separately from checked-in docs until the format stabilizes. -The final durable artifact should be a small JSON result set that can power a -public docs readout later, similar to the existing Sentry security benchmark -data under `packages/docs/src/data/benchmarking/`. - -At minimum, each result record should include: - -```json -{ - "case": "sentry-dashboard-axis-range-existing-widget", - "repository": "getsentry/sentry", - "base": "01217a6efb90cd26f0f8ac8b33c4f255379fe21d", - "head": "1b2028bb7455b62fba59a1eb3b9d00bdcff27d51", - "semantic": true, - "passed": true, - "findings": 1, - "scannerChunks": 3, - "semanticGroups": 1, - "durationMs": 0, - "usage": { - "inputTokens": 0, - "outputTokens": 0, - "costUSD": 0 - } -} -``` - -## Captured Baseline - -Initial smoke run: - -- date: 2026-06-30 -- case: `sentry-dashboard-axis-range-existing-widget` -- repository: `getsentry/sentry` -- synthetic base: `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` -- synthetic head: `4d022227929c4d923cfb9bb2b45721407570cf8c` -- model: `openrouter/anthropic/claude-sonnet-4.6` -- runtime: `pi` -- skill: `code-review` -- verification: disabled - -This smoke run intentionally used a trimmed two-file reverse patch. It proves -that the benchmark mechanics and semantic grouping path work, but it should not -be used as the dataset quality baseline. - -| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| semantic off | 2 | 1 | 11m54s | 445,997 | 39,484 | $1.2213 | -| semantic on | 1 | 1 | 3m03s | 160,096 | 10,079 | $0.4609 | - -The semantic-on run grouped both changed ranges into one scanner chunk and kept -the expected medium-confidence axis-range regression finding. - -Full reverse-patch run: - -- date: 2026-06-30 -- case: `sentry-dashboard-axis-range-existing-widget` -- repository: `getsentry/sentry` -- synthetic base: `01217a6efb90cd26f0f8ac8b33c4f255379fe21d` -- synthetic head: `a4778dc5fcf` -- model: `openrouter/anthropic/claude-sonnet-4.6` -- runtime: `pi` -- skill: `code-review` -- verification: disabled - -| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| non-semantic, partial | 1/13 completed | 1 | 2m53s | 160,637 | 9,024 | $0.3191 | -| semantic on, bounded groups | 5 | 0 | 10m45s | 481,391 | 34,884 | $1.2564 | -| semantic on, bounded groups plus summary anti-bias prompt | 4 | 0 | 8m19s | 472,104 | 26,978 | $1.1046 | -| semantic on, bounded groups plus changed-range cap | 5 | 0 | 7m24s | 432,183 | 23,708 | $0.9862 | -| semantic on, neutral scanner prompt | 4 | 0 | 5m14s | 374,416 | 15,714 | $0.7504 | -| semantic on, neutral prompt plus changed-range cap 2 | 7 | 0 | 13m40s | 807,047 | 45,554 | $2.0509 | - -The full non-semantic run was interrupted after one completed scanner chunk, so -it is not a clean cost comparison. That completed chunk did find the expected -style of regression: removing `axisRange: 'auto'` assertions masked a behavior -break. The semantic full-patch runs preserved the real patch shape but did not -find the expected axis-range regression. This makes the case useful as a recall -warning: semantic grouping can make a bad behavior look like a coherent -migration when tests are changed to match it. The scanner prompt now states -that semantic summaries are grouping hints, not evidence of correctness. - -The changed-range cap run is incomplete for score comparison because one -test-heavy scanner chunk hit `turn_limit`. It is still useful operationally: it -split one broad semantic group into two scanner chunks and reduced recorded -cost, but the run cannot be treated as a clean recall result. - -The neutral scanner prompt removed semantic summaries from scanner prompts. It -reduced cost again but still missed recall, and a stricter changed-range cap of -2 increased scanner calls and cost without recovering the finding. This points -away from size alone. The stronger hypothesis is that semantically grouping -implementation and test-update chunks makes the scanner evaluate a coherent -intentional migration, while raw hunk mode caught an isolated assertion removal -as masking a regression. - -Rejected follow-up experiments isolated changed test assertions as scanner -chunks. Those runs restored findings, but the rule was too benchmark-specific -and worked against the broader goal: semantic grouping should collapse repeated -small similar hunks into precise scanner slices, not special-case test syntax. - -## Acceptance - -Before using this benchmark to make decisions: - -- every initial case must run both modes from the same synthetic branch -- the runner must preserve enough logs to debug planner grouping decisions -- expected finding judgments must reuse the existing eval judge or equivalent - semantic matching -- semantic-on must expose planner summaries in the artifact -- interrupted or timed-out runs must be marked incomplete, not compared diff --git a/specs/semantic-review-chunks.md b/specs/semantic-review-chunks.md index b0068e96..e8c10e40 100644 --- a/specs/semantic-review-chunks.md +++ b/specs/semantic-review-chunks.md @@ -60,7 +60,7 @@ Git hunks remain the evidence and anchoring primitive. Review chunks become the scanner-facing primitive. Benchmarking for this work is tracked in -[`semantic-chunk-benchmark.md`](semantic-chunk-benchmark.md). +[`chunking-strategy-benchmark.md`](chunking-strategy-benchmark.md). ## Module Boundary From 3247b553c5226996e4474c6141eaa4e97372c735 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 17:51:28 -0700 Subject: [PATCH 13/18] test(chunking): Add recall benchmark candidates Add PR-shaped recall candidates to the chunking benchmark runner and record measured outcomes. Keep the stable Slack security case separate from unstable or negative candidates so future chunking changes compare against honest recall data. Refs GH-313 Co-Authored-By: GPT-5 Codex --- benchmarks/chunking/runner.ts | 94 ++++++++++++++----- .../chunking-strategy-benchmark-results.json | 93 ++++++++++++++++++ specs/chunking-strategy-benchmark.md | 48 ++++++++++ 3 files changed, 213 insertions(+), 22 deletions(-) diff --git a/benchmarks/chunking/runner.ts b/benchmarks/chunking/runner.ts index adb569f3..45842404 100644 --- a/benchmarks/chunking/runner.ts +++ b/benchmarks/chunking/runner.ts @@ -19,6 +19,29 @@ const DEFAULT_CASES = [ 'sentry-workflow-status-missing-foreign-key', ]; +const RECALL_CASES = [ + { + name: 'warden-error-cause-chain-regression', + repository: 'getsentry/warden', + skill: 'code-review', + fixCommit: '9bf777889e554ac30216ebfbd1f15a68abf92529', + vulnerableCommit: '9273d82be6582d78c4809c3b4317dba09bf1b58b', + expectedFindings: [ + 'Removing Error cause propagation loses original error details when git and runtime errors are rethrown.', + ], + }, + { + name: 'sentry-slack-options-load-unscoped-group', + repository: 'getsentry/sentry', + skill: 'security-review', + fixCommit: '0f09491755f71a95343285cbe17c93bf272a0d62', + vulnerableCommit: 'c1bc01ad419ac251e153c81f212628221f8c0628', + expectedFindings: [ + 'Slack options-load resolves a Group by caller-controlled id without binding it to the requesting Slack integration organization.', + ], + }, +] as const; + const PERFORMANCE_CASES = [ { name: 'sentry-mcp-search-issues-period-30d', @@ -150,6 +173,7 @@ interface Args { interface BenchmarkCase { name: string; repository: string; + skill: string; fixCommit: string; vulnerableCommit: string; expectedFindings: string[]; @@ -158,6 +182,7 @@ interface BenchmarkCase { interface PerformanceCase { name: string; repository: string; + skill?: string; base: string; head: string; } @@ -243,7 +268,6 @@ function parseArgs(argv: string[]): Args { else usage(); } - if (args.suite === 'historical' && !args.sentryRepo) usage(); if (args.cases.length === 0) { args.cases = args.suite === 'historical' ? [...DEFAULT_CASES] @@ -265,6 +289,9 @@ function fixCommitFromSource(source?: string): string | undefined { } function loadBenchmarkCase(name: string): BenchmarkCase { + const recallCase = RECALL_CASES.find((item) => item.name === name); + if (recallCase) return { ...recallCase }; + const scenarioDir = resolve(import.meta.dirname, '../../packages/evals/code-review'); const filePath = readdirSync(scenarioDir) .map((file) => join(scenarioDir, file)) @@ -285,6 +312,7 @@ function loadBenchmarkCase(name: string): BenchmarkCase { return { name, repository, + skill: 'code-review', fixCommit, vulnerableCommit, expectedFindings: scenario.should_find.map((item) => item.finding), @@ -294,10 +322,6 @@ function loadBenchmarkCase(name: string): BenchmarkCase { function createBenchmarkWorktree(sourceRepo: string, benchmarkCase: BenchmarkCase): string { const worktree = mkdtempSync(join(tmpdir(), `warden-chunking-${benchmarkCase.name}-`)); execGit(sourceRepo, ['worktree', 'add', '--detach', worktree, benchmarkCase.fixCommit]); - execGit(worktree, ['config', 'commit.gpgsign', 'false']); - execGit(worktree, ['config', 'tag.gpgsign', 'false']); - execGit(worktree, ['config', 'user.name', 'Warden Benchmark']); - execGit(worktree, ['config', 'user.email', 'warden-benchmark@example.com']); const reverse = spawnSync('git', ['diff', `${benchmarkCase.vulnerableCommit}..${benchmarkCase.fixCommit}`, '--binary'], { cwd: worktree, encoding: 'utf8', @@ -315,12 +339,24 @@ function createBenchmarkWorktree(sourceRepo: string, benchmarkCase: BenchmarkCas if (apply.status !== 0) { throw new Error(`Failed to apply reverse diff for ${benchmarkCase.name}: ${apply.stderr}`); } - execGit(worktree, ['commit', '--no-verify', '-m', `benchmark: reintroduce ${benchmarkCase.name}`]); + execGit(worktree, [ + '-c', + 'commit.gpgsign=false', + '-c', + 'user.name=Warden Benchmark', + '-c', + 'user.email=warden-benchmark@example.com', + 'commit', + '--no-verify', + '-m', + `benchmark: reintroduce ${benchmarkCase.name}`, + ]); return worktree; } -function writeBenchmarkConfig(path: string, semantic: boolean, model: string, runtime: string): string { +function writeBenchmarkConfig(path: string, semantic: boolean, model: string, runtime: string, skill: string): string { const configPath = join(path, semantic ? 'warden.semantic.toml' : 'warden.nonsemantic.toml'); + const findingThreshold = skill === 'security-review' ? 'low' : 'medium'; writeFileSync(configPath, [ 'version = 1', '', @@ -328,8 +364,8 @@ function writeBenchmarkConfig(path: string, semantic: boolean, model: string, ru `runtime = "${runtime}"`, `model = "${model}"`, 'failOn = "off"', - 'reportOn = "medium"', - 'minConfidence = "medium"', + `reportOn = "${findingThreshold}"`, + `minConfidence = "${findingThreshold}"`, '', '[defaults.verification]', 'enabled = false', @@ -346,7 +382,7 @@ function writeBenchmarkConfig(path: string, semantic: boolean, model: string, ru 'preferWholeFileBelowLines = 800', '', '[[skills]]', - 'name = "code-review"', + `name = "${skill}"`, 'paths = ["**/*"]', '', ].join('\n')); @@ -415,8 +451,13 @@ function summarizeJsonl(path: string, exitCode: number): RunSummary { }; } -function runWarden(args: Args, worktree: string, benchmarkCase: { name: string; base: string }, semantic: boolean): RunSummary { - const config = writeBenchmarkConfig(worktree, semantic, args.model, args.runtime); +function runWarden( + args: Args, + worktree: string, + benchmarkCase: { name: string; base: string; skill: string }, + semantic: boolean, +): RunSummary { + const config = writeBenchmarkConfig(worktree, semantic, args.model, args.runtime, benchmarkCase.skill); const outputPath = join(args.artifactsDir, `${benchmarkCase.name}.${semantic ? 'semantic' : 'nonsemantic'}.jsonl`); const cliArgs = [ 'cli', @@ -426,7 +467,7 @@ function runWarden(args: Args, worktree: string, benchmarkCase: { name: string; '-C', worktree, '--skill', - 'code-review', + benchmarkCase.skill, '-c', config, '--runtime', @@ -456,9 +497,17 @@ function selectedModes(args: Args): boolean[] { } function sourceRepoForPerformanceCase(args: Args, benchmarkCase: PerformanceCase): string { - if (benchmarkCase.repository === 'getsentry/sentry-mcp') return args.sentryMcpRepo; - if (benchmarkCase.repository === 'getsentry/warden') return args.wardenRepo; - throw new Error(`No source repo configured for ${benchmarkCase.repository}`); + return sourceRepoForRepository(args, benchmarkCase.repository); +} + +function sourceRepoForRepository(args: Args, repository: string): string { + if (repository === 'getsentry/sentry') { + if (!args.sentryRepo) throw new Error('Missing --sentry-repo for getsentry/sentry benchmark case'); + return args.sentryRepo; + } + if (repository === 'getsentry/sentry-mcp') return args.sentryMcpRepo; + if (repository === 'getsentry/warden') return args.wardenRepo; + throw new Error(`No source repo configured for ${repository}`); } function loadPerformanceCase(name: string): PerformanceCase { @@ -478,25 +527,25 @@ mkdirSync(args.artifactsDir, { recursive: true }); const results = []; if (args.suite === 'historical') { - const sentryRepo = args.sentryRepo; - if (!sentryRepo) usage(); const cases = args.cases.map(loadBenchmarkCase); for (const benchmarkCase of cases) { console.log(`\n=== ${benchmarkCase.name} ===`); - const worktree = createBenchmarkWorktree(sentryRepo, benchmarkCase); + const sourceRepo = sourceRepoForRepository(args, benchmarkCase.repository); + const worktree = createBenchmarkWorktree(sourceRepo, benchmarkCase); console.log(`Worktree: ${worktree}`); const runs: Record = {}; for (const semantic of selectedModes(args)) { runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( args, worktree, - { name: benchmarkCase.name, base: benchmarkCase.fixCommit }, + { name: benchmarkCase.name, base: benchmarkCase.fixCommit, skill: benchmarkCase.skill }, semantic, ); } results.push({ case: benchmarkCase.name, repository: benchmarkCase.repository, + skill: benchmarkCase.skill, base: benchmarkCase.fixCommit, head: benchmarkCase.vulnerableCommit, expectedFindings: benchmarkCase.expectedFindings, @@ -504,7 +553,7 @@ if (args.suite === 'historical') { worktree: args.keepWorktrees ? worktree : undefined, }); if (!args.keepWorktrees) { - execGit(sentryRepo, ['worktree', 'remove', '--force', worktree]); + execGit(sourceRepo, ['worktree', 'remove', '--force', worktree]); } } } else { @@ -519,13 +568,14 @@ if (args.suite === 'historical') { runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( args, worktree, - { name: benchmarkCase.name, base: benchmarkCase.base }, + { name: benchmarkCase.name, base: benchmarkCase.base, skill: benchmarkCase.skill ?? 'code-review' }, semantic, ); } results.push({ case: benchmarkCase.name, repository: benchmarkCase.repository, + skill: benchmarkCase.skill ?? 'code-review', base: benchmarkCase.base, head: benchmarkCase.head, expectedFindings: [], diff --git a/specs/chunking-strategy-benchmark-results.json b/specs/chunking-strategy-benchmark-results.json index 09065e92..cb725f0e 100644 --- a/specs/chunking-strategy-benchmark-results.json +++ b/specs/chunking-strategy-benchmark-results.json @@ -466,6 +466,99 @@ ] } ], + "candidateBaselines": [ + { + "case": "sentry-slack-options-load-unscoped-group", + "repository": "getsentry/sentry", + "suite": "historical", + "mode": "nonsemantic", + "skill": "security-review", + "base": "0f09491755f71a95343285cbe17c93bf272a0d62", + "head": "synthetic reverse of c1bc01ad419ac251e153c81f212628221f8c0628..0f09491755f71a95343285cbe17c93bf272a0d62", + "complete": true, + "expectedFindingMatched": true, + "rawArtifact": { + "path": "/tmp/warden-slack-options-low-runner-artifacts/sentry-slack-options-load-unscoped-group.nonsemantic.jsonl", + "sha256": "d89d157b158fc854ad092598f77dda0109a280c3a8a4bde15e3725f146b9ddfc" + }, + "scannerChunks": 4, + "completedScannerChunks": 4, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Removed cross-organization authorization check allows IDOR on group assignment data", + "severity": "high", + "confidence": "high", + "location": { + "path": "src/sentry/integrations/slack/webhooks/options_load.py", + "startLine": 99, + "endLine": 104 + } + } + ], + "durationMs": 24815, + "usage": { + "inputTokens": 20437, + "outputTokens": 1065, + "costUSD": 0.06328665 + } + }, + { + "case": "warden-error-cause-chain-regression", + "repository": "getsentry/warden", + "suite": "historical", + "mode": "nonsemantic", + "skill": "code-review", + "base": "9bf777889e554ac30216ebfbd1f15a68abf92529", + "head": "synthetic reverse of 9273d82be6582d78c4809c3b4317dba09bf1b58b..9bf777889e554ac30216ebfbd1f15a68abf92529", + "complete": true, + "expectedFindingMatched": false, + "note": "Manual one-off produced the expected Error cause finding once, but the benchmark runner rerun produced no findings. Keep as an unstable candidate, not an accepted recall baseline.", + "rawArtifact": { + "path": "/tmp/warden-pr215-runner-artifacts/warden-error-cause-chain-regression.nonsemantic.jsonl", + "sha256": "edd93d525bca6e237fe45a912d344de6f7f8f7ca7e7ba73e7fcba5e2563e1a3e" + }, + "scannerChunks": 8, + "completedScannerChunks": 8, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 203827, + "usage": { + "inputTokens": 345903, + "outputTokens": 26543, + "costUSD": 0.83203875 + } + }, + { + "case": "sentry-mcp-ai-conversation-absolute-range-period-conflict", + "repository": "getsentry/sentry-mcp", + "suite": "performance", + "mode": "nonsemantic", + "skill": "code-review", + "base": "df680f28fa705c447679bb8e0afa3f24e72387e0", + "head": "e5df63d64bef3140deb88c8c80701f0583348afc", + "complete": true, + "expectedFindingMatched": false, + "note": "High-fragmentation branch-prefix candidate with a later fix commit, but current code-review produced no findings.", + "rawArtifact": { + "path": "/tmp/warden-branch-recall-absolute-artifacts/sentry-mcp-ai-conversation-absolute-range-period-conflict.nonsemantic.jsonl", + "sha256": "78e59e7e32ec461968f34df18cf0df1967c826e958cd458942936394ec38a7ae" + }, + "scannerChunks": 95, + "completedScannerChunks": 94, + "skippedScannerChunks": 1, + "failedScannerChunks": 0, + "findings": [], + "durationMs": 1691226, + "usage": { + "inputTokens": 7056116, + "outputTokens": 245823, + "costUSD": 14.9409432 + } + } + ], "priorOneOffRuns": [ { "name": "axis-range-trimmed-semantic-off", diff --git a/specs/chunking-strategy-benchmark.md b/specs/chunking-strategy-benchmark.md index 84e67383..83802e62 100644 --- a/specs/chunking-strategy-benchmark.md +++ b/specs/chunking-strategy-benchmark.md @@ -95,6 +95,42 @@ Treat these as the historical recall suite. They answer whether a chunking strategy still finds known bugs. They are not enough to answer whether a chunking strategy handles slow, high-fragmentation real pull requests. +## PR-Shaped Recall Cases + +Use real fix PRs in reverse when they produce current Warden findings. These are +not evals; they are benchmark fixtures that keep the full PR-shaped diff and +use the fix PR body or linked Warden issue as the oracle. + +| Case | Repository | PR | Vulnerable commit | Fix commit | Shape | Skill | Expected finding | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `sentry-slack-options-load-unscoped-group` | `getsentry/sentry` | `#114185` | `c1bc01ad419ac251e153c81f212628221f8c0628` | `0f09491755f71a95343285cbe17c93bf272a0d62` | 2 files, 4 raw chunks, +29/-0 when fixed | `security-review` | Slack options-load can resolve a caller-controlled group id without binding it to the requesting Slack integration's organization. | +| `warden-error-cause-chain-regression` | `getsentry/warden` | `#215` | `9273d82be6582d78c4809c3b4317dba09bf1b58b` | `9bf777889e554ac30216ebfbd1f15a68abf92529` | 7 files, 9 raw chunks, +18/-13 when fixed | `code-review` | Dropping `Error` causes loses original git/runtime error diagnostics. Candidate only; finding was not stable across reruns. | + +Captured non-semantic baseline: + +- date: 2026-06-30 +- model: `openrouter/anthropic/claude-sonnet-4.6` +- runtime: `pi` +- verification: disabled +- raw artifact for Slack case: `/tmp/warden-slack-options-low-runner-artifacts/sentry-slack-options-load-unscoped-group.nonsemantic.jsonl` +- raw artifact for Warden #215 candidate: `/tmp/warden-pr215-runner-artifacts/warden-error-cause-chain-regression.nonsemantic.jsonl` + +| Case | Complete | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-slack-options-load-unscoped-group` | yes | 4 | 1 | 24.8s | 20,437 | 1,065 | $0.0633 | +| `warden-error-cause-chain-regression` | yes | 8 | 0 | 3m24s | 345,903 | 26,543 | $0.8320 | + +Finding produced by current `security-review`: + +- `src/sentry/integrations/slack/webhooks/options_load.py:99-104` - + Removed cross-organization authorization check allows IDOR on group assignment + data + +The Slack case is a small known-positive case. It is not large enough to prove +the high-fragmentation performance path. The Warden #215 case remains useful as +a repeated small-hunk candidate, but it is not accepted as a recall baseline +until the finding reproduces reliably. + ## Performance Shape Cases Add a separate performance suite made of real pull requests that were slow or @@ -205,6 +241,18 @@ These are not replacements for the historical Sentry eval cases. They are a middle ground: real branch context, known later fix, and high-fragmentation PR shape. +Current baseline attempt: + +- `sentry-mcp-ai-conversation-absolute-range-period-conflict` + - non-semantic run completed against the buggy prefix + - 95 scanner chunks, 94 completed, 1 skipped + - 0 findings + - 28m11s, 7,056,116 input tokens, 245,823 output tokens, $14.9409 + +This case remains a strong performance stress case, but it is not a +known-positive recall benchmark for current `code-review`. The scanner did not +find the later fixed `start`/`end` versus `period` conflict in the baseline run. + ## Expansion Cases Add these after the initial four work. From 61b26dfcd39886114d84a5055d354d17502c3ad0 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 18:07:52 -0700 Subject: [PATCH 14/18] test(chunking): Add security recall baselines Add additional PR-shaped security recall cases that current Warden actually finds. Record the measured non-semantic baselines separately from performance-only cases so chunking comparisons have real recall controls. Refs GH-313 Co-Authored-By: GPT-5 Codex --- benchmarks/chunking/runner.ts | 30 +++++ .../chunking-strategy-benchmark-results.json | 109 ++++++++++++++++++ specs/chunking-strategy-benchmark.md | 27 ++++- 3 files changed, 160 insertions(+), 6 deletions(-) diff --git a/benchmarks/chunking/runner.ts b/benchmarks/chunking/runner.ts index 45842404..3b617b69 100644 --- a/benchmarks/chunking/runner.ts +++ b/benchmarks/chunking/runner.ts @@ -40,6 +40,36 @@ const RECALL_CASES = [ 'Slack options-load resolves a Group by caller-controlled id without binding it to the requesting Slack integration organization.', ], }, + { + name: 'sentry-preprod-snapshot-project-access', + repository: 'getsentry/sentry', + skill: 'security-review', + fixCommit: '8fac324d82c903c8022b99dcd4329f3944e57196', + vulnerableCommit: 'c1bc01ad419ac251e153c81f212628221f8c0628', + expectedFindings: [ + 'Preprod snapshot GET and DELETE load artifacts by organization only and do not check project access.', + ], + }, + { + name: 'sentry-release-threshold-empty-project-filter', + repository: 'getsentry/sentry', + skill: 'security-review', + fixCommit: '8a93913509441a0c8e7d035f9c4bc24dabed2d86', + vulnerableCommit: '8f9fe309854228051dabac985fb813476a2a5b24', + expectedFindings: [ + 'ReleaseThreshold query omits project and organization scoping when accessible projects is empty.', + ], + }, + { + name: 'sentry-replay-delete-read-scope', + repository: 'getsentry/sentry', + skill: 'security-review', + fixCommit: '9bf0ea738cd7847438d4a2cfb1fbdbb326426e01', + vulnerableCommit: 'c1bc01ad419ac251e153c81f212628221f8c0628', + expectedFindings: [ + 'Replay DELETE accepts project:read, allowing read-only project users to delete replay data.', + ], + }, ] as const; const PERFORMANCE_CASES = [ diff --git a/specs/chunking-strategy-benchmark-results.json b/specs/chunking-strategy-benchmark-results.json index cb725f0e..04a2e8ac 100644 --- a/specs/chunking-strategy-benchmark-results.json +++ b/specs/chunking-strategy-benchmark-results.json @@ -504,6 +504,115 @@ "costUSD": 0.06328665 } }, + { + "case": "sentry-preprod-snapshot-project-access", + "repository": "getsentry/sentry", + "suite": "historical", + "mode": "nonsemantic", + "skill": "security-review", + "base": "8fac324d82c903c8022b99dcd4329f3944e57196", + "head": "synthetic reverse of c1bc01ad419ac251e153c81f212628221f8c0628..8fac324d82c903c8022b99dcd4329f3944e57196", + "complete": true, + "expectedFindingMatched": true, + "rawArtifact": { + "path": "/tmp/warden-security-recall-batch-artifacts/sentry-preprod-snapshot-project-access.nonsemantic.jsonl", + "sha256": "301d555e94c4876b1b4c2ce8a712447df2ca1b59ee8f35e0f170556cf82744d7" + }, + "scannerChunks": 4, + "completedScannerChunks": 4, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Project-level access check removed from DELETE endpoint, enabling cross-project artifact deletion", + "severity": "high", + "confidence": "high", + "location": { + "path": "src/sentry/preprod/api/endpoints/preprod_artifact_snapshot.py", + "startLine": 140 + } + } + ], + "durationMs": 36045, + "usage": { + "inputTokens": 21179, + "outputTokens": 1320, + "costUSD": 0.06993555 + } + }, + { + "case": "sentry-release-threshold-empty-project-filter", + "repository": "getsentry/sentry", + "suite": "historical", + "mode": "nonsemantic", + "skill": "security-review", + "base": "8a93913509441a0c8e7d035f9c4bc24dabed2d86", + "head": "synthetic reverse of 8f9fe309854228051dabac985fb813476a2a5b24..8a93913509441a0c8e7d035f9c4bc24dabed2d86", + "complete": true, + "expectedFindingMatched": true, + "rawArtifact": { + "path": "/tmp/warden-security-recall-batch-artifacts/sentry-release-threshold-empty-project-filter.nonsemantic.jsonl", + "sha256": "fb53cbf8a7324f7c96d72379bb0d4b7c436abb2c63943cf9ccbefdfab033b1c2" + }, + "scannerChunks": 2, + "completedScannerChunks": 2, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [ + { + "title": "Authorization bypass: empty projects_list removes all project-scoping from ReleaseThreshold query", + "severity": "high", + "confidence": "high", + "location": { + "path": "src/sentry/api/endpoints/release_thresholds/release_threshold_index.py", + "startLine": 50, + "endLine": 58 + } + } + ], + "durationMs": 19481, + "usage": { + "inputTokens": 10631, + "outputTokens": 1042, + "costUSD": 0.05549175 + } + }, + { + "case": "sentry-replay-delete-read-scope", + "repository": "getsentry/sentry", + "suite": "historical", + "mode": "nonsemantic", + "skill": "security-review", + "base": "9bf0ea738cd7847438d4a2cfb1fbdbb326426e01", + "head": "synthetic reverse of c1bc01ad419ac251e153c81f212628221f8c0628..9bf0ea738cd7847438d4a2cfb1fbdbb326426e01", + "complete": true, + "expectedFindingMatched": true, + "rawArtifact": { + "path": "/tmp/warden-security-recall-batch-artifacts/sentry-replay-delete-read-scope.nonsemantic.jsonl", + "sha256": "4fd4bea3b9d7fc3a34945002cb4f8d8c8d6a2db77e6cf4374d352890652a63ff" + }, + "scannerChunks": 2, + "completedScannerChunks": 2, + "skippedScannerChunks": 0, + "failedScannerChunks": 0, + "findings": [ + { + "title": "DELETE permission expanded to allow project:read scope", + "severity": "medium", + "confidence": "high", + "location": { + "path": "src/sentry/replays/endpoints/project_replay_details.py", + "startLine": 26 + } + } + ], + "durationMs": 6754, + "usage": { + "inputTokens": 10365, + "outputTokens": 306, + "costUSD": 0.04345425 + } + }, { "case": "warden-error-cause-chain-regression", "repository": "getsentry/warden", diff --git a/specs/chunking-strategy-benchmark.md b/specs/chunking-strategy-benchmark.md index 83802e62..c8228681 100644 --- a/specs/chunking-strategy-benchmark.md +++ b/specs/chunking-strategy-benchmark.md @@ -104,6 +104,9 @@ use the fix PR body or linked Warden issue as the oracle. | Case | Repository | PR | Vulnerable commit | Fix commit | Shape | Skill | Expected finding | | --- | --- | --- | --- | --- | --- | --- | --- | | `sentry-slack-options-load-unscoped-group` | `getsentry/sentry` | `#114185` | `c1bc01ad419ac251e153c81f212628221f8c0628` | `0f09491755f71a95343285cbe17c93bf272a0d62` | 2 files, 4 raw chunks, +29/-0 when fixed | `security-review` | Slack options-load can resolve a caller-controlled group id without binding it to the requesting Slack integration's organization. | +| `sentry-preprod-snapshot-project-access` | `getsentry/sentry` | `#114169` | `c1bc01ad419ac251e153c81f212628221f8c0628` | `8fac324d82c903c8022b99dcd4329f3944e57196` | 2 files, 4 raw chunks, +66/-1 when fixed | `security-review` | Preprod snapshot GET and DELETE can access artifacts by organization without checking project access. | +| `sentry-release-threshold-empty-project-filter` | `getsentry/sentry` | `#114049` | `8f9fe309854228051dabac985fb813476a2a5b24` | `8a93913509441a0c8e7d035f9c4bc24dabed2d86` | 2 files, 2 raw chunks, +39/-5 when fixed | `security-review` | ReleaseThreshold query can drop project scoping when the accessible-project list is empty. | +| `sentry-replay-delete-read-scope` | `getsentry/sentry` | `#114159` | `c1bc01ad419ac251e153c81f212628221f8c0628` | `9bf0ea738cd7847438d4a2cfb1fbdbb326426e01` | 2 files, 2 raw chunks, +9/-4 when fixed | `security-review` | Replay DELETE can accept `project:read`, allowing read-only users to delete replay data. | | `warden-error-cause-chain-regression` | `getsentry/warden` | `#215` | `9273d82be6582d78c4809c3b4317dba09bf1b58b` | `9bf777889e554ac30216ebfbd1f15a68abf92529` | 7 files, 9 raw chunks, +18/-13 when fixed | `code-review` | Dropping `Error` causes loses original git/runtime error diagnostics. Candidate only; finding was not stable across reruns. | Captured non-semantic baseline: @@ -113,23 +116,35 @@ Captured non-semantic baseline: - runtime: `pi` - verification: disabled - raw artifact for Slack case: `/tmp/warden-slack-options-low-runner-artifacts/sentry-slack-options-load-unscoped-group.nonsemantic.jsonl` +- raw artifacts for additional security cases: `/tmp/warden-security-recall-batch-artifacts/*.nonsemantic.jsonl` - raw artifact for Warden #215 candidate: `/tmp/warden-pr215-runner-artifacts/warden-error-cause-chain-regression.nonsemantic.jsonl` | Case | Complete | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | `sentry-slack-options-load-unscoped-group` | yes | 4 | 1 | 24.8s | 20,437 | 1,065 | $0.0633 | +| `sentry-preprod-snapshot-project-access` | yes | 4 | 1 | 36.0s | 21,179 | 1,320 | $0.0699 | +| `sentry-release-threshold-empty-project-filter` | yes | 2 | 1 | 19.5s | 10,631 | 1,042 | $0.0555 | +| `sentry-replay-delete-read-scope` | yes | 2 | 1 | 6.8s | 10,365 | 306 | $0.0435 | | `warden-error-cause-chain-regression` | yes | 8 | 0 | 3m24s | 345,903 | 26,543 | $0.8320 | -Finding produced by current `security-review`: +Findings produced by current `security-review`: - `src/sentry/integrations/slack/webhooks/options_load.py:99-104` - Removed cross-organization authorization check allows IDOR on group assignment data - -The Slack case is a small known-positive case. It is not large enough to prove -the high-fragmentation performance path. The Warden #215 case remains useful as -a repeated small-hunk candidate, but it is not accepted as a recall baseline -until the finding reproduces reliably. +- `src/sentry/preprod/api/endpoints/preprod_artifact_snapshot.py:140` - + Project-level access check removed from DELETE endpoint, enabling + cross-project artifact deletion +- `src/sentry/api/endpoints/release_thresholds/release_threshold_index.py:50-58` - + Authorization bypass: empty projects list removes all project scoping +- `src/sentry/replays/endpoints/project_replay_details.py:26` - + DELETE permission expanded to allow `project:read` scope + +These are small known-positive security cases. They fix the benchmark's recall +control gap, but they are not large enough to prove the high-fragmentation +performance path. The Warden #215 case remains useful as a repeated small-hunk +candidate, but it is not accepted as a recall baseline until the finding +reproduces reliably. ## Performance Shape Cases From 651ed13157c7de5b1efd515c201b211307cb1eef Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 1 Jul 2026 13:34:27 -0700 Subject: [PATCH 15/18] test(chunking): Refine benchmark strategy Add deterministic profile support to the chunking benchmark runner and document the current benchmark evidence. The benchmark now treats findings equivalence as the quality gate, with wall time and total cost as the primary efficiency metrics. Semantic grouping remains recorded as a rejected efficiency strategy unless new evidence justifies revisiting it. Co-Authored-By: GPT-5 Codex --- benchmarks/chunking/runner.ts | 272 ++++++++++-- specs/chunking-strategy-benchmark.md | 594 +++++++++++++++++++++++++-- 2 files changed, 792 insertions(+), 74 deletions(-) diff --git a/benchmarks/chunking/runner.ts b/benchmarks/chunking/runner.ts index 3b617b69..100d9110 100644 --- a/benchmarks/chunking/runner.ts +++ b/benchmarks/chunking/runner.ts @@ -11,6 +11,8 @@ import { import { join, resolve } from 'node:path'; import { tmpdir } from 'node:os'; import { z } from 'zod'; +import { prepareFiles } from '../../packages/warden/src/sdk/prepare.js'; +import type { EventContext, FileChange } from '../../packages/warden/src/types/index.js'; const DEFAULT_CASES = [ 'sentry-dashboard-axis-range-existing-widget', @@ -197,7 +199,10 @@ interface Args { artifactsDir: string; model: string; runtime: string; + effort?: string; keepWorktrees: boolean; + profile: boolean; + traces: boolean; } interface BenchmarkCase { @@ -236,6 +241,27 @@ interface RunSummary { }; } +interface ChunkProfile { + fileCount: number; + skippedFileCount: number; + scannerChunks: number; + changedRanges: number; + contentChars: number; + files: { + filename: string; + chunks: number; + changedRanges: number; + contentChars: number; + lineRanges: string[]; + contentModes: string[]; + }[]; + skippedFiles: { + filename: string; + reason: string; + pattern?: string; + }[]; +} + function usage(exitCode = 2): never { const text = [ 'usage: pnpm exec tsx benchmarks/chunking/runner.ts [options]', @@ -251,6 +277,9 @@ function usage(exitCode = 2): never { ' --artifacts-dir Raw JSONL artifact directory (default: /tmp/warden-chunking-benchmark-artifacts)', ' --model Model override', ' --runtime Runtime override', + ' --effort Effort override for scanner runs', + ' --profile Only prepare and summarize scanner chunks; do not run Warden', + ' --traces Capture per-chunk runtime traces in JSONL artifacts', ' --keep-worktrees Keep temporary synthetic worktrees', ' -h, --help Show this help', ].join('\n'); @@ -271,6 +300,8 @@ function parseArgs(argv: string[]): Args { model: process.env['WARDEN_BENCHMARK_MODEL'] ?? DEFAULT_MODEL, runtime: process.env['WARDEN_BENCHMARK_RUNTIME'] ?? DEFAULT_RUNTIME, keepWorktrees: false, + profile: false, + traces: false, }; for (let i = 0; i < argv.length; i++) { @@ -294,6 +325,9 @@ function parseArgs(argv: string[]): Args { else if (arg === '--artifacts-dir') args.artifactsDir = resolve(argv[++i] ?? usage()); else if (arg === '--model') args.model = argv[++i] ?? usage(); else if (arg === '--runtime') args.runtime = argv[++i] ?? usage(); + else if (arg === '--effort') args.effort = argv[++i] ?? usage(); + else if (arg === '--profile') args.profile = true; + else if (arg === '--traces') args.traces = true; else if (arg === '--keep-worktrees') args.keepWorktrees = true; else usage(); } @@ -509,6 +543,12 @@ function runWarden( '--log', '--no-color', ]; + if (args.traces) { + cliArgs.push('--traces'); + } + if (args.effort) { + cliArgs.push('--effort', args.effort); + } const result = spawnSync('pnpm', cliArgs, { cwd: resolve(import.meta.dirname, '../..'), encoding: 'utf8', @@ -552,6 +592,136 @@ function createPerformanceWorktree(sourceRepo: string, benchmarkCase: Performanc return worktree; } +function parseNumstat(repo: string, base: string): Map { + const stats = new Map(); + const output = execGit(repo, ['diff', '--numstat', '--find-renames', `${base}..HEAD`]); + if (!output) return stats; + + for (const line of output.split('\n')) { + const [additionsRaw, deletionsRaw, ...pathParts] = line.split('\t'); + const filename = pathParts.at(-1); + if (!filename) continue; + stats.set(filename, { + additions: Number.parseInt(additionsRaw ?? '0', 10) || 0, + deletions: Number.parseInt(deletionsRaw ?? '0', 10) || 0, + }); + } + return stats; +} + +function fileChangeStatus(rawStatus: string): FileChange['status'] { + const status = rawStatus[0]; + if (status === 'A') return 'added'; + if (status === 'D') return 'removed'; + if (status === 'R') return 'renamed'; + if (status === 'C') return 'copied'; + return 'modified'; +} + +function readDiffFiles(repo: string, base: string): FileChange[] { + const nameStatus = execGit(repo, ['diff', '--name-status', '--find-renames', `${base}..HEAD`]); + if (!nameStatus) return []; + + const stats = parseNumstat(repo, base); + return nameStatus.split('\n').flatMap((line): FileChange[] => { + const parts = line.split('\t'); + const statusRaw = parts[0]; + if (!statusRaw) return []; + const filename = statusRaw.startsWith('R') || statusRaw.startsWith('C') ? parts[2] : parts[1]; + if (!filename) return []; + const patch = execGit(repo, ['diff', '--unified=3', `${base}..HEAD`, '--', filename]); + const fileStats = stats.get(filename) ?? { additions: 0, deletions: 0 }; + return [{ + filename, + status: fileChangeStatus(statusRaw), + additions: fileStats.additions, + deletions: fileStats.deletions, + patch, + }]; + }); +} + +function makeProfileContext( + worktree: string, + repository: string, + base: string, + files: FileChange[], +): EventContext { + const [owner = 'getsentry', name = repository] = repository.split('/'); + return { + eventType: 'pull_request', + action: 'opened', + repository: { + owner, + name, + fullName: repository, + defaultBranch: 'main', + }, + pullRequest: { + number: 1, + title: 'Chunking benchmark profile', + body: null, + author: 'warden-benchmark', + baseBranch: 'main', + headBranch: 'benchmark', + headSha: execGit(worktree, ['rev-parse', 'HEAD']), + baseSha: base, + files, + }, + repoPath: worktree, + diffContextSource: { type: 'working-tree' }, + }; +} + +function lineRangeForChunk(chunk: { changedLineMap: { start: number; end: number }[] }): string { + return chunk.changedLineMap + .map((range) => range.start === range.end ? `${range.start}` : `${range.start}-${range.end}`) + .join(','); +} + +function profileChunks(worktree: string, repository: string, base: string): ChunkProfile { + const files = readDiffFiles(worktree, base); + const context = makeProfileContext(worktree, repository, base, files); + const prepared = prepareFiles(context); + const profileFiles = prepared.files.map((file) => { + const contentChars = file.chunks.reduce( + (total, chunk) => total + chunk.files.reduce((sum, chunkFile) => sum + chunkFile.content.length, 0), + 0, + ); + return { + filename: file.filename, + chunks: file.chunks.length, + changedRanges: file.chunks.reduce((total, chunk) => total + chunk.changedLineMap.length, 0), + contentChars, + lineRanges: file.chunks.map(lineRangeForChunk), + contentModes: [...new Set(file.chunks.flatMap((chunk) => chunk.files.map((chunkFile) => chunkFile.contentMode)))], + }; + }); + + return { + fileCount: profileFiles.length, + skippedFileCount: prepared.skippedFiles.length, + scannerChunks: profileFiles.reduce((total, file) => total + file.chunks, 0), + changedRanges: profileFiles.reduce((total, file) => total + file.changedRanges, 0), + contentChars: profileFiles.reduce((total, file) => total + file.contentChars, 0), + files: profileFiles.sort((a, b) => b.chunks - a.chunks || b.contentChars - a.contentChars), + skippedFiles: prepared.skippedFiles, + }; +} + +function printProfile(profile: ChunkProfile): void { + console.log([ + `Prepared files: ${profile.fileCount}`, + `Scanner chunks: ${profile.scannerChunks}`, + `Changed ranges: ${profile.changedRanges}`, + `Content chars: ${profile.contentChars}`, + `Skipped files: ${profile.skippedFileCount}`, + ].join('\n')); + for (const file of profile.files.slice(0, 10)) { + console.log(`- ${file.filename}: ${file.chunks} chunks, ${file.changedRanges} ranges, ${file.contentChars} chars`); + } +} + const args = parseArgs(process.argv.slice(2)); mkdirSync(args.artifactsDir, { recursive: true }); @@ -563,25 +733,40 @@ if (args.suite === 'historical') { const sourceRepo = sourceRepoForRepository(args, benchmarkCase.repository); const worktree = createBenchmarkWorktree(sourceRepo, benchmarkCase); console.log(`Worktree: ${worktree}`); - const runs: Record = {}; - for (const semantic of selectedModes(args)) { - runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( - args, - worktree, - { name: benchmarkCase.name, base: benchmarkCase.fixCommit, skill: benchmarkCase.skill }, - semantic, - ); + if (args.profile) { + const profile = profileChunks(worktree, benchmarkCase.repository, benchmarkCase.fixCommit); + printProfile(profile); + results.push({ + case: benchmarkCase.name, + repository: benchmarkCase.repository, + skill: benchmarkCase.skill, + base: benchmarkCase.fixCommit, + head: benchmarkCase.vulnerableCommit, + expectedFindings: benchmarkCase.expectedFindings, + profile, + worktree: args.keepWorktrees ? worktree : undefined, + }); + } else { + const runs: Record = {}; + for (const semantic of selectedModes(args)) { + runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( + args, + worktree, + { name: benchmarkCase.name, base: benchmarkCase.fixCommit, skill: benchmarkCase.skill }, + semantic, + ); + } + results.push({ + case: benchmarkCase.name, + repository: benchmarkCase.repository, + skill: benchmarkCase.skill, + base: benchmarkCase.fixCommit, + head: benchmarkCase.vulnerableCommit, + expectedFindings: benchmarkCase.expectedFindings, + runs, + worktree: args.keepWorktrees ? worktree : undefined, + }); } - results.push({ - case: benchmarkCase.name, - repository: benchmarkCase.repository, - skill: benchmarkCase.skill, - base: benchmarkCase.fixCommit, - head: benchmarkCase.vulnerableCommit, - expectedFindings: benchmarkCase.expectedFindings, - runs, - worktree: args.keepWorktrees ? worktree : undefined, - }); if (!args.keepWorktrees) { execGit(sourceRepo, ['worktree', 'remove', '--force', worktree]); } @@ -593,25 +778,40 @@ if (args.suite === 'historical') { const sourceRepo = sourceRepoForPerformanceCase(args, benchmarkCase); const worktree = createPerformanceWorktree(sourceRepo, benchmarkCase); console.log(`Worktree: ${worktree}`); - const runs: Record = {}; - for (const semantic of selectedModes(args)) { - runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( - args, - worktree, - { name: benchmarkCase.name, base: benchmarkCase.base, skill: benchmarkCase.skill ?? 'code-review' }, - semantic, - ); + if (args.profile) { + const profile = profileChunks(worktree, benchmarkCase.repository, benchmarkCase.base); + printProfile(profile); + results.push({ + case: benchmarkCase.name, + repository: benchmarkCase.repository, + skill: benchmarkCase.skill ?? 'code-review', + base: benchmarkCase.base, + head: benchmarkCase.head, + expectedFindings: [], + profile, + worktree: args.keepWorktrees ? worktree : undefined, + }); + } else { + const runs: Record = {}; + for (const semantic of selectedModes(args)) { + runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( + args, + worktree, + { name: benchmarkCase.name, base: benchmarkCase.base, skill: benchmarkCase.skill ?? 'code-review' }, + semantic, + ); + } + results.push({ + case: benchmarkCase.name, + repository: benchmarkCase.repository, + skill: benchmarkCase.skill ?? 'code-review', + base: benchmarkCase.base, + head: benchmarkCase.head, + expectedFindings: [], + runs, + worktree: args.keepWorktrees ? worktree : undefined, + }); } - results.push({ - case: benchmarkCase.name, - repository: benchmarkCase.repository, - skill: benchmarkCase.skill ?? 'code-review', - base: benchmarkCase.base, - head: benchmarkCase.head, - expectedFindings: [], - runs, - worktree: args.keepWorktrees ? worktree : undefined, - }); if (!args.keepWorktrees) { execGit(sourceRepo, ['worktree', 'remove', '--force', worktree]); } diff --git a/specs/chunking-strategy-benchmark.md b/specs/chunking-strategy-benchmark.md index c8228681..07b78c3a 100644 --- a/specs/chunking-strategy-benchmark.md +++ b/specs/chunking-strategy-benchmark.md @@ -11,10 +11,11 @@ after the runner and result format are stable. ## Current Takeaway -The evidence so far does not support semantic grouping as the default -performance fix. Semantic grouping preserved recall on some historical cases, -but it did not reduce cost in the paired baseline and did not recover cases that -non-semantic chunking missed. +The evidence so far rejects semantic grouping as an efficiency strategy. +Semantic grouping preserved recall on some historical cases, but it did not +reduce cost in the paired baseline and did not recover cases that non-semantic +chunking missed. Do not include semantic grouping in the main efficiency +comparison unless a new, separate hypothesis justifies it. The better next bet is a precision-preserving chunk optimizer: @@ -22,27 +23,27 @@ The better next bet is a precision-preserving chunk optimizer: - merge sequential same-file hunks under strict size/range caps - keep risky production chunks bounded - treat generated/schema/test churn carefully because it dominates cost -- use semantic labels only as neutral metadata, not as authority to make giant - scanner prompts +- avoid semantic regrouping for scanner efficiency until new evidence changes + this conclusion The benchmark suite now has three roles: - historical recall cases with known expected findings - branch-evolution recall cases where later branch commits fix earlier branch bugs -- real performance-shape PRs that measure cost, chunk count, failures, and - finding quality +- real performance-shape PRs that measure wall time and total cost behind a + same-or-better findings gate ## Goal Answer two questions: -- Does a chunking strategy preserve or improve known-finding recall? -- Does it reduce scanner calls, runtime, tokens, or cost on fragmented changes? +- Does a chunking strategy produce at least the same findings as the baseline? +- If yes, does it reduce wall time or total cost on fragmented changes? -The benchmark should not only prove that a chunking strategy produces fewer -chunks. It must prove that scanner behavior improves or stays correct when the -scanner receives optimized chunks instead of atomic git-hunk chunks. +The benchmark should not treat scanner chunk count as a success metric. Chunk +count, prompt size, and changed-range density are diagnostics for explaining +outcomes. The score is quality-gated wall time and total cost. ## Dataset Shape @@ -150,8 +151,9 @@ reproduces reliably. Add a separate performance suite made of real pull requests that were slow or pathologically fragmented. These cases do not need a known expected finding. -They measure scanner-call count, runtime, token cost, failed chunks, duplicate -findings, and whether reported findings are valid. +They measure wall time and total cost after verifying that reported findings are +at least equivalent to the baseline. Scanner-call count, failed chunks, prompt +size, and duplicate findings are diagnostic fields only. | Case | Repository | Base | Head | Shape | Why it matters | | --- | --- | --- | --- | --- | --- | @@ -178,9 +180,10 @@ iteration, use a smaller smoke subset: Performance cases should be judged differently from historical recall cases: - valid findings per run, adjudicated manually until a judge exists -- scanner chunks and failed/interrupted chunks -- wall time and model usage +- wall time +- total cost - duplicate or near-duplicate findings +- scanner chunks and failed/interrupted chunks, for diagnosis only - whether chunk grouping made findings more vague or less actionable - whether mechanical repeated edits were reviewed once instead of many times @@ -300,9 +303,9 @@ For each historical recall case: 2. Check out the fixing commit. 3. Create a benchmark branch. 4. Apply the full inverse fixing diff back to the vulnerable commit. -5. Run Warden with semantic chunking disabled. +5. Run the baseline strategy. 6. Reset to the same benchmark branch. -7. Run Warden with semantic chunking enabled. +7. Run the candidate strategy. 8. Judge both reports against the existing `should_find` assertion. For each performance shape case: @@ -311,8 +314,8 @@ For each performance shape case: 2. Check out the PR base SHA. 3. Apply or check out the PR head SHA. 4. Run each chunking strategy against `base..head`. -5. Record scanner calls, runtime, tokens, cost, failed chunks, findings, and - manual finding validity. +5. Record findings, wall time, total cost, and enough diagnostic data + (scanner calls, tokens, failed chunks) to explain regressions. The runner lives under `benchmarks/chunking/` because this is maintainer benchmark infrastructure, not the eval framework. It may read historical eval @@ -335,14 +338,63 @@ bug-introducing worktrees. Performance cases use explicit base/head SHAs from real PR-shaped diffs. Both modes write paired summaries plus raw JSONL artifacts. +Use `--profile` before model-backed experiments. It runs the same worktree +setup and `prepareFiles` path, then writes prepared scanner chunk counts, +changed ranges, content size, top files, and skipped files without invoking the +scanner. Use `--traces` when debugging model variance and `--effort high` for +recall controls that are noisy at default effort. + +## Research Signals + +Current external evidence matches the local benchmark failures: code-review +quality is sensitive to context shape, and "more context" is often worse. + +- SWE-PRBench evaluates 350 real pull requests against human review comments. + Its reported result is especially relevant: all tested frontier models got + worse as context expanded from diff-only to richer file/full-context setups, + even with structured semantic layers such as AST-extracted function context + and import graph resolution. The authors attribute the dominant failure mode + to attention dilution, and a smaller structured diff-with-summary prompt beat + a richer full-context prompt. +- Long-context divide-and-conquer research frames failures as a balance between + task noise, model noise, and aggregation noise. Chunking helps when context + size creates model noise, but it fails when the issue requires cross-chunk + dependence or when the aggregator loses signal. +- Dynamic chunking work suggests fixed-size chunking is brittle, but its + successful pattern is not "merge everything semantically." It separates + content adaptively and selects question-relevant chunks. For Warden, the + "question" is the skill/risk class, so chunk selection should be risk-aware. +- Repository-level vulnerability benchmarks increasingly use paired + introducing/fixing commits and agentic inspection because real findings often + require interprocedural or cross-file evidence. This supports keeping + security controls as paired commits with stable expected findings, not just + no-finding performance PRs. +- Code long-context recall work distinguishes lexical recall from semantic + recall and shows code reasoning can degrade by position and granularity. + This is a warning against giant stitched chunks: a model can "see" the lines + and still fail to reason about the relevant statement. + +Design constraints from this: + +- Do not optimize for scanner chunk count alone. +- Treat prompt size, changed-range density, and per-file long-pole runtime as + first-class metrics. +- Prefer small structured diff prompts with targeted context over whole-file or + broad semantic bundles. +- Add benchmark cases with known human or security findings across different + shapes: direct diff issue, same-file contextual issue, cross-file issue, and + generated-artifact-heavy no-finding case. +- Run repeated trials or high-effort controls before declaring recall changes. +- Compare strategies with frozen context configurations, not ad hoc prompt + changes mixed with chunking changes. + ## Strategies To Compare Use the same cases across each strategy: - current non-semantic chunking -- current semantic grouping - deterministic precision-preserving optimizer -- optimizer plus optional neutral semantic labels, if implemented +- generated/derived artifact policy, if implemented The optimizer is not a replacement for all chunking. It should activate only for pathological patch shapes: high raw chunk count, high hunk count in one @@ -350,27 +402,25 @@ file, many tiny hunks, or large total diff size. ## Metrics -Record these for each semantic-off and semantic-on run: +Primary benchmark metrics: - pass/fail against expected finding - total findings +- wall time +- recorded cost + +Diagnostic fields: + - duplicate or near-duplicate findings - failed chunks - failed extractions -- wall time - scanner chunk count -- semantic planner group count -- scanner chunks per semantic group -- semantic planner summaries - input tokens - output tokens -- recorded cost -The semantic-on run is successful when it keeps the expected finding and reduces -operational load. A lower chunk count is not enough if recall regresses. -Semantic groups are allowed to contain multiple scanner chunks; the target is -fewer, better bounded scanner calls than raw git chunks, not one giant scanner -prompt per logical change. +A strategy is successful only when its findings are at least equivalent to the +baseline and it reduces wall time or total cost. Lower chunk count, lower input +tokens, or cleaner grouping are explanatory details, not pass criteria. ## Output @@ -517,6 +567,474 @@ test-specific rule by materializing repeated tiny hunks in a compact form. It is not yet a clear cost win over earlier semantic runs, but it is the first generic semantic variant that recovered the expected finding on the full patch. +## Failed Attempts + +These approaches are recorded as rejected unless new evidence changes the +result. Do not reintroduce them without rerunning the recall controls and at +least one finding-bearing performance case. + +### Trimmed Reverse-Patch Semantic Grouping + +What we tried: + +- Case: `sentry-dashboard-axis-range-existing-widget` +- Patch shape: trimmed two-file reverse patch +- Semantic planner grouped both changed ranges into one scanner chunk. + +Result: + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| semantic off | 2 | 1 | 11m54s | 445,997 | 39,484 | $1.2213 | +| semantic on | 1 | 1 | 3m03s | 160,096 | 10,079 | $0.4609 | + +Why it failed as evidence: + +- It used a trimmed patch, not the full PR-shaped diff. +- It proved the mechanics worked, but it made the problem easier by removing + surrounding branch noise. +- Later full-patch runs did not preserve this apparent win. + +Takeaway: + +- Do not use trimmed patches to validate chunking quality. They can be useful + smoke tests, but they are not recall or performance proof. + +### Full-Patch Semantic Grouping + +What we tried: + +- Case: `sentry-dashboard-axis-range-existing-widget` +- Patch shape: full reverse fixing diff +- Variants: + - bounded semantic groups + - summary anti-bias prompt + - changed-range caps + - neutral scanner prompt + - neutral prompt plus stricter changed-range cap + +Result: + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| non-semantic, partial | 1/13 completed | 1 | 2m53s | 160,637 | 9,024 | $0.3191 | +| semantic on, bounded groups | 5 | 0 | 10m45s | 481,391 | 34,884 | $1.2564 | +| semantic on, bounded groups plus summary anti-bias prompt | 4 | 0 | 8m19s | 472,104 | 26,978 | $1.1046 | +| semantic on, bounded groups plus changed-range cap | 5 | 0 | 7m24s | 432,183 | 23,708 | $0.9862 | +| semantic on, neutral scanner prompt | 4 | 0 | 5m14s | 374,416 | 15,714 | $0.7504 | +| semantic on, neutral prompt plus changed-range cap 2 | 7 | 0 | 13m40s | 807,047 | 45,554 | $2.0509 | + +Why it failed: + +- All full-patch semantic variants missed the expected axis-range regression. +- Lower chunk count did not translate into preserved recall. +- Changed-range caps could reduce prompt size in some cases, but one test-heavy + scanner chunk still hit `turn_limit`. +- Removing semantic summaries from scanner prompts reduced cost but did not + recover the finding. + +Takeaway: + +- Size alone was not the root problem. Grouping implementation and updated test + hunks into one coherent semantic change can make a regression look + intentional. +- Semantic summaries are grouping metadata, not correctness evidence. + +### Test-Assertion-Specific Isolation + +What we tried: + +- Isolated changed test assertions into their own scanner chunks. +- This targeted the axis-range case where removed assertions were the main + signal. + +Result: + +- The axis-range finding came back in the experimental run. + +Why it failed: + +- It was overfit to one benchmark shape. +- It encoded domain-specific syntax heuristics into the chunker instead of + solving the general "many tiny hunks" problem. +- It worked against the goal of collapsing small similar hunks safely. + +Takeaway: + +- Do not add literal test-assertion rules to the chunker. +- If tests need different treatment, make that a general risk/role signal, not + a syntax-specific escape hatch. + +### Semantic Compact Similar Tiny Hunks + +What we tried: + +- Kept semantic grouping, but materialized repeated tiny hunks in compact form. +- The goal was to avoid giant scanner prompts while still grouping repeated + edit shapes. + +Result: + +| Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| semantic on, compact similar tiny hunks | 6 | 1 | 7m23s | 758,900 | 50,400 | $1.9500 | + +Why it failed as a default: + +- It recovered the axis-range finding, so it was better than earlier full-patch + semantic variants. +- It was still expensive relative to the earlier semantic variants and did not + establish a cost win over non-semantic baselines. +- Evidence came from one difficult case, not the broader benchmark suite. + +Takeaway: + +- Compact repeated-hunk materialization is worth remembering, but not enough to + make semantic grouping the default performance fix. + +### Paired Semantic Benchmark Baseline + +What we tried: + +- Ran the initial four historical code-review cases through the maintainer + runner with semantic mode on and off. + +Result: + +| Case | Mode | Expected found | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| axis range | semantic off | no | 9 | 0 | 2m06s | 119,788 | 8,589 | $0.3428 | +| axis range | semantic on | no | 5 | 0 | 2m03s | 207,575 | 7,501 | $0.3904 | +| cursor service account | semantic off | yes | 7 | 3 | 4m49s | 217,727 | 16,902 | $0.5860 | +| cursor service account | semantic on | yes | 7 | 2 | 4m23s | 384,304 | 20,268 | $0.7174 | +| fixability summary | semantic off | no | 5 | 1 | 1m12s | 127,987 | 11,044 | $0.3586 | +| fixability summary | semantic on | no | 4 | 1 | 3m09s | 602,384 | 23,791 | $1.0903 | +| workflow FK | semantic off | yes | 5 | 2 | 7m24s | 535,681 | 23,518 | $1.1353 | +| workflow FK | semantic on | yes | 5 | 2 | 7m46s | 665,740 | 38,759 | $1.4570 | + +Why it failed: + +- Semantic mode preserved recall only where non-semantic mode already found the + issue. +- Semantic mode did not recover missed cases. +- Semantic mode reduced scanner chunks on two cases, but cost increased on + every completed paired run. + +Takeaway: + +- Semantic grouping is not currently justified as the default performance + strategy. It may still be useful as metadata or as an opt-in planner for + specific pathological shapes. + +### Branch-Evolution Recall From MCP #1130 + +What we tried: + +- Case: `sentry-mcp-ai-conversation-absolute-range-period-conflict` +- Ran the buggy branch prefix before the later fix commit. +- Expected finding: absolute `start`/`end` searches conflict with injected or + default `period`. + +Result: + +| Mode | Scanner chunks | Completed | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| non-semantic | 95 | 94 | 0 | 28m11s | 7,056,116 | 245,823 | $14.9409 | + +Why it failed: + +- It did not produce the known later-fix finding. +- It is too expensive to use as the first recall guardrail for iteration. + +Takeaway: + +- Keep it as a performance stress case. +- Do not count it as a known-positive recall case unless current Warden can + reliably find the later-fixed issue. + +### Exact Distant Tiny-Hunk Coalescing + +What we tried: + +- Deterministic non-semantic optimizer. +- Merged distant same-file tiny hunks only when their changed diff lines were + exactly identical. +- Capped each merged group at 4 hunks and the existing coalescing chunk size. +- Exposed only through a temporary benchmark flag; implementation was backed + out after the test. + +Recall guardrail result: + +| Case | Scanner chunks | Findings | Duration | Input tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | +| `sentry-slack-options-load-unscoped-group` | 4 | 1 | 21.9s | 20,445 | $0.0627 | +| `sentry-preprod-snapshot-project-access` | 4 | 1 | 39.2s | 21,179 | $0.0756 | +| `sentry-release-threshold-empty-project-filter` | 2 | 1 | 20.2s | 10,627 | $0.0411 | +| `sentry-replay-delete-read-scope` | 2 | 1 | 14.0s | 10,369 | $0.0492 | + +Performance result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | baseline | 95 | 0 | 7m01s | 2,064,917 | 54,596 | $3.8790 | +| `sentry-mcp-search-issues-period-30d` | exact tiny merge | 67 | 0 | 5m35s | 1,977,998 | 47,867 | $3.7740 | + +Finding-bearing control result: + +| Case | Mode | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-node-pnpm-baseline` | current baseline rerun | 1 | 1m46s | 229,197 | 5,815 | $0.4580 | +| `sentry-mcp-node-pnpm-baseline` | exact tiny merge | 0 | 1m20s | 127,549 | 4,330 | $0.2680 | +| `sentry-mcp-node-pnpm-baseline` | exact tiny merge rerun | 0 | 3m43s | 247,700 | 12,200 | $0.5900 | + +Why it failed: + +- It improved the slow no-finding MCP case, but missed the only finding-bearing + performance control twice. +- The `pnpm-workspace.yaml:4-23` scanner chunk was still isolated, so the + failure was not a visible line-range merge error. +- The missed run spent much less effort on the important chunk than the + successful baseline rerun. The optimization appears to change enough run + shape, ordering, cache behavior, or agent context to reduce thoroughness. + +Takeaway: + +- Fewer scanner calls and faster wall time are not sufficient. +- Even conservative distant-hunk merging can harm recall indirectly. +- Do not merge distant hunks as the next default strategy. The next safer + experiment should prefer sequential same-file coalescing under strict caps and + only activate for pathological hunk counts. + +### Broad Sequential Gap Increase + +What we tried: + +- Used current same-file sequential coalescing, but increased + `maxGapLines` from 30 to 120 for every file in the benchmark config. +- Kept the existing `maxChunkSize` cap. + +Result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| `sentry-slack-options-load-unscoped-group` | baseline | 4 | 1 | 24.8s | 20,437 | $0.0633 | +| `sentry-slack-options-load-unscoped-group` | `maxGapLines=120` | 2 | 0 | 2.4s | 10,500 | $0.0400 | + +Why it failed: + +- It merged both production hunks in `options_load.py` into one scanner chunk: + `14-19, 99-104`. +- Warden missed the high-signal Slack IDOR finding that baseline finds. +- This is a direct recall regression, not merely stochastic benchmark noise, + because the chunk shape changed on the failing file. + +Takeaway: + +- Increasing the sequential gap globally is too aggressive. +- Small security-sensitive files need the existing conservative gap behavior. + +### Adaptive High-Hunk Sequential Gap + +What we tried: + +- Widened same-file sequential coalescing only when a file started with at least + 12 hunks. +- Files below that threshold kept the existing 30-line gap. +- Kept the existing `maxChunkSize` cap. +- Implementation was backed out after testing. + +Recall-control result: + +| Case | Scanner chunks | Findings | Duration | Input tokens | Cost | +| --- | ---: | ---: | ---: | ---: | ---: | +| `sentry-slack-options-load-unscoped-group` | 4 | 0 | 3.7s | 20,437 | $0.0478 | +| `sentry-preprod-snapshot-project-access` | 4 | 1 | 51.7s | 21,179 | $0.0754 | +| `sentry-release-threshold-empty-project-filter` | 2 | 1 | 22.5s | 10,631 | $0.0576 | +| `sentry-replay-delete-read-scope` | 2 | 1 | 13.4s | 10,361 | $0.0481 | + +Performance result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | baseline | 95 | 0 | 7m01s | 2,064,917 | 54,596 | $3.8790 | +| `sentry-mcp-search-issues-period-30d` | adaptive high-hunk gap | 51 | 0 | 5m00s | 1,659,025 | 41,407 | $2.9629 | + +Finding-bearing control result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-node-pnpm-baseline` | baseline | 19 | 1 | 3m34s | 559,623 | 10,929 | $1.2962 | +| `sentry-mcp-node-pnpm-baseline` | current baseline rerun | 19 | 1 | 1m46s | 229,197 | 5,815 | $0.4580 | +| `sentry-mcp-node-pnpm-baseline` | adaptive high-hunk gap | 19 | 0 | 10m13s | 381,049 | 34,753 | $1.3056 | + +Why it failed: + +- Operationally, it was the best result so far on the slow MCP no-finding case: + scanner chunks dropped from 95 to 51 and cost dropped about 24%. +- It did not pass the finding-bearing control. The pnpm case kept the same + scanner chunk count and same `pnpm-workspace.yaml:4-23` chunk shape, but the + run still missed the known finding. +- The Slack recall miss also kept the old 4-chunk shape, so this run exposed + benchmark/model variance in addition to chunking behavior. +- Because the requirement is no precision or recall loss, a promising + performance win is not enough. + +Takeaway: + +- Adaptive sequential same-file coalescing remains the most promising + performance theory, but it needs a stronger and more stable quality gate + before productizing. +- The benchmark should run repeated trials or use a cheaper deterministic + scanner-summary check for finding-bearing controls; single LLM runs are too + noisy to prove "no precision loss." +- If this is revisited, restrict activation to files with high hunk counts and + consider generated/test/docs files first, but do not ship it until + finding-bearing controls pass repeatedly. + +### Skip Generated Definition Artifacts + +What we tried: + +- Skipped generated MCP definition files in the benchmark config: + - `**/skillDefinitions.json` + - `**/toolDefinitions.json` +- Left existing hunk coalescing unchanged. + +Result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | baseline | 95 | 0 | 7m01s | 2,064,917 | 54,596 | $3.8790 | +| `sentry-mcp-search-issues-period-30d` | skip generated definitions | 83 | 0 | 7m26s | 1,552,548 | 46,992 | $2.6260 | + +Why it failed as a standalone strategy: + +- It reduced cost, but wall time got worse. +- The generated files were expensive, but the long pole became + `search-events.test.ts`, which still ran as 32 scanner chunks. +- Skipping generated files is a policy decision, not a general chunking + strategy. It may be valid for derived artifacts, but it does not solve "many + tiny hunks in one source/test file." + +Takeaway: + +- Generated artifact skipping is useful as a multiplier with better chunking, + not sufficient by itself. +- If productized, it should be framed as generated-file policy with clear + defaults and override paths, not hidden in semantic chunking. + +### Skip Generated Definitions Plus Adaptive High-Hunk Gap + +What we tried: + +- Combined the two best operational levers: + - skip generated MCP definition JSON files + - widen same-file sequential coalescing only for files with at least 12 hunks +- Implementation was temporary and backed out after the run. + +Result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | baseline | 95 | 0 | 7m01s | 2,064,917 | 54,596 | $3.8790 | +| `sentry-mcp-search-issues-period-30d` | adaptive high-hunk gap | 51 | 0 | 5m00s | 1,659,025 | 41,407 | $2.9629 | +| `sentry-mcp-search-issues-period-30d` | skip generated definitions | 83 | 0 | 7m26s | 1,552,548 | 46,992 | $2.6260 | +| `sentry-mcp-search-issues-period-30d` | combined | 44 | 0 | 4m24s | 805,895 | 30,757 | $1.3774 | + +Why it is promising: + +- It is the strongest performance result so far on the slow MCP case: + - scanner chunks down 54% + - wall time down 37% + - input tokens down 61% + - cost down 64% +- It avoids cross-file semantic grouping. +- It keeps small files on the existing conservative coalescing behavior. +- It removes derived generated artifacts from scanner review instead of asking + the scanner to re-review source and generated output for the same change. + +Why it is not accepted yet: + +- The performance case has no findings, so it proves efficiency only. +- The current finding-bearing controls are unstable: + - Slack missed twice with unchanged chunk shape and very short runs. + - The pnpm case missed with unchanged chunk shape after a 10m run. +- Because the requirement is no precision or recall loss, this cannot ship until + quality controls pass repeatedly or the benchmark has a more deterministic + recall gate. + +Takeaway: + +- This is the best next implementation candidate, but the next task is improving + the benchmark quality gate, not adding more chunking heuristics. +- A serious follow-up should run repeated trials for controls or use a judge + over saved traces to separate chunking regressions from model/run variance. + +### Pressure-Triggered Same-File Coalescing + +What we tried: + +- Added a temporary `prepareFiles` rule: + - run normal same-file coalescing first + - if one file still produced at least 12 scanner chunks, rerun coalescing for + that file with a 120-line gap + - leave small files on the existing conservative 30-line gap +- Added benchmark runner `--profile` support to measure prepared scanner chunks + without invoking the model. +- Implementation was backed out after the real scanner run. + +Deterministic profile result: + +| Case | Baseline scanner chunks | Pressure scanner chunks | Delta | +| --- | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | 94 | 65 | -29 | +| `sentry-mcp-issue-search-project-period-endpoint` | 91 | 63 | -28 | +| `sentry-mcp-ai-conversation-period-schema-default` | 92 | 64 | -28 | +| `sentry-mcp-ai-conversation-absolute-range-period-conflict` | 94 | 66 | -28 | +| all other profiled performance cases | unchanged | unchanged | 0 | + +Security-control profile result: + +| Case | Baseline scanner chunks | Pressure scanner chunks | +| --- | ---: | ---: | +| `sentry-preprod-snapshot-project-access` | 4 | 4 | +| `sentry-release-threshold-empty-project-filter` | 2 | 2 | +| `sentry-replay-delete-read-scope` | 2 | 2 | +| `sentry-slack-options-load-unscoped-group` | 4 | 4 | + +High-effort recall result under the temporary rule: + +| Case | Findings | Result | +| --- | ---: | --- | +| `sentry-preprod-snapshot-project-access` | 1 high | pass | +| `sentry-release-threshold-empty-project-filter` | 1 high | pass | +| `sentry-replay-delete-read-scope` | 1 high | pass | +| `sentry-slack-options-load-unscoped-group` | 1 high | pass | + +Real performance result: + +| Case | Mode | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `sentry-mcp-search-issues-period-30d` | baseline | 95 | 0 | 7m01s | 2,064,917 | 54,596 | $3.8790 | +| `sentry-mcp-search-issues-period-30d` | pressure coalescing | 66 (65 completed, 1 skipped) | 0 | 9m23s | 2,503,750 | 60,629 | $4.6710 | + +Why it failed: + +- It reduced scanner calls, but larger chunks were more expensive for the model + to reason over. +- The new long pole was `packages/mcp-core/src/toolDefinitions.json`, which + remained 8 scanner chunks and took 9m20s / $2.90 by itself. +- The collapsed `search-events.test.ts` shape was operationally better + (32 chunks became 3), but that win was erased by larger prompts elsewhere. + +Takeaway: + +- Chunk count alone is the wrong optimization target. +- Any strategy that merges hunks must track prompt size and per-file long poles, + not only scanner-call count. +- The next likely lever is explicit generated/derived artifact policy, because + derived definition JSON dominated the failed run. + ## Acceptance Before using this benchmark to make decisions: @@ -526,8 +1044,8 @@ Before using this benchmark to make decisions: - the runner must preserve enough logs to debug planner grouping decisions - expected finding judgments must reuse the existing eval judge or equivalent semantic matching -- semantic-on must expose planner summaries in the artifact - interrupted or timed-out runs must be marked incomplete, not compared -- performance wins must preserve finding quality, not only reduce chunk count -- optimized chunking must beat semantic mode on runtime or cost before it becomes - the default for pathological patches +- performance wins must preserve or improve findings, then reduce wall time or + total cost +- optimized chunking must beat the current baseline on wall time or total cost + before it becomes the default for pathological patches From d266fc6ecbb620d1bdc8d73d6c64bdfdf922b638 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 1 Jul 2026 13:50:29 -0700 Subject: [PATCH 16/18] test(chunking): Fix benchmark gate accounting Use total JSONL usage for benchmark cost summaries and populate expected finding matches for historical cases. Keep semantic chunking opt-in by default, remove the stale whole-file semantic knob, and mark semantic review chunks as historical experiment evidence rather than the accepted efficiency path. Co-Authored-By: GPT-5 Codex --- benchmarks/chunking/runner.ts | 108 +++++++++++++++--- package.json | 3 + packages/evals/src/semantic-chunking.eval.ts | 1 - .../src/action/triggers/executor.test.ts | 4 +- packages/warden/src/cli/output/tasks.test.ts | 1 - packages/warden/src/config/loader.test.ts | 4 +- packages/warden/src/config/schema.ts | 2 - packages/warden/src/sdk/analyze.test.ts | 2 - packages/warden/src/sdk/prepare.test.ts | 1 - packages/warden/src/semantic/planner.test.ts | 3 - skills/warden/references/config-schema.md | 3 +- skills/warden/references/configuration.md | 5 +- specs/chunking-strategy-benchmark.md | 9 +- specs/semantic-review-chunks.md | 6 +- warden.toml | 11 -- 15 files changed, 112 insertions(+), 51 deletions(-) diff --git a/benchmarks/chunking/runner.ts b/benchmarks/chunking/runner.ts index 100d9110..13d96b1b 100644 --- a/benchmarks/chunking/runner.ts +++ b/benchmarks/chunking/runner.ts @@ -158,6 +158,7 @@ const BenchmarkFindingSchema = z.object({ }).passthrough(); const ChunkRecordSchema = z.object({ + type: z.string().optional(), run: z.object({ durationMs: z.number().optional(), }).optional(), @@ -174,6 +175,13 @@ const ChunkRecordSchema = z.object({ costUSD: z.number().optional(), }).optional(), }).optional(), + total: z.object({ + usage: z.object({ + inputTokens: z.number().optional(), + outputTokens: z.number().optional(), + costUSD: z.number().optional(), + }), + }).optional(), }).optional(), }).passthrough(); @@ -241,6 +249,63 @@ interface RunSummary { }; } +function addUsage( + target: RunSummary['usage'], + usage: Partial | undefined +): void { + target.inputTokens += usage?.inputTokens ?? 0; + target.outputTokens += usage?.outputTokens ?? 0; + target.costUSD += usage?.costUSD ?? 0; +} + +function normalizedTokens(text: string): Set { + return new Set( + text + .toLowerCase() + .replace(/[`'"]/g, '') + .split(/[^a-z0-9:_-]+/) + .filter((token) => token.length >= 4) + ); +} + +function findingText(finding: z.infer): string { + const record = finding as Record; + return [ + finding.title, + record['description'], + record['verification'], + finding.location?.path, + ].filter((value): value is string => typeof value === 'string').join(' '); +} + +function findingMatchesExpectation( + finding: z.infer, + expected: string +): boolean { + const expectedText = expected.toLowerCase(); + const actualText = findingText(finding).toLowerCase(); + if (actualText.includes(expectedText)) return true; + + const expectedTokens = normalizedTokens(expected); + if (expectedTokens.size === 0) return false; + const actualTokens = normalizedTokens(actualText); + let matched = 0; + for (const token of expectedTokens) { + if (actualTokens.has(token)) matched += 1; + } + return matched / expectedTokens.size >= 0.45; +} + +function expectedFindingsMatched( + findings: z.infer[], + expectedFindings: string[] +): boolean | null { + if (expectedFindings.length === 0) return null; + return expectedFindings.every((expected) => + findings.some((finding) => findingMatchesExpectation(finding, expected)) + ); +} + interface ChunkProfile { fileCount: number; skippedFileCount: number; @@ -268,7 +333,7 @@ function usage(exitCode = 2): never { '', 'Options:', ' --suite historical or performance (default: historical)', - ' --mode nonsemantic, semantic, or both (default: both)', + ' --mode nonsemantic, semantic, or both (default: nonsemantic)', ' --sentry-repo Existing getsentry/sentry checkout for historical cases', ' --sentry-mcp-repo Existing getsentry/sentry-mcp checkout for performance cases', ' --warden-repo Existing getsentry/warden checkout for performance cases', @@ -291,7 +356,7 @@ function usage(exitCode = 2): never { function parseArgs(argv: string[]): Args { const args: Args = { suite: 'historical', - mode: 'both', + mode: 'nonsemantic', cases: [], sentryMcpRepo: '/home/dcramer/src/sentry-mcp', wardenRepo: resolve(import.meta.dirname, '../..'), @@ -443,7 +508,6 @@ function writeBenchmarkConfig(path: string, semantic: boolean, model: string, ru 'maxEmbeddedDiffChars = 8000', 'maxEmbeddedDiffChunks = 12', 'maxEmbeddedDiffRanges = 12', - 'preferWholeFileBelowLines = 800', '', '[[skills]]', `name = "${skill}"`, @@ -453,9 +517,10 @@ function writeBenchmarkConfig(path: string, semantic: boolean, model: string, ru return configPath; } -function summarizeJsonl(path: string, exitCode: number): RunSummary { +function summarizeJsonl(path: string, exitCode: number, expectedFindings: string[] = []): RunSummary { const findings: z.infer[] = []; const usage = { inputTokens: 0, outputTokens: 0, costUSD: 0 }; + let summaryUsage: RunSummary['usage'] | undefined; let scannerChunks = 0; let completedScannerChunks = 0; let skippedScannerChunks = 0; @@ -469,7 +534,7 @@ function summarizeJsonl(path: string, exitCode: number): RunSummary { complete: exitCode === 0, outputPath: path, exitCode, - expectedFindingMatched: null, + expectedFindingMatched: expectedFindings.length > 0 ? false : null, scannerChunks, completedScannerChunks, skippedScannerChunks, @@ -485,6 +550,19 @@ function summarizeJsonl(path: string, exitCode: number): RunSummary { if (!parsed.success) continue; const record = parsed.data; if (record.run?.durationMs) durationMs = Math.max(durationMs, record.run.durationMs); + if (record.type === 'summary') { + const totalUsage = record.usageBreakdown?.total?.usage; + if (totalUsage) { + summaryUsage = { + inputTokens: totalUsage.inputTokens ?? 0, + outputTokens: totalUsage.outputTokens ?? 0, + costUSD: totalUsage.costUSD ?? 0, + }; + } + continue; + } + + addUsage(usage, record.usageBreakdown?.total?.usage); if (!record.chunk || record.chunk.lineRange === 'post-processing') continue; scannerChunks += 1; @@ -492,11 +570,6 @@ function summarizeJsonl(path: string, exitCode: number): RunSummary { else if (record.status === 'skipped') skippedScannerChunks += 1; else failedScannerChunks += 1; findings.push(...(record.findings ?? [])); - - const scanUsage = record.usageBreakdown?.scan?.usage; - usage.inputTokens += scanUsage?.inputTokens ?? 0; - usage.outputTokens += scanUsage?.outputTokens ?? 0; - usage.costUSD += scanUsage?.costUSD ?? 0; } return { @@ -504,21 +577,21 @@ function summarizeJsonl(path: string, exitCode: number): RunSummary { complete: exitCode === 0 && failedScannerChunks === 0, outputPath: path, exitCode, - expectedFindingMatched: null, + expectedFindingMatched: expectedFindingsMatched(findings, expectedFindings), scannerChunks, completedScannerChunks, skippedScannerChunks, failedScannerChunks, findings, durationMs, - usage, + usage: summaryUsage ?? usage, }; } function runWarden( args: Args, worktree: string, - benchmarkCase: { name: string; base: string; skill: string }, + benchmarkCase: { name: string; base: string; skill: string; expectedFindings?: string[] }, semantic: boolean, ): RunSummary { const config = writeBenchmarkConfig(worktree, semantic, args.model, args.runtime, benchmarkCase.skill); @@ -555,7 +628,7 @@ function runWarden( stdio: 'inherit', }); - const summary = summarizeJsonl(outputPath, result.status ?? 1); + const summary = summarizeJsonl(outputPath, result.status ?? 1, benchmarkCase.expectedFindings); summary.semantic = semantic; return summary; } @@ -752,7 +825,12 @@ if (args.suite === 'historical') { runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( args, worktree, - { name: benchmarkCase.name, base: benchmarkCase.fixCommit, skill: benchmarkCase.skill }, + { + name: benchmarkCase.name, + base: benchmarkCase.fixCommit, + skill: benchmarkCase.skill, + expectedFindings: benchmarkCase.expectedFindings, + }, semantic, ); } diff --git a/package.json b/package.json index 290fa6c0..e6a1df64 100644 --- a/package.json +++ b/package.json @@ -36,6 +36,9 @@ "packages/evals/src/**/*.ts": [ "oxlint --fix" ], + "packages/evals/scripts/**/*.ts": [ + "oxlint --fix" + ], "benchmarks/**/*.ts": [ "oxlint --fix" ], diff --git a/packages/evals/src/semantic-chunking.eval.ts b/packages/evals/src/semantic-chunking.eval.ts index ec56615d..61e90e95 100644 --- a/packages/evals/src/semantic-chunking.eval.ts +++ b/packages/evals/src/semantic-chunking.eval.ts @@ -134,7 +134,6 @@ function createSemanticChunkingHarness(): Harness { chunking: { maxContextFiles: 12, filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], - semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, preferWholeFileBelowLines: 800 }, + semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50 }, }, auxiliaryMaxRetries: 9, }, { @@ -217,7 +217,7 @@ describe('executeTrigger', () => { scan: { maxFiles: 5 }, chunking: { filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], - semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, preferWholeFileBelowLines: 800 }, + semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50 }, }, auxiliaryMaxRetries: 9, }), diff --git a/packages/warden/src/cli/output/tasks.test.ts b/packages/warden/src/cli/output/tasks.test.ts index 27e9b94b..51fef77f 100644 --- a/packages/warden/src/cli/output/tasks.test.ts +++ b/packages/warden/src/cli/output/tasks.test.ts @@ -766,7 +766,6 @@ describe('runSkillTasks', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, postProcessFindings: false, diff --git a/packages/warden/src/config/loader.test.ts b/packages/warden/src/config/loader.test.ts index 5a795944..1e1f7b1d 100644 --- a/packages/warden/src/config/loader.test.ts +++ b/packages/warden/src/config/loader.test.ts @@ -595,7 +595,7 @@ describe('mergeWardenConfigs', () => { chunking: { filePatterns: [{ pattern: '**/*.lock', mode: 'skip' }], coalesce: { enabled: true, maxGapLines: 20, maxChunkSize: 4000 }, - semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, preferWholeFileBelowLines: 800 }, + semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50 }, maxContextFiles: 25, }, }, @@ -623,7 +623,6 @@ describe('mergeWardenConfigs', () => { maxEmbeddedDiffChars: 4000, maxEmbeddedDiffChunks: 8, maxEmbeddedDiffRanges: 8, - preferWholeFileBelowLines: 500, }, maxContextFiles: 10, }, @@ -662,7 +661,6 @@ describe('mergeWardenConfigs', () => { maxEmbeddedDiffChars: 4000, maxEmbeddedDiffChunks: 8, maxEmbeddedDiffRanges: 8, - preferWholeFileBelowLines: 500, }, maxContextFiles: 10, }, diff --git a/packages/warden/src/config/schema.ts b/packages/warden/src/config/schema.ts index a2fad82d..ffdf179e 100644 --- a/packages/warden/src/config/schema.ts +++ b/packages/warden/src/config/schema.ts @@ -197,8 +197,6 @@ export const SemanticChunkingConfigSchema = z.object({ maxEmbeddedDiffChunks: z.number().int().nonnegative().optional(), /** Maximum changed ranges allowed before the semantic planner stops embedding full hunk content */ maxEmbeddedDiffRanges: z.number().int().nonnegative().optional(), - /** Reserved for whole-file materialization below this line count */ - preferWholeFileBelowLines: z.number().int().positive().optional(), }); export type SemanticChunkingConfig = z.infer; diff --git a/packages/warden/src/sdk/analyze.test.ts b/packages/warden/src/sdk/analyze.test.ts index 1aa1d9a6..123f699b 100644 --- a/packages/warden/src/sdk/analyze.test.ts +++ b/packages/warden/src/sdk/analyze.test.ts @@ -837,7 +837,6 @@ describe('runSkill', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, postProcessFindings: false, @@ -914,7 +913,6 @@ describe('runSkill', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, postProcessFindings: false, diff --git a/packages/warden/src/sdk/prepare.test.ts b/packages/warden/src/sdk/prepare.test.ts index b8957315..f0eb2f16 100644 --- a/packages/warden/src/sdk/prepare.test.ts +++ b/packages/warden/src/sdk/prepare.test.ts @@ -234,7 +234,6 @@ describe('prepareFiles', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, }); diff --git a/packages/warden/src/semantic/planner.test.ts b/packages/warden/src/semantic/planner.test.ts index 9ed413b9..3b2ca0f1 100644 --- a/packages/warden/src/semantic/planner.test.ts +++ b/packages/warden/src/semantic/planner.test.ts @@ -101,7 +101,6 @@ describe('planSemanticReviewChunks', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, }); @@ -170,7 +169,6 @@ describe('planSemanticReviewChunks', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, }); @@ -244,7 +242,6 @@ describe('planSemanticReviewChunks', () => { maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50, - preferWholeFileBelowLines: 800, }, }, }); diff --git a/skills/warden/references/config-schema.md b/skills/warden/references/config-schema.md index d45fdc73..0d4f670d 100644 --- a/skills/warden/references/config-schema.md +++ b/skills/warden/references/config-schema.md @@ -62,11 +62,10 @@ maxGapLines = 30 # Lines between hunks to merge maxChunkSize = 8000 # Max chars per chunk [defaults.chunking.semantic] -enabled = true # Group small related review chunks +enabled = false # Experimental semantic grouping opt-in maxChunks = 20 # Target max review chunks after grouping maxChunkChars = 30000 # Max chars per grouped review chunk maxHunksPerChunk = 50 # Max raw hunks per grouped review chunk -preferWholeFileBelowLines = 800 # Reserved for whole-file materialization [[defaults.chunking.filePatterns]] pattern = "*.config.*" # Glob pattern diff --git a/skills/warden/references/configuration.md b/skills/warden/references/configuration.md index 8ec1d829..feae9d8b 100644 --- a/skills/warden/references/configuration.md +++ b/skills/warden/references/configuration.md @@ -84,14 +84,13 @@ pattern = "*.config.*" mode = "whole-file" ``` -**Group small review chunks:** +**Experiment with semantic grouping:** ```toml [defaults.chunking.semantic] -enabled = true +enabled = false maxChunks = 20 maxChunkChars = 30000 maxHunksPerChunk = 50 -preferWholeFileBelowLines = 800 ``` ## Model Lanes diff --git a/specs/chunking-strategy-benchmark.md b/specs/chunking-strategy-benchmark.md index 07b78c3a..7bf0613d 100644 --- a/specs/chunking-strategy-benchmark.md +++ b/specs/chunking-strategy-benchmark.md @@ -448,13 +448,14 @@ At minimum, each result record should include: "outputTokens": 0, "costUSD": 0 }, - "expectedFindingMatched": null + "expectedFindingMatched": true } ``` -`expectedFindingMatched` stays `null` until the benchmark reuses the existing -eval judge for semantic finding matching. Do not use exact title matching as a -substitute; these reports often describe the same bug with different wording. +`expectedFindingMatched` is a deterministic benchmark gate for historical cases. +It is useful for rejecting obvious misses, but final benchmark decisions should +still manually adjudicate finding equivalence until a judge is wired into this +runner. ## Captured Baseline diff --git a/specs/semantic-review-chunks.md b/specs/semantic-review-chunks.md index e8c10e40..d76a3f0e 100644 --- a/specs/semantic-review-chunks.md +++ b/specs/semantic-review-chunks.md @@ -1,5 +1,10 @@ # Semantic Review Chunks +Status: historical experiment. The current chunking benchmark rejects semantic +grouping as an efficiency strategy: it did not produce same-or-better findings +with lower wall time or total cost. Keep this document as design context for +the experiment, not as the accepted optimization plan. + Warden currently prepares code for review from git hunks. Git hunks are useful for changed-line anchoring, but they are a poor unit of review. A single logical change can produce dozens of tiny hunks, especially in tests, generated catalogs, @@ -349,7 +354,6 @@ maxChangedRangesPerChunk = 4 maxEmbeddedDiffChars = 8000 maxEmbeddedDiffChunks = 12 maxEmbeddedDiffRanges = 12 -preferWholeFileBelowLines = 800 ``` Do not expose fallback behavior as config. If semantic planning fails mechanical diff --git a/warden.toml b/warden.toml index e164954c..7439ec1c 100644 --- a/warden.toml +++ b/warden.toml @@ -10,17 +10,6 @@ reportOn = "medium" # Exclude build output and internal eval fixtures from all skills ignorePaths = ["dist/**", "packages/evals/**"] -[defaults.chunking.semantic] -enabled = true -maxChunks = 20 -maxChunkChars = 20000 -maxHunksPerChunk = 4 -maxChangedRangesPerChunk = 4 -maxEmbeddedDiffChars = 8000 -maxEmbeddedDiffChunks = 12 -maxEmbeddedDiffRanges = 12 -preferWholeFileBelowLines = 800 - [[skills]] name = "security-review" paths = [ From b763756420a87fcfb5485df03de45e33f777a808 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 1 Jul 2026 14:00:16 -0700 Subject: [PATCH 17/18] ref(chunking): Remove semantic chunking implementation Keep the benchmark evidence that rejected semantic grouping, but remove the product planner, config, eval, and ReviewChunk plumbing so the branch only carries the baseline benchmark work. The benchmark runner now measures the normal non-semantic path. Co-Authored-By: GPT-5 --- benchmarks/chunking/runner.ts | 94 +-- packages/evals/src/semantic-chunking.eval.ts | 276 -------- .../src/action/triggers/executor.test.ts | 11 +- .../warden/src/action/triggers/executor.ts | 81 +-- .../src/action/workflow/pr-workflow.test.ts | 10 +- .../warden/src/action/workflow/pr-workflow.ts | 35 +- .../warden/src/cli/output/ink-runner.test.tsx | 1 - packages/warden/src/cli/output/ink-runner.tsx | 7 +- packages/warden/src/cli/output/tasks.test.ts | 256 ++------ packages/warden/src/cli/output/tasks.ts | 226 ++----- packages/warden/src/config/loader.test.ts | 56 -- packages/warden/src/config/loader.ts | 11 - packages/warden/src/config/schema.ts | 28 +- packages/warden/src/diff/index.ts | 1 - packages/warden/src/diff/review-chunk.test.ts | 84 --- packages/warden/src/diff/review-chunk.ts | 160 ----- packages/warden/src/sdk/analyze.test.ts | 226 +------ packages/warden/src/sdk/analyze.ts | 316 ++++------ packages/warden/src/sdk/extract.ts | 32 +- packages/warden/src/sdk/prepare.test.ts | 71 +-- packages/warden/src/sdk/prepare.ts | 32 +- packages/warden/src/sdk/prompt.ts | 61 +- packages/warden/src/sdk/runner.test.ts | 54 +- packages/warden/src/sdk/runner.ts | 3 +- packages/warden/src/sdk/runtimes/types.ts | 1 - packages/warden/src/sdk/types.ts | 20 +- packages/warden/src/semantic/index.ts | 8 - packages/warden/src/semantic/planner.test.ts | 593 ------------------ packages/warden/src/semantic/planner.ts | 568 ----------------- packages/warden/src/semantic/tools.test.ts | 93 --- packages/warden/src/semantic/tools.ts | 156 ----- skills/warden/references/config-schema.md | 9 +- skills/warden/references/configuration.md | 9 - specs/chunking-strategy-benchmark.md | 6 +- specs/semantic-review-chunks.md | 590 ----------------- 35 files changed, 338 insertions(+), 3847 deletions(-) delete mode 100644 packages/evals/src/semantic-chunking.eval.ts delete mode 100644 packages/warden/src/diff/review-chunk.test.ts delete mode 100644 packages/warden/src/diff/review-chunk.ts delete mode 100644 packages/warden/src/semantic/index.ts delete mode 100644 packages/warden/src/semantic/planner.test.ts delete mode 100644 packages/warden/src/semantic/planner.ts delete mode 100644 packages/warden/src/semantic/tools.test.ts delete mode 100644 packages/warden/src/semantic/tools.ts delete mode 100644 specs/semantic-review-chunks.md diff --git a/benchmarks/chunking/runner.ts b/benchmarks/chunking/runner.ts index 13d96b1b..1cfbbb5d 100644 --- a/benchmarks/chunking/runner.ts +++ b/benchmarks/chunking/runner.ts @@ -198,7 +198,6 @@ const HistoricalScenarioSchema = z.object({ interface Args { suite: 'historical' | 'performance'; - mode: 'nonsemantic' | 'semantic' | 'both'; sentryRepo?: string; sentryMcpRepo: string; wardenRepo: string; @@ -231,7 +230,6 @@ interface PerformanceCase { } interface RunSummary { - semantic: boolean; complete: boolean; outputPath: string; exitCode: number; @@ -318,7 +316,6 @@ interface ChunkProfile { changedRanges: number; contentChars: number; lineRanges: string[]; - contentModes: string[]; }[]; skippedFiles: { filename: string; @@ -333,7 +330,6 @@ function usage(exitCode = 2): never { '', 'Options:', ' --suite historical or performance (default: historical)', - ' --mode nonsemantic, semantic, or both (default: nonsemantic)', ' --sentry-repo Existing getsentry/sentry checkout for historical cases', ' --sentry-mcp-repo Existing getsentry/sentry-mcp checkout for performance cases', ' --warden-repo Existing getsentry/warden checkout for performance cases', @@ -356,7 +352,6 @@ function usage(exitCode = 2): never { function parseArgs(argv: string[]): Args { const args: Args = { suite: 'historical', - mode: 'nonsemantic', cases: [], sentryMcpRepo: '/home/dcramer/src/sentry-mcp', wardenRepo: resolve(import.meta.dirname, '../..'), @@ -378,10 +373,6 @@ function parseArgs(argv: string[]): Args { const suite = argv[++i]; if (suite !== 'historical' && suite !== 'performance') usage(); args.suite = suite; - } else if (arg === '--mode') { - const mode = argv[++i]; - if (mode !== 'nonsemantic' && mode !== 'semantic' && mode !== 'both') usage(); - args.mode = mode; } else if (arg === '--sentry-repo') args.sentryRepo = resolve(argv[++i] ?? usage()); else if (arg === '--sentry-mcp-repo') args.sentryMcpRepo = resolve(argv[++i] ?? usage()); else if (arg === '--warden-repo') args.wardenRepo = resolve(argv[++i] ?? usage()); @@ -483,8 +474,8 @@ function createBenchmarkWorktree(sourceRepo: string, benchmarkCase: BenchmarkCas return worktree; } -function writeBenchmarkConfig(path: string, semantic: boolean, model: string, runtime: string, skill: string): string { - const configPath = join(path, semantic ? 'warden.semantic.toml' : 'warden.nonsemantic.toml'); +function writeBenchmarkConfig(path: string, model: string, runtime: string, skill: string): string { + const configPath = join(path, 'warden.benchmark.toml'); const findingThreshold = skill === 'security-review' ? 'low' : 'medium'; writeFileSync(configPath, [ 'version = 1', @@ -499,16 +490,6 @@ function writeBenchmarkConfig(path: string, semantic: boolean, model: string, ru '[defaults.verification]', 'enabled = false', '', - '[defaults.chunking.semantic]', - `enabled = ${semantic ? 'true' : 'false'}`, - 'maxChunks = 20', - 'maxChunkChars = 20000', - 'maxHunksPerChunk = 4', - 'maxChangedRangesPerChunk = 4', - 'maxEmbeddedDiffChars = 8000', - 'maxEmbeddedDiffChunks = 12', - 'maxEmbeddedDiffRanges = 12', - '', '[[skills]]', `name = "${skill}"`, 'paths = ["**/*"]', @@ -530,7 +511,6 @@ function summarizeJsonl(path: string, exitCode: number, expectedFindings: string const content = existsSync(path) ? readFileSync(path, 'utf8').trim() : ''; if (!content) { return { - semantic: false, complete: exitCode === 0, outputPath: path, exitCode, @@ -573,7 +553,6 @@ function summarizeJsonl(path: string, exitCode: number, expectedFindings: string } return { - semantic: false, complete: exitCode === 0 && failedScannerChunks === 0, outputPath: path, exitCode, @@ -592,10 +571,9 @@ function runWarden( args: Args, worktree: string, benchmarkCase: { name: string; base: string; skill: string; expectedFindings?: string[] }, - semantic: boolean, ): RunSummary { - const config = writeBenchmarkConfig(worktree, semantic, args.model, args.runtime, benchmarkCase.skill); - const outputPath = join(args.artifactsDir, `${benchmarkCase.name}.${semantic ? 'semantic' : 'nonsemantic'}.jsonl`); + const config = writeBenchmarkConfig(worktree, args.model, args.runtime, benchmarkCase.skill); + const outputPath = join(args.artifactsDir, `${benchmarkCase.name}.nonsemantic.jsonl`); const cliArgs = [ 'cli', '--', @@ -628,15 +606,7 @@ function runWarden( stdio: 'inherit', }); - const summary = summarizeJsonl(outputPath, result.status ?? 1, benchmarkCase.expectedFindings); - summary.semantic = semantic; - return summary; -} - -function selectedModes(args: Args): boolean[] { - if (args.mode === 'nonsemantic') return [false]; - if (args.mode === 'semantic') return [true]; - return [false, true]; + return summarizeJsonl(outputPath, result.status ?? 1, benchmarkCase.expectedFindings); } function sourceRepoForPerformanceCase(args: Args, benchmarkCase: PerformanceCase): string { @@ -746,10 +716,10 @@ function makeProfileContext( }; } -function lineRangeForChunk(chunk: { changedLineMap: { start: number; end: number }[] }): string { - return chunk.changedLineMap - .map((range) => range.start === range.end ? `${range.start}` : `${range.start}-${range.end}`) - .join(','); +function lineRangeForHunk(hunk: { hunk: { newStart: number; newCount: number } }): string { + const start = hunk.hunk.newStart; + const end = start + Math.max(hunk.hunk.newCount - 1, 0); + return start === end ? `${start}` : `${start}-${end}`; } function profileChunks(worktree: string, repository: string, base: string): ChunkProfile { @@ -757,17 +727,17 @@ function profileChunks(worktree: string, repository: string, base: string): Chun const context = makeProfileContext(worktree, repository, base, files); const prepared = prepareFiles(context); const profileFiles = prepared.files.map((file) => { - const contentChars = file.chunks.reduce( - (total, chunk) => total + chunk.files.reduce((sum, chunkFile) => sum + chunkFile.content.length, 0), + const contentChars = file.hunks.reduce( + (total, hunk) => + total + hunk.hunk.content.length + hunk.contextBefore.join('\n').length + hunk.contextAfter.join('\n').length, 0, ); return { filename: file.filename, - chunks: file.chunks.length, - changedRanges: file.chunks.reduce((total, chunk) => total + chunk.changedLineMap.length, 0), + chunks: file.hunks.length, + changedRanges: file.hunks.length, contentChars, - lineRanges: file.chunks.map(lineRangeForChunk), - contentModes: [...new Set(file.chunks.flatMap((chunk) => chunk.files.map((chunkFile) => chunkFile.contentMode)))], + lineRanges: file.hunks.map(lineRangeForHunk), }; }); @@ -821,19 +791,12 @@ if (args.suite === 'historical') { }); } else { const runs: Record = {}; - for (const semantic of selectedModes(args)) { - runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( - args, - worktree, - { - name: benchmarkCase.name, - base: benchmarkCase.fixCommit, - skill: benchmarkCase.skill, - expectedFindings: benchmarkCase.expectedFindings, - }, - semantic, - ); - } + runs['nonsemantic'] = runWarden(args, worktree, { + name: benchmarkCase.name, + base: benchmarkCase.fixCommit, + skill: benchmarkCase.skill, + expectedFindings: benchmarkCase.expectedFindings, + }); results.push({ case: benchmarkCase.name, repository: benchmarkCase.repository, @@ -871,14 +834,11 @@ if (args.suite === 'historical') { }); } else { const runs: Record = {}; - for (const semantic of selectedModes(args)) { - runs[semantic ? 'semantic' : 'nonsemantic'] = runWarden( - args, - worktree, - { name: benchmarkCase.name, base: benchmarkCase.base, skill: benchmarkCase.skill ?? 'code-review' }, - semantic, - ); - } + runs['nonsemantic'] = runWarden( + args, + worktree, + { name: benchmarkCase.name, base: benchmarkCase.base, skill: benchmarkCase.skill ?? 'code-review' }, + ); results.push({ case: benchmarkCase.name, repository: benchmarkCase.repository, @@ -900,7 +860,7 @@ writeFileSync(args.output, JSON.stringify({ schemaVersion: 1, capturedAt: new Date().toISOString(), suite: args.suite, - mode: args.mode, + mode: 'nonsemantic', model: args.model, runtime: args.runtime, cases: results, diff --git a/packages/evals/src/semantic-chunking.eval.ts b/packages/evals/src/semantic-chunking.eval.ts deleted file mode 100644 index 61e90e95..00000000 --- a/packages/evals/src/semantic-chunking.eval.ts +++ /dev/null @@ -1,276 +0,0 @@ -import { mkdtemp, mkdir, rm, writeFile } from 'node:fs/promises'; -import { join } from 'node:path'; -import { tmpdir } from 'node:os'; -import { expect } from 'vitest'; -import { createJudge, describeEval } from 'vitest-evals'; -import type { JudgeContext } from 'vitest-evals'; -import { normalizeContent, toJsonValue, type Harness, type JsonValue } from 'vitest-evals/harness'; -import { z } from 'zod'; -import type { EventContext } from '../../warden/src/types/index.js'; -import { prepareFiles } from '../../warden/src/sdk/prepare.js'; -import { planSemanticReviewChunks } from '../../warden/src/semantic/index.js'; -import { - DEFAULT_EVAL_RUNTIME, - defaultEvalModel, - getEvalProviderApiKey, - getEvalRuntimeApiKey, -} from './auth.js'; - -const model = process.env['WARDEN_SEMANTIC_CHUNK_EVAL_MODEL'] ?? defaultEvalModel(); -const apiKey = getEvalRuntimeApiKey(model); -const providerApiKey = getEvalProviderApiKey(model); - -const SemanticChunkingEvalInputSchema = z.object({ - name: z.string(), -}); -type SemanticChunkingEvalInput = z.infer; - -const SemanticChunkingEvalOutputSchema = z.object({ - atomicChunkCount: z.number().int().nonnegative(), - groups: z.array(z.object({ - displayName: z.string(), - files: z.array(z.string()), - changedLineMap: z.array(z.object({ - path: z.string(), - start: z.number().int(), - end: z.number().int(), - })), - summary: z.string().optional(), - scannerChunkCount: z.number().int().positive(), - })), - chunks: z.array(z.object({ - summary: z.string().optional(), - files: z.array(z.string()), - changedLineMap: z.array(z.object({ - path: z.string(), - start: z.number().int(), - end: z.number().int(), - })), - content: z.string(), - })), -}); - -async function createSemanticChunkingRepo(): Promise { - const repoPath = await mkdtemp(join(tmpdir(), 'warden-semantic-chunking-eval-')); - await mkdir(join(repoPath, 'src'), { recursive: true }); - await mkdir(join(repoPath, 'tests'), { recursive: true }); - await writeFile(join(repoPath, 'src/dashboard.ts'), [ - 'const range = widget.axisRange ?? getDefaultAxisRange(widget);', - 'const chart = convertWidgetToChart(widget, range);', - 'return renderChart(chart, series, range);', - ].join('\n')); - await writeFile(join(repoPath, 'tests/dashboard.test.ts'), [ - 'expect(rendered.range).toEqual(customAxisRange);', - ].join('\n')); - return repoPath; -} - -function makeSemanticChunkingContext(repoPath: string): EventContext { - return { - eventType: 'pull_request', - action: 'opened', - repository: { - owner: 'getsentry', - name: 'semantic-chunking-eval', - fullName: 'getsentry/semantic-chunking-eval', - defaultBranch: 'main', - }, - repoPath, - pullRequest: { - number: 313, - title: 'Update dashboard behavior', - body: '', - author: 'eval', - baseBranch: 'main', - headBranch: 'semantic-axis-range', - headSha: 'head', - baseSha: 'base', - files: [{ - filename: 'src/dashboard.ts', - status: 'modified', - additions: 3, - deletions: 3, - chunks: 3, - patch: [ - '@@ -10,1 +10,1 @@', - '-const range = getDefaultAxisRange(widget);', - '+const range = widget.axisRange ?? getDefaultAxisRange(widget);', - '@@ -80,1 +80,1 @@', - '-const chart = convertWidgetToChart(widget);', - '+const chart = convertWidgetToChart(widget, range);', - '@@ -140,1 +140,1 @@', - '-return renderChart(chart, series);', - '+return renderChart(chart, series, range);', - ].join('\n'), - }, { - filename: 'tests/dashboard.test.ts', - status: 'modified', - additions: 1, - deletions: 1, - chunks: 1, - patch: [ - '@@ -220,1 +220,1 @@', - '-expect(rendered.range).toEqual(defaultAxisRange);', - '+expect(rendered.range).toEqual(customAxisRange);', - ].join('\n'), - }], - }, - }; -} - -function createSemanticChunkingHarness(): Harness { - return { - name: 'semantic-chunking', - run: async (input) => { - SemanticChunkingEvalInputSchema.parse(input); - const startTime = Date.now(); - const repoPath = await createSemanticChunkingRepo(); - try { - const context = makeSemanticChunkingContext(repoPath); - const prepared = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - }); - const atomicChunks = prepared.files.flatMap((file) => file.chunks); - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - apiKey, - runtime: DEFAULT_EVAL_RUNTIME, - model, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - maxEmbeddedDiffChars: 0, - maxEmbeddedDiffChunks: 0, - maxEmbeddedDiffRanges: 0, - }); - const chunks = planned.groups.flatMap((group) => group.chunks); - const output = { - atomicChunkCount: atomicChunks.length, - groups: planned.groups.map((group) => ({ - displayName: group.displayName, - files: group.filenames, - changedLineMap: group.chunks.flatMap((chunk) => chunk.changedLineMap), - summary: group.chunks.find((chunk) => chunk.summary)?.summary, - scannerChunkCount: group.chunks.length, - })), - chunks: chunks.map((chunk) => ({ - summary: chunk.summary, - files: chunk.files.map((file) => file.path), - changedLineMap: chunk.changedLineMap, - content: chunk.files.map((file) => file.content).join('\n'), - })), - }; - - return { - output: toJsonValue(output) as JsonValue, - session: { - messages: [ - { - role: 'user', - content: normalizeContent({ - name: input.name, - goal: 'Group related dashboard range changes semantically across implementation and test files.', - }), - }, - { - role: 'assistant', - content: normalizeContent(output), - }, - ], - provider: DEFAULT_EVAL_RUNTIME, - model, - }, - usage: {}, - timings: { totalMs: Date.now() - startTime }, - artifacts: {}, - errors: [], - }; - } finally { - await rm(repoPath, { recursive: true, force: true }); - } - }, - }; -} - -function createSemanticChunkingJudge() { - return createJudge>('SemanticChunkingJudge', async ({ run }) => { - const output = SemanticChunkingEvalOutputSchema.safeParse(run.output); - if (!output.success) { - return { - score: 0, - metadata: { rationale: `Invalid semantic chunking output: ${output.error.message}` }, - }; - } - - const chunks = output.data.chunks; - const groups = output.data.groups; - const crossFileGroup = groups.find((group) => group.files.length > 1); - const summaryText = groups.map((group) => group.summary ?? '').join(' '); - const hasSemanticSummary = /axisRange|axis range|range/i.test(summaryText) - && /widget|convert|render|chart/i.test(summaryText) - && !/lines?\s+\d+/i.test(summaryText) - && !/\b10\b.*\b80\b.*\b140\b/.test(summaryText); - const hasExpectedLineMap = Boolean(crossFileGroup) - && [ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'src/dashboard.ts', start: 80, end: 80 }, - { path: 'src/dashboard.ts', start: 140, end: 140 }, - { path: 'tests/dashboard.test.ts', start: 220, end: 220 }, - ].every((range) => - crossFileGroup?.changedLineMap.some((actual) => - actual.path === range.path && actual.start === range.start && actual.end === range.end - ) - ); - const allContent = chunks.map((chunk) => chunk.content).join('\n'); - const hasExpectedContent = Boolean(crossFileGroup) - && allContent.includes('convertWidgetToChart(widget, range)') - && allContent.includes('renderChart(chart, series, range)') - && allContent.includes('expect(rendered.range).toEqual(customAxisRange)'); - const reducedChunks = chunks.length < output.data.atomicChunkCount; - const passed = reducedChunks && Boolean(crossFileGroup) && hasExpectedLineMap - && hasSemanticSummary && hasExpectedContent; - - return { - score: passed ? 1 : 0, - metadata: { - rationale: passed - ? 'Planner produced a semantic cross-file chunk with behavior-level summary.' - : 'Planner did not produce the expected semantic cross-file chunk.', - atomicChunkCount: output.data.atomicChunkCount, - chunkCount: chunks.length, - hasCrossFileGroup: Boolean(crossFileGroup), - hasExpectedLineMap, - hasSemanticSummary, - hasExpectedContent, - }, - }; - }); -} - -describeEval( - 'semantic-chunking', - { - harness: createSemanticChunkingHarness(), - judges: [createSemanticChunkingJudge()], - judgeThreshold: 1, - skipIf: () => !providerApiKey, - }, - (it) => { - it('plans a behavior-level delta for many tiny related hunks', { timeout: 120_000 }, async ({ run }) => { - const result = await run({ name: 'dashboard-axis-range' }); - const output = SemanticChunkingEvalOutputSchema.parse(result.output); - const chunks = output.chunks; - - expect(output.atomicChunkCount).toBe(4); - expect(chunks.length).toBeLessThan(output.atomicChunkCount); - expect(output.groups.some((group) => group.files.length > 1)).toBe(true); - }); - }, -); diff --git a/packages/warden/src/action/triggers/executor.test.ts b/packages/warden/src/action/triggers/executor.test.ts index 0be5c0d5..4aa23a6d 100644 --- a/packages/warden/src/action/triggers/executor.test.ts +++ b/packages/warden/src/action/triggers/executor.test.ts @@ -198,11 +198,7 @@ describe('executeTrigger', () => { maxContextFiles: 12, ignore: { paths: ['**/fixtures/**'] }, scan: { maxFiles: 5 }, - chunking: { - maxContextFiles: 12, - filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], - semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50 }, - }, + chunking: { maxContextFiles: 12, filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }] }, auxiliaryMaxRetries: 9, }, { ...mockDeps, @@ -215,10 +211,7 @@ describe('executeTrigger', () => { maxContextFiles: 12, ignore: { paths: ['**/fixtures/**'] }, scan: { maxFiles: 5 }, - chunking: { - filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], - semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50 }, - }, + chunking: { filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }] }, auxiliaryMaxRetries: 9, }), }), diff --git a/packages/warden/src/action/triggers/executor.ts b/packages/warden/src/action/triggers/executor.ts index 93be020a..90affb87 100644 --- a/packages/warden/src/action/triggers/executor.ts +++ b/packages/warden/src/action/triggers/executor.ts @@ -43,11 +43,8 @@ function toAnalysisChunkingConfig( if (chunking.coalesce) { analysisChunking.coalesce = chunking.coalesce; } - if (chunking.semantic) { - analysisChunking.semantic = chunking.semantic; - } - return analysisChunking.filePatterns || analysisChunking.coalesce || analysisChunking.semantic + return analysisChunking.filePatterns || analysisChunking.coalesce ? analysisChunking : undefined; } @@ -108,8 +105,6 @@ export interface TriggerExecutorDeps { circuitBreaker?: ProviderFailureCircuitBreaker; /** Optional context-bound check writer. Omit for analyze mode. */ checks?: TriggerCheckReporter; - /** Prepared review chunks shared by the action workflow across matching triggers. */ - preparedFiles?: SkillTaskOptions['preparedFiles']; } /** @@ -136,47 +131,6 @@ export interface TriggerResult { // Executor // ----------------------------------------------------------------------------- -/** Translate a resolved action trigger into the shared skill-task boundary. */ -export function createTriggerSkillTaskOptions( - trigger: ResolvedTrigger, - deps: TriggerExecutorDeps -): SkillTaskOptions { - const { context, anthropicApiKey, claudePath } = deps; - const failOn = trigger.failOn ?? deps.globalFailOn; - const skillRoot = trigger.useBuiltinSkill ? undefined : (trigger.skillRoot ?? context.repoPath); - - return { - name: trigger.name, - displayName: trigger.skill, - triggerName: trigger.name, - failOn, - resolveSkill: () => resolveSkillAsync(trigger.skill, skillRoot, { - remote: trigger.remote, - }), - context: filterContextByPaths(context, trigger.filters), - preparedFiles: deps.preparedFiles, - runnerOptions: { - apiKey: anthropicApiKey, - model: trigger.model, - runtime: trigger.runtime, - effort: trigger.effort, - auxiliaryModel: trigger.auxiliaryModel, - synthesisModel: trigger.synthesisModel, - maxTurns: trigger.maxTurns, - batchDelayMs: trigger.batchDelayMs, - maxContextFiles: trigger.maxContextFiles, - ignore: trigger.ignore, - scan: trigger.scan, - chunking: toAnalysisChunkingConfig(trigger.chunking), - pathToClaudeCodeExecutable: claudePath, - auxiliaryMaxRetries: trigger.auxiliaryMaxRetries, - verifyFindings: trigger.verifyFindings, - abortController: deps.abortController, - circuitBreaker: deps.circuitBreaker, - }, - }; -} - /** * Execute a single trigger and return results. * @@ -194,7 +148,7 @@ export async function executeTrigger( async (span) => { span.setAttribute('gen_ai.agent.name', trigger.skill); span.setAttribute('warden.trigger.name', trigger.name); - const { context } = deps; + const { context, anthropicApiKey, claudePath } = deps; logGroup(`Running trigger: ${trigger.name} (skill: ${trigger.skill})`); @@ -215,11 +169,40 @@ export async function executeTrigger( const minConfidence = trigger.minConfidence ?? 'medium'; const requestChanges = trigger.requestChanges ?? deps.globalRequestChanges; const failCheck = trigger.failCheck ?? deps.globalFailCheck; + const skillRoot = trigger.useBuiltinSkill ? undefined : (trigger.skillRoot ?? context.repoPath); try { assertValidPiModelSelectors([trigger]); - const taskOptions = createTriggerSkillTaskOptions(trigger, deps); + const taskOptions: SkillTaskOptions = { + name: trigger.name, + displayName: trigger.skill, + triggerName: trigger.name, + failOn, + resolveSkill: () => resolveSkillAsync(trigger.skill, skillRoot, { + remote: trigger.remote, + }), + context: filterContextByPaths(context, trigger.filters), + runnerOptions: { + apiKey: anthropicApiKey, + model: trigger.model, + runtime: trigger.runtime, + effort: trigger.effort, + auxiliaryModel: trigger.auxiliaryModel, + synthesisModel: trigger.synthesisModel, + maxTurns: trigger.maxTurns, + batchDelayMs: trigger.batchDelayMs, + maxContextFiles: trigger.maxContextFiles, + ignore: trigger.ignore, + scan: trigger.scan, + chunking: toAnalysisChunkingConfig(trigger.chunking), + pathToClaudeCodeExecutable: claudePath, + auxiliaryMaxRetries: trigger.auxiliaryMaxRetries, + verifyFindings: trigger.verifyFindings, + abortController: deps.abortController, + circuitBreaker: deps.circuitBreaker, + }, + }; const callbacks = createDefaultCallbacks([taskOptions], CI_OUTPUT_MODE, Verbosity.Normal); const fileConcurrency = deps.semaphore ? Number.MAX_SAFE_INTEGER : DEFAULT_FILE_CONCURRENCY; diff --git a/packages/warden/src/action/workflow/pr-workflow.test.ts b/packages/warden/src/action/workflow/pr-workflow.test.ts index 86f333d9..bb7d2cf0 100644 --- a/packages/warden/src/action/workflow/pr-workflow.test.ts +++ b/packages/warden/src/action/workflow/pr-workflow.test.ts @@ -40,7 +40,6 @@ vi.mock('../../cli/output/tasks.js', async () => { return { ...actual, runSkillTask: vi.fn(), - prepareSemanticPlansForTasks: vi.fn(async (tasks) => tasks), }; }); @@ -116,7 +115,7 @@ vi.mock('./base.js', async () => { }); // Import after mocks -import { prepareSemanticPlansForTasks, runSkillTask } from '../../cli/output/tasks.js'; +import { runSkillTask } from '../../cli/output/tasks.js'; import { fetchExistingComments, deduplicateFindings, processDuplicateActions } from '../../output/dedup.js'; import { evaluateFixAttempts } from '../fix-evaluation/index.js'; import { setFailed, writeFindingsOutput } from './base.js'; @@ -127,7 +126,6 @@ import { buildFindingsOutput } from '../reporting/output.js'; // Type the mocks const mockRunSkillTask = vi.mocked(runSkillTask); -const mockPrepareSemanticPlansForTasks = vi.mocked(prepareSemanticPlansForTasks); const mockFetchExistingComments = vi.mocked(fetchExistingComments); const mockDeduplicateFindings = vi.mocked(deduplicateFindings); const mockProcessDuplicateActions = vi.mocked(processDuplicateActions); @@ -1253,12 +1251,6 @@ describe('runPRWorkflow', () => { name: 'test-skill', displayName: 'test-skill', })); - expect(mockPrepareSemanticPlansForTasks).toHaveBeenCalledWith([ - expect.objectContaining({ - name: 'test-skill', - displayName: 'test-skill', - }), - ]); // When a semaphore is provided, fileConcurrency is unlimited (semaphore is the gate) expect(fileConcurrency).toBe(Number.MAX_SAFE_INTEGER); expect(semaphore).toBeInstanceOf(Semaphore); diff --git a/packages/warden/src/action/workflow/pr-workflow.ts b/packages/warden/src/action/workflow/pr-workflow.ts index 8b1e5e19..7546b383 100644 --- a/packages/warden/src/action/workflow/pr-workflow.ts +++ b/packages/warden/src/action/workflow/pr-workflow.ts @@ -39,9 +39,8 @@ import { formatCost, formatTokens, formatDuration } from '../../cli/output/forma import { findBotReviewState } from '../review-state.js'; import type { BotReviewInfo } from '../review-state.js'; import type { ActionInputs } from '../inputs.js'; -import { createTriggerSkillTaskOptions, executeTrigger } from '../triggers/executor.js'; +import { executeTrigger } from '../triggers/executor.js'; import type { TriggerCheckReporter, TriggerResult } from '../triggers/executor.js'; -import { prepareSemanticPlansForTasks } from '../../cli/output/tasks.js'; import { postTriggerReview } from '../review/poster.js'; import { shouldResolveStaleComments } from '../review/coordination.js'; import type { FindingObservation } from '../reporting/outcomes.js'; @@ -486,32 +485,24 @@ async function executeAllTriggers( const semaphore = new Semaphore(concurrency); const abortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController }); - const baseTriggerDeps = { - context, - anthropicApiKey: inputs.anthropicApiKey, - claudePath: runtimeEnv.pathToClaudeCodeExecutable, - globalFailOn: inputs.failOn, - globalReportOn: inputs.reportOn, - globalMaxFindings: inputs.maxFindings, - globalRequestChanges: inputs.requestChanges, - globalFailCheck: inputs.failCheck, - semaphore, - abortController, - circuitBreaker, - }; - const plannedTaskOptions = await prepareSemanticPlansForTasks( - matchedTriggers.map((trigger) => createTriggerSkillTaskOptions(trigger, baseTriggerDeps)) - ); - const preparedFilesByTriggerIndex = plannedTaskOptions.map((task) => task.preparedFiles); // Limit trigger dispatch too; the semaphore only gates work after a trigger starts. return runPool( matchedTriggers, concurrency, - (trigger, index) => + (trigger) => executeTrigger(trigger, { - ...baseTriggerDeps, - preparedFiles: preparedFilesByTriggerIndex[index], + context, + anthropicApiKey: inputs.anthropicApiKey, + claudePath: runtimeEnv.pathToClaudeCodeExecutable, + globalFailOn: inputs.failOn, + globalReportOn: inputs.reportOn, + globalMaxFindings: inputs.maxFindings, + globalRequestChanges: inputs.requestChanges, + globalFailCheck: inputs.failCheck, + semaphore, + abortController, + circuitBreaker, checks: options.checks, }), { shouldAbort: () => abortController.signal.aborted }, diff --git a/packages/warden/src/cli/output/ink-runner.test.tsx b/packages/warden/src/cli/output/ink-runner.test.tsx index 3fa12ecd..f6583ec5 100644 --- a/packages/warden/src/cli/output/ink-runner.test.tsx +++ b/packages/warden/src/cli/output/ink-runner.test.tsx @@ -36,7 +36,6 @@ vi.mock('ink', () => ({ vi.mock('./tasks.js', () => ({ composeTasksWithFailFast: vi.fn((tasks) => tasks), - prepareSemanticPlansForTasks: vi.fn(async (tasks) => tasks), runComposedSkillTasks: mockRunComposedSkillTasks, })); diff --git a/packages/warden/src/cli/output/ink-runner.tsx b/packages/warden/src/cli/output/ink-runner.tsx index dd9181d1..eacae891 100644 --- a/packages/warden/src/cli/output/ink-runner.tsx +++ b/packages/warden/src/cli/output/ink-runner.tsx @@ -18,7 +18,6 @@ import chalk from 'chalk'; import { composeTasksWithFailFast, runComposedSkillTasks, - prepareSemanticPlansForTasks, type SkillTaskOptions, type SkillTaskResult, type RunTasksOptions, @@ -281,9 +280,8 @@ export async function runSkillTasksWithInk( const semaphore = new Semaphore(concurrency); const circuitAbortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController: circuitAbortController }); - const plannedTasks = await prepareSemanticPlansForTasks(tasks); const composedTasks = composeTasksWithFailFast( - plannedTasks, + tasks, failFastController, circuitBreaker, circuitAbortController, @@ -482,9 +480,8 @@ export async function runSkillTasksWithInk( // Compose per-task abort controllers: fire on SIGINT, fail-fast, or provider circuit breaker. const circuitAbortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController: circuitAbortController }); - const plannedTasks = await prepareSemanticPlansForTasks(tasks); const composedTasks = composeTasksWithFailFast( - plannedTasks, + tasks, failFastController, circuitBreaker, circuitAbortController, diff --git a/packages/warden/src/cli/output/tasks.test.ts b/packages/warden/src/cli/output/tasks.test.ts index 51fef77f..3b7ea1b8 100644 --- a/packages/warden/src/cli/output/tasks.test.ts +++ b/packages/warden/src/cli/output/tasks.test.ts @@ -6,13 +6,12 @@ import type { OutputMode } from './tty.js'; import type { SkillReport, Finding, HunkFailure } from '../../types/index.js'; import type { SkillTaskOptions } from './tasks.js'; import type { FileAnalysisResult } from '../../sdk/types.js'; -import type { ReviewChunk } from '../../diff/index.js'; +import type { HunkWithContext } from '../../diff/index.js'; import type { SkillDefinition } from '../../config/schema.js'; import { Semaphore, runPool } from '../../utils/index.js'; import { SkillRunnerError, WardenAuthenticationError } from '../../sdk/errors.js'; import { ProviderFailureCircuitBreaker } from '../../sdk/circuit-breaker.js'; import * as sdkRunner from '../../sdk/runner.js'; -import * as semanticPlanner from '../../semantic/index.js'; function makeFinding(overrides: Partial = {}): Finding { return { @@ -44,23 +43,6 @@ function makeTask(name: string, displayName?: string): SkillTaskOptions { }; } -function makeReviewChunk(path = 'a.ts', start = 1, end = 10): ReviewChunk { - return { - id: `${path}:${start}-${end}`, - title: `${path}:${start}-${end}`, - summary: `Changes in ${path}`, - files: [{ - path, - language: 'typescript', - changedRanges: [{ path, start, end }], - content: '@@ -1,1 +1,1 @@\n-old\n+new', - contentMode: 'raw-hunks', - sourceLines: [{ line: start, content: 'new' }], - }], - changedLineMap: [{ path, start, end }], - }; -} - function logMode(): OutputMode { return { isTTY: false, supportsColor: false, columns: 80 }; } @@ -681,10 +663,12 @@ describe('runSkillTasks', () => { it('does not fail-fast on findings rejected by post-processing', async () => { const candidate = makeFinding(); const controller = new AbortController(); - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -727,185 +711,15 @@ describe('runSkillTasks', () => { postProcessFindings.mockRestore(); }); - it('shares semantic planning across skill tasks for the same changeset', async () => { - const fakeChunk = makeReviewChunk('a.ts', 1, 1); - const context = { - eventType: 'pull_request', - repository: { owner: 'o', name: 'n', fullName: 'o/n', defaultBranch: 'main' }, - repoPath: '/tmp', - pullRequest: { - number: 1, - title: 't', - body: '', - author: 'octocat', - baseBranch: 'main', - headBranch: 'feature', - headSha: 'abc', - baseSha: 'def', - files: [{ - filename: 'a.ts', - status: 'modified', - additions: 1, - deletions: 1, - patch: '@@ -1,1 +1,1 @@\n-old\n+new', - }], - }, - } as unknown as SkillTaskOptions['context']; - const copiedContext = { - ...context, - repository: { ...context.repository }, - pullRequest: { - ...context.pullRequest, - files: [...(context.pullRequest?.files ?? [])], - }, - } as SkillTaskOptions['context']; - const runnerOptions: SkillTaskOptions['runnerOptions'] = { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - postProcessFindings: false, - }; - - const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], - skippedFiles: [], - }); - const planSemanticReviewChunks = vi.spyOn(semanticPlanner, 'planSemanticReviewChunks').mockResolvedValue({ - groups: [{ displayName: 'a.ts', filenames: ['a.ts'], chunks: [fakeChunk] }], - usage: { inputTokens: 10, outputTokens: 5, costUSD: 0.001 }, - }); - const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ - filename: 'a.ts', - findings: [], - usage: { inputTokens: 1, outputTokens: 1, costUSD: 0.001 }, - failedHunks: 0, - failedExtractions: 0, - hunkFailures: [], - } satisfies FileAnalysisResult); - - const results = await runSkillTasks([ - { - name: 'skill-a', - resolveSkill: async () => ({ name: 'skill-a', description: '', prompt: '' }), - context, - runnerOptions, - }, - { - name: 'skill-b', - resolveSkill: async () => ({ name: 'skill-b', description: '', prompt: '' }), - context: copiedContext, - runnerOptions, - }, - ], { - mode: logMode(), - verbosity: Verbosity.Quiet, - concurrency: 2, - }); - - expect(prepareFiles).toHaveBeenCalledTimes(1); - expect(planSemanticReviewChunks).toHaveBeenCalledTimes(1); - expect(analyzeFile).toHaveBeenCalledTimes(2); - expect(results).toHaveLength(2); - expect(results.filter((result) => - Boolean(result.report?.auxiliaryUsage?.['semantic-chunk-planner']) - )).toHaveLength(1); - - prepareFiles.mockRestore(); - planSemanticReviewChunks.mockRestore(); - analyzeFile.mockRestore(); - }); - - it('keys shared semantic planning by primary model and not auxiliary model', async () => { - const fakeChunk = makeReviewChunk('a.ts', 1, 1); - const context = { - eventType: 'pull_request', - repository: { owner: 'o', name: 'n', fullName: 'o/n', defaultBranch: 'main' }, - repoPath: '/tmp', - pullRequest: { - number: 1, - title: 't', - body: '', - author: 'octocat', - baseBranch: 'main', - headBranch: 'feature', - headSha: 'abc', - baseSha: 'def', - files: [{ - filename: 'a.ts', - status: 'modified', - additions: 1, - deletions: 1, - patch: '@@ -1,1 +1,1 @@\n-old\n+new', - }], - }, - } as unknown as SkillTaskOptions['context']; - const baseRunnerOptions: SkillTaskOptions['runnerOptions'] = { - model: 'primary-a', - chunking: { semantic: { enabled: true } }, - postProcessFindings: false, - }; - const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], - skippedFiles: [], - }); - const planSemanticReviewChunks = vi.spyOn(semanticPlanner, 'planSemanticReviewChunks').mockResolvedValue({ - groups: [{ displayName: 'a.ts', filenames: ['a.ts'], chunks: [fakeChunk] }], - usage: { inputTokens: 10, outputTokens: 5, costUSD: 0.001 }, - }); - const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ - filename: 'a.ts', - findings: [], - usage: { inputTokens: 1, outputTokens: 1, costUSD: 0.001 }, - failedHunks: 0, - failedExtractions: 0, - hunkFailures: [], - } satisfies FileAnalysisResult); - - await runSkillTasks([ - { - name: 'skill-a', - resolveSkill: async () => ({ name: 'skill-a', description: '', prompt: '' }), - context, - runnerOptions: { ...baseRunnerOptions, auxiliaryModel: 'aux-a' }, - }, - { - name: 'skill-b', - resolveSkill: async () => ({ name: 'skill-b', description: '', prompt: '' }), - context, - runnerOptions: { ...baseRunnerOptions, auxiliaryModel: 'aux-b', auxiliaryMaxRetries: 99 }, - }, - { - name: 'skill-c', - resolveSkill: async () => ({ name: 'skill-c', description: '', prompt: '' }), - context, - runnerOptions: { ...baseRunnerOptions, model: 'primary-b', auxiliaryModel: 'aux-a' }, - }, - ], { - mode: logMode(), - verbosity: Verbosity.Quiet, - concurrency: 3, - }); - - expect(prepareFiles).toHaveBeenCalledTimes(2); - expect(planSemanticReviewChunks).toHaveBeenCalledTimes(2); - - prepareFiles.mockRestore(); - planSemanticReviewChunks.mockRestore(); - analyzeFile.mockRestore(); - }); - it('fail-fast aborts after final findings are post-processed', async () => { const finalFinding = makeFinding(); const controller = new AbortController(); - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const prepareFiles = vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); const analyzeFile = vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1007,13 +821,15 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('synthesizes a report with error.code=auth_failed when every hunk fails with auth errors', async () => { - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const hunkFailures: HunkFailure[] = [ { type: 'analysis', filename: 'a.ts', lineRange: '1-10', code: 'auth_failed', message: 'bad key' }, ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); @@ -1061,10 +877,10 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('preserves auth_failed when all analysis failures are auth alongside extraction failures', async () => { - const fakeChunks = [ - makeReviewChunk('a.ts', 1, 10), - makeReviewChunk('a.ts', 20, 24), - ]; + const fakeHunks = [ + { hunk: { newStart: 1, newCount: 10 } }, + { hunk: { newStart: 20, newCount: 5 } }, + ] as unknown as HunkWithContext[]; const hunkFailures: HunkFailure[] = [ { type: 'analysis', filename: 'a.ts', lineRange: '1-10', code: 'auth_failed', message: 'bad key' }, { @@ -1077,7 +893,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: fakeChunks }], + files: [{ filename: 'a.ts', hunks: fakeHunks }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1110,7 +926,9 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('synthesizes provider_unavailable when every hunk fails with provider errors', async () => { - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const hunkFailures: HunkFailure[] = [ { type: 'analysis', @@ -1122,7 +940,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1154,7 +972,9 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('preserves invalid_model_selector when every hunk fails from Pi model validation', async () => { - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const hunkFailures: HunkFailure[] = [ { type: 'analysis', @@ -1166,7 +986,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1198,14 +1018,16 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('ignores unrelated circuit state when this skill completed without failures', async () => { - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const circuitBreaker = new ProviderFailureCircuitBreaker({ maxConsecutiveProviderFailures: 1, }); circuitBreaker.recordFailure('provider_unavailable', 'temporary outage'); vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ @@ -1248,7 +1070,9 @@ describe('runSkillTask all-hunks-fail synthesis', () => { // If every hunk fails extraction (SDK call succeeds, parsing fails) // then failedHunks is 0 — naive `failedHunks === totalHunks` checks // would silently produce a "0 findings" run. Detection must sum both. - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const hunkFailures: HunkFailure[] = [ { type: 'extraction', @@ -1260,7 +1084,7 @@ describe('runSkillTask all-hunks-fail synthesis', () => { ]; vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); @@ -1295,14 +1119,16 @@ describe('runSkillTask all-hunks-fail synthesis', () => { }); it('does not convert user interruption into all_hunks_failed', async () => { - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const hunkFailures: HunkFailure[] = [ { type: 'analysis', filename: 'a.ts', lineRange: '1-10', code: 'aborted', message: 'Analysis aborted' }, ]; const controller = new AbortController(); vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); @@ -1411,11 +1237,13 @@ describe('runSkillTask model lanes', () => { }); it('passes model lanes to shared finding post-processing', async () => { - const fakeChunk = makeReviewChunk(); + const fakeHunk = { + hunk: { newStart: 1, newCount: 10 }, + } as unknown as HunkWithContext; const finding = makeFinding(); vi.spyOn(sdkRunner, 'prepareFiles').mockReturnValue({ - files: [{ filename: 'a.ts', chunks: [fakeChunk] }], + files: [{ filename: 'a.ts', hunks: [fakeHunk] }], skippedFiles: [], }); vi.spyOn(sdkRunner, 'analyzeFile').mockResolvedValue({ diff --git a/packages/warden/src/cli/output/tasks.ts b/packages/warden/src/cli/output/tasks.ts index ef6337ea..095d441a 100644 --- a/packages/warden/src/cli/output/tasks.ts +++ b/packages/warden/src/cli/output/tasks.ts @@ -5,7 +5,7 @@ * Reporter spec: specs/reporters.md */ -import type { SkillReport, SeverityThreshold, ConfidenceThreshold, Finding, UsageStats, EventContext, HunkFailure, AuxiliaryUsageMap, ErrorCode, HunkTrace, SkippedFile } from '../../types/index.js'; +import type { SkillReport, SeverityThreshold, ConfidenceThreshold, Finding, UsageStats, EventContext, HunkFailure, AuxiliaryUsageMap, ErrorCode, HunkTrace } from '../../types/index.js'; import type { SkillDefinition } from '../../config/schema.js'; import { Sentry, emitSkillMetrics, logger } from '../../sentry.js'; import { SkillRunnerError, WardenAuthenticationError, classifyError } from '../../sdk/errors.js'; @@ -19,12 +19,11 @@ import { type AuxiliaryUsageEntry, type SkillRunnerOptions, type FileAnalysisCallbacks, - type ReviewChunkGroup, + type PreparedFile, type PRPromptContext, type ChunkAnalysisResult, type FindingProcessingEvent, } from '../../sdk/runner.js'; -import { planSemanticReviewChunks } from '../../semantic/index.js'; import { ProviderFailureCircuitBreaker } from '../../sdk/circuit-breaker.js'; import { buildFileReports } from '../../sdk/report-files.js'; import chalk from 'chalk'; @@ -197,26 +196,6 @@ export interface SkillTaskOptions { context: EventContext; /** Options passed to the runner */ runnerOptions?: SkillRunnerOptions; - /** Prepared review chunks shared across skill tasks for the same changeset. */ - preparedFiles?: PreparedTaskFiles; -} - -export interface PreparedTaskFiles { - groups: ReviewChunkGroup[]; - skippedFiles: SkippedFile[]; - semanticUsage?: UsageStats; -} - -function fileReportInputsFromGroup(args: { - group: ReviewChunkGroup; - durationMs?: number; - usage?: UsageStats; -}) { - return args.group.filenames.map((filename, index) => ({ - filename, - durationMs: index === 0 ? args.durationMs : undefined, - usage: index === 0 ? args.usage : undefined, - })); } /** @@ -241,7 +220,7 @@ export interface SkillProgressCallbacks { onSkillStart: (skill: SkillState) => void; onSkillUpdate: (name: string, updates: Partial) => void; onFileUpdate: (skillName: string, filename: string, updates: Partial) => void; - /** Called when a chunk analysis starts (one SDK invocation per review chunk) */ + /** Called when a hunk analysis starts (one SDK invocation per hunk) */ onHunkStart?: (skillName: string, filename: string, hunkNum: number, totalHunks: number, lineRange: string) => void; onChunkComplete?: (skillName: string, chunk: ChunkAnalysisResult) => void; onSkillComplete: (name: string, report: SkillReport) => void; @@ -255,7 +234,7 @@ export interface SkillProgressCallbacks { onExtractionResult?: (skillName: string, filename: string, lineRange: string, findingsCount: number, method: 'regex' | 'llm' | 'none') => void; /** Called when findings are dropped, revised, merged, or stripped after analysis */ onFindingProcessing?: (skillName: string, event: FindingProcessingEvent) => void; - /** Called when chunk analysis fails (SDK error, API error, abort) */ + /** Called when hunk analysis fails (SDK error, API error, abort) */ onHunkFailed?: (skillName: string, filename: string, lineRange: string, error: string) => void; /** Called when findings extraction fails (both regex and LLM fallback failed) */ onExtractionFailure?: (skillName: string, filename: string, lineRange: string, error: string, preview: string) => void; @@ -263,37 +242,6 @@ export interface SkillProgressCallbacks { onRetry?: (skillName: string, filename: string, lineRange: string, attempt: number, maxRetries: number, error: string, delayMs: number) => void; } -async function prepareTaskFiles( - context: EventContext, - runnerOptions: SkillRunnerOptions, -): Promise { - const { files: initialPreparedFiles, skippedFiles } = prepareFiles(context, { - contextLines: runnerOptions.contextLines, - ignore: runnerOptions.ignore, - scan: runnerOptions.scan, - chunking: runnerOptions.chunking, - }); - const semanticPlan = await planSemanticReviewChunks(initialPreparedFiles, context, { - enabled: runnerOptions.chunking?.semantic?.enabled, - apiKey: runnerOptions.apiKey, - runtime: runnerOptions.runtime, - model: runnerOptions.model, - maxChunks: runnerOptions.chunking?.semantic?.maxChunks, - maxChunkChars: runnerOptions.chunking?.semantic?.maxChunkChars, - maxHunksPerChunk: runnerOptions.chunking?.semantic?.maxHunksPerChunk, - maxChangedRangesPerChunk: runnerOptions.chunking?.semantic?.maxChangedRangesPerChunk, - maxEmbeddedDiffChars: runnerOptions.chunking?.semantic?.maxEmbeddedDiffChars, - maxEmbeddedDiffChunks: runnerOptions.chunking?.semantic?.maxEmbeddedDiffChunks, - maxEmbeddedDiffRanges: runnerOptions.chunking?.semantic?.maxEmbeddedDiffRanges, - }); - - return { - groups: semanticPlan.groups, - skippedFiles, - semanticUsage: semanticPlan.usage, - }; -} - /** * Run a single skill task. */ @@ -337,12 +285,15 @@ export async function runSkillTask( throw new SkillRunnerError(message, { cause: err, code: 'skill_resolution_failed' }); } - // Prepare files into review chunks before optional semantic planning. - const prepared = options.preparedFiles ?? await prepareTaskFiles(context, runnerOptions); - const chunkGroups = prepared.groups; - const skippedFiles = prepared.skippedFiles; + // Prepare files (parse patches into hunks) + const { files: preparedFiles, skippedFiles } = prepareFiles(context, { + contextLines: runnerOptions.contextLines, + ignore: runnerOptions.ignore, + scan: runnerOptions.scan, + chunking: runnerOptions.chunking, + }); - if (chunkGroups.length === 0) { + if (preparedFiles.length === 0) { // No files to analyze - skip const skippedReport: SkillReport = { skill: skill.name, @@ -367,11 +318,11 @@ export async function runSkillTask( } // Initialize file states - const fileStates: FileState[] = chunkGroups.map((group) => ({ - filename: group.displayName, + const fileStates: FileState[] = preparedFiles.map((file) => ({ + filename: file.filename, status: 'pending', currentHunk: 0, - totalHunks: group.chunks.length, + totalHunks: file.hunks.length, findings: [], })); @@ -387,7 +338,7 @@ export async function runSkillTask( // Build PR context for inclusion in prompts (if available) // For non-PR contexts (CLI file/diff mode), skip the "Other Files" list to avoid - // bloating every chunk prompt with thousands of filenames. + // bloating every hunk prompt with thousands of filenames. const isPullRequest = context.pullRequest ? context.pullRequest.number !== 0 : false; const prContext: PRPromptContext | undefined = context.pullRequest ? { @@ -399,8 +350,8 @@ export async function runSkillTask( : undefined; // Process files with concurrency - const processFile = async (group: ReviewChunkGroup, index: number): Promise => { - const filename = group.displayName; + const processFile = async (prepared: PreparedFile, index: number): Promise => { + const filename = prepared.filename; const fileStartTime = Date.now(); // Update file state to running (local + callback) @@ -486,7 +437,7 @@ export async function runSkillTask( const result = await analyzeFile( skill, - group, + prepared, context.repoPath, runnerOptions, fileCallbacks, @@ -523,7 +474,7 @@ export async function runSkillTask( const processSkippedFile = (index: number): FileProcessResult => { const localState = fileStates[index]; if (localState) localState.status = 'skipped'; - const filename = chunkGroups[index]?.displayName ?? 'unknown'; + const filename = preparedFiles[index]?.filename ?? 'unknown'; callbacks.onFileUpdate(name, filename, { status: 'skipped' }); return { findings: [], durationMs: 0, failedHunks: 0, failedExtractions: 0, hunkFailures: [] }; }; @@ -534,8 +485,8 @@ export async function runSkillTask( // The effective concurrency for batch delay: when a semaphore gates work, // use its permit count (the actual concurrency limit) rather than fileConcurrency. const effectiveConcurrency = semaphore ? semaphore.initialPermits : fileConcurrency; - const allResults = await runPool(chunkGroups, fileConcurrency, - async (group, index) => { + const allResults = await runPool(preparedFiles, fileConcurrency, + async (file, index) => { if (semaphore) await semaphore.acquire(); try { // Check abort after acquiring the semaphore -- the file may have @@ -545,7 +496,7 @@ export async function runSkillTask( if (index >= effectiveConcurrency && batchDelayMs > 0) { await new Promise((resolve) => setTimeout(resolve, batchDelayMs)); } - return await processFile(group, index); + return await processFile(file, index); } finally { if (semaphore) semaphore.release(); } @@ -565,23 +516,15 @@ export async function runSkillTask( const allFindings = allResults.flatMap((r) => r.findings); const allUsage = allResults.map((r) => r.usage).filter((u): u is UsageStats => u !== undefined); const allAuxEntries = allResults.flatMap((r) => r.auxiliaryUsage ?? []); - if (prepared.semanticUsage) { - allAuxEntries.push({ - agent: 'semantic-chunk-planner', - usage: prepared.semanticUsage, - model: runnerOptions.auxiliaryModel ?? runnerOptions.model, - runtime: runnerOptions.runtime, - }); - } const allTraces = allResults.flatMap((r) => r.traces ?? []); const totalFailedHunks = allResults.reduce((sum, r) => sum + r.failedHunks, 0); const totalFailedExtractions = allResults.reduce((sum, r) => sum + r.failedExtractions, 0); const allHunkFailures: HunkFailure[] = allResults.flatMap((r) => r.hunkFailures); - const totalHunks = chunkGroups.reduce((sum, group) => sum + group.chunks.length, 0); - // Each chunk contributes to at most one of failedHunks / failedExtractions + const totalHunks = preparedFiles.reduce((sum, f) => sum + f.hunks.length, 0); + // Each hunk contributes to at most one of failedHunks / failedExtractions // (mutually exclusive in analyzeFile), so summing them gives the total - // failed chunk count. Counting only analysis failures would miss the - // scenario where every chunk's SDK call succeeded but every extraction + // failed-hunk count. Counting only analysis failures would miss the + // scenario where every hunk's SDK call succeeded but every extraction // failed — a silent zero-findings run otherwise. const totalAttemptFailures = totalFailedHunks + totalFailedExtractions; @@ -613,21 +556,19 @@ export async function runSkillTask( durationMs: duration, model: runnerOptions?.model, runtime, - // Preserve per-group metadata (timing, partial usage, attempted + // Preserve per-file metadata (timing, partial usage, attempted // filenames) on failure runs too — `warden runs` and JSONL // consumers iterate this array to count attempted files. Without // it, a failed run shows totalFiles: 0. - files: buildFileReports( - chunkGroups.flatMap((group, i) => { - const r = allResults[i]; - return fileReportInputsFromGroup({ - group, - durationMs: r?.durationMs, - usage: r?.usage, - }); - }), - allFindings, - ), + files: preparedFiles.map((file, i) => { + const r = allResults[i]; + return { + filename: file.filename, + findings: r?.findings.length ?? 0, + durationMs: r?.durationMs, + usage: r?.usage, + }; + }), failedHunks: totalFailedHunks, hunkFailures: allHunkFailures, error: { @@ -689,13 +630,13 @@ export async function runSkillTask( model: runnerOptions?.model, runtime, files: buildFileReports( - chunkGroups.flatMap((group, i) => { + preparedFiles.map((file, i) => { const r = allResults[i]; - return fileReportInputsFromGroup({ - group, + return { + filename: file.filename, durationMs: r?.durationMs, usage: r?.usage, - }); + }; }), finalFindings, ), @@ -924,7 +865,7 @@ export function createDefaultCallbacks( debugLog(mode, formatFindingProcessingEvent(event)); } : undefined, - // Verbose mode: show per-chunk analysis failures (spec: event #16 hunk_failed) + // Verbose mode: show per-hunk analysis failures (spec: event #16 hunk_failed) onHunkFailed: verbosity >= Verbosity.Verbose ? (_skillName, filename, lineRange, error) => { const location = `${filename}:${lineRange}`; @@ -935,7 +876,7 @@ export function createDefaultCallbacks( } } : undefined, - // Verbose mode: show per-chunk extraction failures (spec: event #17 extraction_failure) + // Verbose mode: show per-hunk extraction failures (spec: event #17 extraction_failure) onExtractionFailure: verbosity >= Verbosity.Verbose ? (_skillName, filename, lineRange, error, preview) => { const location = `${filename}:${lineRange}`; @@ -1030,84 +971,6 @@ export function composeTasksWithFailFast( })); } -function semanticContextKey(context: EventContext): object { - const pullRequest = context.pullRequest - ? { - number: context.pullRequest.number, - title: context.pullRequest.title, - body: context.pullRequest.body, - headSha: context.pullRequest.headSha, - baseSha: context.pullRequest.baseSha, - files: [...context.pullRequest.files] - .map((file) => ({ - filename: file.filename, - status: file.status, - additions: file.additions, - deletions: file.deletions, - patch: file.patch, - chunks: file.chunks, - })) - .sort((a, b) => a.filename.localeCompare(b.filename)), - } - : undefined; - - return { - eventType: context.eventType, - action: context.action, - label: context.label, - repoPath: context.repoPath, - diffContextSource: context.diffContextSource, - repository: context.repository, - pullRequest, - }; -} - -function semanticPlanKey(task: SkillTaskOptions): string | undefined { - if (!task.runnerOptions?.chunking?.semantic?.enabled) return undefined; - return JSON.stringify({ - context: semanticContextKey(task.context), - contextLines: task.runnerOptions.contextLines, - ignore: task.runnerOptions.ignore, - scan: task.runnerOptions.scan, - chunking: task.runnerOptions.chunking, - apiKey: Boolean(task.runnerOptions.apiKey), - runtime: task.runnerOptions.runtime, - model: task.runnerOptions.model, - }); -} - -/** Share semantic chunk planning across skill tasks for the same changeset. */ -export async function prepareSemanticPlansForTasks(tasks: SkillTaskOptions[]): Promise { - const plansByKey = new Map>(); - const chargedPlans = new Set(); - - return Promise.all(tasks.map(async (task) => { - const key = semanticPlanKey(task); - if (!key || task.preparedFiles) { - return task; - } - - let plan = plansByKey.get(key); - if (!plan) { - plan = prepareTaskFiles(task.context, task.runnerOptions ?? {}); - plansByKey.set(key, plan); - } - - const chargeSemanticUsage = !chargedPlans.has(key); - chargedPlans.add(key); - - const preparedFiles = await plan; - - return { - ...task, - preparedFiles: { - ...preparedFiles, - semanticUsage: chargeSemanticUsage ? preparedFiles.semanticUsage : undefined, - }, - }; - })); -} - /** * Launch all skill tasks in parallel using a shared semaphore for concurrency. */ @@ -1172,9 +1035,8 @@ export async function runSkillTasks( const circuitAbortController = new AbortController(); const circuitBreaker = new ProviderFailureCircuitBreaker({ abortController: circuitAbortController }); - const plannedTasks = await prepareSemanticPlansForTasks(tasks); const composedTasks = composeTasksWithFailFast( - plannedTasks, + tasks, failFastController, circuitBreaker, circuitAbortController, diff --git a/packages/warden/src/config/loader.test.ts b/packages/warden/src/config/loader.test.ts index 1e1f7b1d..b6a443f6 100644 --- a/packages/warden/src/config/loader.test.ts +++ b/packages/warden/src/config/loader.test.ts @@ -595,7 +595,6 @@ describe('mergeWardenConfigs', () => { chunking: { filePatterns: [{ pattern: '**/*.lock', mode: 'skip' }], coalesce: { enabled: true, maxGapLines: 20, maxChunkSize: 4000 }, - semantic: { enabled: true, maxChunks: 20, maxChunkChars: 30000, maxHunksPerChunk: 50 }, maxContextFiles: 25, }, }, @@ -615,15 +614,6 @@ describe('mergeWardenConfigs', () => { chunking: { filePatterns: [{ pattern: '**/*.snap', mode: 'skip' }], coalesce: { maxGapLines: 5, maxChunkSize: 2000, enabled: true }, - semantic: { - enabled: true, - maxChunks: 10, - maxChunkChars: 12000, - maxHunksPerChunk: 25, - maxEmbeddedDiffChars: 4000, - maxEmbeddedDiffChunks: 8, - maxEmbeddedDiffRanges: 8, - }, maxContextFiles: 10, }, }, @@ -653,58 +643,12 @@ describe('mergeWardenConfigs', () => { { pattern: '**/*.snap', mode: 'skip' }, ], coalesce: { enabled: true, maxGapLines: 5, maxChunkSize: 2000 }, - semantic: { - enabled: true, - maxChunks: 10, - maxChunkChars: 12000, - maxHunksPerChunk: 25, - maxEmbeddedDiffChars: 4000, - maxEmbeddedDiffChunks: 8, - maxEmbeddedDiffRanges: 8, - }, maxContextFiles: 10, }, }); expect(merged.skills.map((skill) => skill.name)).toEqual(['org-skill', 'repo-skill']); }); - it('field-merges semantic chunking defaults across config layers', () => { - const baseConfig = WardenConfigSchema.parse({ - version: 1, - defaults: { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 50000, - maxEmbeddedDiffChars: 8000, - }, - }, - }, - skills: [], - }); - const repoConfig = WardenConfigSchema.parse({ - version: 1, - defaults: { - chunking: { - semantic: { - maxChunks: 8, - maxEmbeddedDiffRanges: 6, - }, - }, - }, - skills: [], - }); - - expect(mergeWardenConfigs(baseConfig, repoConfig).defaults?.chunking?.semantic).toEqual({ - enabled: true, - maxChunks: 8, - maxChunkChars: 50000, - maxEmbeddedDiffChars: 8000, - maxEmbeddedDiffRanges: 6, - }); - }); - it('deep-merges nested default model lanes across layers', () => { const baseConfig: WardenConfig = { version: 1, diff --git a/packages/warden/src/config/loader.ts b/packages/warden/src/config/loader.ts index b21a03b7..20e43d5a 100644 --- a/packages/warden/src/config/loader.ts +++ b/packages/warden/src/config/loader.ts @@ -13,7 +13,6 @@ import { type Defaults, type ChunkingConfig, type CoalesceConfig, - type SemanticChunkingConfig, type IgnoreConfig, type ScanConfig, type RunnerConfig, @@ -98,15 +97,6 @@ function mergeCoalesceConfig( return { ...base, ...overlay }; } -function mergeSemanticChunkingConfig( - base?: SemanticChunkingConfig, - overlay?: SemanticChunkingConfig -): SemanticChunkingConfig | undefined { - if (!base) return overlay; - if (!overlay) return base; - return { ...base, ...overlay }; -} - function mergeChunkingConfig( base?: ChunkingConfig, overlay?: ChunkingConfig @@ -118,7 +108,6 @@ function mergeChunkingConfig( ...overlay, filePatterns: mergeArray(base.filePatterns, overlay.filePatterns), coalesce: mergeCoalesceConfig(base.coalesce, overlay.coalesce), - semantic: mergeSemanticChunkingConfig(base.semantic, overlay.semantic), }; } diff --git a/packages/warden/src/config/schema.ts b/packages/warden/src/config/schema.ts index ffdf179e..245c30df 100644 --- a/packages/warden/src/config/schema.ts +++ b/packages/warden/src/config/schema.ts @@ -144,7 +144,7 @@ export const SkillConfigSchema = z.object({ failCheck: z.boolean().optional(), /** Model to use for this skill (e.g., 'openai/gpt-5.5'). Uses SDK default if not specified. */ model: z.string().optional(), - /** Maximum agentic turns (API round-trips) per chunk analysis. Overrides defaults.maxTurns. */ + /** Maximum agentic turns (API round-trips) per hunk analysis. Overrides defaults.maxTurns. */ maxTurns: z.number().int().positive().optional(), /** Minimum confidence level for findings. Findings below this are filtered from output. */ minConfidence: ConfidenceThresholdSchema.optional(), @@ -180,35 +180,13 @@ export const CoalesceConfigSchema = z.object({ }); export type CoalesceConfig = z.infer; -export const SemanticChunkingConfigSchema = z.object({ - /** Enable semantic review chunk materialization (default: false) */ - enabled: z.boolean().optional(), - /** Maximum number of scanner review chunks to emit after semantic grouping */ - maxChunks: z.number().int().positive().optional(), - /** Target max size per scanner review chunk in characters */ - maxChunkChars: z.number().int().positive().optional(), - /** Maximum atomic hunks to place in one scanner review chunk inside a semantic change */ - maxHunksPerChunk: z.number().int().positive().optional(), - /** Maximum changed line ranges to place in one scanner review chunk inside a semantic change */ - maxChangedRangesPerChunk: z.number().int().positive().optional(), - /** Maximum total hunk content characters to embed in the semantic planner prompt */ - maxEmbeddedDiffChars: z.number().int().nonnegative().optional(), - /** Maximum prepared chunks allowed before the semantic planner stops embedding full hunk content */ - maxEmbeddedDiffChunks: z.number().int().nonnegative().optional(), - /** Maximum changed ranges allowed before the semantic planner stops embedding full hunk content */ - maxEmbeddedDiffRanges: z.number().int().nonnegative().optional(), -}); -export type SemanticChunkingConfig = z.infer; - // Chunking configuration for controlling how files are processed export const ChunkingConfigSchema = z.object({ /** Patterns to control file processing mode */ filePatterns: z.array(FilePatternSchema).optional(), /** Coalescing options for merging nearby hunks */ coalesce: CoalesceConfigSchema.optional(), - /** Semantic review chunk options */ - semantic: SemanticChunkingConfigSchema.optional(), - /** Max number of "other files" to list in chunk prompts for PR context. 0 disables the section entirely. Default: 50 */ + /** Max number of "other files" to list in hunk prompts for PR context. 0 disables the section entirely. Default: 50 */ maxContextFiles: z.number().int().nonnegative().default(50), }); export type ChunkingConfig = z.infer; @@ -253,7 +231,7 @@ export const DefaultsSchema = z.object({ failCheck: z.boolean().optional(), /** Default model for all skills (e.g., 'openai/gpt-5.5') */ model: z.string().optional(), - /** Maximum agentic turns (API round-trips) per chunk analysis. Default: 50 */ + /** Maximum agentic turns (API round-trips) per hunk analysis. Default: 50 */ maxTurns: z.number().int().positive().optional(), /** Runtime backend for all model-backed execution. Default: pi */ runtime: RuntimeNameSchema.optional(), diff --git a/packages/warden/src/diff/index.ts b/packages/warden/src/diff/index.ts index 019df42d..3dd9078e 100644 --- a/packages/warden/src/diff/index.ts +++ b/packages/warden/src/diff/index.ts @@ -1,6 +1,5 @@ export * from './parser.js'; export * from './apply.js'; export * from './context.js'; -export * from './review-chunk.js'; export * from './classify.js'; export * from './coalesce.js'; diff --git a/packages/warden/src/diff/review-chunk.test.ts b/packages/warden/src/diff/review-chunk.test.ts deleted file mode 100644 index 5037cc46..00000000 --- a/packages/warden/src/diff/review-chunk.test.ts +++ /dev/null @@ -1,84 +0,0 @@ -import { describe, expect, it } from 'vitest'; -import { coalesceHunks } from './coalesce.js'; -import { parsePatch } from './parser.js'; -import { reviewChunkFromHunk } from './review-chunk.js'; -import type { HunkWithContext } from './context.js'; - -function makeHunkContext(overrides: Partial = {}): HunkWithContext { - return { - filename: 'src/example.ts', - hunk: { - oldStart: 10, - oldCount: 1, - newStart: 10, - newCount: 0, - header: '@@ -10,1 +10,0 @@', - lines: ['-const removed = true;'], - content: '-const removed = true;', - }, - contextBefore: [], - contextAfter: [], - contextStartLine: 10, - language: 'typescript', - ...overrides, - }; -} - -describe('reviewChunkFromHunk', () => { - it('uses hunk coordinates and an anchor range for deletion-only chunks', () => { - const chunk = reviewChunkFromHunk(makeHunkContext()); - - expect(chunk.changedLineMap).toEqual([{ path: 'src/example.ts', start: 10, end: 10 }]); - expect(chunk.id).toBe('src/example.ts:old10-10:new10-9'); - }); - - it('resets changed-line ranges at embedded hunk headers from coalesced content', () => { - const parsedHunks = parsePatch([ - '@@ -10,1 +10,1 @@', - '-const first = oldValue;', - '+const first = newValue;', - '@@ -100,1 +100,1 @@', - '-const second = oldValue;', - '+const second = newValue;', - ].join('\n')); - const [hunk] = coalesceHunks(parsedHunks, { maxGapLines: 100 }); - expect(hunk?.lines).toEqual([ - '-const first = oldValue;', - '+const first = newValue;', - '-const second = oldValue;', - '+const second = newValue;', - ]); - - const chunk = reviewChunkFromHunk(makeHunkContext({ - hunk, - })); - - expect(chunk.changedLineMap).toEqual([ - { path: 'src/example.ts', start: 10, end: 10 }, - { path: 'src/example.ts', start: 100, end: 100 }, - ]); - expect(chunk.files[0].sourceLines).toEqual([ - { line: 10, content: 'const first = newValue;' }, - { line: 100, content: 'const second = newValue;' }, - ]); - }); - - it('anchors deletion-only embedded hunk sections independently', () => { - const parsedHunks = parsePatch([ - '@@ -10,1 +10,0 @@', - '-const first = removed;', - '@@ -100,1 +100,0 @@', - '-const second = removed;', - ].join('\n')); - const [hunk] = coalesceHunks(parsedHunks, { maxGapLines: 100 }); - - const chunk = reviewChunkFromHunk(makeHunkContext({ - hunk, - })); - - expect(chunk.changedLineMap).toEqual([ - { path: 'src/example.ts', start: 10, end: 10 }, - { path: 'src/example.ts', start: 100, end: 100 }, - ]); - }); -}); diff --git a/packages/warden/src/diff/review-chunk.ts b/packages/warden/src/diff/review-chunk.ts deleted file mode 100644 index 5ac2a85a..00000000 --- a/packages/warden/src/diff/review-chunk.ts +++ /dev/null @@ -1,160 +0,0 @@ -import type { SourceSnippetLine } from '../types/index.js'; -import type { HunkWithContext } from './context.js'; -import { formatHunkForAnalysis } from './context.js'; - -export type ReviewChunkContentMode = 'whole-file' | 'stitched-file' | 'raw-hunks'; - -export interface ChangedLineRange { - path: string; - start: number; - end: number; -} - -export interface ReviewChunkFile { - path: string; - language: string; - changedRanges: ChangedLineRange[]; - content: string; - contentMode: ReviewChunkContentMode; - sourceLines: SourceSnippetLine[]; -} - -export type ReviewChunkFiles = [ReviewChunkFile, ...ReviewChunkFile[]]; - -export interface ReviewChunk { - id: string; - title: string; - summary?: string; - files: ReviewChunkFiles; - changedLineMap: ChangedLineRange[]; -} - -function parseEmbeddedHunkHeader(line: string): { newStart: number; newCount: number } | undefined { - const match = line.match(/^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/); - if (!match?.[1]) return undefined; - return { - newStart: parseInt(match[1], 10), - newCount: parseInt(match[2] ?? '1', 10), - }; -} - -function linesWithHunkBoundaries(hunkCtx: HunkWithContext): string[] { - const contentLines = hunkCtx.hunk.content.split('\n'); - const hunkHeaderCount = contentLines.filter((line) => parseEmbeddedHunkHeader(line)).length; - return hunkHeaderCount > 1 ? contentLines : hunkCtx.hunk.lines; -} - -function changedRangesFromHunk(path: string, hunkCtx: HunkWithContext): ChangedLineRange[] { - const ranges: ChangedLineRange[] = []; - let newLine = hunkCtx.hunk.newStart; - let currentStart: number | undefined; - let currentEnd: number | undefined; - let segmentStart = hunkCtx.hunk.newStart; - let segmentEnd = Math.max(segmentStart, segmentStart + hunkCtx.hunk.newCount - 1); - let segmentHasAddedLine = false; - let segmentHasDeletedLine = false; - - function flush(): void { - if (currentStart === undefined || currentEnd === undefined) return; - ranges.push({ path, start: currentStart, end: currentEnd }); - currentStart = undefined; - currentEnd = undefined; - } - - function flushDeletionOnlySegment(): void { - flush(); - if (!segmentHasAddedLine && segmentHasDeletedLine) { - ranges.push({ path, start: segmentStart, end: segmentEnd }); - } - segmentHasAddedLine = false; - segmentHasDeletedLine = false; - } - - for (const diffLine of linesWithHunkBoundaries(hunkCtx)) { - const embeddedHeader = parseEmbeddedHunkHeader(diffLine); - if (embeddedHeader) { - flushDeletionOnlySegment(); - newLine = embeddedHeader.newStart; - segmentStart = embeddedHeader.newStart; - segmentEnd = Math.max(segmentStart, segmentStart + embeddedHeader.newCount - 1); - continue; - } - - if (diffLine.startsWith('+')) { - segmentHasAddedLine = true; - if (currentStart === undefined) { - currentStart = newLine; - } - currentEnd = newLine; - newLine += 1; - continue; - } - - flush(); - if (diffLine.startsWith('-')) { - segmentHasDeletedLine = true; - } else if (diffLine.startsWith(' ')) { - newLine += 1; - } - } - - flushDeletionOnlySegment(); - return ranges; -} - -function hunkSourceLines(hunkCtx: HunkWithContext): SourceSnippetLine[] { - const lines: SourceSnippetLine[] = []; - for (const [index, content] of hunkCtx.contextBefore.entries()) { - lines.push({ line: hunkCtx.contextStartLine + index, content }); - } - - let newLine = hunkCtx.hunk.newStart; - for (const diffLine of linesWithHunkBoundaries(hunkCtx)) { - const embeddedHeader = parseEmbeddedHunkHeader(diffLine); - if (embeddedHeader) { - newLine = embeddedHeader.newStart; - continue; - } - - if (diffLine.startsWith('-')) continue; - if (!diffLine.startsWith('+') && !diffLine.startsWith(' ')) continue; - const content = diffLine.slice(1); - lines.push({ line: newLine, content }); - newLine += 1; - } - - const afterStart = hunkCtx.hunk.newStart + hunkCtx.hunk.newCount; - for (const [index, content] of hunkCtx.contextAfter.entries()) { - lines.push({ line: afterStart + index, content }); - } - - return lines; -} - -/** Convert an expanded git hunk into Warden's scanner-facing review chunk. */ -export function reviewChunkFromHunk(hunkCtx: HunkWithContext): ReviewChunk { - const changedRanges = changedRangesFromHunk(hunkCtx.filename, hunkCtx); - const hasAddedLines = hunkCtx.hunk.lines.some((line) => line.startsWith('+')); - const lineRange = changedRanges - .map((range) => range.start === range.end ? `${range.start}` : `${range.start}-${range.end}`) - .join(','); - const hunkCoordinate = [ - `old${hunkCtx.hunk.oldStart}-${hunkCtx.hunk.oldStart + hunkCtx.hunk.oldCount - 1}`, - `new${hunkCtx.hunk.newStart}-${hunkCtx.hunk.newStart + hunkCtx.hunk.newCount - 1}`, - ].join(':'); - const chunkKey = hasAddedLines ? lineRange : hunkCoordinate; - - return { - id: `${hunkCtx.filename}:${chunkKey}`, - title: `${hunkCtx.filename}:${chunkKey}`, - files: [{ - path: hunkCtx.filename, - language: hunkCtx.language, - changedRanges, - content: formatHunkForAnalysis(hunkCtx), - contentMode: 'raw-hunks', - sourceLines: hunkSourceLines(hunkCtx), - }], - changedLineMap: changedRanges, - }; -} diff --git a/packages/warden/src/sdk/analyze.test.ts b/packages/warden/src/sdk/analyze.test.ts index 123f699b..086ed47e 100644 --- a/packages/warden/src/sdk/analyze.test.ts +++ b/packages/warden/src/sdk/analyze.test.ts @@ -1,7 +1,7 @@ import { describe, it, expect, vi, afterEach, beforeAll, afterAll } from 'vitest'; import { APIError } from '@anthropic-ai/sdk'; import type { SkillDefinition } from '../config/schema.js'; -import { reviewChunkFromHunk, type HunkWithContext } from '../diff/index.js'; +import type { HunkWithContext } from '../diff/index.js'; import type { EventContext, Finding, UsageStats } from '../types/index.js'; import { analyzeFile, buildSourceSnippet, filterOutOfRangeFindings, runSkill } from './analyze.js'; import type { PreparedFile } from './types.js'; @@ -30,14 +30,14 @@ afterAll(async () => { await Sentry.close(0); }); -function makeFinding(startLine: number, id = `f-${startLine}`, path = 'file.ts'): Finding { +function makeFinding(startLine: number, id = `f-${startLine}`): Finding { return { id, severity: 'medium', confidence: 'high', title: `Finding at line ${startLine}`, description: 'test', - location: { path, startLine }, + location: { path: 'file.ts', startLine }, }; } @@ -81,7 +81,7 @@ function makePreparedFile(hunkCount = 1): PreparedFile { }); return { filename: 'src/example.ts', - chunks: hunks.map(reviewChunkFromHunk), + hunks, }; } @@ -187,48 +187,6 @@ function makeContextWithOneHunk(): EventContext { }; } -function makeContextWithCrossFileHunks(): EventContext { - return { - eventType: 'pull_request', - action: 'opened', - repository: { owner: 'o', name: 'r', fullName: 'o/r', defaultBranch: 'main' }, - repoPath: '/tmp/repo', - pullRequest: { - number: 1, - title: 'Test PR', - body: '', - author: 'test', - baseBranch: 'main', - headBranch: 'feature', - headSha: 'head', - baseSha: 'base', - files: [{ - filename: 'src/example.ts', - status: 'modified', - additions: 1, - deletions: 1, - patch: [ - '@@ -10,1 +10,1 @@', - '-old10', - '+new10', - ].join('\n'), - chunks: 1, - }, { - filename: 'tests/example.test.ts', - status: 'modified', - additions: 1, - deletions: 1, - patch: [ - '@@ -50,1 +50,1 @@', - '-old50', - '+new50', - ].join('\n'), - chunks: 1, - }], - }, - }; -} - describe('filterOutOfRangeFindings', () => { const hunkRange = { start: 10, end: 20 }; @@ -279,32 +237,6 @@ describe('filterOutOfRangeFindings', () => { expect(dropped).toEqual([belowRange, aboveRange]); }); - it('filters findings against multi-file changed line maps', () => { - const firstFileFinding = makeFinding(15, 'first-file', 'src/one.ts'); - const secondFileFinding = makeFinding(42, 'second-file', 'src/two.ts'); - const wrongFileFinding = makeFinding(15, 'wrong-file', 'src/three.ts'); - const partialRangeFinding: Finding = { - ...makeFinding(42, 'partial-range', 'src/two.ts'), - location: { path: 'src/two.ts', startLine: 42, endLine: 45 }, - }; - const crossRangeFinding: Finding = { - ...makeFinding(42, 'cross-range', 'src/two.ts'), - location: { path: 'src/two.ts', startLine: 42, endLine: 50 }, - }; - - const { filtered, dropped } = filterOutOfRangeFindings( - [firstFileFinding, secondFileFinding, wrongFileFinding, partialRangeFinding, crossRangeFinding], - [ - { path: 'src/one.ts', start: 10, end: 20 }, - { path: 'src/two.ts', start: 40, end: 43 }, - { path: 'src/two.ts', start: 50, end: 50 }, - ], - ); - - expect(filtered).toEqual([firstFileFinding, secondFileFinding, crossRangeFinding]); - expect(dropped).toEqual([wrongFileFinding, partialRangeFinding]); - }); - it('returns empty arrays for empty input', () => { const { filtered, dropped } = filterOutOfRangeFindings([], hunkRange); expect(filtered).toEqual([]); @@ -336,10 +268,10 @@ describe('buildSourceSnippet', () => { language: 'typescript', }; - const snippet = buildSourceSnippet(makeFinding(11, 'f-11', 'src/example.ts'), reviewChunkFromHunk(hunk), 2); + const snippet = buildSourceSnippet(makeFinding(11), hunk, 2); expect(snippet).toEqual({ - path: 'src/example.ts', + path: 'file.ts', language: 'typescript', startLine: 9, endLine: 13, @@ -372,7 +304,7 @@ describe('buildSourceSnippet', () => { language: 'typescript', }; - const snippet = buildSourceSnippet(makeFinding(10, 'f-10', 'src/example.ts'), reviewChunkFromHunk(hunk), 1); + const snippet = buildSourceSnippet(makeFinding(10), hunk, 1); expect(snippet?.lines).toEqual([ { line: 10, content: 'newCall();', highlighted: true }, @@ -790,146 +722,6 @@ describe('runSkill', () => { expect(report.runtime).toBe('pi'); }); - it('sends planner-provided semantic chunks to the scanner', async () => { - const runSkillMock = vi.fn().mockResolvedValue({ - result: { - status: 'success', - text: JSON.stringify({ findings: [] }), - errors: [], - usage: makeUsage(), - }, - }); - const runAuxiliaryMock = vi.fn().mockResolvedValue({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart conversion, rendering, and tests.', - chunkIds: [ - 'src/example.ts:10', - 'src/example.ts:100', - 'src/example.ts:200', - ], - }], - }, - usage: makeUsage(), - }); - vi.mocked(getRuntime).mockReturnValue({ - name: 'pi', - runSkill: runSkillMock, - runAuxiliary: runAuxiliaryMock, - runSynthesis: vi.fn(), - } as unknown as Runtime); - - const report = await runSkill( - { - name: 'security-review', - description: 'Security review.', - prompt: 'Return findings as JSON.', - }, - makeContextWithThreeHunks(), - { - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - postProcessFindings: false, - }, - ); - - expect(runAuxiliaryMock).toHaveBeenCalledWith(expect.objectContaining({ - task: 'semantic_chunking', - agentName: 'semantic-chunk-planner', - })); - expect(runSkillMock).toHaveBeenCalledTimes(1); - const request = runSkillMock.mock.calls[0]?.[0]; - expect(request.userPrompt).toContain('## Review Chunk: semantic scanner slice'); - expect(request.userPrompt).not.toContain('## Semantic Summary:'); - expect(request.userPrompt).not.toContain('Dashboard charts now carry a widget-provided axis range through chart conversion, rendering, and tests.'); - expect(request.userPrompt).toContain('- src/example.ts:10-10'); - expect(request.userPrompt).toContain('- src/example.ts:100-100'); - expect(request.userPrompt).toContain('- src/example.ts:200-200'); - expect(report.auxiliaryUsageAttribution?.['semantic-chunk-planner']).toBeDefined(); - }); - - it('infers missing finding paths from cross-file semantic chunk line maps', async () => { - const runSkillMock = vi.fn().mockResolvedValue({ - result: { - status: 'success', - text: JSON.stringify({ - findings: [{ - id: 'pathless-finding', - severity: 'medium', - confidence: 'high', - title: 'Pathless finding', - description: 'test', - location: { startLine: 50 }, - }], - }), - errors: [], - usage: makeUsage(), - }, - }); - const runAuxiliaryMock = vi.fn().mockResolvedValue({ - success: true, - data: { - groups: [{ - title: 'Preserve behavior and test expectation', - summary: 'The implementation and test update the same behavior contract.', - chunkIds: [ - 'src/example.ts:10', - 'tests/example.test.ts:50', - ], - }], - }, - usage: makeUsage(), - }); - vi.mocked(getRuntime).mockReturnValue({ - name: 'pi', - runSkill: runSkillMock, - runAuxiliary: runAuxiliaryMock, - runSynthesis: vi.fn(), - } as unknown as Runtime); - - const report = await runSkill( - { - name: 'security-review', - description: 'Security review.', - prompt: 'Return findings as JSON.', - }, - makeContextWithCrossFileHunks(), - { - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - postProcessFindings: false, - }, - ); - - expect(report.findings).toEqual([ - expect.objectContaining({ - title: 'Pathless finding', - location: { - path: 'tests/example.test.ts', - startLine: 50, - }, - }), - ]); - }); - it('preserves candidate findings when verification is interrupted', async () => { const controller = new AbortController(); const runSkillMock = vi.fn() @@ -937,7 +729,7 @@ describe('runSkill', () => { result: { status: 'success', text: JSON.stringify({ - findings: [makeFinding(10, 'candidate-finding', 'src/example.ts')], + findings: [makeFinding(10, 'candidate-finding')], }), errors: [], usage: makeUsage(), @@ -985,7 +777,7 @@ describe('runSkill', () => { result: { status: 'success', text: JSON.stringify({ - findings: [makeFinding(10, 'first-finding', 'src/example.ts')], + findings: [makeFinding(10, 'first-finding')], }), errors: [], usage: makeUsage(), diff --git a/packages/warden/src/sdk/analyze.ts b/packages/warden/src/sdk/analyze.ts index 735b8f83..14bcc5a1 100644 --- a/packages/warden/src/sdk/analyze.ts +++ b/packages/warden/src/sdk/analyze.ts @@ -1,15 +1,14 @@ import type { Span } from '@sentry/node'; import type { SkillDefinition } from '../config/schema.js'; import type { ErrorCode, Finding, RetryConfig } from '../types/index.js'; -import type { ChangedLineRange, ReviewChunk } from '../diff/index.js'; +import { getHunkLineRange, type HunkWithContext } from '../diff/index.js'; import { Sentry, emitExtractionMetrics, emitRetryMetric, emitSkillMetrics, ensureLocalTracing } from '../sentry.js'; import { SkillRunnerError, WardenAuthenticationError, isRetryableError, isAuthenticationError, isAuthenticationErrorMessage, isSubprocessError, classifyError, mapExtractionErrorCode, sanitizeErrorMessage } from './errors.js'; import type { CircuitBreakerReason } from './circuit-breaker.js'; import { DEFAULT_RETRY_CONFIG, calculateRetryDelay, sleep } from './retry.js'; import { aggregateUsage, emptyUsage, estimateTokens, aggregateAuxiliaryUsage, aggregateAuxiliaryUsageAttribution } from './usage.js'; -import { buildHunkSystemPrompt, buildReviewChunkUserPrompt, type PRPromptContext } from './prompt.js'; +import { buildHunkSystemPrompt, buildHunkUserPrompt, type PRPromptContext } from './prompt.js'; import { extractFindingsJson, extractFindingsWithLLM, validateFindings } from './extract.js'; -import type { FindingPathResolver } from './extract.js'; import { postProcessFindings } from './post-process.js'; import { buildFileReports } from './report-files.js'; import { getRuntime, getRuntimeProviderOptions } from './runtimes/index.js'; @@ -22,19 +21,17 @@ import { type HunkAnalysisCallbacks, type SkillRunnerOptions, type PreparedFile, - type ReviewChunkGroup, type FileAnalysisCallbacks, type FileAnalysisResult, type ChunkAnalysisResult, } from './types.js'; import { prepareFiles } from './prepare.js'; -import { planSemanticReviewChunks } from '../semantic/index.js'; import type { EventContext, SkillReport, UsageStats, HunkFailure, HunkTrace } from '../types/index.js'; -import type { SourceSnippet } from '../types/index.js'; +import type { SourceSnippet, SourceSnippetLine } from '../types/index.js'; import { runPool } from '../utils/index.js'; import { getSpanContext, startTraceRecorder, withTraceRecorder, type TraceRecorder } from '../sentry-trace.js'; -/** Result from parsing review chunk output */ +/** Result from parsing hunk output */ interface ParseHunkOutputResult { findings: Finding[]; /** Whether extraction failed (both regex and LLM fallback) */ @@ -105,30 +102,6 @@ function allHunksFailedGuidance(runtime: SkillRunnerOptions['runtime'] | undefin return "Verify WARDEN_ANTHROPIC_API_KEY is set correctly, or run 'claude login' when using the Claude runtime without an API key."; } -function normalizeReviewChunkGroup(input: PreparedFile | ReviewChunkGroup): ReviewChunkGroup { - if ('filenames' in input) { - return input; - } - - return { - displayName: input.filename, - filenames: [input.filename], - chunks: input.chunks, - }; -} - -function fileReportInputsFromGroup(args: { - group: ReviewChunkGroup; - durationMs?: number; - usage?: UsageStats; -}) { - return args.group.filenames.map((filename, index) => ({ - filename, - durationMs: index === 0 ? args.durationMs : undefined, - usage: index === 0 ? args.usage : undefined, - })); -} - function buildHunkTrace(args: { enabled: boolean | undefined; span: Span; @@ -164,14 +137,14 @@ function buildHunkTrace(args: { } /** - * Parse findings from a review chunk analysis result. + * Parse findings from a hunk analysis result. * Uses a two-tier extraction strategy: * 1. Regex-based extraction (fast, handles well-formed output) * 2. LLM fallback using haiku (handles malformed output gracefully) */ async function parseHunkOutput( result: SkillRunResult, - defaultFilename: string | FindingPathResolver | undefined, + filename: string, skillName: string, options: SkillRunnerOptions ): Promise { @@ -182,10 +155,9 @@ async function parseHunkOutput( // Tier 1: Try regex-based extraction first (fast) const extracted = extractFindingsJson(result.text); - const filenameOrResolver = defaultFilename ?? (() => undefined); if (extracted.success) { - return { findings: validateFindings(extracted.findings, filenameOrResolver), extractionFailed: false, extractionMethod: 'regex' }; + return { findings: validateFindings(extracted.findings, filename), extractionFailed: false, extractionMethod: 'regex' }; } // Tier 2: Try LLM fallback for malformed output @@ -198,7 +170,7 @@ async function parseHunkOutput( }); if (fallback.success) { - return { findings: validateFindings(fallback.findings, filenameOrResolver), extractionFailed: false, extractionMethod: 'llm', extractionUsage: fallback.usage }; + return { findings: validateFindings(fallback.findings, filename), extractionFailed: false, extractionMethod: 'llm', extractionUsage: fallback.usage }; } // Both tiers failed - return extraction failure info @@ -212,71 +184,21 @@ async function parseHunkOutput( }; } -function numberFromRecord(record: Record, key: string): number | undefined { - const value = record[key]; - return typeof value === 'number' && Number.isInteger(value) && value > 0 ? value : undefined; -} - -function resolveFindingPathFromChangedLines( - changedLineMap: ChangedLineRange[], - fallbackFilename: string | undefined, - finding: Record -): string | undefined { - const location = finding['location']; - if (!location || typeof location !== 'object') { - return fallbackFilename; - } - - const locationRecord = location as Record; - const explicitPath = locationRecord['path']; - if (typeof explicitPath === 'string' && explicitPath.length > 0) { - return explicitPath; - } - - if (fallbackFilename) { - return fallbackFilename; - } - - const startLine = numberFromRecord(locationRecord, 'startLine'); - if (!startLine) { - return undefined; - } - - const endLine = numberFromRecord(locationRecord, 'endLine') ?? startLine; - const matchingPaths = new Set(); - for (const range of changedLineMap) { - if (startLine >= range.start && endLine <= range.end) { - matchingPaths.add(range.path); - } - } - - return matchingPaths.size === 1 ? [...matchingPaths][0] : undefined; -} - /** - * Filter findings whose location falls outside the changed line map. + * Filter findings whose startLine falls outside the hunk line range. * Findings without a location are kept (general findings). */ export function filterOutOfRangeFindings( findings: Finding[], - changedLineMap: ChangedLineRange[] | { start: number; end: number } + hunkRange: { start: number; end: number } ): { filtered: Finding[]; dropped: Finding[] } { - const ranges: (ChangedLineRange | { path?: undefined; start: number; end: number })[] = Array.isArray(changedLineMap) - ? changedLineMap - : [{ start: changedLineMap.start, end: changedLineMap.end }]; const filtered: Finding[] = []; const dropped: Finding[] = []; function isWithinHunk(finding: Finding): boolean { if (!finding.location) return true; - const { path, startLine } = finding.location; - const endLine = finding.location.endLine ?? startLine; - const lineInRange = (line: number): boolean => ranges.some((range) => - (range.path === undefined || range.path === path) - && line >= range.start - && line <= range.end - ); - return lineInRange(startLine) && lineInRange(endLine); + const { startLine } = finding.location; + return startLine >= hunkRange.start && startLine <= hunkRange.end; } for (const finding of findings) { @@ -289,21 +211,41 @@ export function filterOutOfRangeFindings( return { filtered, dropped }; } -/** Build a source snippet for a finding from the matching review chunk file. */ +function hunkSourceLines(hunkCtx: HunkWithContext): SourceSnippetLine[] { + const lines: SourceSnippetLine[] = []; + for (const [index, content] of hunkCtx.contextBefore.entries()) { + lines.push({ line: hunkCtx.contextStartLine + index, content }); + } + + let newLine = hunkCtx.hunk.newStart; + for (const diffLine of hunkCtx.hunk.lines) { + if (diffLine.startsWith('-')) continue; + if (!diffLine.startsWith('+') && !diffLine.startsWith(' ')) continue; + const content = diffLine.slice(1); + lines.push({ line: newLine, content }); + newLine += 1; + } + + const afterStart = hunkCtx.hunk.newStart + hunkCtx.hunk.newCount; + for (const [index, content] of hunkCtx.contextAfter.entries()) { + lines.push({ line: afterStart + index, content }); + } + + return lines; +} + export function buildSourceSnippet( finding: Finding, - chunk: ReviewChunk, + hunkCtx: HunkWithContext, contextLines = 3 ): SourceSnippet | undefined { if (!finding.location) return undefined; - const chunkFile = chunk.files.find((file) => file.path === finding.location?.path); - if (!chunkFile) return undefined; const targetStartLine = finding.location.startLine; const targetEndLine = finding.location.endLine ?? targetStartLine; const startLine = Math.max(1, targetStartLine - contextLines); const endLine = targetEndLine + contextLines; - const lines = chunkFile.sourceLines + const lines = hunkSourceLines(hunkCtx) .filter((line) => line.line >= startLine && line.line <= endLine) .map((line) => ({ ...line, @@ -317,7 +259,7 @@ export function buildSourceSnippet( return { path: finding.location.path, - language: chunkFile.language, + language: hunkCtx.language, startLine: firstLine.line, endLine: lastLine.line, targetStartLine, @@ -326,20 +268,20 @@ export function buildSourceSnippet( }; } -function attachSourceSnippets(findings: Finding[], chunk: ReviewChunk): Finding[] { +function attachSourceSnippets(findings: Finding[], hunkCtx: HunkWithContext): Finding[] { return findings.map((finding) => { if (!finding.location) return finding; - const sourceSnippet = buildSourceSnippet(finding, chunk); + const sourceSnippet = buildSourceSnippet(finding, hunkCtx); return sourceSnippet ? { ...finding, sourceSnippet } : finding; }); } /** - * Analyze a single review chunk with retry logic for transient failures. + * Analyze a single hunk with retry logic for transient failures. */ -async function analyzeReviewChunk( +async function analyzeHunk( skill: SkillDefinition, - chunk: ReviewChunk, + hunkCtx: HunkWithContext, repoPath: string, options: SkillRunnerOptions, callbacks?: HunkAnalysisCallbacks, @@ -349,16 +291,15 @@ async function analyzeReviewChunk( ensureLocalTracing(); } - const lineRange = callbacks?.lineRange ?? formatChunkLineRange(chunk); - const primaryFile = chunk.files[0]?.path ?? 'unknown'; + const lineRange = callbacks?.lineRange ?? formatHunkLineRange(hunkCtx); return Sentry.startSpan( { op: 'skill.analyze_hunk', - name: `analyze chunk ${primaryFile}:${lineRange}`, + name: `analyze hunk ${hunkCtx.filename}:${lineRange}`, attributes: { 'gen_ai.agent.name': skill.name, - 'code.file.path': primaryFile, + 'code.file.path': hunkCtx.filename, 'warden.hunk.line_range': lineRange, }, }, @@ -368,7 +309,7 @@ async function analyzeReviewChunk( const traceRecorder = options.captureTraces ? startTraceRecorder(span) : undefined; const systemPrompt = buildHunkSystemPrompt(skill); - const userPrompt = buildReviewChunkUserPrompt(skill, chunk, prContext); + const userPrompt = buildHunkUserPrompt(skill, hunkCtx, prContext); // Report prompt size information const systemChars = systemPrompt.length; @@ -404,7 +345,7 @@ async function analyzeReviewChunk( buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: circuitReason.code, @@ -427,7 +368,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: 'aborted', @@ -477,7 +418,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: 'missing_result', @@ -525,7 +466,7 @@ async function analyzeReviewChunk( buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: resultMessage.status, @@ -545,7 +486,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: resultMessage.status, @@ -558,30 +499,22 @@ async function analyzeReviewChunk( options.circuitBreaker?.recordSuccess(); const parseResult = await withTraceRecorder( traceRecorder, - () => parseHunkOutput( - resultMessage, - (finding) => resolveFindingPathFromChangedLines( - chunk.changedLineMap, - chunk.files.length === 1 ? primaryFile : undefined, - finding, - ), - skill.name, - options, - ), + () => parseHunkOutput(resultMessage, hunkCtx.filename, skill.name, options), ); - // Filter findings outside changed line ranges (defense-in-depth) - const { filtered, dropped } = filterOutOfRangeFindings(parseResult.findings, chunk.changedLineMap); - const filteredFindings = attachSourceSnippets(filtered, chunk); + // Filter findings outside hunk line range (defense-in-depth) + const hunkRange = getHunkLineRange(hunkCtx.hunk); + const { filtered, dropped } = filterOutOfRangeFindings(parseResult.findings, hunkRange); + const filteredFindings = attachSourceSnippets(filtered, hunkCtx); if (dropped.length > 0) { Sentry.addBreadcrumb({ category: 'finding.out_of_range', - message: `Dropped ${dropped.length} finding(s) outside changed line map`, + message: `Dropped ${dropped.length} finding(s) outside hunk range ${hunkRange.start}-${hunkRange.end}`, level: 'warning', data: { skill: skill.name, - filename: primaryFile, - changedLineMap: chunk.changedLineMap, + filename: hunkCtx.filename, + hunkRange, droppedLines: dropped.map((f) => f.location?.startLine), }, }); @@ -627,7 +560,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: resultMessage.status, @@ -651,7 +584,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: 'aborted', @@ -698,7 +631,7 @@ async function analyzeReviewChunk( Sentry.addBreadcrumb({ category: 'retry', - message: `Retrying review chunk analysis`, + message: `Retrying hunk analysis`, data: { attempt: attempt + 1, error: errorMessage, delayMs }, level: 'warning', }); @@ -729,7 +662,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: 'aborted', @@ -773,7 +706,7 @@ async function analyzeReviewChunk( buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: retryCode, @@ -792,7 +725,7 @@ async function analyzeReviewChunk( trace: buildHunkTrace({ enabled: options.captureTraces, span, - filename: primaryFile, + filename: hunkCtx.filename, lineRange, runtime: runtimeName, status: retryCode, @@ -804,15 +737,12 @@ async function analyzeReviewChunk( } /** - * Format a review chunk's changed ranges as a display string. + * Format a hunk's line range as a display string (e.g. "10-20" or "10"). */ -function formatChunkLineRange(chunk: ReviewChunk): string { - return chunk.changedLineMap - .map((range) => { - const lineRange = range.start === range.end ? `${range.start}` : `${range.start}-${range.end}`; - return chunk.files.length === 1 ? lineRange : `${range.path}:${lineRange}`; - }) - .join(', '); +function formatHunkLineRange(hunk: HunkWithContext): string { + const start = hunk.hunk.newStart; + const end = start + hunk.hunk.newCount - 1; + return start === end ? `${start}` : `${start}-${end}`; } /** @@ -827,25 +757,24 @@ function attachElapsedTime(findings: Finding[], skillStartTime: number | undefin } /** - * Analyze a single prepared file's review chunks. + * Analyze a single prepared file's hunks. */ export async function analyzeFile( skill: SkillDefinition, - file: PreparedFile | ReviewChunkGroup, + file: PreparedFile, repoPath: string, options: SkillRunnerOptions = {}, callbacks?: FileAnalysisCallbacks, prContext?: PRPromptContext ): Promise { - const group = normalizeReviewChunkGroup(file); return Sentry.startSpan( { op: 'skill.analyze_file', - name: `analyze chunk group ${group.displayName}`, + name: `analyze file ${file.filename}`, attributes: { 'gen_ai.agent.name': skill.name, - 'code.file.path': group.displayName, - 'warden.hunk.count': group.chunks.length, + 'code.file.path': file.filename, + 'warden.hunk.count': file.hunks.length, }, }, async (span) => { @@ -858,11 +787,11 @@ export async function analyzeFile( let failedHunks = 0; let failedExtractions = 0; - for (const [chunkIndex, chunk] of group.chunks.entries()) { + for (const [hunkIndex, hunk] of file.hunks.entries()) { if (abortController?.signal.aborted) break; - const lineRange = formatChunkLineRange(chunk); - callbacks?.onHunkStart?.(chunkIndex + 1, group.chunks.length, lineRange); + const lineRange = formatHunkLineRange(hunk); + callbacks?.onHunkStart?.(hunkIndex + 1, file.hunks.length, lineRange); const hunkCallbacks: HunkAnalysisCallbacks | undefined = callbacks ? { @@ -877,7 +806,7 @@ export async function analyzeFile( : undefined; const hunkStartTime = Date.now(); - const result = await analyzeReviewChunk(skill, chunk, repoPath, options, hunkCallbacks, prContext); + const result = await analyzeHunk(skill, hunk, repoPath, options, hunkCallbacks, prContext); const hunkDurationMs = Date.now() - hunkStartTime; // `failed` and `extractionFailed` are conceptually mutually exclusive: @@ -889,7 +818,7 @@ export async function analyzeFile( failedHunks++; hunkFailures.push({ type: 'analysis', - filename: group.displayName, + filename: file.filename, lineRange, code: result.failureCode ?? 'unknown', message: result.failureMessage ?? 'unknown error', @@ -899,7 +828,7 @@ export async function analyzeFile( failedExtractions++; hunkFailures.push({ type: 'extraction', - filename: group.displayName, + filename: file.filename, lineRange, code: mapExtractionErrorCode(result.extractionError), message: result.extractionError ?? 'unknown extraction error', @@ -908,15 +837,15 @@ export async function analyzeFile( } attachElapsedTime(result.findings, callbacks?.skillStartTime); - callbacks?.onHunkComplete?.(chunkIndex + 1, result.findings, result.usage); + callbacks?.onHunkComplete?.(hunkIndex + 1, result.findings, result.usage); if (result.trace) { hunkTraces.push(result.trace); } const chunkResult: ChunkAnalysisResult = { - filename: group.displayName, + filename: file.filename, model: options.model, - index: chunkIndex + 1, - total: group.chunks.length, + index: hunkIndex + 1, + total: file.hunks.length, lineRange, findings: result.findings, usage: result.usage, @@ -944,8 +873,7 @@ export async function analyzeFile( span.setAttribute('warden.extraction.failed_count', failedExtractions); return { - filename: group.displayName, - filenames: group.filenames, + filename: file.filename, findings: fileFindings, usage: aggregateUsage(fileUsage), failedHunks, @@ -980,7 +908,7 @@ export function generateSummary(skillName: string, findings: Finding[]): string } /** - * Run a skill on a PR, analyzing each prepared review chunk separately. + * Run a skill on a PR, analyzing each hunk separately. */ export async function runSkill( skill: SkillDefinition, @@ -1023,28 +951,14 @@ async function runSkillAnalysis( throw new SkillRunnerError('Pull request context required for skill execution'); } - const { files: initialPreparedFiles, skippedFiles } = prepareFiles(context, { + const { files: fileHunks, skippedFiles } = prepareFiles(context, { contextLines: options.contextLines, ignore: options.ignore, scan: options.scan, chunking: options.chunking, }); - const semanticPlan = await planSemanticReviewChunks(initialPreparedFiles, context, { - enabled: options.chunking?.semantic?.enabled, - apiKey: options.apiKey, - runtime: options.runtime, - model: options.model, - maxChunks: options.chunking?.semantic?.maxChunks, - maxChunkChars: options.chunking?.semantic?.maxChunkChars, - maxHunksPerChunk: options.chunking?.semantic?.maxHunksPerChunk, - maxChangedRangesPerChunk: options.chunking?.semantic?.maxChangedRangesPerChunk, - maxEmbeddedDiffChars: options.chunking?.semantic?.maxEmbeddedDiffChars, - maxEmbeddedDiffChunks: options.chunking?.semantic?.maxEmbeddedDiffChunks, - maxEmbeddedDiffRanges: options.chunking?.semantic?.maxEmbeddedDiffRanges, - }); - const chunkGroups = semanticPlan.groups; - if (chunkGroups.length === 0) { + if (fileHunks.length === 0) { const report: SkillReport = { skill: skill.name, summary: 'No code changes to analyze', @@ -1060,21 +974,13 @@ async function runSkillAnalysis( return report; } - const totalFiles = chunkGroups.length; - const totalHunks = chunkGroups.reduce((sum, group) => sum + group.chunks.length, 0); + const totalFiles = fileHunks.length; + const totalHunks = fileHunks.reduce((sum, file) => sum + file.hunks.length, 0); const allFindings: Finding[] = []; // Track all usage stats for aggregation const allUsage: UsageStats[] = []; const allAuxiliaryUsage: AuxiliaryUsageEntry[] = []; - if (semanticPlan.usage) { - allAuxiliaryUsage.push({ - agent: 'semantic-chunk-planner', - usage: semanticPlan.usage, - model: options.model, - runtime: options.runtime, - }); - } const allTraces: HunkTrace[] = []; // Track failed hunks across all files @@ -1083,7 +989,7 @@ async function runSkillAnalysis( // Build PR context for inclusion in prompts (helps LLM understand the full scope of changes) // For non-PR contexts (CLI file/diff mode), skip the "Other Files" list to avoid - // bloating every chunk prompt with thousands of filenames. + // bloating every hunk prompt with thousands of filenames. const isPullRequest = context.pullRequest.number !== 0; const prContext: PRPromptContext = { repository: context.repository.fullName, @@ -1094,14 +1000,14 @@ async function runSkillAnalysis( }; /** - * Process all review chunks for a single file sequentially. + * Process all hunks for a single file sequentially. * Wraps analyzeFile with progress callbacks. */ async function processFile( - group: ReviewChunkGroup, + fileHunkEntry: PreparedFile, fileIndex: number ): Promise { - const filename = group.displayName; + const { filename } = fileHunkEntry; callbacks?.onFileStart?.(filename, fileIndex, totalFiles); @@ -1145,7 +1051,7 @@ async function runSkillAnalysis( : undefined, }; - const result = await analyzeFile(skill, group, context.repoPath, options, fileCallbacks, prContext); + const result = await analyzeFile(skill, fileHunkEntry, context.repoPath, options, fileCallbacks, prContext); callbacks?.onFileComplete?.(filename, fileIndex, totalFiles); @@ -1153,15 +1059,15 @@ async function runSkillAnalysis( } /** Process a file with timing, returning a self-contained result. */ - async function processFileWithTiming(group: ReviewChunkGroup, fileIndex: number) { + async function processFileWithTiming(fileHunkEntry: PreparedFile, fileIndex: number) { const fileStart = Date.now(); - const result = await processFile(group, fileIndex); + const result = await processFile(fileHunkEntry, fileIndex); const durationMs = Date.now() - fileStart; - return { group, result, durationMs }; + return { filename: fileHunkEntry.filename, result, durationMs }; } // Collect results in input order (Promise.all preserves order) - const fileResults: { group: ReviewChunkGroup; result: FileAnalysisResult; durationMs: number }[] = []; + const fileResults: { filename: string; result: FileAnalysisResult; durationMs: number }[] = []; // Process files - parallel or sequential based on options if (parallel) { @@ -1169,23 +1075,23 @@ async function runSkillAnalysis( const fileConcurrency = options.concurrency ?? DEFAULT_FILE_CONCURRENCY; const batchDelayMs = options.batchDelayMs ?? 0; - fileResults.push(...await runPool(chunkGroups, fileConcurrency, - async (group, index) => { + fileResults.push(...await runPool(fileHunks, fileConcurrency, + async (fileHunkEntry, index) => { // Rate-limit: delay items beyond the first concurrent wave if (index >= fileConcurrency && batchDelayMs > 0) { await new Promise((resolve) => setTimeout(resolve, batchDelayMs)); } - return processFileWithTiming(group, index); + return processFileWithTiming(fileHunkEntry, index); }, { shouldAbort: () => abortController?.signal.aborted ?? false } )); } else { // Process files sequentially - for (const [fileIndex, group] of chunkGroups.entries()) { + for (const [fileIndex, fileHunkEntry] of fileHunks.entries()) { // Check for abort before starting new file if (abortController?.signal.aborted) break; - fileResults.push(await processFileWithTiming(group, fileIndex)); + fileResults.push(await processFileWithTiming(fileHunkEntry, fileIndex)); } } @@ -1207,11 +1113,11 @@ async function runSkillAnalysis( } } - // All chunks failed — typically a systemic problem (auth, subprocess, etc). + // All hunks failed — typically a systemic problem (auth, subprocess, etc). // Throw so direct SDK consumers (evals, scheduled workflows) keep their // prior exception-based contract. The CLI path (tasks.ts) has its own // all-hunks-fail detection that emits a structured JSONL record instead. - // Count both analysis and extraction failures: each chunk contributes to + // Count both analysis and extraction failures: each hunk contributes to // at most one (analyzeFile makes them mutually exclusive), and an // extraction-only failure scenario would otherwise slip through silently. const totalAttemptFailures = totalFailedHunks + totalFailedExtractions; @@ -1271,7 +1177,7 @@ async function runSkillAnalysis( // Generate summary const summary = generateSummary(skill.name, finalFindings); - // Aggregate usage across all chunks + // Aggregate usage across all hunks const totalUsage = aggregateUsage(allUsage); const report: SkillReport = { @@ -1282,8 +1188,8 @@ async function runSkillAnalysis( durationMs: Date.now() - startTime, model: options.model, files: buildFileReports( - fileResults.flatMap((fr) => fileReportInputsFromGroup({ - group: fr.group, + fileResults.map((fr) => ({ + filename: fr.filename, durationMs: fr.durationMs, usage: fr.result.usage, })), diff --git a/packages/warden/src/sdk/extract.ts b/packages/warden/src/sdk/extract.ts index a2da1ac6..b7ef0229 100644 --- a/packages/warden/src/sdk/extract.ts +++ b/packages/warden/src/sdk/extract.ts @@ -16,8 +16,6 @@ import { const ExtractedFindingSchema = FindingSchema.omit({ sourceSnippet: true }); -export type FindingPathResolver = (finding: Record) => string | undefined; - /** Pattern to match the start of findings JSON (allows whitespace after brace) */ export const FINDINGS_JSON_START = /\{\s*"findings"/; @@ -276,46 +274,28 @@ export function generateShortId(): string { * Validate and normalize findings from extracted JSON. * Replaces the LLM-provided ID with a short ID for cross-referencing. */ -export function validateFindings( - findings: unknown[], - defaultFilenameOrResolver: string | FindingPathResolver -): Finding[] { +export function validateFindings(findings: unknown[], filename: string): Finding[] { const validated: Finding[] = []; for (const f of findings) { const candidate = typeof f === 'object' && f !== null ? { ...(f as Record) } : f; - const defaultFilename = typeof defaultFilenameOrResolver === 'function' - && typeof candidate === 'object' - && candidate !== null - ? defaultFilenameOrResolver(candidate as Record) - : defaultFilenameOrResolver; - - // Normalize missing location paths before validation. Multi-file review - // chunks may provide explicit paths, so preserve them when present. + + // Normalize location path before validation if (typeof candidate === 'object' && candidate !== null && 'location' in candidate) { const loc = (candidate as Record)['location']; if (loc && typeof loc === 'object') { - const location = loc as Record; (candidate as Record)['location'] = { - ...location, - path: typeof location['path'] === 'string' && location['path'].length > 0 - ? location['path'] - : defaultFilename, + ...(loc as Record), + path: filename, }; } } const result = ExtractedFindingSchema.safeParse(candidate); if (result.success) { - const locationPath = result.data.location?.path || defaultFilename; - if (result.data.location && !locationPath) { - continue; - } - const location = result.data.location - ? { ...result.data.location, path: locationPath as string } - : undefined; + const location = result.data.location ? { ...result.data.location, path: filename } : undefined; validated.push({ ...result.data, id: generateShortId(), diff --git a/packages/warden/src/sdk/prepare.test.ts b/packages/warden/src/sdk/prepare.test.ts index f0eb2f16..3bb58799 100644 --- a/packages/warden/src/sdk/prepare.test.ts +++ b/packages/warden/src/sdk/prepare.test.ts @@ -208,67 +208,6 @@ describe('prepareFiles', () => { ]); }); - it('leaves distant same-file hunks atomic for semantic planning', () => { - const context = makeContext([ - { - filename: 'src/example.ts', - status: 'modified', - patch: [ - '@@ -10,1 +10,1 @@', - '-old10', - '+new10', - '@@ -100,1 +100,1 @@', - '-old100', - '+new100', - '@@ -200,1 +200,1 @@', - '-old200', - '+new200', - ].join('\n'), - }, - ]); - - const result = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - }); - - expect(result.files).toHaveLength(1); - expect(result.files[0]?.chunks).toHaveLength(3); - expect(result.files[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ - [{ path: 'src/example.ts', start: 10, end: 10 }], - [{ path: 'src/example.ts', start: 100, end: 100 }], - [{ path: 'src/example.ts', start: 200, end: 200 }], - ]); - }); - - it('maps only changed added lines, not unchanged hunk context', () => { - const context = makeContext([ - { - filename: 'src/example.ts', - status: 'modified', - patch: [ - '@@ -10,3 +10,3 @@', - ' const before = true;', - '-const value = oldValue;', - '+const value = newValue;', - ' const after = true;', - ].join('\n'), - }, - ]); - - const result = prepareFiles(context); - - expect(result.files[0]?.chunks[0]?.changedLineMap).toEqual([ - { path: 'src/example.ts', start: 11, end: 11 }, - ]); - }); - it('does not skip go.mod as a built-in ignored file', () => { const context = makeContext([ { filename: 'go.mod', patch: '@@ -0,0 +1,1 @@\n+module example.com/app' }, @@ -412,10 +351,7 @@ describe('prepareFiles', () => { const context = buildLocalEventContext({ base: 'HEAD^', head: 'HEAD', cwd: repoPath }); const result = prepareFiles(context, { contextLines: 1 }); - expect(result.files[0]?.chunks[0]?.files[0]?.sourceLines).toContainEqual({ - line: 8, - content: 'clean context', - }); + expect(result.files[0]?.hunks[0]?.contextAfter).toEqual(['clean context']); }, 30_000); it('applies git-ref scan limits to the analyzed commit, not the dirty working tree', () => { @@ -460,9 +396,6 @@ describe('prepareFiles', () => { const context = buildLocalEventContext({ cwd: repoPath, staged: true }); const result = prepareFiles(context, { contextLines: 1 }); - expect(result.files[0]?.chunks[0]?.files[0]?.sourceLines).toContainEqual({ - line: 8, - content: 'index context', - }); + expect(result.files[0]?.hunks[0]?.contextAfter).toEqual(['index context']); }, 30_000); }); diff --git a/packages/warden/src/sdk/prepare.ts b/packages/warden/src/sdk/prepare.ts index abe0632e..84808b91 100644 --- a/packages/warden/src/sdk/prepare.ts +++ b/packages/warden/src/sdk/prepare.ts @@ -5,8 +5,7 @@ import { classifyFile, coalesceHunks, splitLargeHunks, - reviewChunkFromHunk, - type ReviewChunk, + type HunkWithContext, } from '../diff/index.js'; import type { PreparedFile, PrepareFilesOptions, PrepareFilesResult } from './types.js'; import { applyScanPolicy } from './scan-policy.js'; @@ -18,26 +17,26 @@ function matchingChunkingSkipPattern( return patterns?.find((pattern) => classifyFile(filename, [pattern]) === 'skip')?.pattern; } -/** Adapt atomic review chunks into the per-file shape used before semantic planning. */ -export function groupChunksByFile(chunks: ReviewChunk[]): PreparedFile[] { - const fileMap = new Map(); +/** + * Group hunks by filename into PreparedFile entries. + */ +export function groupHunksByFile(hunks: HunkWithContext[]): PreparedFile[] { + const fileMap = new Map(); - for (const chunk of chunks) { - const filename = chunk.files[0]?.path; - if (!filename) continue; - const existing = fileMap.get(filename); + for (const hunk of hunks) { + const existing = fileMap.get(hunk.filename); if (existing) { - existing.push(chunk); + existing.push(hunk); } else { - fileMap.set(filename, [chunk]); + fileMap.set(hunk.filename, [hunk]); } } - return Array.from(fileMap, ([filename, chunks]) => ({ filename, chunks })); + return Array.from(fileMap, ([filename, fileHunks]) => ({ filename, hunks: fileHunks })); } /** - * Prepare files for analysis by parsing patches into review chunks with context. + * Prepare files for analysis by parsing patches into hunks with context. * Returns files that have changes to analyze and files that were skipped. */ export function prepareFiles( @@ -51,7 +50,7 @@ export function prepareFiles( } const pr = context.pullRequest; - const allChunks: ReviewChunk[] = []; + const allHunks: HunkWithContext[] = []; const skippedFiles: SkippedFile[] = []; const scanPolicy = applyScanPolicy(pr.files, { @@ -110,12 +109,11 @@ export function prepareFiles( contextLines, contentSource: context.diffContextSource, }); - const rawChunks = hunksWithContext.map(reviewChunkFromHunk); - allChunks.push(...rawChunks); + allHunks.push(...hunksWithContext); } return { - files: groupChunksByFile(allChunks), + files: groupHunksByFile(allHunks), skippedFiles, }; } diff --git a/packages/warden/src/sdk/prompt.ts b/packages/warden/src/sdk/prompt.ts index d043d353..8c0e92b4 100644 --- a/packages/warden/src/sdk/prompt.ts +++ b/packages/warden/src/sdk/prompt.ts @@ -1,7 +1,7 @@ import { existsSync } from 'node:fs'; import { join } from 'node:path'; import type { SkillDefinition } from '../config/schema.js'; -import type { ReviewChunk } from '../diff/index.js'; +import { formatHunkForAnalysis, type HunkWithContext } from '../diff/index.js'; import { buildChangedFilesSection, buildJsonOutputSection, @@ -12,38 +12,11 @@ import { export type PRPromptContext = PromptPRContext; -function formatChangedLineMap(chunk: ReviewChunk): string { - return chunk.changedLineMap - .map((range) => `- ${range.path}:${range.start}-${range.end}`) - .join('\n'); -} - -/** Format one semantic review chunk with its summary, line map, and file content. */ -export function formatReviewChunkForAnalysis(chunk: ReviewChunk): string { - const lines: string[] = []; - - lines.push(`## Review Chunk: ${chunk.summary ? 'semantic scanner slice' : chunk.title}`); - lines.push(''); - lines.push('## Changed Line Map'); - lines.push(formatChangedLineMap(chunk)); - - for (const file of chunk.files) { - lines.push(''); - lines.push(`## File: ${file.path}`); - lines.push(`## Language: ${file.language}`); - lines.push(`## Content Mode: ${file.contentMode}`); - lines.push(''); - lines.push(file.content); - } - - return lines.join('\n'); -} - /** - * Builds the system prompt for review chunk analysis. + * Builds the system prompt for hunk-based analysis. * * Future enhancement: Could have the agent output a structured `contextAssessment` - * (applicationType, trustBoundaries, filesChecked) to cache across chunks, allow + * (applicationType, trustBoundaries, filesChecked) to cache across hunks, allow * user overrides, or build analytics. Not implemented since we don't consume it yet. */ export function buildHunkSystemPrompt(skill: SkillDefinition): string { @@ -57,7 +30,7 @@ Before reporting a finding: 1. Read the relevant source code to understand the full context 2. Trace through the code path — follow imports, base classes, and indirect references, not just the immediate file 3. Verify your assumptions — confirm the issue exists, don't infer from incomplete information -4. Ensure each finding's location starts on a line listed in the review chunk's changed-line map +4. Ensure the finding references lines within the hunk being analyzed 5. Document the evidence trace in the 'verification' field of each finding `, @@ -93,15 +66,15 @@ Full schema: Requirements: - Return valid JSON starting with {"findings": - "findings" array can be empty if no issues found -- "location.path" must be one of the files in the changed line map. For single-file chunks, use that file path. -- "location.startLine" MUST be within one of the changed line map ranges. If "location.endLine" is present, it must also be within one of the changed line map ranges for the same file. If the issue originates in surrounding code, anchor to the nearest changed line in the changed line map and note the actual location in the description. +- "location.path" is auto-filled from context - just provide startLine (and optionally endLine). Omit location entirely for general findings not about a specific line. +- "location.startLine" MUST be within the hunk line range (shown in the "## Hunk" header). If the issue originates in surrounding code, anchor to the nearest changed line in the hunk and note the actual location in the description. - "confidence" reflects how certain you are this is a real issue given the codebase context - "description" is rendered directly in GitHub inline comments. Keep it brief and actionable, usually one sentence. - Put the concrete evidence trace in "verification", not "description". - Write "verification" as evidence, not reasoning: facts from the code path, guards, conditions, and observed behavior that make the finding believable. - Do not format "verification" as any labeled checklist or template. - Do not include severity, confidence, finding ID, skill name, or generic review framing in "description". -- Focus your analysis on the code changes in the review chunk. Surrounding context and tool results are for understanding only -- all findings must reference lines within the changed line map. +- Focus your analysis on the code changes in the hunk. Surrounding context and tool results are for understanding only -- all findings must reference lines within the hunk range. `), ]; @@ -122,25 +95,23 @@ You can read files from ${dirList} subdirectories using the Read tool with the f return sections.join('\n\n'); } -/** Build the scanner prompt around one semantic chunk while preserving location constraints. */ -export function buildReviewChunkUserPrompt( +/** + * Builds the user prompt for a single hunk. + */ +export function buildHunkUserPrompt( skill: SkillDefinition, - chunk: ReviewChunk, + hunkCtx: HunkWithContext, prContext?: PRPromptContext ): string { - const currentFile = chunk.files.length === 1 ? chunk.files[0]?.path : undefined; return joinPromptSections([ ` -Analyze this semantic review chunk according to the "${skill.name}" skill criteria. +Analyze this code change according to the "${skill.name}" skill criteria. `, buildPullRequestContextSection(prContext), - buildChangedFilesSection(prContext, currentFile), - formatReviewChunkForAnalysis(chunk), + buildChangedFilesSection(prContext, hunkCtx.filename), + formatHunkForAnalysis(hunkCtx), ` -Only report findings that are explicitly covered by the skill instructions. Do not report general code quality issues, bugs, or improvements unless the skill specifically asks for them. Locations must be inside the changed line map. Return an empty findings array if no issues match the skill's criteria. +Only report findings that are explicitly covered by the skill instructions. Do not report general code quality issues, bugs, or improvements unless the skill specifically asks for them. Return an empty findings array if no issues match the skill's criteria. `, ]); } - -/** Legacy alias for callers that still use the old hunk prompt name. */ -export const buildHunkUserPrompt = buildReviewChunkUserPrompt; diff --git a/packages/warden/src/sdk/runner.test.ts b/packages/warden/src/sdk/runner.test.ts index b304fc27..17de9dd2 100644 --- a/packages/warden/src/sdk/runner.test.ts +++ b/packages/warden/src/sdk/runner.test.ts @@ -944,7 +944,7 @@ describe('validateFindings', () => { expect(validated[0]!.id).not.toBe(validated[1]!.id); }); - it('preserves explicit location paths for multi-file chunks', () => { + it('normalizes location path to the provided filename', () => { const rawFindings = [ { id: 'id-1', @@ -955,62 +955,10 @@ describe('validateFindings', () => { }, ]; - const validated = validateFindings(rawFindings, 'correct-path.ts'); - expect(validated[0]!.location!.path).toBe('wrong-path.ts'); - }); - - it('defaults missing location paths to the provided filename', () => { - const rawFindings = [ - { - id: 'id-1', - severity: 'medium', - title: 'Issue', - description: 'Details', - location: { startLine: 5 }, - }, - ]; - const validated = validateFindings(rawFindings, 'correct-path.ts'); expect(validated[0]!.location!.path).toBe('correct-path.ts'); }); - it('defaults missing location paths with a resolver', () => { - const rawFindings = [ - { - id: 'id-1', - severity: 'medium', - title: 'Issue', - description: 'Details', - location: { startLine: 42 }, - }, - ]; - - const validated = validateFindings(rawFindings, (finding) => { - const location = finding['location']; - if (!location || typeof location !== 'object') return undefined; - return (location as Record)['startLine'] === 42 - ? 'src/cross-file.ts' - : undefined; - }); - - expect(validated[0]!.location!.path).toBe('src/cross-file.ts'); - }); - - it('drops findings when the resolver cannot infer a missing path', () => { - const rawFindings = [ - { - id: 'id-1', - severity: 'medium', - title: 'Issue', - description: 'Details', - location: { startLine: 42 }, - }, - ]; - - const validated = validateFindings(rawFindings, () => undefined); - expect(validated).toHaveLength(0); - }); - it('strips model-provided source snippets during extraction', () => { const rawFindings = [ { diff --git a/packages/warden/src/sdk/runner.ts b/packages/warden/src/sdk/runner.ts index e2aafd44..22b0dac0 100644 --- a/packages/warden/src/sdk/runner.ts +++ b/packages/warden/src/sdk/runner.ts @@ -29,7 +29,7 @@ export { anthropicUsageToStats, apiUsageToStats } from './pricing.js'; export type { AnthropicApiUsage } from './pricing.js'; // Re-export prompt building (with legacy alias) -export { buildHunkSystemPrompt, buildHunkUserPrompt, buildReviewChunkUserPrompt, formatReviewChunkForAnalysis } from './prompt.js'; +export { buildHunkSystemPrompt, buildHunkUserPrompt } from './prompt.js'; export type { PRPromptContext } from './prompt.js'; // Legacy export for backwards compatibility export { buildHunkSystemPrompt as buildSystemPrompt } from './prompt.js'; @@ -105,7 +105,6 @@ export type { SkillRunnerCallbacks, SkillRunnerOptions, PreparedFile, - ReviewChunkGroup, PrepareFilesOptions, PrepareFilesResult, FileAnalysisCallbacks, diff --git a/packages/warden/src/sdk/runtimes/types.ts b/packages/warden/src/sdk/runtimes/types.ts index 0bbedad8..812f7282 100644 --- a/packages/warden/src/sdk/runtimes/types.ts +++ b/packages/warden/src/sdk/runtimes/types.ts @@ -90,7 +90,6 @@ export type AuxiliaryTask = | 'extraction' | 'deduplication' | 'fix_evaluation' - | 'semantic_chunking' | 'eval_judge'; export type SynthesisTask = diff --git a/packages/warden/src/sdk/types.ts b/packages/warden/src/sdk/types.ts index 4d7a2dee..7c773562 100644 --- a/packages/warden/src/sdk/types.ts +++ b/packages/warden/src/sdk/types.ts @@ -1,5 +1,5 @@ import type { Finding, UsageStats, SkippedFile, RetryConfig, ErrorCode, HunkFailure, HunkTrace } from '../types/index.js'; -import type { ReviewChunk } from '../diff/index.js'; +import type { HunkWithContext } from '../diff/index.js'; import type { ChunkingConfig, Effort, IgnoreConfig, ScanConfig } from '../config/schema.js'; import type { RuntimeName } from './runtimes/index.js'; import type { ProviderFailureCircuitBreaker } from './circuit-breaker.js'; @@ -146,27 +146,18 @@ export interface SkillRunnerOptions { postProcessFindings?: boolean; /** Trigger name to attach to skill-level telemetry when the caller has one. */ telemetryTriggerName?: string; - /** Capture per-chunk runtime traces in structured run output. Defaults to false. */ + /** Capture per-hunk runtime traces in structured run output. Defaults to false. */ captureTraces?: boolean; } -export type AnalysisChunkingConfig = Pick; +export type AnalysisChunkingConfig = Pick; /** - * A file prepared for analysis with its review chunks. + * A file prepared for analysis with its hunks. */ export interface PreparedFile { filename: string; - chunks: ReviewChunk[]; -} - -/** - * One scanner-facing group of review chunks. - */ -export interface ReviewChunkGroup { - displayName: string; - filenames: string[]; - chunks: ReviewChunk[]; + hunks: HunkWithContext[]; } /** @@ -220,7 +211,6 @@ export interface FileAnalysisCallbacks { */ export interface FileAnalysisResult { filename: string; - filenames?: string[]; findings: Finding[]; usage: UsageStats; /** Number of hunks that failed to analyze */ diff --git a/packages/warden/src/semantic/index.ts b/packages/warden/src/semantic/index.ts deleted file mode 100644 index 8772b598..00000000 --- a/packages/warden/src/semantic/index.ts +++ /dev/null @@ -1,8 +0,0 @@ -export { - planSemanticReviewChunks, -} from './planner.js'; - -export type { - SemanticChunkPlanningOptions, - SemanticChunkPlanningResult, -} from './planner.js'; diff --git a/packages/warden/src/semantic/planner.test.ts b/packages/warden/src/semantic/planner.test.ts deleted file mode 100644 index 3b2ca0f1..00000000 --- a/packages/warden/src/semantic/planner.test.ts +++ /dev/null @@ -1,593 +0,0 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest'; -import type { EventContext } from '../types/index.js'; -import { prepareFiles } from '../sdk/prepare.js'; -import { planSemanticReviewChunks } from './planner.js'; -import type { Runtime } from '../sdk/runtimes/index.js'; -import type * as RuntimeModule from '../sdk/runtimes/index.js'; - -const runAuxiliary = vi.fn(); - -vi.mock('../sdk/runtimes/index.js', async (importOriginal) => { - const actual = await importOriginal(); - return { - ...actual, - getRuntime: vi.fn(() => ({ - name: 'pi', - runSkill: vi.fn(), - runAuxiliary, - runSynthesis: vi.fn(), - } satisfies Runtime)), - }; -}); - -beforeEach(() => { - runAuxiliary.mockReset(); -}); - -function makeContext(): EventContext { - return { - eventType: 'pull_request', - action: 'opened', - repository: { - owner: 'qa', - name: 'repo', - fullName: 'qa/repo', - defaultBranch: 'main', - }, - repoPath: '/tmp/warden-semantic-planner-test', - pullRequest: { - number: 1, - title: 'Preserve user-selected dashboard axis range', - body: '', - author: 'qa', - baseBranch: 'main', - headBranch: 'feature', - headSha: 'head', - baseSha: 'base', - files: [{ - filename: 'src/dashboard.ts', - status: 'modified', - additions: 3, - deletions: 3, - chunks: 3, - patch: [ - '@@ -10,1 +10,1 @@', - '-const range = getDefaultRange(widget);', - '+const range = widget.axisRange ?? getDefaultRange(widget);', - '@@ -100,1 +100,1 @@', - '-return buildChart(series);', - '+return buildChart(series, range);', - '@@ -200,1 +200,1 @@', - '-expect(chart.range).toEqual(defaultRange);', - '+expect(chart.range).toEqual(customAxisRange);', - ].join('\n'), - }], - }, - }; -} - -describe('planSemanticReviewChunks', () => { - it('groups atomic chunks using a model-provided semantic delta', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: [ - 'src/dashboard.ts:10', - 'src/dashboard.ts:100', - 'src/dashboard.ts:200', - ], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - }); - expect(prepared.files[0]?.chunks).toHaveLength(3); - - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }); - - expect(planned.groups).toHaveLength(1); - expect(planned.groups[0]?.chunks).toHaveLength(1); - expect(planned.groups[0]?.chunks[0]).toMatchObject({ - id: 'semantic:1', - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - changedLineMap: [ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'src/dashboard.ts', start: 100, end: 100 }, - { path: 'src/dashboard.ts', start: 200, end: 200 }, - ], - }); - expect(runAuxiliary).toHaveBeenCalledWith(expect.objectContaining({ - task: 'semantic_chunking', - agentName: 'semantic-chunk-planner', - model: 'anthropic/claude-sonnet-4-6', - tools: expect.arrayContaining([ - expect.objectContaining({ name: 'read_review_chunk' }), - expect.objectContaining({ name: 'read_changed_file' }), - expect.objectContaining({ name: 'search_repo' }), - ]), - executeTool: expect.any(Function), - maxIterations: 5, - })); - const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; - expect(prompt).toContain('Changed-line preview:'); - expect(prompt).toContain('Embedded small diff:'); - expect(prompt).toContain('@@ -10,1 +10,1 @@'); - }); - - it('keeps many-hunk planner prompts metadata-first and tool-backed', async () => { - const context = makeContext(); - context.pullRequest!.files = [{ - filename: 'src/dashboard.ts', - status: 'modified', - additions: 13, - deletions: 13, - chunks: 13, - patch: Array.from({ length: 13 }, (_, index) => { - const line = index + 1; - return [ - `@@ -${line},1 +${line},1 @@`, - `-const value${line} = getDefaultRange(widget);`, - `+const value${line} = widget.axisRange ?? getDefaultRange(widget);`, - ].join('\n'); - }).join('\n'), - }]; - const prepared = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - }); - const chunkIds = prepared.files.flatMap((file) => file.chunks.map((chunk) => chunk.id)); - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard range values now prefer widget-provided axis ranges across repeated dashboard call sites.', - chunkIds, - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - model: 'anthropic/claude-sonnet-4-6', - }); - - const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; - expect(prompt).toContain('Changed-line preview:'); - expect(prompt).not.toContain('Embedded small diff:'); - expect(prompt).not.toContain('@@ -1,1 +1,1 @@'); - const executeTool = runAuxiliary.mock.calls[0]?.[0].executeTool as ( - name: string, - input: Record, - ) => Promise; - await expect(executeTool('read_review_chunk', { chunkId: chunkIds[0] })) - .resolves.toContain('@@ -1,1 +1,1 @@'); - }); - - it('uses configured embedded diff limits for smaller-context models', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context, { - chunking: { - semantic: { - enabled: true, - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }, - }, - }); - - await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - model: 'small-context-model', - maxEmbeddedDiffChars: 0, - maxEmbeddedDiffChunks: 0, - maxEmbeddedDiffRanges: 0, - }); - - const prompt = runAuxiliary.mock.calls[0]?.[0].prompt as string; - expect(prompt).toContain('0 chars, 0 chunks, 0 changed ranges'); - expect(prompt).toContain('Changed-line preview:'); - expect(prompt).not.toContain('Embedded small diff:'); - expect(prompt).not.toContain('@@ -10,1 +10,1 @@'); - }); - - it('splits planned semantic changes into bounded scanner chunks', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context); - - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxChunkChars: 500, - }); - - expect(planned.groups).toHaveLength(1); - expect(planned.groups[0]?.displayName).toBe('Preserve dashboard axis range'); - expect(planned.groups[0]?.chunks.map((chunk) => chunk.title)).toEqual([ - 'Preserve dashboard axis range (1/2)', - 'Preserve dashboard axis range (2/2)', - ]); - expect(planned.groups[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ - [ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'src/dashboard.ts', start: 100, end: 100 }, - ], - [{ path: 'src/dashboard.ts', start: 200, end: 200 }], - ]); - }); - - it('rejects materialized scanner chunks that exceed maxChunks after splitting', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context); - - await expect(planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxChunks: 1, - maxChunkChars: 500, - })).rejects.toThrow('materialized 2 chunks, exceeding maxChunks 1'); - }); - - it('preserves one semantic group when maxHunksPerChunk splits scanner chunks', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context); - - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxHunksPerChunk: 2, - }); - - expect(planned.groups).toHaveLength(1); - expect(planned.groups[0]?.chunks).toHaveLength(2); - expect(planned.groups[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ - [ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'src/dashboard.ts', start: 100, end: 100 }, - ], - [{ path: 'src/dashboard.ts', start: 200, end: 200 }], - ]); - }); - - it('preserves one semantic group when changed ranges split scanner chunks', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10', 'src/dashboard.ts:100', 'src/dashboard.ts:200'], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context); - - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxChangedRangesPerChunk: 2, - }); - - expect(planned.groups).toHaveLength(1); - expect(planned.groups[0]?.chunks).toHaveLength(2); - expect(planned.groups[0]?.chunks.map((chunk) => chunk.changedLineMap)).toEqual([ - [ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'src/dashboard.ts', start: 100, end: 100 }, - ], - [{ path: 'src/dashboard.ts', start: 200, end: 200 }], - ]); - }); - - it('collapses repeated tiny similar hunks into one compact scanner chunk', async () => { - const context = makeContext(); - context.pullRequest!.files = [{ - filename: 'src/repeated.ts', - status: 'modified', - additions: 5, - deletions: 5, - chunks: 5, - patch: [10, 60, 110, 160, 210].map((line) => [ - `@@ -${line},1 +${line},1 @@`, - "-params.push({axisRange: 'auto'});", - '+params.push({axisRange: undefined});', - ].join('\n')).join('\n'), - }]; - - const prepared = prepareFiles(context); - const chunkIds = prepared.files.flatMap((file) => file.chunks.map((chunk) => chunk.id)); - expect(chunkIds).toHaveLength(5); - - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Update repeated axisRange defaults', - summary: 'Repeated call sites now pass an undefined axis range instead of the auto default.', - chunkIds, - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxHunksPerChunk: 2, - maxChangedRangesPerChunk: 2, - }); - - const chunk = planned.groups[0]?.chunks[0]; - expect(planned.groups).toHaveLength(1); - expect(planned.groups[0]?.chunks).toHaveLength(1); - expect(chunk?.changedLineMap).toHaveLength(5); - expect(chunk?.files[0]?.content).toContain("params.push({axisRange: 'auto'});"); - expect(chunk?.files[0]?.content).not.toContain('### Context Before'); - }); - - it('rejects atomic chunks that exceed maxChunkChars', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry a widget-provided axis range through chart construction and assert that custom range in tests.', - chunkIds: ['src/dashboard.ts:10'], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - const prepared = prepareFiles(context); - - await expect(planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxChunkChars: 10, - })).rejects.toThrow('cannot split atomic chunk src/dashboard.ts:10 under maxChunkChars 10'); - }); - - it('materializes semantic groups across multiple files', async () => { - runAuxiliary.mockResolvedValueOnce({ - success: true, - data: { - groups: [{ - title: 'Preserve dashboard axis range', - summary: 'Dashboard charts now carry widget axis ranges through rendering and assert that behavior in tests.', - chunkIds: [ - 'src/dashboard.ts:10', - 'tests/dashboard.test.ts:20', - ], - }], - }, - usage: { - inputTokens: 100, - outputTokens: 25, - cacheReadInputTokens: 0, - cacheCreationInputTokens: 0, - cacheCreation5mInputTokens: 0, - cacheCreation1hInputTokens: 0, - webSearchRequests: 0, - costUSD: 0.001, - }, - }); - - const context = makeContext(); - context.pullRequest!.files = [ - { - filename: 'src/dashboard.ts', - status: 'modified', - additions: 1, - deletions: 1, - chunks: 1, - patch: [ - '@@ -10,1 +10,1 @@', - '-const range = getDefaultRange(widget);', - '+const range = widget.axisRange ?? getDefaultRange(widget);', - ].join('\n'), - }, - { - filename: 'tests/dashboard.test.ts', - status: 'modified', - additions: 1, - deletions: 1, - chunks: 1, - patch: [ - '@@ -20,1 +20,1 @@', - '-expect(chart.range).toEqual(defaultRange);', - '+expect(chart.range).toEqual(customAxisRange);', - ].join('\n'), - }, - ]; - - const prepared = prepareFiles(context); - const planned = await planSemanticReviewChunks(prepared.files, context, { - enabled: true, - runtime: 'pi', - maxChunks: 20, - maxChunkChars: 30000, - maxHunksPerChunk: 50, - }); - - const chunk = planned.groups[0]?.chunks[0]; - expect(planned.groups).toHaveLength(1); - expect(chunk).toMatchObject({ - id: 'semantic:1', - title: 'Preserve dashboard axis range', - changedLineMap: [ - { path: 'src/dashboard.ts', start: 10, end: 10 }, - { path: 'tests/dashboard.test.ts', start: 20, end: 20 }, - ], - }); - expect(chunk?.files).toHaveLength(2); - expect(chunk?.files.map((file) => file.path)).toEqual([ - 'src/dashboard.ts', - 'tests/dashboard.test.ts', - ]); - }); -}); diff --git a/packages/warden/src/semantic/planner.ts b/packages/warden/src/semantic/planner.ts deleted file mode 100644 index a9661d9f..00000000 --- a/packages/warden/src/semantic/planner.ts +++ /dev/null @@ -1,568 +0,0 @@ -import { z } from 'zod'; -import type { EventContext, UsageStats } from '../types/index.js'; -import type { ChangedLineRange, ReviewChunk } from '../diff/index.js'; -import { SkillRunnerError } from '../sdk/errors.js'; -import { getRuntime } from '../sdk/runtimes/index.js'; -import type { RuntimeName } from '../sdk/runtimes/index.js'; -import type { PreparedFile, ReviewChunkGroup } from '../sdk/types.js'; -import { createSemanticPlannerToolExecutor, SEMANTIC_PLANNER_TOOLS } from './tools.js'; - -const SemanticChunkPlanSchema = z.object({ - groups: z.array(z.object({ - title: z.string().min(5), - summary: z.string().min(20), - chunkIds: z.array(z.string().min(1)).min(1), - })), -}); - -const MAX_EMBEDDED_DIFF_CHARS = 8000; -const MAX_EMBEDDED_DIFF_CHUNKS = 12; -const MAX_EMBEDDED_DIFF_RANGES = 12; -const MAX_CHUNK_CHARS = 20000; -const MAX_HUNKS_PER_CHUNK = 4; -const MAX_CHANGED_RANGES_PER_CHUNK = 4; -const SIMILAR_TINY_RANGE_MULTIPLIER = 3; -const MAX_TINY_CHANGED_LINES = 6; - -export interface SemanticChunkPlanningOptions { - enabled?: boolean; - apiKey?: string; - runtime?: RuntimeName; - model?: string; - maxChunks?: number; - maxChunkChars?: number; - maxHunksPerChunk?: number; - maxChangedRangesPerChunk?: number; - maxEmbeddedDiffChars?: number; - maxEmbeddedDiffChunks?: number; - maxEmbeddedDiffRanges?: number; -} - -interface AtomicHunkSummary { - id: string; - path: string; - language?: string; - changedRanges: ChangedLineRange[]; - title: string; - contentMode: ReviewChunk['files'][number]['contentMode']; - additions: number; - deletions: number; - changedLinePreview: string[]; - embeddedDiff?: string; -} - -interface SemanticChunkLimits { - maxChunks: number; - maxChunkChars: number; - maxHunksPerChunk: number; - maxChangedRangesPerChunk: number; - maxEmbeddedDiffChars: number; - maxEmbeddedDiffChunks: number; - maxEmbeddedDiffRanges: number; -} - -interface SemanticChunkPlannerInput { - context: EventContext; - chunks: AtomicHunkSummary[]; - limits: SemanticChunkLimits; -} - -export interface SemanticChunkPlanningResult { - groups: ReviewChunkGroup[]; - usage?: UsageStats; -} - -function countChangedLines(chunk: ReviewChunk, prefix: '+' | '-'): number { - return chunk.files.reduce((sum, file) => { - const matches = file.content.match(new RegExp(`^\\${prefix}(?!\\${prefix}|\\+\\+)`, 'gm')); - return sum + (matches?.length ?? 0); - }, 0); -} - -function compactChangedLinePreview(chunk: ReviewChunk, maxLines = 8): string[] { - const preview: string[] = []; - for (const file of chunk.files) { - for (const line of file.content.split('\n')) { - if ((!line.startsWith('+') && !line.startsWith('-')) || line.startsWith('+++') || line.startsWith('---')) { - continue; - } - preview.push(line.length > 180 ? `${line.slice(0, 177)}...` : line); - if (preview.length >= maxLines) return preview; - } - } - return preview; -} - -function atomicHunkSummaryFromChunk(chunk: ReviewChunk): AtomicHunkSummary { - const firstFile = chunk.files[0]; - return { - id: chunk.id, - path: firstFile.path, - language: firstFile.language, - changedRanges: chunk.changedLineMap, - title: chunk.title, - contentMode: firstFile.contentMode, - additions: countChangedLines(chunk, '+'), - deletions: countChangedLines(chunk, '-'), - changedLinePreview: compactChangedLinePreview(chunk), - }; -} - -function shouldEmbedDiffs(chunks: ReviewChunk[], limits: SemanticChunkLimits): boolean { - if (chunks.length > limits.maxEmbeddedDiffChunks) return false; - const changedRanges = chunks.reduce((sum, chunk) => sum + chunk.changedLineMap.length, 0); - if (changedRanges > limits.maxEmbeddedDiffRanges) return false; - const totalChars = chunks.reduce( - (sum, chunk) => sum + chunk.files.reduce((fileSum, file) => fileSum + file.content.length, 0), - 0, - ); - return totalChars <= limits.maxEmbeddedDiffChars; -} - -function atomicHunkSummariesFromChunks(chunks: ReviewChunk[], limits: SemanticChunkLimits): AtomicHunkSummary[] { - const embedDiffs = shouldEmbedDiffs(chunks, limits); - return chunks.map((chunk) => { - const summary = atomicHunkSummaryFromChunk(chunk); - if (!embedDiffs) return summary; - - return { - ...summary, - embeddedDiff: chunk.files.map((file) => [ - `File: ${file.path}`, - file.content, - ].join('\n')).join('\n\n'), - }; - }); -} - -function formatFileInventory(context: EventContext): string { - return (context.pullRequest?.files ?? []).map((file) => [ - `${file.filename} (${file.status})`, - ` additions: ${file.additions}`, - ` deletions: ${file.deletions}`, - ` hunks: ${file.chunks}`, - ].join('\n')).join('\n'); -} - -function semanticChunkLimits(options: SemanticChunkPlanningOptions): SemanticChunkLimits { - return { - maxChunks: options.maxChunks ?? 20, - maxChunkChars: options.maxChunkChars ?? MAX_CHUNK_CHARS, - maxHunksPerChunk: options.maxHunksPerChunk ?? MAX_HUNKS_PER_CHUNK, - maxChangedRangesPerChunk: options.maxChangedRangesPerChunk ?? MAX_CHANGED_RANGES_PER_CHUNK, - maxEmbeddedDiffChars: options.maxEmbeddedDiffChars ?? MAX_EMBEDDED_DIFF_CHARS, - maxEmbeddedDiffChunks: options.maxEmbeddedDiffChunks ?? MAX_EMBEDDED_DIFF_CHUNKS, - maxEmbeddedDiffRanges: options.maxEmbeddedDiffRanges ?? MAX_EMBEDDED_DIFF_RANGES, - }; -} - -function buildSemanticChunkPlannerInput( - context: EventContext, - chunks: ReviewChunk[], - options: SemanticChunkPlanningOptions, -): SemanticChunkPlannerInput { - const limits = semanticChunkLimits(options); - return { - context, - chunks: atomicHunkSummariesFromChunks(chunks, limits), - limits, - }; -} - -function formatPlannerInput(input: SemanticChunkPlannerInput): string { - const pr = input.context.pullRequest; - const sections = [ - `Repository: ${input.context.repository.fullName}`, - pr ? `Pull request title: ${pr.title}` : undefined, - pr?.body ? `Pull request body: ${pr.body}` : undefined, - formatFileInventory(input.context) - ? ['Changed files:', formatFileInventory(input.context)].join('\n') - : undefined, - [ - 'Atomic chunks:', - input.chunks.map((chunk) => [ - `Chunk ID: ${chunk.id}`, - `Path: ${chunk.path}`, - chunk.language ? `Language: ${chunk.language}` : undefined, - `Title: ${chunk.title}`, - `Changed ranges: ${chunk.changedRanges.map((range) => `${range.path}:${range.start}-${range.end}`).join(', ')}`, - `Content mode: ${chunk.contentMode}`, - `Additions: ${chunk.additions}`, - `Deletions: ${chunk.deletions}`, - chunk.changedLinePreview.length > 0 - ? ['Changed-line preview:', ...chunk.changedLinePreview].join('\n') - : undefined, - chunk.embeddedDiff - ? ['Embedded small diff:', chunk.embeddedDiff].join('\n') - : undefined, - ].filter((line): line is string => Boolean(line)).join('\n')).join('\n\n---\n\n'), - ].join('\n'), - ]; - - return sections.filter((section): section is string => Boolean(section)).join('\n\n'); -} - -function buildSemanticChunkPlanningPrompt( - context: EventContext, - chunks: ReviewChunk[], - options: SemanticChunkPlanningOptions -): string { - const plannerInput = buildSemanticChunkPlannerInput(context, chunks, options); - const { maxChunks, maxHunksPerChunk, maxChangedRangesPerChunk, maxChunkChars } = plannerInput.limits; - const header = [ - 'You are planning code review chunks for Warden.', - '', - 'Group atomic git chunks into semantic changes.', - 'A semantic change should contain chunks that a reviewer should understand together: one behavior change, API contract, data flow, migration, validation rule, or test expectation.', - 'Collapse repeated small similar hunks into one semantic change when they apply the same transformation, update the same contract, or repeat the same expectation across call sites.', - 'For each planned group, write a semantic delta summary: what behavior, contract, data flow, API, or test expectation changed.', - 'Do not restate filenames or line ranges. Do not say "lines 10, 100, 200".', - 'Do not summarize the input shape. Explain the product, security, data-flow, API, or test behavior being changed.', - 'Keep each summary one sentence. Be concrete enough that a scanner knows why the grouped changes belong together.', - 'Inspect the changed code with tools before finalizing the plan, especially when grouping across files or distant hunks.', - 'Use read_review_chunk to inspect exact hunk content for non-embedded chunks; read_changed_file only shows current head content.', - 'If embedded small diffs are present, use them as initial evidence and use tools only for missing relationships or context.', - `Target at most ${maxChunks} planned groups.`, - `Warden may split a planned semantic change into bounded scanner chunks after planning.`, - `Each scanner chunk will use at most ${maxHunksPerChunk} atomic chunks.`, - `Each scanner chunk will use at most ${maxChangedRangesPerChunk} changed line ranges.`, - `Each scanner chunk will stay under roughly ${maxChunkChars} characters once materialized.`, - `Embedded small diffs are only present when the changeset fits these planner limits: ${plannerInput.limits.maxEmbeddedDiffChars} chars, ${plannerInput.limits.maxEmbeddedDiffChunks} chunks, ${plannerInput.limits.maxEmbeddedDiffRanges} changed ranges.`, - 'Every input Chunk ID must appear in exactly one group. Do not invent Chunk IDs.', - ].filter((line): line is string => Boolean(line)); - - return [ - ...header, - '', - 'Return JSON with this exact shape:', - '{"groups":[{"title":"semantic title","summary":"semantic delta summary","chunkIds":["chunk id"]}]}', - '', - formatPlannerInput(plannerInput), - ].join('\n'); -} - -function mergeSourceLines(chunks: ReviewChunk[], path: string) { - const byLine = new Map(); - for (const chunk of chunks) { - for (const file of chunk.files.filter((file) => file.path === path)) { - for (const line of file.sourceLines) { - byLine.set(line.line, line.content); - } - } - } - - return Array.from(byLine, ([line, content]) => ({ line, content })) - .sort((a, b) => a.line - b.line); -} - -function changedDiffLines(content: string): string[] { - return content.split('\n').filter((line) => - (line.startsWith('+') || line.startsWith('-')) && !line.startsWith('+++') && !line.startsWith('---') - ); -} - -function countChangedDiffLines(chunk: ReviewChunk): number { - return chunk.files.reduce((sum, file) => sum + changedDiffLines(file.content).length, 0); -} - -function normalizeChangedLine(line: string): string { - const sign = line[0]; - return `${sign}${line - .slice(1) - .replace(/(['"`])(?:\\.|(?!\1).)*\1/g, '') - .replace(/\b\d+(?:\.\d+)?\b/g, '') - .replace(/\s+/g, ' ') - .trim()}`; -} - -function similarityKey(chunk: ReviewChunk): string | undefined { - if (countChangedDiffLines(chunk) > MAX_TINY_CHANGED_LINES) return undefined; - - const firstFile = chunk.files[0]; - if (!firstFile) return undefined; - - const normalized = chunk.files - .flatMap((file) => changedDiffLines(file.content).map(normalizeChangedLine)) - .filter(Boolean); - if (normalized.length === 0) return undefined; - - return [ - firstFile.language, - normalized.join('\n'), - ].join(':'); -} - -function orderedSimilarityBatches(chunks: ReviewChunk[]): ReviewChunk[][] { - const batches: ReviewChunk[][] = []; - const indexByKey = new Map(); - - for (const chunk of chunks) { - const key = similarityKey(chunk); - if (!key) { - batches.push([chunk]); - continue; - } - - const existingIndex = indexByKey.get(key); - if (existingIndex !== undefined) { - batches[existingIndex]?.push(chunk); - continue; - } - - indexByKey.set(key, batches.length); - batches.push([chunk]); - } - - return batches; -} - -function compactRepeatedHunkContent(files: ReviewChunk['files'][number][]): string { - return files.map((file) => { - const lines = file.content.split('\n'); - const compactLines = lines.filter((line) => - line.startsWith('## Hunk:') || - line.startsWith('## Scope:') || - line.startsWith('@@') || - ((line.startsWith('+') || line.startsWith('-')) && !line.startsWith('+++') && !line.startsWith('---')) - ); - return compactLines.join('\n'); - }).join('\n\n---\n\n'); -} - -function materializePlannedChunk( - index: number, - title: string, - summary: string, - chunks: ReviewChunk[], - options: { compact?: boolean } = {} -): ReviewChunk { - const changedLineMap = chunks.flatMap((chunk) => chunk.changedLineMap); - const paths = [...new Set(chunks.flatMap((chunk) => chunk.files.map((file) => file.path)))]; - const files = paths.map((path) => { - const chunkFiles = chunks.flatMap((chunk) => chunk.files.filter((file) => file.path === path)); - const first = chunkFiles[0]; - if (!first) { - throw new Error(`Cannot materialize planned chunk without file content for ${path}`); - } - - return { - path, - language: first.language, - changedRanges: changedLineMap.filter((range) => range.path === path), - content: options.compact - ? compactRepeatedHunkContent(chunkFiles) - : chunkFiles.map((file) => file.content).join('\n\n---\n\n'), - contentMode: 'raw-hunks' as const, - sourceLines: mergeSourceLines(chunks, path), - }; - }); - const firstFile = files[0]; - if (!firstFile) { - throw new Error('Cannot materialize planned chunk without files'); - } - - return { - id: `semantic:${index + 1}`, - title, - summary, - files: [firstFile, ...files.slice(1)], - changedLineMap, - }; -} - -function reviewChunkContentChars(chunk: ReviewChunk): number { - return chunk.files.reduce((sum, file) => sum + file.content.length, 0); -} - -function plannedChunkTitle(title: string, partIndex: number, totalParts: number): string { - return totalParts <= 1 ? title : `${title} (${partIndex}/${totalParts})`; -} - -function splitPlannedChunks( - startIndex: number, - title: string, - summary: string, - chunks: ReviewChunk[], - limits: { maxChunkChars: number; maxHunksPerChunk: number; maxChangedRangesPerChunk: number } -): ReviewChunk[] { - const batches: ReviewChunk[][] = []; - let current: ReviewChunk[] = []; - const plannedBatches = orderedSimilarityBatches(chunks); - - const pushCurrent = () => { - if (current.length > 0) { - batches.push(current); - current = []; - } - }; - - const addChunk = (chunk: ReviewChunk, maxHunks: number, maxChangedRanges: number) => { - const singleChunk = materializePlannedChunk(startIndex + batches.length, title, summary, [chunk]); - if (reviewChunkContentChars(singleChunk) > limits.maxChunkChars) { - throw new SkillRunnerError( - `Semantic chunk planning cannot split atomic chunk ${chunk.id} under maxChunkChars ${limits.maxChunkChars}`, - { code: 'sdk_error' }, - ); - } - - if (current.length === 0) { - current = [chunk]; - return; - } - - const candidate = [...current, chunk]; - const candidateChunk = materializePlannedChunk(startIndex + batches.length, title, summary, candidate); - const exceedsHunks = candidate.length > maxHunks; - const exceedsChangedRanges = candidateChunk.changedLineMap.length > maxChangedRanges; - const exceedsChars = reviewChunkContentChars(candidateChunk) > limits.maxChunkChars; - - if (exceedsHunks || exceedsChangedRanges || exceedsChars) { - pushCurrent(); - current = [chunk]; - } else { - current = candidate; - } - }; - - for (const plannedBatch of plannedBatches) { - const compactSimilar = plannedBatch.length > 1; - if (compactSimilar) pushCurrent(); - - const maxHunks = compactSimilar - ? limits.maxHunksPerChunk * SIMILAR_TINY_RANGE_MULTIPLIER - : limits.maxHunksPerChunk; - const maxChangedRanges = compactSimilar - ? limits.maxChangedRangesPerChunk * SIMILAR_TINY_RANGE_MULTIPLIER - : limits.maxChangedRangesPerChunk; - - for (const chunk of plannedBatch) { - addChunk(chunk, maxHunks, maxChangedRanges); - } - - if (compactSimilar) pushCurrent(); - } - - pushCurrent(); - - return batches.map((batch, batchIndex) => { - const compact = batch.length > 1 && new Set(batch.map(similarityKey)).size === 1; - return materializePlannedChunk( - startIndex + batchIndex, - plannedChunkTitle(title, batchIndex + 1, batches.length), - summary, - batch, - { compact }, - ); - }); -} - -function groupFromPreparedFile(file: PreparedFile): ReviewChunkGroup { - const filenames = [...new Set(file.chunks.flatMap((chunk) => chunk.files.map((chunkFile) => chunkFile.path)))]; - return { - displayName: file.filename, - filenames: filenames.length > 0 ? filenames : [file.filename], - chunks: file.chunks, - }; -} - -function groupFromPlannedChunks(title: string, chunks: ReviewChunk[]): ReviewChunkGroup { - const filenames = [...new Set(chunks.flatMap((chunk) => chunk.files.map((file) => file.path)))]; - return { - displayName: title, - filenames, - chunks, - }; -} - -/** Plan and materialize semantic review chunk groups with a model-backed planning pass. */ -export async function planSemanticReviewChunks( - files: PreparedFile[], - context: EventContext, - options: SemanticChunkPlanningOptions = {} -): Promise { - const chunks = files.flatMap((file) => file.chunks); - if (!options.enabled || chunks.length === 0) { - return { groups: files.map(groupFromPreparedFile) }; - } - - const runtimeName = options.runtime ?? 'pi'; - const result = await getRuntime(runtimeName).runAuxiliary({ - task: 'semantic_chunking', - agentName: 'semantic-chunk-planner', - apiKey: options.apiKey, - prompt: buildSemanticChunkPlanningPrompt(context, chunks, options), - schema: SemanticChunkPlanSchema, - tools: SEMANTIC_PLANNER_TOOLS, - executeTool: createSemanticPlannerToolExecutor(context, chunks), - maxIterations: 5, - model: options.model, - }); - - if (!result.success) { - throw new SkillRunnerError(`Semantic chunk planning failed: ${result.error}`, { - code: 'sdk_error', - }); - } - - const chunksById = new Map(chunks.map((chunk) => [chunk.id, chunk])); - const seen = new Set(); - const plannedGroups: ReviewChunkGroup[] = []; - let plannedChunkCount = 0; - const maxChunks = options.maxChunks ?? 20; - const maxHunksPerChunk = options.maxHunksPerChunk ?? MAX_HUNKS_PER_CHUNK; - const maxChangedRangesPerChunk = options.maxChangedRangesPerChunk ?? MAX_CHANGED_RANGES_PER_CHUNK; - const maxChunkChars = options.maxChunkChars ?? MAX_CHUNK_CHARS; - - if (result.data.groups.length > maxChunks) { - throw new SkillRunnerError( - `Semantic chunk planning returned ${result.data.groups.length} groups, exceeding maxChunks ${maxChunks}`, - { code: 'sdk_error' }, - ); - } - - for (const group of result.data.groups) { - const groupChunks: ReviewChunk[] = []; - for (const chunkId of group.chunkIds) { - if (seen.has(chunkId)) { - throw new SkillRunnerError(`Semantic chunk planning assigned chunk ${chunkId} more than once`, { - code: 'sdk_error', - }); - } - const chunk = chunksById.get(chunkId); - if (!chunk) { - throw new SkillRunnerError(`Semantic chunk planning returned unknown chunk ${chunkId}`, { - code: 'sdk_error', - }); - } - seen.add(chunkId); - groupChunks.push(chunk); - } - const chunksForGroup = splitPlannedChunks(plannedChunkCount, group.title, group.summary, groupChunks, { - maxChunkChars, - maxHunksPerChunk, - maxChangedRangesPerChunk, - }); - plannedChunkCount += chunksForGroup.length; - plannedGroups.push(groupFromPlannedChunks(group.title, chunksForGroup)); - - if (plannedChunkCount > maxChunks) { - throw new SkillRunnerError( - `Semantic chunk planning materialized ${plannedChunkCount} chunks, exceeding maxChunks ${maxChunks}`, - { code: 'sdk_error' }, - ); - } - } - - const missing = chunks.filter((chunk) => !seen.has(chunk.id)); - if (missing.length > 0) { - throw new SkillRunnerError( - `Semantic chunk planning omitted ${missing.length} chunk${missing.length === 1 ? '' : 's'}`, - { code: 'sdk_error' }, - ); - } - - return { - groups: plannedGroups, - usage: result.usage, - }; -} diff --git a/packages/warden/src/semantic/tools.test.ts b/packages/warden/src/semantic/tools.test.ts deleted file mode 100644 index 8c8fa193..00000000 --- a/packages/warden/src/semantic/tools.test.ts +++ /dev/null @@ -1,93 +0,0 @@ -import { mkdtemp, rm, writeFile, mkdir } from 'node:fs/promises'; -import { join } from 'node:path'; -import { tmpdir } from 'node:os'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; -import type { EventContext } from '../types/index.js'; -import type { ReviewChunk } from '../diff/index.js'; -import { createSemanticPlannerToolExecutor } from './tools.js'; - -describe('createSemanticPlannerToolExecutor', () => { - let tempDir: string; - - beforeEach(async () => { - tempDir = await mkdtemp(join(tmpdir(), 'warden-semantic-tools-')); - await mkdir(join(tempDir, 'src'), { recursive: true }); - await writeFile(join(tempDir, 'src/dashboard.ts'), 'export const axisRange = widget.axisRange;\n'); - await writeFile(join(tempDir, 'src/secret.ts'), 'export const unrelated = true;\n'); - }); - - afterEach(async () => { - await rm(tempDir, { recursive: true, force: true }); - }); - - function makeContext(): EventContext { - return { - eventType: 'pull_request', - action: 'opened', - repository: { - owner: 'qa', - name: 'repo', - fullName: 'qa/repo', - defaultBranch: 'main', - }, - repoPath: tempDir, - pullRequest: { - number: 1, - title: 'Preserve dashboard range', - body: '', - author: 'qa', - baseBranch: 'main', - headBranch: 'feature', - headSha: 'head', - baseSha: 'base', - files: [{ - filename: 'src/dashboard.ts', - status: 'modified', - additions: 1, - deletions: 1, - patch: '@@ -1,1 +1,1 @@\n-old\n+new', - }], - }, - }; - } - - it('only reads changed files inside the repository', async () => { - const executeTool = createSemanticPlannerToolExecutor(makeContext(), []); - - await expect(executeTool('read_changed_file', { path: 'src/dashboard.ts' })) - .resolves.toContain('axisRange'); - await expect(executeTool('read_changed_file', { path: 'src/secret.ts' })) - .resolves.toBe('Refusing to read a file that is not in the changed file list'); - await expect(executeTool('read_changed_file', { path: '../outside.ts' })) - .resolves.toBe('Refusing to read a file that is not in the changed file list'); - }); - - it('searches the repository with capped output', async () => { - const executeTool = createSemanticPlannerToolExecutor(makeContext(), []); - - await expect(executeTool('search_repo', { query: 'axisRange' })) - .resolves.toContain('src/dashboard.ts:1'); - }); - - it('reads exact review chunk diffs by chunk id', async () => { - const chunk: ReviewChunk = { - id: 'src/dashboard.ts:1', - title: 'src/dashboard.ts:1', - changedLineMap: [{ path: 'src/dashboard.ts', start: 1, end: 1 }], - files: [{ - path: 'src/dashboard.ts', - language: 'typescript', - changedRanges: [{ path: 'src/dashboard.ts', start: 1, end: 1 }], - content: '@@ -1,1 +1,1 @@\n-old\n+new', - contentMode: 'raw-hunks', - sourceLines: [{ line: 1, content: 'new' }], - }], - }; - const executeTool = createSemanticPlannerToolExecutor(makeContext(), [chunk]); - - await expect(executeTool('read_review_chunk', { chunkId: 'src/dashboard.ts:1' })) - .resolves.toContain('-old'); - await expect(executeTool('read_review_chunk', { chunkId: 'missing' })) - .resolves.toBe('Unknown review chunk ID'); - }); -}); diff --git a/packages/warden/src/semantic/tools.ts b/packages/warden/src/semantic/tools.ts deleted file mode 100644 index ae8bfb80..00000000 --- a/packages/warden/src/semantic/tools.ts +++ /dev/null @@ -1,156 +0,0 @@ -import { execFile } from 'node:child_process'; -import { readFile } from 'node:fs/promises'; -import { resolve, relative, isAbsolute } from 'node:path'; -import { promisify } from 'node:util'; -import { z } from 'zod'; -import type { EventContext } from '../types/index.js'; -import type { ReviewChunk } from '../diff/index.js'; -import type { AuxiliaryTool } from '../sdk/runtimes/index.js'; - -const execFileAsync = promisify(execFile); -const MAX_FILE_CHARS = 20000; -const MAX_SEARCH_CHARS = 12000; -const MAX_CHUNK_DIFF_CHARS = 12000; - -export const SEMANTIC_PLANNER_TOOLS: AuxiliaryTool[] = [ - { - name: 'read_review_chunk', - description: 'Read exact diff content for one atomic review chunk by Chunk ID. Use this to inspect the actual delta without embedding the whole patch.', - inputSchema: { - type: 'object' as const, - properties: { - chunkId: { type: 'string', description: 'Atomic review chunk ID from the planner input' }, - }, - required: ['chunkId'], - }, - }, - { - name: 'read_changed_file', - description: 'Read the current head content for a changed file in this pull request. Output is capped.', - inputSchema: { - type: 'object' as const, - properties: { - path: { type: 'string', description: 'Changed file path to read' }, - }, - required: ['path'], - }, - }, - { - name: 'search_repo', - description: 'Search the repository with ripgrep for imports, symbols, and usage relationships. Output is capped.', - inputSchema: { - type: 'object' as const, - properties: { - query: { type: 'string', description: 'Literal or regex query to search for' }, - path: { type: 'string', description: 'Optional path or directory to limit the search' }, - }, - required: ['query'], - }, - }, -]; - -const ReadChangedFileInput = z.object({ - path: z.string().min(1), -}); - -const ReadReviewChunkInput = z.object({ - chunkId: z.string().min(1), -}); - -const SearchRepoInput = z.object({ - query: z.string().min(1).max(200), - path: z.string().min(1).optional(), -}); - -function safeRepoPath(repoPath: string, inputPath: string): string | undefined { - if (isAbsolute(inputPath)) return undefined; - const absolute = resolve(repoPath, inputPath); - const repoRelative = relative(repoPath, absolute); - if (repoRelative.startsWith('..') || isAbsolute(repoRelative)) return undefined; - return absolute; -} - -function capText(text: string, maxChars: number): string { - if (text.length <= maxChars) return text; - return `${text.slice(0, maxChars)}\n[truncated at ${maxChars} chars]`; -} - -function errorStdout(error: unknown): string { - if ( - typeof error === 'object' - && error !== null - && 'stdout' in error - && ((typeof error.stdout === 'string') || Buffer.isBuffer(error.stdout)) - ) { - return String(error.stdout); - } - return ''; -} - -/** Create the read-only tool executor used by the semantic chunk planner. */ -export function createSemanticPlannerToolExecutor( - context: EventContext, - chunks: ReviewChunk[], -): (name: string, input: Record) => Promise { - const changedFiles = new Set((context.pullRequest?.files ?? []).map((file) => file.filename)); - const chunksById = new Map(chunks.map((chunk) => [chunk.id, chunk])); - - return async (name: string, input: Record): Promise => { - if (name === 'read_review_chunk') { - const parsed = ReadReviewChunkInput.safeParse(input); - if (!parsed.success) return `Invalid input: ${parsed.error.message}`; - - const chunk = chunksById.get(parsed.data.chunkId); - if (!chunk) return 'Unknown review chunk ID'; - - return capText(chunk.files.map((file) => [ - `File: ${file.path}`, - `Changed ranges: ${file.changedRanges.map((range) => `${range.path}:${range.start}-${range.end}`).join(', ')}`, - file.content, - ].join('\n')).join('\n\n---\n\n'), MAX_CHUNK_DIFF_CHARS); - } - - if (name === 'read_changed_file') { - const parsed = ReadChangedFileInput.safeParse(input); - if (!parsed.success) return `Invalid input: ${parsed.error.message}`; - if (!changedFiles.has(parsed.data.path)) return 'Refusing to read a file that is not in the changed file list'; - - const absolutePath = safeRepoPath(context.repoPath, parsed.data.path); - if (!absolutePath) return 'Invalid path'; - - try { - return capText(await readFile(absolutePath, 'utf8'), MAX_FILE_CHARS); - } catch (error) { - return `Unable to read file: ${error instanceof Error ? error.message : String(error)}`; - } - } - - if (name === 'search_repo') { - const parsed = SearchRepoInput.safeParse(input); - if (!parsed.success) return `Invalid input: ${parsed.error.message}`; - - const searchPath = parsed.data.path ? safeRepoPath(context.repoPath, parsed.data.path) : context.repoPath; - if (!searchPath) return 'Invalid path'; - - try { - const { stdout } = await execFileAsync('rg', [ - '--line-number', - '--max-count', - '40', - parsed.data.query, - searchPath, - ], { - cwd: context.repoPath, - maxBuffer: MAX_SEARCH_CHARS * 2, - }); - return capText(stdout, MAX_SEARCH_CHARS); - } catch (error) { - const maybeStdout = errorStdout(error); - if (maybeStdout) return capText(maybeStdout, MAX_SEARCH_CHARS); - return 'No matches found'; - } - } - - return `Unknown tool: ${name}`; - }; -} diff --git a/skills/warden/references/config-schema.md b/skills/warden/references/config-schema.md index 0d4f670d..dfb829c7 100644 --- a/skills/warden/references/config-schema.md +++ b/skills/warden/references/config-schema.md @@ -56,17 +56,14 @@ maxRetries = 5 # Retries for auxiliary structured calls [defaults.synthesis] model = "anthropic/claude-opus-4-5" # Consolidation and generated-skill build model +[defaults.chunking] +enabled = true # Enable hunk-based chunking + [defaults.chunking.coalesce] enabled = true # Merge nearby hunks maxGapLines = 30 # Lines between hunks to merge maxChunkSize = 8000 # Max chars per chunk -[defaults.chunking.semantic] -enabled = false # Experimental semantic grouping opt-in -maxChunks = 20 # Target max review chunks after grouping -maxChunkChars = 30000 # Max chars per grouped review chunk -maxHunksPerChunk = 50 # Max raw hunks per grouped review chunk - [[defaults.chunking.filePatterns]] pattern = "*.config.*" # Glob pattern mode = "whole-file" # per-hunk | whole-file | skip diff --git a/skills/warden/references/configuration.md b/skills/warden/references/configuration.md index feae9d8b..2ddb81b6 100644 --- a/skills/warden/references/configuration.md +++ b/skills/warden/references/configuration.md @@ -84,15 +84,6 @@ pattern = "*.config.*" mode = "whole-file" ``` -**Experiment with semantic grouping:** -```toml -[defaults.chunking.semantic] -enabled = false -maxChunks = 20 -maxChunkChars = 30000 -maxHunksPerChunk = 50 -``` - ## Model Lanes Warden uses different model lanes for different kinds of work: diff --git a/specs/chunking-strategy-benchmark.md b/specs/chunking-strategy-benchmark.md index 7bf0613d..ad51d727 100644 --- a/specs/chunking-strategy-benchmark.md +++ b/specs/chunking-strategy-benchmark.md @@ -71,8 +71,8 @@ call-site edits. The point of this benchmark is to preserve the real fragmentation shape. The benchmark should run against a real `getsentry/sentry` checkout, not the -current fixture-only eval repository. The semantic planner has read-only tools, -and those tools are only meaningful when surrounding repository context exists. +current fixture-only eval repository. Chunking changes need surrounding +repository context to expose the same prompt shapes Warden sees in production. ## Initial Cases @@ -85,7 +85,7 @@ Start with these cases from `packages/evals/code-review/`. | `sentry-fixability-missing-issue-summary` | `4199c6aeed84c7c359aa7ad6863534174769d436` | `62125c6514958cd89aa3cf7374be32f984adb683` | 4 files, 20 hunks | Fixability calculation misses the cached issue summary needed by Seer. | | `sentry-workflow-status-missing-foreign-key` | `e36f46a85cf4a6c9a6ae0e5e545a7c13b789d478` | `f4cc09c52e73c2ab60a3b14291c60dd0db5458a7` | 2 files, 17 hunks | Workflow status processing fails hard on missing or deleted foreign keys. | -These four cover the core semantic chunking risks: +These four cover the core chunking risks: - many tiny hunks that describe one behavior change - cross-file implementation and test updates diff --git a/specs/semantic-review-chunks.md b/specs/semantic-review-chunks.md deleted file mode 100644 index d76a3f0e..00000000 --- a/specs/semantic-review-chunks.md +++ /dev/null @@ -1,590 +0,0 @@ -# Semantic Review Chunks - -Status: historical experiment. The current chunking benchmark rejects semantic -grouping as an efficiency strategy: it did not produce same-or-better findings -with lower wall time or total cost. Keep this document as design context for -the experiment, not as the accepted optimization plan. - -Warden currently prepares code for review from git hunks. Git hunks are useful -for changed-line anchoring, but they are a poor unit of review. A single logical -change can produce dozens of tiny hunks, especially in tests, generated catalogs, -or repeated call-site updates. Reviewing each tiny hunk independently repeats -prompt setup, repeats codebase exploration, increases cost, and can hide the -shape of the actual change. - -Semantic review chunks make the scanner read a coherent change unit while -Warden still uses git hunks as the source of truth for where inline comments may -land. - -## Current Behavior - -The current pipeline is: - -```text -git patch - -> parse file diffs into hunks - -> split large hunks - -> coalesce nearby hunks by line distance and size - -> expand context around each hunk - -> run one scanner call per hunk-like chunk -``` - -This is simple and safe, but it only fixes nearby fragmentation. If a file has -50 small hunks spread across distant test cases, Warden still makes many scanner -calls for one logical change. - -## Desired Outcome - -The scanner should receive a larger review packet when that better matches how a -human would review the change. - -Examples: - -- one test file with many small related changes becomes one review chunk with - the whole file or a stitched excerpt -- one implementation change plus related tests becomes one semantic chunk when - both sides are needed to understand the behavior -- one behavior change touching several files remains one semantic chunk with - multiple file payloads under the same summary -- unrelated edits in the same file stay separate chunks -- very large or generated files stay governed by scan policy and existing skip - behavior - -The target pipeline is: - -```text -git patch - -> atomic hunk inventory - -> semantic planning module - -> ReviewChunk materialization - -> run one scanner call per ReviewChunk - -> validate findings against changedLineMap -``` - -Git hunks remain the evidence and anchoring primitive. Review chunks become the -scanner-facing primitive. - -Benchmarking for this work is tracked in -[`chunking-strategy-benchmark.md`](chunking-strategy-benchmark.md). - -## Module Boundary - -Semantic planning should be cleanly encapsulated in its own module because it is -an input/output planning pass, not scanner logic. The rest of Warden should not -know about planner prompts, tool definitions, model selection details, or -planner validation internals. - -Current layout: - -```text -packages/warden/src/semantic/ - index.ts - planner.ts - tools.ts -``` - -Split `inventory.ts`, `materialize.ts`, or `types.ts` out later only if the -module grows enough that the separation removes real complexity. - -Public surface: - -```ts -export interface SemanticChunkPlannerInput { - context: EventContext; - chunks: AtomicHunkSummary[]; - limits: SemanticChunkLimits; -} - -``` - -The integration point remains thin: - -```text -prepareFiles() - -> atomic ReviewChunks -semantic.planSemanticReviewChunks(...) - -> semantic ReviewChunkGroups -run scanner on groups -``` - -`prepareFiles` remains responsible for parsing diffs and creating deterministic -atomic chunks. The semantic module owns only the decision of which atomic chunks -belong together and why. - -## ReviewChunk Contract - -```ts -export interface ReviewChunk { - id: string; - title: string; - summary?: string; - files: ReviewChunkFile[]; - changedLineMap: ChangedLineRange[]; -} - -export interface ReviewChunkFile { - path: string; - changedRanges: ChangedLineRange[]; - content: string; - contentMode: 'whole-file' | 'stitched-file' | 'raw-hunks'; -} - -export interface ChangedLineRange { - path: string; - start: number; - end: number; -} -``` - -`title` is a stable label for progress, logging, and trace output. Deterministic -chunking may use filenames and changed ranges. - -`summary` is optional and planner-owned. It must only be set when a semantic -planner has described the logical change. Deterministic grouping must not -populate `summary` with filenames or changed ranges and call that semantic. -Scanner prompts must treat summaries as grouping hints, not evidence that the -change is intended or correct. - -`files[].content` is the readable review packet. It can be larger than a hunk -and may include unchanged surrounding code. This content is for understanding. -One semantic change may include multiple files when the same logical change -spans implementation, tests, config, or call sites. That semantic change may -still materialize as multiple bounded `ReviewChunk` scanner calls when the -group would otherwise be too large. The chunk title and summary describe the -shared change; each `files[]` entry carries the file-local content needed to -review that scanner slice. - -`changedLineMap` is the hard validation boundary. A scanner finding may only -anchor to a line inside this map. Surrounding content can explain a finding but -cannot be used as the comment location. - -## Atomic Hunk Inventory - -Before planning, Warden should normalize parsed hunks into stable atomic units: - -```ts -export interface AtomicHunkSummary { - id: string; - path: string; - language?: string; - oldStart: number; - oldEnd: number; - newStart: number; - newEnd: number; - header?: string; - symbolHint?: string; - additions: number; - deletions: number; - changedLinePreview?: string[]; -} -``` - -The planner operates on this inventory. It does not invent changed lines. Each -atomic hunk is either assigned to exactly one review chunk or excluded by an -existing scan/ignore policy before planning. - -The planner should be tool-backed on every semantic run. It should have enough -compact metadata to decide where to inspect, then use read-only tools to inspect -the changed code before finalizing the grouping. - -Full content is still primarily for scanner execution after semantic grouping. -Planning input should stay small enough that the planner can reason over the -whole changeset without reproducing the same token cost that semantic chunking -is meant to avoid. - -Small diffs can be embedded directly as an optimization. The cutoff should be -tunable because model context windows and price/performance profiles vary. A -conservative default is: - -- embed full atomic hunk content only when the total planner diff payload is at - most 8k characters -- embed only when there are at most 12 atomic chunks -- embed only when there are at most 12 changed ranges after deterministic - preparation -- otherwise send compact metadata plus capped changed-line previews and require - tool inspection - -This keeps tiny PRs cheap while preserving the main goal for pathological PRs: -avoid passing a huge fragmented patch to the planner. - -The knobs belong under semantic chunking config, for example: - -```toml -[defaults.chunking.semantic] -maxEmbeddedDiffChars = 8000 -maxEmbeddedDiffChunks = 12 -maxEmbeddedDiffRanges = 12 -``` - -Smaller-context or expensive models can set these lower, including `0` to force -metadata-plus-tools planning. - -Allowed compact inputs include: - -- repository metadata -- PR title and body -- commit messages, when available -- changed file list and file statuses -- atomic chunk ids -- file paths and languages -- changed ranges -- hunk headers and symbol hints -- additions, deletions, and hunk counts -- small capped changed-line previews when useful -- full atomic hunk content only when the whole semantic planning input is below - the small-diff threshold - -Disallowed by default: - -- full rendered hunk content for medium or large changesets -- whole changed files -- scanner skill prompts -- full repository context - -## Planner - -The semantic chunk planner groups atomic hunks into review chunks and writes a -semantic summary for each planned group. - -Planner input: - -- repository and PR metadata -- PR title and body -- commit messages, when available -- changed file list -- atomic hunk inventory with compact metadata -- file sizes and line counts where available -- hard limits for chunk size and count - -Planner output: - -```ts -export interface PlannedReviewChunk { - title: string; - summary: string; - chunkIds: string[]; -} -``` - -Current materialization keeps planned groups as `raw-hunks` review chunks. Whole -file and stitched-file materializers remain future work. - -The planner should optimize for scanner usefulness, not for diff prettiness. -Good chunks are cohesive enough that the scanner can understand the change in -one pass and small enough that the scanner can stay precise. -Cross-file groups are allowed and expected when the files are part of the same -semantic delta. A source change and its test assertion should usually stay -together if reviewing them independently would hide the behavior change. - -The planner should use the primary scanner model by default, not the auxiliary -model. Semantic grouping is a hard reasoning task: it must infer relationships -between files, tests, call sites, and behavior. Auxiliary models remain -appropriate for extraction repair and other bounded post-processing work. - -### Tool-Using Planning Agent - -The planner should be a tool-using read-only agent, not only a single schema -call over a rendered prompt. It should always have tools available and should -inspect code before finalizing a plan. Embedded small diffs can reduce tool -turns, but tools remain the primary way to resolve cross-file relationships, -symbol usage, and behavior. - -Read-only tools should include: - -- read changed file content at head -- read base content or changed-range context -- inspect nearby symbols/functions -- search usages/imports with bounded `rg` -- inspect PR metadata and commit messages - -The planner must not have write tools. It should also not receive an unbounded -repo dump. Tool calls should be driven by grouping uncertainty: inspect enough -code to decide which chunks belong together, then stop. - -The scanner remains responsible for finding issues. The planner is responsible -only for grouping and writing semantic summaries that explain why the grouped -changes should be reviewed together. - -## Materialization - -Materialization turns a planned chunk into the scanner-facing `ReviewChunk`. -For cross-file groups, materialization keeps one shared `ReviewChunk` and -creates one `ReviewChunkFile` per path. It must not flatten file contents into a -single synthetic file, because extraction and changed-line validation depend on -real paths. - -Content modes: - -| Mode | Use When | Content | -|------|----------|---------| -| `whole-file` | File is below configured line/byte limits and many small hunks are spread across it | Current file contents | -| `stitched-file` | File is too large for whole-file but related hunks need broad structure | Ordered excerpts around changed ranges, with omitted sections marked | -| `raw-hunks` | Planning is unnecessary or the change is already compact | Existing formatted hunks with context | - -For test files with many tiny hunks, `whole-file` should usually be preferred -when file limits allow it. Tests often need the surrounding `describe`/`it` -structure to make the change reviewable. - -## Finding Validation - -Scanner prompts must say that findings can only anchor to changed lines in the -review chunk's changed-line map. - -Warden must also enforce that rule after extraction: - -- a finding with no location may remain a general finding -- a finding with a location is accepted only if `location.path` and - `location.startLine` fall inside `changedLineMap` -- multi-line findings must be fully contained by a changed range -- out-of-range findings are dropped and recorded in telemetry - -This replaces the current single hunk-range check with a multi-range check. - -## Configuration - -Semantic review chunks should be configured with a small public surface: - -```toml -[defaults.chunking.semantic] -enabled = true -maxChunks = 20 -maxChunkChars = 20000 -maxHunksPerChunk = 4 -maxChangedRangesPerChunk = 4 -maxEmbeddedDiffChars = 8000 -maxEmbeddedDiffChunks = 12 -maxEmbeddedDiffRanges = 12 -``` - -Do not expose fallback behavior as config. If semantic planning fails mechanical -validation or cannot run, any recovery behavior should remain internal. Users -should not need to choose recovery strategy. - -## Validation Rules - -Planner output must pass deterministic validation before scanner execution: - -- every planned `hunkId` exists -- no hunk is assigned to more than one chunk -- every included hunk is assigned to a chunk -- every planned path exists in the changed file set -- each chunk respects hard size and hunk-count limits after materialization -- each `changedLineMap` range comes from assigned atomic hunks -- chunk ids are stable within the run and unique - -Invalid plans are not partially trusted. - -## Prompt Changes - -The scanner task should move from hunk-specific language to chunk-specific -language: - -```text -Analyze this review chunk according to the skill criteria. - -If a semantic summary is present, use it as planner context for why these -changed ranges are being reviewed together. File content may include unchanged -surrounding code for context. Only report findings covered by the skill -instructions, and only anchor locations to lines listed in the changed-line map. -``` - -The JSON output schema can stay mostly unchanged. The location rules need to -reference changed-line maps instead of a single hunk range. - -## Telemetry - -Warden should record enough data to prove whether semantic chunking helps: - -- original atomic hunk count -- planned review chunk count -- materialized chunk count -- content mode counts -- planner duration and usage -- scanner duration and usage -- internal recovery reason when semantic chunking is not used -- number of dropped out-of-range findings - -This should make cost and latency changes visible without reading logs line by -line. - -## User-Facing Visibility - -Semantic segments may be useful beyond internal execution. Warden should be able -to expose them in debug or verbose output, and possibly in pull request reports, -without making them noisy by default. - -Useful fields to expose: - -- segment title -- semantic summary -- files included -- changed ranges included -- atomic chunk count -- scanner chunk count -- content mode per file -- scanner result count for the segment - -CLI presentation can show a compact plan before scanning when verbose output is -enabled: - -```text -Semantic plan: 7 review chunks from 42 atomic hunks - 1. Enforce project access in token lookup - api/tokens.ts, auth/permissions.ts, api/tokens.test.ts - 2. Update dashboard range validation - dashboard/range.ts, dashboard/range.test.ts -``` - -Pull request presentation should be more conservative. A collapsed report -section can help reviewers understand why findings came from a cross-file -chunk, but inline comments should remain focused on findings. The plan should -not become another review surface unless it clearly helps explain Warden's -coverage and cost. - -## Benchmarking and Evals - -Semantic chunking needs more than a chunk-count benchmark. We need fixtures -where semantic grouping should materially improve or preserve scanner quality. - -Good fixtures: - -- real commit or PR SHA -- known security regression or confirmed product bug -- objective expected finding -- issue spans multiple files or many tiny hunks -- baseline hunk mode is worse in recall, duplicates, cost, or model calls - -Preferred security examples: - -- removed authorization check -- unsafe command execution -- path traversal -- SSRF -- SQL/query injection -- secret exposure -- unsafe deserialization - -Each fixture should run the same scanner twice: - -```text -same fixture + same skill + semantic off -same fixture + same skill + semantic on -``` - -Compare: - -- expected finding found or missed -- severity and confidence -- location correctness -- duplicate findings -- total findings -- chunk count -- model calls -- input and output tokens -- cost -- wall time - -The eval should also inspect planner output directly: - -- at least one expected semantic group exists -- group summary describes behavior, not files or line numbers -- cross-file changes can be grouped when they are one logical change -- unrelated edits remain separate - -The benchmark corpus already supports known Sentry vulnerabilities. We should -reuse that machinery where possible, but semantic chunking needs fixtures -selected for cross-file or many-hunk behavior, not just historical security -coverage. - -## Prior Art Notes - -This space has enough prior art to shape the implementation, but not enough -public detail to copy an architecture directly. - -- CodeRabbit documents context-aware PR review, path-specific instructions, - linked issue context, MCP context, multi-repo analysis, walkthrough comments, - and a review interface that reorganizes a PR into a logical walkthrough - rather than a flat file list: - -- CodeRabbit's path instructions and filters are a useful reminder that semantic - grouping should respect review scope and file-specific guidance. Generated - files, binaries, and lockfiles should be filtered before semantic planning, - and path-scoped review instructions should remain scanner inputs rather than - planner inputs: - -- Qodo describes its current review product as multi-agent PR analysis with - diff plus repository-aware reasoning, ticket-aware context, and a separate - semantic code intelligence layer for repository retrieval and multi-step - reasoning: - -- SWE-PRBench is the strongest warning against "just add more context." Its - results show AI review quality degrading as context expands, even with - structured semantic layers such as AST function context and import graph - resolution. It also found that a compact diff-with-summary context beat a - richer full-context prompt in their setup: - -- MutaGReP supports the same direction from code-use tasks: instead of putting - the whole repo into context, it builds grounded natural-language plans using - retrieval over relevant symbols. The reported plans use a small fraction of a - large context window while preserving task performance: - -- Semantic-aware AST diff work, especially RefactoringMiner-based approaches, - suggests a future deterministic enhancement: detect moved/renamed/refactored - code so the semantic planner starts with better symbol and relationship hints: - -- Empirical AI review studies suggest concise, actionable comments matter. - One study of GitHub Actions review tools found concise comments with code - snippets and hunk-level review tools were more likely to lead to code changes: - -- Industrial Qodo PR Agent studies found automated review can help bug - detection and awareness, but can also increase review time and produce faulty - or irrelevant comments: - - -Implementation lessons for Warden: - -- Keep planner context compact. The planner should operate over an inventory and - retrieve extra context through bounded tools only when needed. -- Treat semantic summaries as routing/context, not evidence. Findings still - need changed-line anchors and scanner evidence. -- Separate planning from scanning. A planning agent can inspect the repo to - group work; skill scanners still own domain-specific finding generation. -- Prefer explicit semantic segments over invisible behavior. Showing the plan in - verbose CLI output, JSONL, or a collapsed PR section can make Warden's work - auditable without overwhelming inline review. -- Evaluate against expected findings, not only cost. A cheaper run that misses a - known bug is worse. A more expensive planner may be justified only when it - improves or preserves recall while reducing scanner fragmentation. -- Avoid unbounded full-context prompts. Prior work suggests attention dilution - is a real failure mode; semantic chunking should reduce cognitive load, not - move the entire PR into an earlier prompt. - -## Rollout - -1. Add `AtomicHunkSummary`, `ReviewChunk`, and multi-range finding validation. -2. Adapt the existing hunk/coalescing flow to emit `ReviewChunk` values using - `raw-hunks`. -3. Update scanner prompt construction to accept `ReviewChunk`. -4. Move semantic planning into `packages/warden/src/semantic/` with a narrow - public API. -5. Replace the simple auxiliary schema call with a primary-model, read-only - tool-using planning agent. -6. Keep planner input compact and make code inspection explicit through tools. -7. Add materializers for `whole-file`, `stitched-file`, and `raw-hunks`. -8. Add telemetry for chunk counts, cost, duration, and planner failures. -9. Add regression fixtures for tiny-hunk pathological changes, including the - `getsentry/sentry-mcp` style case from Warden issue 313. -10. Add semantic-vs-hunk eval fixtures with known expected findings. -11. Add optional verbose CLI and collapsed PR visibility for semantic segments. -12. Enable selectively, compare eval recall and cost, then consider making it - default. - -## Non-Goals - -- replacing git diff parsing -- letting the planner decide where comments may land -- exposing planner recovery strategy as user config -- building a full AST differ -- reviewing unchanged lines as primary finding locations -- passing full diffs or full repository context to the planner by default -- making semantic segment display a required PR review surface From 4d2ba31e7ba3225af7c76f6c2673d65978f5a286 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 1 Jul 2026 14:14:14 -0700 Subject: [PATCH 18/18] docs(chunking): Move benchmark notes under benchmarks Keep the chunking benchmark plan and captured results next to the benchmark runner instead of treating them as general specs. Co-Authored-By: GPT-5 --- .../chunking/README.md | 2 +- .../chunking/results.json | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename specs/chunking-strategy-benchmark.md => benchmarks/chunking/README.md (99%) rename specs/chunking-strategy-benchmark-results.json => benchmarks/chunking/results.json (100%) diff --git a/specs/chunking-strategy-benchmark.md b/benchmarks/chunking/README.md similarity index 99% rename from specs/chunking-strategy-benchmark.md rename to benchmarks/chunking/README.md index ad51d727..48173098 100644 --- a/specs/chunking-strategy-benchmark.md +++ b/benchmarks/chunking/README.md @@ -468,7 +468,7 @@ Paired maintainer-runner baseline: - runtime: `pi` - skill: `code-review` - verification: disabled -- results: `specs/chunking-strategy-benchmark-results.json` +- results: `benchmarks/chunking/results.json` | Case | Mode | Expected found | Scanner chunks | Findings | Duration | Input tokens | Output tokens | Cost | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | diff --git a/specs/chunking-strategy-benchmark-results.json b/benchmarks/chunking/results.json similarity index 100% rename from specs/chunking-strategy-benchmark-results.json rename to benchmarks/chunking/results.json