feat(evaluation-system): add eval framework with graders, runner, report - #36
Conversation
…ponse cache Foundational layer of the prompt-eval suite (D1). Adds src/eval/ with a single-shot Claude Code CLI (`claude -p`) substrate — never the Anthropic SDK: - types.ts: EvalRequest/EvalResult/Envelope + injectable Spawn and Cache seams. - engine.ts: buildArgs (pure print-mode argv: --print --output-format json --json-schema --model --strict-mcp-config --settings + conditional system-prompt/tool-policy flags; prompt on stdin), parseEnvelope (graded, never throws — prefers structured_output, falls back to result, shallow schema check), runClaude (cache-first, spawn, cache-on-success), isAvailable and a skip signal distinct from graded failure for an absent CLI. - cache.ts: content-addressed ResponseCache keyed on a canonical SHA-256 over every output-affecting field. - tests/helpers/fake-claude.ts + tests/eval/*: hermetic engine/cache coverage via the injected Spawn stub, no real claude. Not a plugin deployable — no build.ts entry, no version bump.
…x runner, report, CLI Grading + orchestration layer on top of the D1 `claude -p` engine (D2). - src/eval/case.ts: EvalCase / Variant types, defineCase, toRequest lowering. - src/eval/grade.ts: Grader/GradeResult, built-in schemaConforms/assert/judge (judge routes through the D1 engine, cached/stubbable), weighted gradeCase. - src/eval/runner.ts: runCase, bounded-parallel runSuite (mapPool), runAB delta record (Δcorrectness/Δcost/Δlatency/Δturns), runMatrix per-model grid; skips with notice when the CLI is unavailable. - src/eval/report.ts: aligned text table + JSON over all three axes, optional clearly-labelled projected-cost column from a static pricing table. - src/eval/cli.ts: `bun src/eval/cli.ts` — suite loading, --ab/--matrix/--filter/ --models/--json/--tolerance; report-only exit 0 except an A/B correctness regression past tolerance → exit 1; absent claude → notice + exit 0. - package.json: `eval` script (kept OUT of `test`; test stays hermetic/offline). - CLAUDE.md + conventions.md: run the eval suite alongside `bun test` when changing harness prompts (docs-only, no generated-artifact churn, no bump). - tests/eval/framework.test.ts: graders, gradeCase, runAB/runMatrix deltas, bounded pool, report text/JSON, CLI exit-code semantics — all via fakeSpawn.
| export function runSuite(cases: readonly EvalCase[], opts: SuiteOptions = {}): Promise<CaseResult[]> { | ||
| const jobs: { c: EvalCase; model: string }[] = [] | ||
| for (const c of cases) { | ||
| const models = opts.models ?? c.models ?? [DEFAULT_MODEL] |
There was a problem hiding this comment.
runSuite resolves models with opts.models ?? c.models ?? [DEFAULT_MODEL]. Because ?? only substitutes null/undefined, a case authored with models: [] yields zero jobs and is silently dropped — no error, no notice. Treat an empty array as "unset" (or reject it loudly). Surfaced by the D2 code review; flagging for the review loop.
| * faster, tighter candidate reads as POSITIVE across the board; correctness is | ||
| * candidate−baseline so a regression reads as NEGATIVE (what the CLI gate keys on). | ||
| */ | ||
| export function runAB(c: EvalCase, baseline: Variant, candidate: Variant, opts: ABOptions = {}): ABResult { |
There was a problem hiding this comment.
runAB is single-draw (N=1 per variant). claude -p is stochastic (temperature can't be set on opus/sonnet), and the D4 report empirically showed the same prompt's cost/latency swinging sign run-to-run (input context 86k→180k tokens). A flat --tolerance over one sample can't separate signal from noise. Recommend n-samples per variant + a variance-aware band (report mean ± CI), so the correctness verdict is trustworthy even where sub-noise-floor cost deltas stay unattributable.
| const results: ABResult[] = [] | ||
| for (const c of cases) { | ||
| if (!c.variants) continue | ||
| results.push(runAB(c, c.variants.baseline, c.variants.candidate, { model: flags.models?.[0] ?? DEFAULT_MODEL, spawn, cache: deps.cache })) |
There was a problem hiding this comment.
In --ab mode the model is flags.models?.[0] ?? DEFAULT_MODEL, which overrides a case's own c.models[0]. A case declaring models: ['claude-opus-4-8'] is silently A/B'd at haiku when --models is omitted. Fall back to the case's declared model before DEFAULT_MODEL. (D2 review finding.)
| return { code, output } | ||
| } | ||
|
|
||
| if (import.meta.main) { |
There was a problem hiding this comment.
The import.meta.main entrypoint calls main(...) with no deps, so no ResponseCache is wired — a live bun run eval re-spends on every run even though D1 built a content-addressed cache. Wire a default file-backed ResponseCache here (with --no-cache to bypass). Flagged as a side-discovery in the D4 report.
1 participant
Add the grading and orchestration layer on top of the D1 engine: a case/suite
abstraction, three metric graders (correctness / performance / cost), a
bounded-parallel runner with A/B (baseline vs candidate prompt) and
model-matrix modes, and a human + JSON report. A correctness dip that buys a
large cost or latency win is treated as a win — A/B mode reports Δcorrectness /
Δcost / Δlatency side by side so a human judges the trade. Also wires the
process integration so agents know to run the suite alongside
bun testwhenchanging harness prompts.
Summary
New
src/eval/files:case.ts,grade.ts(schemaConforms/assert/judge,gradeCase),runner.ts(runSuite/runAB/runMatrix, skips-with-notice whenclaudeis unavailable),report.ts(aligned text table +toJSON, plus aclearly-labelled projected-cost column), and
cli.ts(bun run eval, exitsnon-zero only on an A/B regression past
--tolerance). Adds theevalnpmscript (not part of
test), a CLAUDE.md guideline, and a conventions.mdsubsection. All framework tests run through the D1
fakeSpawnstub.Acceptance criteria
schemaConforms/assert/judgeeach return aGradeResult;judgeroutes through the engine (cached/stubbable).runABreports Δcorrectness / Δcost / Δlatency / Δturns at a fixed model;runMatrixreports a per-model grid.claudeis unavailable.bun run evalis its own script (not part ofbun run test); report prints a text table,--jsonemits machine JSON, A/B exits non-zero only on a regression past--tolerance.bun teston prompt changes; conventions.md documents the suite.bun run typecheck && bun run lint && bun testgreen; no new runtime dependency.Stack
All deliverables target the
strappedrepo (linear stack, each PR based on its parent's branch).strapped/evaluation-system/D1-eval-enginestrapped/evaluation-system/D2-eval-frameworkstrapped/evaluation-system/D3-harness-eval-suitestrapped/evaluation-system/D4-verified-optimizationsDepends on #35