Skip to content

feat(evaluation-system): add claude -p eval engine and response cache - #35

Merged
cmschuetz merged 1 commit into
mainfrom
strapped/evaluation-system/D1-eval-engine
Jul 28, 2026
Merged

feat(evaluation-system): add claude -p eval engine and response cache#35
cmschuetz merged 1 commit into
mainfrom
strapped/evaluation-system/D1-eval-engine

Conversation

@cmschuetz

@cmschuetz cmschuetz commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Introduce the foundational eval-engine layer that runs a single-shot model
invocation through the Claude Code CLI in print mode (claude -p) rather than
the Anthropic SDK, so the harness's real output schemas are exercised end to
end. The engine forces schemas with --json-schema, reads cost/latency/usage
from the --output-format json envelope, content-addresses responses in a
file-backed cache, and stays fully hermetic under bun test by faking the
subprocess boundary. This is the substrate every later deliverable builds on.

Summary

New src/eval/ module: engine.ts (buildArgs/parseEnvelope/runClaude/
isAvailable), types.ts (EvalRequest/EvalResult/Envelope/Spawn), and
cache.ts (sha256 content-addressed response cache). Envelope parsing prefers
structured_output, falls back to JSON.parse(result), grades error/mismatch/
unparseable cases as ok:false without throwing, and surfaces an absent CLI as
a distinct skip signal. All tests inject a fake Spawn — no real claude, no
new runtime dependency.

Acceptance criteria

  • runClaude invokes only claude; argv includes --print --output-format json --json-schema … --model … --strict-mcp-config --settings … plus system-prompt flags when provided.
  • Success envelope with structured_output yields EvalResult{ok:true,…}; missing structured_output falls back to JSON.parse(result).
  • is_error, non-success subtype, unparseable stdout, and schema-nonconforming output each yield ok:false with a descriptive error and no throw.
  • Response cache keys on prompt+systemPrompt+model+schema+tools+settings: identical request hits, any keyed-field change misses.
  • Absent claude CLI → isAvailable() false and runClaude returns a skipped result; nothing throws.
  • bun run typecheck && bun run lint && bun test green; no new runtime dependency.

Stack

All deliverables target the strapped repo (linear stack, each PR based on its parent's branch).

Deliverable Branch PR
D1 — eval engine strapped/evaluation-system/D1-eval-engine #35
D2 — eval framework strapped/evaluation-system/D2-eval-framework #36
D3 — harness eval suite strapped/evaluation-system/D3-harness-eval-suite #37
D4 — verified optimizations strapped/evaluation-system/D4-verified-optimizations #38

…ponse cache

Foundational layer of the prompt-eval suite (D1). Adds src/eval/ with a
single-shot Claude Code CLI (`claude -p`) substrate — never the Anthropic SDK:

- types.ts: EvalRequest/EvalResult/Envelope + injectable Spawn and Cache seams.
- engine.ts: buildArgs (pure print-mode argv: --print --output-format json
  --json-schema --model --strict-mcp-config --settings + conditional
  system-prompt/tool-policy flags; prompt on stdin), parseEnvelope (graded,
  never throws — prefers structured_output, falls back to result, shallow
  schema check), runClaude (cache-first, spawn, cache-on-success), isAvailable
  and a skip signal distinct from graded failure for an absent CLI.
- cache.ts: content-addressed ResponseCache keyed on a canonical SHA-256 over
  every output-affecting field.
- tests/helpers/fake-claude.ts + tests/eval/*: hermetic engine/cache coverage
  via the injected Spawn stub, no real claude.

Not a plugin deployable — no build.ts entry, no version bump.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant