Skip to content

bench: evaluate agent continuation with --json-envelope - #55

Open
yearth wants to merge 1 commit into
yusukebe:mainfrom
yearth:codex/agent-continuation-bench-pr
Open

bench: evaluate agent continuation with --json-envelope#55
yearth wants to merge 1 commit into
yusukebe:mainfrom
yearth:codex/agent-continuation-bench-pr

Conversation

@yearth

@yearth yearth commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a deterministic 120-record HTML fixture with ground truth kept outside the agent workspace
  • add conformance and guided/ablated adoption prompts with a fixed multi-page budget
  • normalize Claude Code stream-json, correlate standalone ax calls through a benchmark-local shim, and grade answer, continuation protocol, access isolation, and run metrics
  • add a guarded 3+3+3 matrix runner and publish sanitized Sonnet 5 results in bench/RESULTS.md

Results

Condition Exact Protocol pass Envelope adoption
Conformance 2/3 3/3
Adoption guided 0/3 2/3 2/3
Adoption ablated 0/3 0/3 0/3

The report keeps answer correctness, protocol correctness, and active envelope adoption separate. It also discloses three earlier stopped matrix attempts and the resulting methodology changes, so these results are presented as adaptive and exploratory rather than confirmatory.

Raw session records, telemetry, generated fixtures, and ground truth remain untracked. No src/ files are changed.

Validation

  • bun test — 255 pass, 0 fail
  • bunx tsc --noEmit
  • bun run format
  • bun run build
  • replay grading of all nine retained runs remained gradable with access checks passing and published scores unchanged
  • independent final code review: ready to merge

Closes #53

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench: evaluate agent continuation with --json-envelope

1 participant