bench: evaluate agent continuation with --json-envelope

[yearth]

Context

--json-envelope shipped in v0.1.22, and its CLI contract is covered by unit tests. What is not yet measured is whether a real coding agent can follow meta.next_offset across multiple pages without restarting, overlapping, missing data, or continuing past a terminal state.

This follows the resumable-output work discussed in #39 and implemented further in #51.

Proposed benchmark

Would an agent-behavior benchmark like this fit the repository's existing bench/ suite?

The initial idea is:

• generate a deterministic HTML fixture with verifiable ground truth;
• use a small output budget that forces several continuation calls;
• grade the final answer exactly for missing or duplicate records;
• validate offset progression and stopping behavior from the tool trace;
• first run an explicit protocol-conformance task that requires --json-envelope;
• optionally run an adoption A/B comparing the current Skill with guidance that does not mention --json-envelope;
• record correctness, turns, tokens, duration, and cost;
• make no changes to src/.

Generated fixtures and raw session traces would remain untracked. The committed artifacts would be limited to the deterministic generator, prompts, grader, any minimal trace-runner support, and summarized results.

Questions

1. Should the first version focus only on protocol conformance, or also include the Skill adoption A/B?
2. Would a Claude Code stream-json trace be acceptable for recording and grading tool calls?
3. Should the summarized results be appended to bench/RESULTS.md?

If this direction is useful, I can turn it into a small, reproducible benchmark PR after agreeing on the scope and methodology here.

[yusukebe]

Hi @yearth

What is not yet measured is whether a real coding agent can follow meta.next_offset across multiple pages without restarting, overlapping, missing data, or continuing past a terminal state.

You are right. We have to measure real-world use cases. The answers are below:

1. Both.
2. Yes. stream-json is fine.
3. Yes. Append to bench/RESULTS.md

Looking forward to your PRs!