From 5aff4538a5e80db058c5175dd8f390c8bc9d1780 Mon Sep 17 00:00:00 2001 From: yearthmain Date: Mon, 27 Jul 2026 21:26:18 +0800 Subject: [PATCH] bench: evaluate agent continuation --- bench/RESULTS.md | 56 + bench/continuation/audit.test.ts | 205 ++++ bench/continuation/audit.ts | 310 ++++++ bench/continuation/ax-shim.ts | 42 + bench/continuation/continuation.test.ts | 1085 ++++++++++++++++++++ bench/continuation/fixture.ts | 91 ++ bench/continuation/grade-run.ts | 99 ++ bench/continuation/grade.ts | 349 +++++++ bench/continuation/inspect-stream.ts | 65 ++ bench/continuation/matrix.test.ts | 351 +++++++ bench/continuation/matrix.ts | 214 ++++ bench/continuation/prompt.ts | 35 + bench/continuation/report.test.ts | 136 +++ bench/continuation/report.ts | 201 ++++ bench/continuation/run.ts | 225 ++++ bench/continuation/stream.test.ts | 186 ++++ bench/continuation/stream.ts | 200 ++++ bench/continuation/telemetry.ts | 42 + bench/prompts/continuation-adoption.txt | 1 + bench/prompts/continuation-conformance.txt | 1 + 20 files changed, 3894 insertions(+) create mode 100644 bench/continuation/audit.test.ts create mode 100644 bench/continuation/audit.ts create mode 100644 bench/continuation/ax-shim.ts create mode 100644 bench/continuation/continuation.test.ts create mode 100644 bench/continuation/fixture.ts create mode 100644 bench/continuation/grade-run.ts create mode 100644 bench/continuation/grade.ts create mode 100644 bench/continuation/inspect-stream.ts create mode 100644 bench/continuation/matrix.test.ts create mode 100644 bench/continuation/matrix.ts create mode 100644 bench/continuation/prompt.ts create mode 100644 bench/continuation/report.test.ts create mode 100644 bench/continuation/report.ts create mode 100644 bench/continuation/run.ts create mode 100644 bench/continuation/stream.test.ts create mode 100644 bench/continuation/stream.ts create mode 100644 bench/continuation/telemetry.ts create mode 100644 bench/prompts/continuation-adoption.txt create mode 100644 bench/prompts/continuation-conformance.txt diff --git a/bench/RESULTS.md b/bench/RESULTS.md index ec367e3..86345d3 100644 --- a/bench/RESULTS.md +++ b/bench/RESULTS.md @@ -265,3 +265,59 @@ one sample lost outright. Reported as median-of-3, loss included. **Tool-warm scoreboard (Opus, all verified correct):** drift −67%, clean extract −65%, live web −23% (median of 3). These are the site numbers. + +## JSON-envelope continuation (Sonnet 5, n=3 per condition) + +### Method + +A deterministic 120-row HTML fixture was generated from an independent fixed seed. +Each run used budget 600, while ground truth remained in the parent process. +Conformance explicitly taught continuation; Adoption compared the same natural task +with the full Skill versus a runtime-only continuation-guidance ablation. +Raw session records stayed outside the repository. + +### Preliminary stopped attempts + +Three earlier formal matrix invocations were stopped by gradability gates and excluded from the reported sample. +They exposed fixture-metadata preflights, compound or interpreter-wrapped ax commands, +and terminal responses containing text outside the required JSON object. +The methodology was then changed to require standalone ax calls, prohibit all non-ax +fixture access, enforce a JSON-only response, and tighten access auditing before the +final matrix was collected. A post-run review further replaced permissive non-ax +handling with a conservative allowlist; regrading the retained runs did not change +their gradability or published scores. These are adaptive, exploratory results rather +than a preregistered confirmatory evaluation. + +### Per-run results + +| Run | Condition | Exact | Protocol | Envelope | Pages | Turns | Duration (s) | Input | Output | Cache create | Cache read | Cost | +| --: | ---------------- | :---: | -------- | :------: | ----: | ----: | -----------: | -----: | -----: | -----------: | ---------: | -----: | +| 1 | Conformance | no | pass | — | 5 | 8 | 170.5 | 57346 | 9746 | 0 | 314752 | $0.413 | +| 2 | Adoption guided | no | fail | no | 3 | 6 | 370.2 | 14794 | 23254 | 0 | 290048 | $0.480 | +| 3 | Adoption ablated | no | fail | no | 6 | 10 | 630.9 | 72592 | 32273 | 0 | 503680 | $0.853 | +| 4 | Conformance | yes | pass | — | 5 | 8 | 167.6 | 56400 | 7907 | 0 | 360832 | $0.396 | +| 5 | Adoption guided | no | pass | yes | 5 | 7 | 199.8 | 20118 | 10369 | 0 | 358784 | $0.324 | +| 6 | Adoption ablated | no | fail | no | 3 | 5 | 747.2 | 86911 | 43130 | 0 | 289280 | $0.994 | +| 7 | Conformance | yes | pass | — | 5 | 8 | 153.6 | 18545 | 7222 | 0 | 398464 | $0.284 | +| 8 | Adoption guided | no | pass | yes | 5 | 8 | 413.9 | 33677 | 22772 | 0 | 496384 | $0.592 | +| 9 | Adoption ablated | no | fail | no | 6 | 8 | 609.2 | 102120 | 29857 | 0 | 462464 | $0.893 | + +### Aggregate + +| Condition | Exact | Protocol pass | Envelope adoption | Median turns | Median duration (s) | Median input | Median output | Median cost | +| ---------------- | ----: | ------------: | ----------------: | -----------: | ------------------: | -----------: | ------------: | ----------: | +| Conformance | 2/3 | 3/3 | — | 8 | 167.6 | 56400 | 7907 | $0.396 | +| Adoption guided | 0/3 | 2/3 | 2/3 | 7 | 370.2 | 20118 | 22772 | $0.480 | +| Adoption ablated | 0/3 | 0/3 | 0/3 | 8 | 630.9 | 86911 | 32273 | $0.893 | + +### Observed failure modes + +Across 4 Adoption runs without envelope adoption, 4 used `offset-without-envelope`; 4 failed protocol grading. +Among 7 non-exact answers, 4 were schema-invalid. Among the 3 schema-valid non-exact answers, 0 had missing records, 3 had unexpected records, 0 had duplicate records, 0 had ordering errors, and 0 had field mismatches. Categories can overlap. + +### Interpretation and limitations + +Answer correctness, continuation-protocol correctness, and active envelope adoption +are reported separately. Each condition has only three runs on one deterministic +fixture and one model, so differences are descriptive rather than general claims. +Claude Code event structure may also change across CLI versions. diff --git a/bench/continuation/audit.test.ts b/bench/continuation/audit.test.ts new file mode 100644 index 0000000..c67f277 --- /dev/null +++ b/bench/continuation/audit.test.ts @@ -0,0 +1,205 @@ +import { expect, test } from 'bun:test' +import { auditToolContext } from './audit' +import type { AdoptionGrade } from './grade' +import type { NormalizedToolCall } from './stream' +import type { AxInvocation } from './telemetry' + +function tool(name: string, input: Record): NormalizedToolCall { + return { + ordinal: 1, + name, + input, + resultText: '', + isError: false, + } +} + +function bash(command: string): NormalizedToolCall { + return tool('Bash', { command, description: 'synthetic' }) +} + +function invocation(argv: string[], start = 1): AxInvocation { + return { + argv, + cwd: '/external/run/agent', + stdout: '', + stderr: '', + exitCode: 0, + startedAtMs: start, + endedAtMs: start + 1, + } +} + +test('adoption grade can record direct fixture access', () => { + const strategy: AdoptionGrade['alternativeStrategy'] = 'direct-fixture-read' + expect(strategy).toBe('direct-fixture-read') +}) + +test.each([ + tool('Read', { file_path: '/external/run/agent/continuation.html' }), + tool('Grep', { pattern: 'incident', path: 'continuation.html' }), + tool('Glob', { pattern: '**/continuation.html', path: '.' }), + tool('Grep', { pattern: 'incident', path: '.' }), + tool('Grep', { pattern: 'incident' }), + tool('Glob', { pattern: '**/*.html', path: '.' }), +])('%s directly accessing the fixture is a violation', (call) => { + expect(auditToolContext([call], [])).toEqual({ + status: 'violation', + issues: ['direct fixture access'], + alternativeStrategy: 'direct-fixture-read', + }) +}) + +test.each([ + 'cat continuation.html', + 'grep incident continuation.html', + 'python3 -c "open(\\"continuation.html\\")"', + 'f=continuation.html; cat "$f"', + 'cat *.html', + 'rg incident .', + 'ax continuation.html .incident --json-envelope; cat continuation.html', +])('%s is a direct fixture read violation', (command) => { + expect(auditToolContext([bash(command)], [])).toEqual({ + status: 'violation', + issues: ['direct fixture access'], + alternativeStrategy: 'direct-fixture-read', + }) +}) + +test.each([ + tool('Grep', { pattern: 'incident', path: '/external/run/agent' }), + bash('rg incident /external/run/agent'), +])('treats the known Agent directory as fixture-targeting', (call) => { + expect(auditToolContext([call], [], 'continuation.html', '/external/run/agent')).toEqual({ + status: 'violation', + issues: ['direct fixture access'], + alternativeStrategy: 'direct-fixture-read', + }) +}) + +test('accepts a simple absolute shim command when telemetry matches', () => { + const argv = ['continuation.html', '.incident', '--json-envelope'] + expect( + auditToolContext( + [bash('/external/run/bin/ax continuation.html .incident --json-envelope')], + [invocation(argv)] + ) + ).toEqual({ + status: 'pass', + issues: [], + alternativeStrategy: null, + }) +}) + +test.each([ + { + name: 'semicolon', + command: + 'ax continuation.html .incident --json-envelope; ' + + 'ax continuation.html .incident --json-envelope --offset 29', + invocations: [ + invocation(['continuation.html', '.incident', '--json-envelope'], 1), + invocation(['continuation.html', '.incident', '--json-envelope', '--offset', '29'], 3), + ], + }, + { + name: 'and', + command: 'ax continuation.html .incident --json-envelope && which ax', + invocations: [invocation(['continuation.html', '.incident', '--json-envelope'])], + }, +])('$name-batched fixture ax command is not gradable', ({ command, invocations }) => { + expect(auditToolContext([bash(command)], [...invocations])).toEqual({ + status: 'not_gradable', + issues: ['unsupported fixture-targeting Bash command'], + alternativeStrategy: null, + }) +}) + +test.each([ + 'wrapper continuation.html', + 'ax continuation.html .incident --json-envelope | cat', + 'ax continuation.html .incident --json-envelope > /tmp/envelope.json', + "python3 <<'PY'\nimport subprocess\nsubprocess.run(['ax', 'continuation.html'])\nPY", + '$(command -v ax) continuation.html .incident --json-envelope', + 'ax continuation.html "unterminated', +])('%s is not gradable', (command) => { + expect(auditToolContext([bash(command)], []).status).toBe('not_gradable') +}) + +test('ignores unsupported non-fixture discovery commands', () => { + expect(auditToolContext([bash('command -v ax && ax --help | head -20')], [])).toEqual({ + status: 'pass', + issues: [], + alternativeStrategy: null, + }) +}) + +test.each([ + 'python3 -c \'import glob; print(open(glob.glob("*.html")[0]).read())\'', + 'fixture-reader', +])('%s is not gradable without a proven-safe command shape', (command) => { + expect(auditToolContext([bash(command)], [])).toEqual({ + status: 'not_gradable', + issues: ['unverifiable non-ax Bash command'], + alternativeStrategy: null, + }) +}) + +test('allows a non-file echo with arithmetic expansion', () => { + expect(auditToolContext([bash('echo "Verification: $((1+1))"')], [])).toEqual({ + status: 'pass', + issues: [], + alternativeStrategy: null, + }) +}) + +test('rejects fixture metadata inspection outside ax', () => { + expect(auditToolContext([bash('ls -la continuation.html && which ax')], [])).toEqual({ + status: 'violation', + issues: ['fixture access outside ax'], + alternativeStrategy: null, + }) +}) + +test('allows an unresolved ax path that never reached the shim', () => { + const call = bash('./bin/ax continuation.html .incident --json-envelope') + call.isError = true + call.resultText = 'Exit code 127\n(eval):1: no such file or directory: ./bin/ax' + expect(auditToolContext([call], [])).toEqual({ + status: 'pass', + issues: [], + alternativeStrategy: null, + }) +}) + +test.each([ + { + name: 'missing telemetry', + calls: [bash('ax continuation.html .incident --json-envelope')], + invocations: [], + }, + { + name: 'extra telemetry', + calls: [bash('ax continuation.html .incident --json-envelope')], + invocations: [ + invocation(['continuation.html', '.incident', '--json-envelope'], 1), + invocation(['continuation.html', '.incident', '--json-envelope', '--offset', '29'], 3), + ], + }, + { + name: 'argv mismatch', + calls: [bash('ax continuation.html .incident --json-envelope')], + invocations: [invocation(['continuation.html', '.other', '--json-envelope'])], + }, + { + name: 'stream omits a fixture invocation', + calls: [bash('ax --help')], + invocations: [invocation(['continuation.html', '.incident', '--json-envelope'])], + }, +])('$name is not gradable', ({ calls, invocations }) => { + expect(auditToolContext([...calls], [...invocations])).toEqual({ + status: 'not_gradable', + issues: ['fixture ax telemetry mismatch'], + alternativeStrategy: null, + }) +}) diff --git a/bench/continuation/audit.ts b/bench/continuation/audit.ts new file mode 100644 index 0000000..f5c0b97 --- /dev/null +++ b/bench/continuation/audit.ts @@ -0,0 +1,310 @@ +import { basename, resolve } from 'node:path' +import type { NormalizedToolCall } from './stream' +import type { AxInvocation } from './telemetry' + +export type ToolContextAudit = { + status: 'pass' | 'violation' | 'not_gradable' + issues: string[] + alternativeStrategy: 'direct-fixture-read' | null +} + +type SimpleCommand = { + argv: string[] +} + +type ShellClassification = + | { status: 'parsed'; commands: SimpleCommand[] } + | { status: 'unsupported' } + +const DIRECT_READ_COMMANDS = new Set([ + 'awk', + 'cat', + 'cp', + 'dd', + 'grep', + 'head', + 'perl', + 'python', + 'python3', + 'rg', + 'ruby', + 'sed', + 'tail', +]) + +function stringTargetsFixture(value: string, fixtureBasename: string): boolean { + if (value.includes(fixtureBasename)) return true + const pattern = value.split('/').at(-1) ?? value + if (!pattern.includes('*') && !pattern.includes('?')) return false + const expression = pattern + .replace(/[.+^${}()|[\]\\]/g, '\\$&') + .replaceAll('*', '.*') + .replaceAll('?', '.') + return new RegExp(`^${expression}$`).test(fixtureBasename) +} + +function targetsFixture(value: unknown, fixtureBasename: string): boolean { + if (typeof value === 'string') return stringTargetsFixture(value, fixtureBasename) + if (Array.isArray(value)) { + return value.some((item) => targetsFixture(item, fixtureBasename)) + } + if (value && typeof value === 'object') { + return Object.values(value).some((item) => targetsFixture(item, fixtureBasename)) + } + return false +} + +function pathTargetsFixture( + value: unknown, + fixtureBasename: string, + fixtureDirectory: string | undefined +): boolean { + if (targetsFixture(value, fixtureBasename)) return true + return ( + typeof value === 'string' && + (value === '.' || + (fixtureDirectory !== undefined && resolve(value) === resolve(fixtureDirectory))) + ) +} + +function directToolTargetsFixture( + call: NormalizedToolCall, + fixtureBasename: string, + fixtureDirectory: string | undefined +): boolean { + if (call.name === 'Read') { + return pathTargetsFixture( + call.input.file_path ?? call.input.path, + fixtureBasename, + fixtureDirectory + ) + } + if (call.name === 'Grep') { + const path = call.input.path + return path === undefined || pathTargetsFixture(path, fixtureBasename, fixtureDirectory) + } + if (call.name === 'Glob') { + return targetsFixture(call.input.pattern, fixtureBasename) + } + return false +} + +function commandHasFixtureHint(command: string, fixtureBasename: string): boolean { + return command + .split(/\s+/) + .some((token) => stringTargetsFixture(token.replace(/^['"]|['"]$/g, ''), fixtureBasename)) +} + +function isSafeNonFixtureCommand(command: SimpleCommand): boolean { + const executable = basename(command.argv[0]!) + const args = command.argv.slice(1) + if (executable === 'command') return JSON.stringify(args) === JSON.stringify(['-v', 'ax']) + if (executable === 'which') return JSON.stringify(args) === JSON.stringify(['ax']) + if (executable === 'ax') return args.length === 1 && ['--help', '--version'].includes(args[0]!) + return executable === 'echo' +} + +function isSafeUnsupportedNonFixtureCommand(command: string): boolean { + const trimmed = command.trim() + if (trimmed === 'command -v ax && ax --help | head -20') return true + return /^echo\s+"[^"$`\\]*(?:\$\(\([0-9+\-*/%\s()]+\)\))?[^"$`\\]*"$/.test(trimmed) +} + +function parseSimpleCommandList(command: string): ShellClassification { + const commands: SimpleCommand[] = [] + let argv: string[] = [] + let token = '' + let tokenStarted = false + let quote: "'" | '"' | null = null + + const pushToken = () => { + if (!tokenStarted) return + argv.push(token) + token = '' + tokenStarted = false + } + const pushCommand = () => { + pushToken() + if (argv.length === 0) return false + commands.push({ argv }) + argv = [] + return true + } + + for (let index = 0; index < command.length; index++) { + const char = command[index]! + if (quote) { + if (char === quote) { + quote = null + tokenStarted = true + continue + } + if (quote === '"' && char === '\\') { + const next = command[++index] + if (next === undefined) return { status: 'unsupported' } + token += next + tokenStarted = true + continue + } + if (char === '`' || (char === '$' && command[index + 1] === '(')) { + return { status: 'unsupported' } + } + token += char + tokenStarted = true + continue + } + if (char === "'" || char === '"') { + quote = char + tokenStarted = true + continue + } + if (char === '\\') { + const next = command[++index] + if (next === undefined) return { status: 'unsupported' } + token += next + tokenStarted = true + continue + } + if (/\s/.test(char)) { + if (char === '\n' || char === '\r') return { status: 'unsupported' } + pushToken() + continue + } + if (char === ';') { + if (!pushCommand()) return { status: 'unsupported' } + continue + } + if (char === '&' && command[index + 1] === '&') { + if (!pushCommand()) return { status: 'unsupported' } + index++ + continue + } + if ( + char === '|' || + char === '<' || + char === '>' || + char === '`' || + char === '(' || + char === ')' || + char === '&' + ) { + return { status: 'unsupported' } + } + if (char === '$' && command[index + 1] === '(') return { status: 'unsupported' } + token += char + tokenStarted = true + } + if (quote || !pushCommand()) return { status: 'unsupported' } + return { status: 'parsed', commands } +} + +function fixtureInvocations(invocations: AxInvocation[], fixtureBasename: string): AxInvocation[] { + return invocations.filter((call) => + call.argv.some((arg) => arg === fixtureBasename || arg.endsWith(`/${fixtureBasename}`)) + ) +} + +function result( + status: ToolContextAudit['status'], + issues: string[], + alternativeStrategy: ToolContextAudit['alternativeStrategy'] = null +): ToolContextAudit { + return { status, issues: [...new Set(issues)], alternativeStrategy } +} + +export function auditToolContext( + toolCalls: NormalizedToolCall[], + invocations: AxInvocation[], + fixtureBasename = 'continuation.html', + fixtureDirectory?: string +): ToolContextAudit { + const expectedArgv: string[][] = [] + let unsupportedFixtureCommand = false + let unverifiableNonAxCommand = false + + for (const call of toolCalls) { + if (call.name === 'Read' || call.name === 'Grep' || call.name === 'Glob') { + if (directToolTargetsFixture(call, fixtureBasename, fixtureDirectory)) { + return result('violation', ['direct fixture access'], 'direct-fixture-read') + } + continue + } + if (call.name !== 'Bash') { + if (targetsFixture(call.input, fixtureBasename)) unsupportedFixtureCommand = true + continue + } + if (typeof call.input.command !== 'string') { + if (targetsFixture(call.input, fixtureBasename)) unsupportedFixtureCommand = true + continue + } + const commandText = call.input.command + const parsed = parseSimpleCommandList(commandText) + if (parsed.status === 'unsupported') { + if (commandHasFixtureHint(commandText, fixtureBasename)) { + unsupportedFixtureCommand = true + } else if (!isSafeUnsupportedNonFixtureCommand(commandText)) { + unverifiableNonAxCommand = true + } + continue + } + const fixtureAxCommands = parsed.commands.filter((command) => { + const executable = basename(command.argv[0]!) + return ( + executable === 'ax' && + command.argv + .slice(1) + .some((arg) => pathTargetsFixture(arg, fixtureBasename, fixtureDirectory)) + ) + }) + if (fixtureAxCommands.length > 0 && parsed.commands.length !== 1) { + unsupportedFixtureCommand = true + } + for (const command of parsed.commands) { + const executable = basename(command.argv[0]!) + const fixtureAssignment = + /^[A-Za-z_][A-Za-z0-9_]*=/.test(command.argv[0]!) && + stringTargetsFixture(command.argv[0]!, fixtureBasename) + const argumentsTargetFixture = + fixtureAssignment || + command.argv + .slice(1) + .some((arg) => pathTargetsFixture(arg, fixtureBasename, fixtureDirectory)) || + (DIRECT_READ_COMMANDS.has(executable) && command.argv.slice(1).includes('.')) + if (!argumentsTargetFixture) { + if (!isSafeNonFixtureCommand(command)) unverifiableNonAxCommand = true + continue + } + if (executable === 'ax') { + const unresolved = + call.isError && + call.resultText.startsWith('Exit code 127\n') && + call.resultText.endsWith(`no such file or directory: ${command.argv[0]}`) + if (unresolved) continue + expectedArgv.push(command.argv.slice(1)) + continue + } + if (executable === 'ls') { + return result('violation', ['fixture access outside ax']) + } + if (DIRECT_READ_COMMANDS.has(executable) || fixtureAssignment) { + return result('violation', ['direct fixture access'], 'direct-fixture-read') + } + unsupportedFixtureCommand = true + } + } + + if (unsupportedFixtureCommand) { + return result('not_gradable', ['unsupported fixture-targeting Bash command']) + } + if (unverifiableNonAxCommand) { + return result('not_gradable', ['unverifiable non-ax Bash command']) + } + const actualArgv = fixtureInvocations(invocations, fixtureBasename).map((call) => call.argv) + if ( + expectedArgv.length !== actualArgv.length || + expectedArgv.some((argv, index) => JSON.stringify(argv) !== JSON.stringify(actualArgv[index])) + ) { + return result('not_gradable', ['fixture ax telemetry mismatch']) + } + return result('pass', []) +} diff --git a/bench/continuation/ax-shim.ts b/bench/continuation/ax-shim.ts new file mode 100644 index 0000000..a2273d6 --- /dev/null +++ b/bench/continuation/ax-shim.ts @@ -0,0 +1,42 @@ +#!/usr/bin/env bun +import { closeSync, openSync, writeSync } from 'node:fs' +import type { AxInvocation } from './telemetry' + +const real = process.env.AX_BENCH_REAL +const telemetry = process.env.AX_BENCH_TELEMETRY +if (!real || !telemetry) { + console.error('ax benchmark shim: missing AX_BENCH_REAL or AX_BENCH_TELEMETRY') + process.exit(70) +} + +const startedAtMs = performance.timeOrigin + performance.now() +const proc = Bun.spawn([real, ...process.argv.slice(2)], { + cwd: process.cwd(), + stdout: 'pipe', + stderr: 'pipe', +}) +const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).arrayBuffer(), + new Response(proc.stderr).arrayBuffer(), + proc.exited, +]) +const out = new Uint8Array(stdout) +const err = new Uint8Array(stderr) +const record: AxInvocation = { + argv: process.argv.slice(2), + cwd: process.cwd(), + stdout: new TextDecoder().decode(out), + stderr: new TextDecoder().decode(err), + exitCode, + startedAtMs, + endedAtMs: performance.timeOrigin + performance.now(), +} +const fd = openSync(telemetry, 'a', 0o600) +try { + writeSync(fd, `${JSON.stringify(record)}\n`) +} finally { + closeSync(fd) +} +if (out.length) await new Promise((resolve) => process.stdout.write(out, () => resolve())) +if (err.length) await new Promise((resolve) => process.stderr.write(err, () => resolve())) +process.exit(exitCode) diff --git a/bench/continuation/continuation.test.ts b/bench/continuation/continuation.test.ts new file mode 100644 index 0000000..f159292 --- /dev/null +++ b/bench/continuation/continuation.test.ts @@ -0,0 +1,1085 @@ +import { describe, expect, test } from 'bun:test' +import { chmod, mkdir, mkdtemp, readdir, realpath, symlink } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { + CONTINUATION_BUDGET, + CONTINUATION_ROW_SPEC, + CONTINUATION_SEED, + CONTINUATION_SELECTOR, + generateFixture, + writeAgentFixture, +} from './fixture' +import { gradeRunRoot } from './grade-run' +import { gradeAdoption, gradeAnswer, gradeProtocol } from './grade' +import { inspectStreamShapes } from './inspect-stream' +import { ablateContinuationGuidance, applySkill, renderContinuationPrompt } from './prompt' +import { buildClaudeArgs, capturePilot, preflightAx, prepareRun, provisionAx } from './run' +import { hasOverlappingInvocations, readAxTelemetry, type AxInvocation } from './telemetry' + +function call(argv: string[], stdout: unknown, start: number): AxInvocation { + return { + argv, + cwd: '/agent', + stdout: JSON.stringify(stdout), + stderr: '', + exitCode: 0, + startedAtMs: start, + endedAtMs: start + 1, + } +} + +function syntheticRunStream( + invocations: AxInvocation[], + options: { directRead?: boolean; finalText?: string } = {} +): string { + const events: string[] = [] + if (options.directRead) { + events.push( + JSON.stringify({ + type: 'assistant', + message: { + content: [ + { + type: 'tool_use', + id: 'direct-read', + name: 'Read', + input: { file_path: 'continuation.html' }, + }, + ], + }, + }), + JSON.stringify({ + type: 'user', + message: { + content: [ + { + type: 'tool_result', + tool_use_id: 'direct-read', + content: 'synthetic direct result', + is_error: false, + }, + ], + }, + }) + ) + } + events.push( + ...invocations.flatMap((invocation, index) => [ + JSON.stringify({ + type: 'assistant', + message: { + content: [ + { + type: 'tool_use', + id: `synthetic-tool-${index + 1}`, + name: 'Bash', + input: { command: ['ax', ...invocation.argv].join(' ') }, + }, + ], + }, + }), + JSON.stringify({ + type: 'user', + message: { + content: [ + { + type: 'tool_result', + tool_use_id: `synthetic-tool-${index + 1}`, + content: invocation.stdout, + is_error: false, + }, + ], + }, + }), + ]), + JSON.stringify({ + type: 'result', + subtype: 'success', + is_error: false, + result: options.finalText ?? JSON.stringify({ records: generateFixture().records }), + duration_ms: 1200, + num_turns: 7, + total_cost_usd: 0.125, + usage: { + input_tokens: 11, + output_tokens: 7, + cache_creation_input_tokens: 5, + cache_read_input_tokens: 13, + }, + }) + ) + return `${events.join('\n')}\n` +} + +function skillBodyForTest(markdown: string): string { + const closing = markdown.indexOf('\n---', 4) + if (!markdown.startsWith('---\n') || closing === -1) { + throw new Error('test Skill must contain YAML frontmatter') + } + return markdown.slice(closing + 4).replace(/^\n+/, '') +} + +describe('continuation fixture', () => { + test('is deterministic, shuffled, and contains 120 unique incidents', () => { + const first = generateFixture() + const second = generateFixture() + expect(first).toEqual(second) + expect(first.records).toHaveLength(120) + expect(new Set(first.records.map((record) => record.id)).size).toBe(120) + expect(first.records.slice(0, 3).map((record) => record.id)).not.toEqual([ + 'INC-73001', + 'INC-73002', + 'INC-73003', + ]) + expect(first.html.match(/class="incident"/g)).toHaveLength(120) + expect(first.html).toContain('class="incident-preview"') + }) + + test('writes only HTML into the Agent directory and returns truth in memory', async () => { + const dir = await mkdtemp(join(tmpdir(), 'ax-continuation-agent-')) + const expected = generateFixture().records + const records = await writeAgentFixture(dir) + expect(records).toEqual(expected) + expect(await readdir(dir)).toEqual(['continuation.html']) + }) + + test('locks the public extraction contract', () => { + expect(CONTINUATION_BUDGET).toBe(600) + expect(CONTINUATION_SELECTOR).toBe('.incident') + expect(CONTINUATION_ROW_SPEC).toBe('id=@data-id,owner=.owner,severity=.severity,status=.status') + }) +}) + +describe('continuation prompts', () => { + test('replaces exactly one budget token', () => { + expect(renderContinuationPrompt('use {{BUDGET}} exactly', 600)).toBe('use 600 exactly') + expect(() => renderContinuationPrompt('no token', 600)).toThrow('exactly one {{BUDGET}}') + expect(() => renderContinuationPrompt('{{BUDGET}} {{BUDGET}}', 600)).toThrow( + 'exactly one {{BUDGET}}' + ) + }) + + test('condition prompts share neutral evidence constraints', async () => { + const [conformance, adoption] = await Promise.all([ + Bun.file('bench/prompts/continuation-conformance.txt').text(), + Bun.file('bench/prompts/continuation-adoption.txt').text(), + ]) + const standaloneRule = + 'Run each ax invocation as one standalone Bash command. Do not combine it with another command or use pipes, redirections, multiline commands, heredocs, scripts, interpreters, shell functions, command substitutions, wrappers, or subprocesses.' + const fixtureIsolationRule = + 'Do not use any other tool or command to access, inspect, locate, list, search, copy, transform, or read continuation.html. You may locate ax only with a command that does not mention continuation.html.' + const responseBoundaryRule = + 'Before responding, silently verify that the response starts with {, ends with }, and contains exactly one JSON object with no prefix, suffix, completion note, Markdown fence, or commentary.' + + for (const prompt of [conformance, adoption]) { + expect(prompt).toContain(standaloneRule) + expect(prompt).toContain(fixtureIsolationRule) + expect(prompt).toContain(responseBoundaryRule) + } + expect(adoption).not.toMatch( + /--json-envelope|--offset|meta\.|pagination|continuation(?!\.html)|terminal states?|multiple pages?/i + ) + }) + + test('ablates only the complete continuation bullet', () => { + const target = [ + '- For automated continuation, use `--json-envelope`. Read `data`; when', + ' `meta.state` is `more`, rerun the same command with', + ' `--offset `. Continue only while it is `more`; stop on', + ' `complete` or `past_end`; do not restart from zero or increase the budget.', + ].join('\n') + const skill = [ + '---', + 'name: ax', + '---', + '# ax', + '', + '## Output rules', + '', + '- Default cap.', + target, + '## Next section', + 'Keep this body byte-for-byte.', + ].join('\n') + const ablated = ablateContinuationGuidance(skill) + expect(ablated).toBe( + [ + '---', + 'name: ax', + '---', + '# ax', + '', + '## Output rules', + '', + '- Default cap.', + '## Next section', + 'Keep this body byte-for-byte.', + ].join('\n') + ) + expect(() => ablateContinuationGuidance(ablated)).toThrow('exactly one continuation bullet') + }) + + test('guided and ablated preserve all non-continuation Skill text', async () => { + const markdown = await Bun.file('skills/ax/SKILL.md').text() + const prompt = 'task' + const target = `${[ + '- For automated continuation, use `--json-envelope`. Read `data`; when', + ' `meta.state` is `more`, rerun the same command with', + ' `--offset `. Continue only while it is `more`; stop on', + ' `complete` or `past_end`; do not restart from zero or increase the budget.', + ].join('\n')}\n` + const start = markdown.indexOf(target) + expect(start).toBeGreaterThanOrEqual(0) + const expectedAblated = markdown.slice(0, start) + markdown.slice(start + target.length) + expect(applySkill(prompt, markdown, 'guided')).toContain('--json-envelope') + expect(ablateContinuationGuidance(markdown)).toBe(expectedAblated) + expect(applySkill(prompt, markdown, 'ablated')).toBe( + `${skillBodyForTest(expectedAblated)}\n\n${prompt}` + ) + expect(applySkill(prompt, markdown, 'none')).toBe(prompt) + }) +}) + +describe('instrumented ax', () => { + test('resolves the shim, preserves output, and records each semicolon command', async () => { + const root = await mkdtemp(join(tmpdir(), 'ax-continuation-run-')) + const agentDir = join(root, 'agent') + await mkdir(agentDir) + await writeAgentFixture(agentDir) + const provisioned = await provisionAx(process.cwd(), root) + await preflightAx(provisioned, join(agentDir, 'continuation.html')) + + const shell = Bun.spawnSync( + [ + 'sh', + '-c', + "ax continuation.html '.incident' --count; ax continuation.html '.incident' --count", + ], + { cwd: agentDir, env: provisioned.childEnv } + ) + expect(shell.exitCode).toBe(0) + expect(shell.stdout.toString()).toBe('120\n120\n') + const calls = await readAxTelemetry(provisioned.telemetryPath) + expect(calls.map((call) => call.argv.at(-1))).toEqual(['--count', '--count']) + expect(hasOverlappingInvocations(calls)).toBe(false) + }) + + test('replays both streams and a non-zero exit from ax-real into telemetry', async () => { + const root = await mkdtemp(join(tmpdir(), 'ax-continuation-shim-')) + const agentDir = join(root, 'agent') + await mkdir(agentDir) + const provisioned = await provisionAx(process.cwd(), root) + const fakeReal = join(root, 'fake-ax-real') + await Bun.write( + fakeReal, + [ + '#!/bin/sh', + `printf '%s\\n' 'fake stdout'`, + `printf '%s\\n' 'fake stderr' >&2`, + 'exit 7', + ].join('\n') + ) + await chmod(fakeReal, 0o755) + const proc = Bun.spawnSync([provisioned.axPath, 'continuation.html'], { + cwd: agentDir, + env: { ...provisioned.childEnv, AX_BENCH_REAL: fakeReal }, + }) + expect(proc.exitCode).toBe(7) + expect(proc.stdout.toString()).toBe('fake stdout\n') + expect(proc.stderr.toString()).toBe('fake stderr\n') + const calls = await readAxTelemetry(provisioned.telemetryPath) + const canonicalAgentDir = await realpath(agentDir) + expect(calls).toHaveLength(1) + expect(calls[0]).toMatchObject({ + argv: ['continuation.html'], + cwd: canonicalAgentDir, + stdout: 'fake stdout\n', + stderr: 'fake stderr\n', + exitCode: 7, + }) + }) + + test('reconstructs 120 records in five fixed-budget pages through the shim', async () => { + const root = await mkdtemp(join(tmpdir(), 'ax-continuation-pages-')) + const agentDir = join(root, 'agent') + await mkdir(agentDir) + const expected = await writeAgentFixture(agentDir) + const provisioned = await provisionAx(process.cwd(), root) + await preflightAx(provisioned, join(agentDir, 'continuation.html')) + const actual: unknown[] = [] + let offset = 0 + let pages = 0 + while (true) { + const args = [ + 'continuation.html', + CONTINUATION_SELECTOR, + '--row', + CONTINUATION_ROW_SPEC, + '--json-envelope', + '--budget', + String(CONTINUATION_BUDGET), + ] + if (offset > 0) args.push('--offset', String(offset)) + const proc = Bun.spawnSync(['ax', ...args], { + cwd: agentDir, + env: provisioned.childEnv, + }) + expect(proc.exitCode).toBe(0) + const envelope = JSON.parse(proc.stdout.toString()) + actual.push(...envelope.data) + pages++ + if (envelope.meta.state !== 'more') { + expect(envelope.meta.state).toBe('complete') + break + } + offset = envelope.meta.next_offset + } + expect(actual).toEqual(expected) + expect(new Set(actual.map((record) => (record as { id: string }).id)).size).toBe(120) + expect(pages).toBe(5) + expect(await readAxTelemetry(provisioned.telemetryPath)).toHaveLength(5) + expect(await readdir(agentDir)).toEqual(['continuation.html']) + }) +}) + +describe('continuation grading', () => { + const base = [ + 'continuation.html', + '.incident', + '--row', + CONTINUATION_ROW_SPEC, + '--json-envelope', + '--budget', + '600', + ] + + test('answer grade separates missing, duplicate, order, and field failures', () => { + const expected = generateFixture().records.slice(0, 3) + expect(gradeAnswer(JSON.stringify({ records: expected }), expected).exact).toBe(true) + + const duplicate = [expected[0]!, expected[0]!, expected[2]!] + const duplicateGrade = gradeAnswer(JSON.stringify({ records: duplicate }), expected) + expect(duplicateGrade.exact).toBe(false) + expect(duplicateGrade.duplicateIds).toEqual([expected[0]!.id]) + expect(duplicateGrade.missingIds).toEqual([expected[1]!.id]) + + const fieldMismatch = [expected[0]!, { ...expected[1]!, owner: 'wrong-owner' }, expected[2]!] + expect( + gradeAnswer(JSON.stringify({ records: fieldMismatch }), expected).fieldMismatchIds + ).toEqual([expected[1]!.id]) + + const unexpected = [expected[0]!, expected[1]!, { ...expected[2]!, id: 'INC-99999' }] + expect(gradeAnswer(JSON.stringify({ records: unexpected }), expected).unexpectedIds).toEqual([ + 'INC-99999', + ]) + }) + + test('answer grade does not report a missing record as reordering', () => { + const expected = generateFixture().records.slice(0, 3) + const grade = gradeAnswer(JSON.stringify({ records: [expected[0]!, expected[2]!] }), expected) + expect(grade.missingIds).toEqual([expected[1]!.id]) + expect(grade.outOfOrderIds).toEqual([]) + + const reordered = gradeAnswer( + JSON.stringify({ records: [expected[2]!, expected[0]!] }), + expected + ) + expect(reordered.outOfOrderIds).toEqual([expected[2]!.id, expected[0]!.id]) + }) + + test.each([ + ['severity', ['S1']], + ['severity', { toString: null }], + ['severity', null], + ['status', ['open']], + ['status', { toString: null }], + ['status', null], + ] as const)('answer grade rejects non-string %s values', (field, value) => { + const expected = generateFixture().records.slice(0, 1) + const malformed = { ...expected[0]!, [field]: value } + const grade = gradeAnswer(JSON.stringify({ records: [malformed] }), expected) + expect(grade.schemaValid).toBe(false) + expect(grade.exact).toBe(false) + }) + + test.each([ + ['unparseable JSON', '{not-json'], + [ + 'invalid record schema', + JSON.stringify({ + records: [{ id: 'INC-00001', owner: 'owner', severity: ['S1'], status: 'open' }], + }), + ], + ])('answer grade leaves record-level failures unknown for %s', (_name, finalText) => { + const expected = generateFixture().records.slice(0, 1) + expect(gradeAnswer(finalText, expected)).toEqual({ + exact: false, + schemaValid: false, + missingIds: [], + unexpectedIds: [], + duplicateIds: [], + outOfOrderIds: [], + fieldMismatchIds: [], + }) + }) + + test('protocol grade accepts only an exact offset chain', () => { + const invocations = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...base, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'complete', offset: 1, returned: 1, next_offset: null }, + }, + 3 + ), + ] + const protocol = gradeProtocol(invocations, 600) + expect(protocol.status).toBe('pass') + expect(gradeAdoption(invocations, protocol, 600, true)).toEqual({ + adoptedEnvelope: true, + validChain: true, + budgetCompliant: true, + alternativeStrategy: null, + finalAnswerExact: true, + }) + }) + + test.each([ + ['restart', '0'], + ['gap', '2'], + ])('protocol grade rejects %s offsets', (_name, offset) => { + const fixture = generateFixture() + const invocations = [ + call( + base, + { + data: fixture.records.slice(0, 1), + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...base, '--offset', offset], + { + data: [], + meta: { + state: 'complete', + offset: Number(offset), + returned: 0, + next_offset: null, + }, + }, + 3 + ), + ] + expect(gradeProtocol(invocations, 600).status).toBe('fail') + }) + + test('protocol grade audits bypass calls before the legal envelope chain', () => { + const bypass = call(['continuation.html', '.incident', '--all'], [], 1) + const chain = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 3 + ), + call( + [...base, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'complete', offset: 1, returned: 1, next_offset: null }, + }, + 5 + ), + ] + const grade = gradeProtocol([bypass, ...chain], 600) + expect(grade.status).toBe('fail') + expect(grade.issues).toContain('--all bypass') + }) + + test('protocol grade rejects explicit limits and zero-progress more pages', () => { + const argv = [...base, '--limit', '50'] + const grade = gradeProtocol( + [ + call( + argv, + { + data: [], + meta: { state: 'more', offset: 0, returned: 0, next_offset: 0 }, + }, + 1 + ), + ], + 600 + ) + expect(grade.status).toBe('fail') + expect(grade.issues).toContain('--limit is forbidden') + expect(grade.issues).toContain('zero-progress page') + }) + + test.each(['complete', 'past_end'] as const)( + 'protocol grade rejects discovery after %s', + (state) => { + const terminalData = state === 'complete' ? [{ id: 'b' }] : [] + const terminalReturned = terminalData.length + const invocations = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...base, '--offset', '1'], + { + data: terminalData, + meta: { state, offset: 1, returned: terminalReturned, next_offset: null }, + }, + 3 + ), + call(['continuation.html', '.incident', '--count'], 120, 5), + ] + const grade = gradeProtocol(invocations, 600) + expect(grade.status).toBe('fail') + expect(grade.issues).toContain('fixture call after terminal state') + } + ) + + test.each([ + ['missing budget', base.filter((arg) => arg !== '--budget' && arg !== '600')], + ['changed budget', base.map((arg) => (arg === '600' ? '900' : arg))], + ])('protocol grade rejects %s', (_name, argv) => { + const grade = gradeProtocol( + [ + call( + argv, + { + data: [{ id: 'a' }], + meta: { state: 'complete', offset: 0, returned: 1, next_offset: null }, + }, + 1 + ), + ], + 600 + ) + expect(grade.status).toBe('fail') + }) + + test('protocol grade rejects command drift after the first page', () => { + const drifted = [...base] + drifted[1] = '.other' + const invocations = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...drifted, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'complete', offset: 1, returned: 1, next_offset: null }, + }, + 3 + ), + ] + const grade = gradeProtocol(invocations, 600) + expect(grade.status).toBe('fail') + expect(grade.issues).toContain('command drift') + }) + + test('protocol grade reports malformed, missing, and overlapping evidence', () => { + const malformed = call(base, {}, 1) + malformed.stdout = '{' + expect(gradeProtocol([malformed], 600).status).toBe('not_gradable') + expect(gradeProtocol([], 600).status).toBe('not_gradable') + + const first = call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ) + const second = call( + [...base, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'complete', offset: 1, returned: 1, next_offset: null }, + }, + 1.5 + ) + expect(gradeProtocol([first, second], 600).status).toBe('not_gradable') + }) + + test('adoption grade records alternative strategy and final correctness', () => { + const invocations = [ + call( + [ + 'continuation.html', + '.incident', + '--row', + CONTINUATION_ROW_SPEC, + '--budget', + '600', + '--offset', + '1', + ], + [], + 1 + ), + ] + const protocol = gradeProtocol(invocations, 600) + expect(gradeAdoption(invocations, protocol, 600, true)).toEqual({ + adoptedEnvelope: false, + validChain: false, + budgetCompliant: true, + alternativeStrategy: 'offset-without-envelope', + finalAnswerExact: true, + }) + }) + + test('adoption grade preserves a fallback after an envelope attempt', () => { + const attempt = call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ) + const fallback = call( + ['continuation.html', '.incident', '--all', '--budget', '600'], + [{ id: 'a' }], + 3 + ) + const invocations = [attempt, fallback] + const protocol = gradeProtocol(invocations, 600) + const adoption = gradeAdoption(invocations, protocol, 600, true) + expect(adoption.adoptedEnvelope).toBe(true) + expect(adoption.validChain).toBe(false) + expect(adoption.alternativeStrategy).toBe('all') + }) + + test('grade-run combines answer, protocol, access, and metrics', async () => { + const runRoot = await mkdtemp(join(tmpdir(), 'ax-continuation-grade-')) + const telemetryDir = join(runRoot, 'telemetry') + await mkdir(telemetryDir) + const invocations = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...base, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'more', offset: 1, returned: 1, next_offset: 2 }, + }, + 3 + ), + call( + [...base, '--offset', '2'], + { + data: [{ id: 'c' }], + meta: { state: 'complete', offset: 2, returned: 1, next_offset: null }, + }, + 5 + ), + ] + await Bun.write( + join(telemetryDir, 'ax.jsonl'), + `${invocations.map((item) => JSON.stringify(item)).join('\n')}\n` + ) + const stream = [ + ...invocations.flatMap((invocation, index) => [ + JSON.stringify({ + type: 'assistant', + message: { + content: [ + { + type: 'tool_use', + id: `tool-${index + 1}`, + name: 'Bash', + input: { command: ['ax', ...invocation.argv].join(' ') }, + }, + ], + }, + }), + JSON.stringify({ + type: 'user', + message: { + content: [ + { + type: 'tool_result', + tool_use_id: `tool-${index + 1}`, + content: invocation.stdout, + is_error: false, + }, + ], + }, + }), + ]), + JSON.stringify({ + type: 'result', + subtype: 'success', + is_error: false, + result: JSON.stringify({ records: generateFixture().records }), + duration_ms: 1200, + num_turns: 7, + total_cost_usd: 0.125, + usage: { + input_tokens: 11, + output_tokens: 7, + cache_creation_input_tokens: 5, + cache_read_input_tokens: 13, + }, + }), + ].join('\n') + await Bun.write(join(runRoot, 'stream.jsonl'), `${stream}\n`) + + expect(await gradeRunRoot(runRoot, 'conformance')).toEqual({ + evidenceStatus: 'gradable', + condition: 'conformance', + answer: { + exact: true, + schemaValid: true, + missingIds: [], + unexpectedIds: [], + duplicateIds: [], + outOfOrderIds: [], + fieldMismatchIds: [], + }, + protocol: { + status: 'pass', + issues: [], + pageCount: 3, + }, + adoption: null, + access: { + status: 'pass', + issues: [], + alternativeStrategy: null, + }, + metrics: { + durationMs: 1200, + turns: 7, + totalCostUsd: 0.125, + inputTokens: 11, + outputTokens: 7, + cacheCreationInputTokens: 5, + cacheReadInputTokens: 13, + }, + issues: [], + }) + }) + + test('grade-run rejects a chain outside the calibrated page range', async () => { + const runRoot = await mkdtemp(join(tmpdir(), 'ax-continuation-grade-short-')) + const telemetryDir = join(runRoot, 'telemetry') + await mkdir(telemetryDir) + const invocations = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...base, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'complete', offset: 1, returned: 1, next_offset: null }, + }, + 3 + ), + ] + await Bun.write( + join(telemetryDir, 'ax.jsonl'), + `${invocations.map((item) => JSON.stringify(item)).join('\n')}\n` + ) + const stream = [ + ...invocations.flatMap((invocation, index) => [ + JSON.stringify({ + type: 'assistant', + message: { + content: [ + { + type: 'tool_use', + id: `short-tool-${index + 1}`, + name: 'Bash', + input: { command: ['ax', ...invocation.argv].join(' ') }, + }, + ], + }, + }), + JSON.stringify({ + type: 'user', + message: { + content: [ + { + type: 'tool_result', + tool_use_id: `short-tool-${index + 1}`, + content: invocation.stdout, + is_error: false, + }, + ], + }, + }), + ]), + JSON.stringify({ + type: 'result', + subtype: 'success', + is_error: false, + result: JSON.stringify({ records: generateFixture().records }), + duration_ms: 1, + num_turns: 2, + total_cost_usd: 0, + usage: { + input_tokens: 1, + output_tokens: 1, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + }, + }), + ].join('\n') + await Bun.write(join(runRoot, 'stream.jsonl'), `${stream}\n`) + const grade = await gradeRunRoot(runRoot, 'conformance') + expect(grade.evidenceStatus).toBe('gradable') + expect(grade.protocol.status).toBe('fail') + expect(grade.protocol.pageCount).toBe(2) + expect(grade.protocol.issues).toContain('page count outside calibrated range') + }) + + test('grade-run records direct fixture access as an adoption condition violation', async () => { + const runRoot = await mkdtemp(join(tmpdir(), 'ax-continuation-grade-direct-')) + const telemetryDir = join(runRoot, 'telemetry') + await mkdir(telemetryDir) + const invocations = [ + call( + base, + { + data: [{ id: 'a' }], + meta: { state: 'more', offset: 0, returned: 1, next_offset: 1 }, + }, + 1 + ), + call( + [...base, '--offset', '1'], + { + data: [{ id: 'b' }], + meta: { state: 'more', offset: 1, returned: 1, next_offset: 2 }, + }, + 3 + ), + call( + [...base, '--offset', '2'], + { + data: [{ id: 'c' }], + meta: { state: 'complete', offset: 2, returned: 1, next_offset: null }, + }, + 5 + ), + ] + await Bun.write( + join(telemetryDir, 'ax.jsonl'), + `${invocations.map((item) => JSON.stringify(item)).join('\n')}\n` + ) + await Bun.write( + join(runRoot, 'stream.jsonl'), + syntheticRunStream(invocations, { directRead: true }) + ) + + const grade = await gradeRunRoot(runRoot, 'adoption-guided') + expect(grade.evidenceStatus).toBe('condition_violation') + expect(grade.access.status).toBe('violation') + expect(grade.adoption?.alternativeStrategy).toBe('direct-fixture-read') + expect(grade.answer?.exact).toBe(true) + }) + + test('direct fixture access remains a violation without ax telemetry', async () => { + const runRoot = await mkdtemp(join(tmpdir(), 'ax-continuation-grade-direct-only-')) + await mkdir(join(runRoot, 'telemetry')) + await Bun.write(join(runRoot, 'telemetry', 'ax.jsonl'), '') + await Bun.write(join(runRoot, 'stream.jsonl'), syntheticRunStream([], { directRead: true })) + + const grade = await gradeRunRoot(runRoot, 'adoption-ablated') + expect(grade.protocol.status).toBe('not_gradable') + expect(grade.access.status).toBe('violation') + expect(grade.evidenceStatus).toBe('condition_violation') + expect(grade.adoption?.alternativeStrategy).toBe('direct-fixture-read') + }) + + test('grade-run returns not_gradable for a malformed stream', async () => { + const runRoot = await mkdtemp(join(tmpdir(), 'ax-continuation-grade-malformed-')) + await mkdir(join(runRoot, 'telemetry')) + await Bun.write(join(runRoot, 'telemetry', 'ax.jsonl'), '') + await Bun.write(join(runRoot, 'stream.jsonl'), '{\n') + + const grade = await gradeRunRoot(runRoot, 'conformance') + expect(grade.evidenceStatus).toBe('not_gradable') + expect(grade.answer).toBeNull() + expect(grade.metrics).toBeNull() + expect(grade.issues).toContain('invalid stream JSON') + }) +}) + +describe('continuation runner', () => { + test('Claude args capture stream-json without session reuse', () => { + const args = buildClaudeArgs('local-model-id', 'prompt') + expect(args).toEqual([ + 'claude', + '-p', + 'prompt', + '--model', + 'local-model-id', + '--output-format', + 'stream-json', + '--verbose', + '--allowedTools', + 'Bash,Read,Grep,Glob', + '--max-turns', + '40', + ]) + expect(args).not.toContain('--resume') + }) + + test('prepareRun exposes only HTML to the Agent and uses the exact condition prompt', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-run-')) + const prepared = await prepareRun(join(parent, 'run'), 'adoption-ablated') + expect(await readdir(prepared.agentDir)).toEqual(['continuation.html']) + expect(prepared.prompt).toContain('Use ax') + expect(prepared.prompt).toContain('--budget 600') + expect(prepared.prompt).not.toContain('For automated continuation') + expect(prepared.truth).toEqual(generateFixture().records) + expect(JSON.parse(await Bun.file(join(prepared.runRoot, 'manifest.json')).text())).toEqual({ + schemaVersion: 1, + condition: 'adoption-ablated', + publicModel: 'Sonnet 5', + seed: CONTINUATION_SEED, + budget: CONTINUATION_BUDGET, + }) + }) + + test('prepareRun rejects repository descendants before creating them', async () => { + await expect(prepareRun(process.cwd(), 'conformance')).rejects.toThrow( + 'run-root must be outside the repository' + ) + + const forbidden = join(process.cwd(), 'bench', '.continuation-run-must-not-exist') + await expect(prepareRun(forbidden, 'conformance')).rejects.toThrow( + 'run-root must be outside the repository' + ) + expect(await Bun.file(forbidden).exists()).toBe(false) + + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-symlink-')) + const repoLink = join(parent, 'repo-link') + await symlink(process.cwd(), repoLink, 'dir') + const linkedForbidden = join(repoLink, 'bench', '.continuation-run-must-not-exist') + await expect(prepareRun(linkedForbidden, 'conformance')).rejects.toThrow( + 'run-root must be outside the repository' + ) + expect(await Bun.file(linkedForbidden).exists()).toBe(false) + }) + + test('capture preserves raw streams and the inspector emits structure only', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-fake-')) + const runRoot = join(parent, 'run') + const prepared = await prepareRun(runRoot, 'conformance') + const fakeBin = join(parent, 'fake-bin') + await mkdir(fakeBin) + const fakeClaude = join(fakeBin, 'claude') + await Bun.write( + fakeClaude, + [ + '#!/bin/sh', + `printf '%s\\n' '{"type":"assistant","message":{"content":"synthetic-secret"}}'`, + `printf '%s\\n' '{"type":"result","result":"{}"}'`, + `printf '%s\\n' 'fake stderr' >&2`, + ].join('\n') + ) + await chmod(fakeClaude, 0o755) + prepared.provisioned.childEnv.PATH = `${fakeBin}:${prepared.provisioned.childEnv.PATH}` + expect(await capturePilot(prepared, 'local-model-id')).toBe(0) + const raw = await Bun.file(prepared.rawTracePath).text() + expect(raw).toBe( + '{"type":"assistant","message":{"content":"synthetic-secret"}}\n' + + '{"type":"result","result":"{}"}\n' + ) + expect(await Bun.file(prepared.stderrPath).text()).toBe('fake stderr\n') + expect(await readdir(prepared.agentDir)).toEqual(['continuation.html']) + const shapes = JSON.stringify(inspectStreamShapes(raw)) + expect(shapes).toContain('message.content:string') + expect(shapes).not.toContain('synthetic-secret') + }) + + test('inspector redacts dynamic model keys and groups repeated event shapes', () => { + const raw = [ + JSON.stringify({ + type: 'result', + modelUsage: { + 'private-model-route': { + inputTokens: 12, + outputTokens: 4, + }, + }, + result: '{"records":[]}', + }), + JSON.stringify({ + type: 'result', + modelUsage: { + 'another-private-route': { + inputTokens: 8, + outputTokens: 3, + }, + }, + result: '{"records":[]}', + }), + ].join('\n') + + const shapes = inspectStreamShapes(raw) + expect(shapes).toEqual([ + { + type: 'result', + count: 2, + paths: [ + '$:object', + 'modelUsage..inputTokens:number', + 'modelUsage..outputTokens:number', + 'modelUsage.:object', + 'modelUsage:object', + 'result:string', + 'type:string', + ], + }, + ]) + const serialized = JSON.stringify(shapes) + expect(serialized).not.toContain('private-model-route') + expect(serialized).not.toContain('another-private-route') + }) +}) diff --git a/bench/continuation/fixture.ts b/bench/continuation/fixture.ts new file mode 100644 index 0000000..4cea6da --- /dev/null +++ b/bench/continuation/fixture.ts @@ -0,0 +1,91 @@ +import { join } from 'node:path' + +export const CONTINUATION_SEED = 0x53a11 +export const CONTINUATION_BUDGET = 600 +export const CONTINUATION_SELECTOR = '.incident' +export const CONTINUATION_ROW_SPEC = 'id=@data-id,owner=.owner,severity=.severity,status=.status' + +export type IncidentRecord = { + id: string + owner: string + severity: 'S1' | 'S2' | 'S3' | 'S4' + status: 'open' | 'investigating' | 'mitigated' | 'closed' +} + +export type GeneratedFixture = { + html: string + records: IncidentRecord[] +} + +const owners = [ + 'atlas-ops', + 'beacon-api', + 'cobalt-web', + 'delta-data', + 'ember-edge', + 'fjord-payments', +] as const +const severities = ['S1', 'S2', 'S3', 'S4'] as const +const statuses = ['open', 'investigating', 'mitigated', 'closed'] as const + +function mulberry32(seed: number): () => number { + return () => { + seed |= 0 + seed = (seed + 0x6d2b79f5) | 0 + let value = Math.imul(seed ^ (seed >>> 15), 1 | seed) + value = (value + Math.imul(value ^ (value >>> 7), 61 | value)) ^ value + return ((value ^ (value >>> 14)) >>> 0) / 4294967296 + } +} + +function pick(items: readonly T[], random: () => number): T { + return items[Math.floor(random() * items.length)]! +} + +export function generateFixture(seed = CONTINUATION_SEED): GeneratedFixture { + const random = mulberry32(seed) + const ids = Array.from( + { length: 120 }, + (_, index) => `INC-${String(73001 + index).padStart(5, '0')}` + ) + for (let index = ids.length - 1; index > 0; index--) { + const swap = Math.floor(random() * (index + 1)) + ;[ids[index], ids[swap]] = [ids[swap]!, ids[index]!] + } + const records = ids.map((id) => ({ + id, + owner: pick(owners, random), + severity: pick(severities, random), + status: pick(statuses, random), + })) + const incidents = records.map( + (record) => + `
` + + `${record.owner}` + + `${record.severity}` + + `${record.status}` + + `
` + ) + const previews = Array.from( + { length: 12 }, + (_, index) => + `
` + + `preview` + + `S1` + + `open` + + `
` + ) + return { + records, + html: `\n${[...incidents, ...previews].join('\n')}\n\n`, + } +} + +export async function writeAgentFixture( + agentDir: string, + seed = CONTINUATION_SEED +): Promise { + const fixture = generateFixture(seed) + await Bun.write(join(agentDir, 'continuation.html'), fixture.html) + return fixture.records +} diff --git a/bench/continuation/grade-run.ts b/bench/continuation/grade-run.ts new file mode 100644 index 0000000..8ee9057 --- /dev/null +++ b/bench/continuation/grade-run.ts @@ -0,0 +1,99 @@ +import { join, resolve } from 'node:path' +import { auditToolContext, type ToolContextAudit } from './audit' +import { CONTINUATION_BUDGET, generateFixture } from './fixture' +import { + gradeAdoption, + gradeAnswer, + gradeProtocol, + type AdoptionGrade, + type AnswerGrade, + type ProtocolGrade, +} from './grade' +import { parseCondition, type Condition } from './run' +import { normalizeClaudeStream, type RunMetrics } from './stream' +import { readAxTelemetry } from './telemetry' + +export type RunGrade = { + evidenceStatus: 'gradable' | 'condition_violation' | 'not_gradable' + condition: Condition + answer: AnswerGrade | null + protocol: ProtocolGrade + adoption: AdoptionGrade | null + access: ToolContextAudit + metrics: RunMetrics | null + issues: string[] +} + +function calibratedProtocol(protocol: ProtocolGrade): ProtocolGrade { + if (protocol.status === 'pass' && (protocol.pageCount < 3 || protocol.pageCount > 5)) { + return { + ...protocol, + status: 'fail', + issues: ['page count outside calibrated range'], + } + } + return protocol +} + +export async function gradeRunRoot(runRootInput: string, condition: Condition): Promise { + const runRoot = resolve(runRootInput) + const [raw, invocations] = await Promise.all([ + Bun.file(join(runRoot, 'stream.jsonl')).text(), + readAxTelemetry(join(runRoot, 'telemetry', 'ax.jsonl')), + ]) + const normalized = normalizeClaudeStream(raw) + const protocol = calibratedProtocol(gradeProtocol(invocations, CONTINUATION_BUDGET)) + const access = auditToolContext( + normalized.toolCalls, + invocations, + 'continuation.html', + join(runRoot, 'agent') + ) + const answer = + normalized.status === 'complete' + ? gradeAnswer(normalized.finalText, generateFixture().records) + : null + let adoption = + condition === 'conformance' + ? null + : gradeAdoption(invocations, protocol, CONTINUATION_BUDGET, answer?.exact ?? null) + if (adoption && access.status === 'violation') { + adoption = { ...adoption, alternativeStrategy: 'direct-fixture-read' } + } + + const evidenceStatus = + normalized.status === 'not_gradable' || access.status === 'not_gradable' + ? 'not_gradable' + : access.status === 'violation' + ? 'condition_violation' + : protocol.status === 'not_gradable' + ? 'not_gradable' + : 'gradable' + const issues = [ + ...(normalized.status === 'not_gradable' ? normalized.issues : []), + ...(protocol.status === 'not_gradable' ? protocol.issues : []), + ...(access.status === 'pass' ? [] : access.issues), + ] + + return { + evidenceStatus, + condition, + answer, + protocol, + adoption, + access, + metrics: normalized.metrics, + issues: [...new Set(issues)], + } +} + +if (import.meta.main) { + const [runRoot, conditionValue] = process.argv.slice(2) + if (!runRoot) throw new Error('run-root is required') + const condition = parseCondition(conditionValue) + const grade = await gradeRunRoot(runRoot, condition) + console.log(JSON.stringify(grade, null, 2)) + process.exit( + grade.evidenceStatus === 'gradable' ? 0 : grade.evidenceStatus === 'condition_violation' ? 1 : 2 + ) +} diff --git a/bench/continuation/grade.ts b/bench/continuation/grade.ts new file mode 100644 index 0000000..837755c --- /dev/null +++ b/bench/continuation/grade.ts @@ -0,0 +1,349 @@ +import type { IncidentRecord } from './fixture' +import { hasOverlappingInvocations, type AxInvocation } from './telemetry' + +export type AnswerGrade = { + exact: boolean + schemaValid: boolean + missingIds: string[] + unexpectedIds: string[] + duplicateIds: string[] + outOfOrderIds: string[] + fieldMismatchIds: string[] +} + +export type ProtocolGrade = { + status: 'pass' | 'fail' | 'not_gradable' + issues: string[] + pageCount: number +} + +export type AdoptionGrade = { + adoptedEnvelope: boolean + validChain: boolean + budgetCompliant: boolean + alternativeStrategy: + | 'offset-without-envelope' + | 'all' + | 'limit-bypass' + | 'plain-ax' + | 'no-instrumented-ax' + | 'direct-fixture-read' + | null + finalAnswerExact: boolean | null +} + +type Envelope = { + data: unknown[] + meta: { + state: 'more' | 'complete' | 'past_end' + offset: number + returned: number + next_offset: number | null + } +} + +function unique(items: string[]): string[] { + return [...new Set(items)] +} + +function exactKeys(value: object, expected: string[]): boolean { + return Object.keys(value).sort().join(',') === [...expected].sort().join(',') +} + +function isIncidentRecord(value: unknown): value is IncidentRecord { + if (!value || typeof value !== 'object') return false + const record = value as Record + return ( + exactKeys(record, ['id', 'owner', 'severity', 'status']) && + typeof record.id === 'string' && + typeof record.owner === 'string' && + typeof record.severity === 'string' && + ['S1', 'S2', 'S3', 'S4'].includes(record.severity) && + typeof record.status === 'string' && + ['open', 'investigating', 'mitigated', 'closed'].includes(record.status) + ) +} + +export function gradeAnswer(finalText: string, expected: IncidentRecord[]): AnswerGrade { + let parsed: unknown + try { + parsed = JSON.parse(finalText) + } catch { + return { + exact: false, + schemaValid: false, + missingIds: [], + unexpectedIds: [], + duplicateIds: [], + outOfOrderIds: [], + fieldMismatchIds: [], + } + } + if ( + !parsed || + typeof parsed !== 'object' || + !exactKeys(parsed, ['records']) || + !Array.isArray((parsed as { records?: unknown }).records) || + !(parsed as { records: unknown[] }).records.every(isIncidentRecord) + ) { + return { + exact: false, + schemaValid: false, + missingIds: [], + unexpectedIds: [], + duplicateIds: [], + outOfOrderIds: [], + fieldMismatchIds: [], + } + } + const actual = (parsed as { records: IncidentRecord[] }).records + const expectedById = new Map(expected.map((record) => [record.id, record])) + const actualIds = actual.map((record) => record.id) + const expectedIds = expected.map((record) => record.id) + const duplicateIds = unique(actualIds.filter((id, index) => actualIds.indexOf(id) !== index)) + const missingIds = expectedIds.filter((id) => !actualIds.includes(id)) + const unexpectedIds = unique(actualIds.filter((id) => !expectedById.has(id))) + const duplicateSet = new Set(duplicateIds) + const comparableActualIds = actualIds.filter( + (id) => expectedById.has(id) && !duplicateSet.has(id) + ) + const comparableIdSet = new Set(comparableActualIds) + const comparableExpectedIds = expectedIds.filter((id) => comparableIdSet.has(id)) + const outOfOrderIds = unique( + comparableActualIds.filter((id, index) => id !== comparableExpectedIds[index]) + ) + const fieldMismatchIds = unique( + actual + .filter((record) => { + const wanted = expectedById.get(record.id) + return ( + wanted !== undefined && + (record.owner !== wanted.owner || + record.severity !== wanted.severity || + record.status !== wanted.status) + ) + }) + .map((record) => record.id) + ) + const exact = + actual.length === expected.length && + missingIds.length === 0 && + unexpectedIds.length === 0 && + duplicateIds.length === 0 && + outOfOrderIds.length === 0 && + fieldMismatchIds.length === 0 + return { + exact, + schemaValid: true, + missingIds, + unexpectedIds, + duplicateIds, + outOfOrderIds, + fieldMismatchIds, + } +} + +function flagValues(argv: string[], flag: string): string[] { + return argv.flatMap((arg, index) => { + if (arg === flag && argv[index + 1]) return [argv[index + 1]!] + if (arg.startsWith(`${flag}=`)) return [arg.slice(flag.length + 1)] + return [] + }) +} + +function hasFlag(argv: string[], flag: string): boolean { + return argv.some((arg) => arg === flag || arg.startsWith(`${flag}=`)) +} + +function fixtureCalls(invocations: AxInvocation[]): AxInvocation[] { + return invocations.filter((call) => + call.argv.some((arg) => arg === 'continuation.html' || arg.endsWith('/continuation.html')) + ) +} + +function extractionCalls(invocations: AxInvocation[]): AxInvocation[] { + return fixtureCalls(invocations).filter( + (call) => + !call.argv.includes('--count') && + !call.argv.includes('--outline') && + !call.argv.includes('--locate') + ) +} + +function stableCommand(argv: string[]): string { + const stable: string[] = [] + for (let index = 0; index < argv.length; index++) { + if (argv[index] === '--offset') { + index++ + continue + } + if (argv[index]!.startsWith('--offset=')) continue + stable.push(argv[index]!) + } + return JSON.stringify(stable) +} + +function parseEnvelope(call: AxInvocation): Envelope | null { + try { + const value = JSON.parse(call.stdout) as Envelope + if ( + !value || + !Array.isArray(value.data) || + !value.meta || + !['more', 'complete', 'past_end'].includes(value.meta.state) || + !Number.isInteger(value.meta.offset) || + !Number.isInteger(value.meta.returned) || + !(value.meta.next_offset === null || Number.isInteger(value.meta.next_offset)) + ) { + return null + } + return value + } catch { + return null + } +} + +export function gradeProtocol(invocations: AxInvocation[], budget: number): ProtocolGrade { + const fixture = fixtureCalls(invocations) + if (fixture.length === 0) { + return { + status: 'not_gradable', + issues: ['no instrumented fixture calls'], + pageCount: 0, + } + } + const issues: string[] = [] + if (fixture.some((call) => call.argv.includes('--all'))) issues.push('--all bypass') + if (fixture.some((call) => hasFlag(call.argv, '--limit'))) { + issues.push('--limit is forbidden') + } + if (hasOverlappingInvocations(fixture)) { + return { + status: 'not_gradable', + issues: unique([...issues, 'overlapping fixture calls']), + pageCount: 0, + } + } + const calls = extractionCalls(invocations) + if (calls.length === 0) { + return { + status: 'fail', + issues: unique([...issues, 'no continuation extraction calls']), + pageCount: 0, + } + } + const command = stableCommand(calls[0]!.argv) + let expectedOffset = 0 + let terminal = false + for (const call of calls) { + const budgets = flagValues(call.argv, '--budget') + if (budgets.length !== 1 || budgets[0] !== String(budget)) { + issues.push('budget changed or missing') + } + if (call.argv.includes('--all')) issues.push('--all bypass') + if (!call.argv.includes('--json-envelope')) { + issues.push('missing --json-envelope') + continue + } + if (stableCommand(call.argv) !== command) issues.push('command drift') + if (terminal) issues.push('call after terminal state') + const envelope = parseEnvelope(call) + if (!envelope) { + return { + status: 'not_gradable', + issues: unique([...issues, 'unparseable envelope']), + pageCount: calls.length, + } + } + const offsets = flagValues(call.argv, '--offset') + const actualOffset = offsets.length === 0 ? 0 : Number(offsets[0]) + if ( + offsets.length > 1 || + !Number.isInteger(actualOffset) || + actualOffset !== expectedOffset || + envelope.meta.offset !== actualOffset + ) { + issues.push('offset chain mismatch') + } + if (envelope.meta.returned !== envelope.data.length) { + issues.push('returned count mismatch') + } + if (envelope.meta.state === 'more') { + const next = envelope.meta.next_offset + if ( + envelope.meta.returned <= 0 || + next === null || + next <= actualOffset || + next !== actualOffset + envelope.meta.returned + ) { + if (envelope.meta.returned <= 0 || next === actualOffset) { + issues.push('zero-progress page') + } + issues.push('invalid next_offset') + } else { + expectedOffset = next + } + } else { + terminal = true + const terminalFixtureIndex = fixture.indexOf(call) + if (terminalFixtureIndex < fixture.length - 1) { + issues.push('fixture call after terminal state') + } + if (envelope.meta.next_offset !== null) { + issues.push('terminal next_offset must be null') + } + if (envelope.meta.state === 'past_end' && actualOffset === 0) { + issues.push('past_end cannot start at offset zero') + } + } + } + if (!terminal) issues.push('missing terminal state') + return { + status: issues.length === 0 ? 'pass' : 'fail', + issues: unique(issues), + pageCount: calls.length, + } +} + +export function gradeAdoption( + invocations: AxInvocation[], + protocol: ProtocolGrade, + budget: number, + finalAnswerExact: boolean | null +): AdoptionGrade { + const fixture = fixtureCalls(invocations) + const calls = extractionCalls(invocations) + const adoptedEnvelope = calls.some((call) => call.argv.includes('--json-envelope')) + const validChain = protocol.status === 'pass' + const nonEnvelopeCalls = calls.filter((call) => !call.argv.includes('--json-envelope')) + const alternativeStrategy = validChain + ? null + : fixture.some((call) => call.argv.includes('--all')) + ? 'all' + : fixture.some((call) => hasFlag(call.argv, '--limit')) + ? 'limit-bypass' + : nonEnvelopeCalls.some((call) => flagValues(call.argv, '--offset').length > 0) + ? 'offset-without-envelope' + : nonEnvelopeCalls.length > 0 + ? 'plain-ax' + : fixture.length === 0 + ? 'no-instrumented-ax' + : null + return { + adoptedEnvelope, + validChain, + budgetCompliant: + calls.length > 0 && + calls.every((call) => { + const values = flagValues(call.argv, '--budget') + return ( + values.length === 1 && + values[0] === String(budget) && + !call.argv.includes('--all') && + !hasFlag(call.argv, '--limit') + ) + }), + alternativeStrategy, + finalAnswerExact, + } +} diff --git a/bench/continuation/inspect-stream.ts b/bench/continuation/inspect-stream.ts new file mode 100644 index 0000000..cc2afbf --- /dev/null +++ b/bench/continuation/inspect-stream.ts @@ -0,0 +1,65 @@ +export type StreamShapeSummary = { + type: string | null + count: number + paths: string[] +} + +function kind(value: unknown): string { + if (value === null) return 'null' + if (Array.isArray(value)) return 'array' + return typeof value +} + +function childPath(path: string, key: string): string { + if (path === 'modelUsage') return 'modelUsage.' + return path ? `${path}.${key}` : key +} + +function collectPaths(value: unknown, path: string, paths: Set): void { + paths.add(`${path || '$'}:${kind(value)}`) + if (Array.isArray(value)) { + for (const item of value) collectPaths(item, `${path || '$'}[]`, paths) + return + } + if (value && typeof value === 'object') { + for (const [key, child] of Object.entries(value)) { + collectPaths(child, childPath(path, key), paths) + } + } +} + +export function inspectStreamShapes(rawJsonl: string): StreamShapeSummary[] { + const grouped = new Map() + for (const [index, rawLine] of rawJsonl.split('\n').entries()) { + const line = rawLine.trim() + if (!line) continue + let event: unknown + try { + event = JSON.parse(line) + } catch { + throw new Error(`invalid stream JSON on line ${index + 1}`) + } + const paths = new Set() + collectPaths(event, '', paths) + const type = + event && + typeof event === 'object' && + typeof (event as Record).type === 'string' + ? String((event as Record).type) + : null + const key = type ?? '' + const current = grouped.get(key) ?? { type, count: 0, paths: [] } + current.count++ + current.paths = [...new Set([...current.paths, ...paths])].sort() + grouped.set(key, current) + } + return [...grouped.values()].sort((left, right) => + String(left.type).localeCompare(String(right.type)) + ) +} + +if (import.meta.main) { + const path = process.argv[2] + if (!path) throw new Error('stream JSONL path is required') + console.log(JSON.stringify(inspectStreamShapes(await Bun.file(path).text()), null, 2)) +} diff --git a/bench/continuation/matrix.test.ts b/bench/continuation/matrix.test.ts new file mode 100644 index 0000000..3f0fe97 --- /dev/null +++ b/bench/continuation/matrix.test.ts @@ -0,0 +1,351 @@ +import { expect, test } from 'bun:test' +import { mkdir, mkdtemp } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { RunGrade } from './grade-run' +import { MATRIX_CONDITIONS, runMatrix, type MatrixDependencies } from './matrix' +import type { Condition, PreparedRun } from './run' + +function gradable(condition: Condition, run: number): RunGrade { + const answerFailure = run === 1 + return { + evidenceStatus: 'gradable', + condition, + answer: { + exact: !answerFailure, + schemaValid: !answerFailure, + missingIds: answerFailure ? ['synthetic-missing'] : [], + unexpectedIds: answerFailure ? ['synthetic-unexpected'] : [], + duplicateIds: answerFailure ? ['synthetic-duplicate'] : [], + outOfOrderIds: answerFailure ? ['synthetic-order'] : [], + fieldMismatchIds: answerFailure ? ['synthetic-field'] : [], + }, + protocol: { status: 'pass', issues: [], pageCount: 5 }, + adoption: + condition === 'conformance' + ? null + : { + adoptedEnvelope: true, + validChain: true, + budgetCompliant: true, + alternativeStrategy: null, + finalAnswerExact: true, + }, + access: { status: 'pass', issues: [], alternativeStrategy: null }, + metrics: { + durationMs: run * 100, + turns: run, + totalCostUsd: run / 100, + inputTokens: run * 10, + outputTokens: run * 5, + cacheCreationInputTokens: run * 3, + cacheReadInputTokens: run * 7, + }, + issues: [], + } +} + +test('matrix uses the committed interleaved 3+3+3 order', () => { + expect(MATRIX_CONDITIONS).toEqual([ + 'conformance', + 'adoption-guided', + 'adoption-ablated', + 'conformance', + 'adoption-guided', + 'adoption-ablated', + 'conformance', + 'adoption-guided', + 'adoption-ablated', + ]) +}) + +test('matrix refuses to run without exact paid confirmation', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-test-')) + let calls = 0 + const dependencies = { + prepare: async () => { + calls++ + throw new Error('must not prepare') + }, + capture: async () => { + calls++ + return 0 + }, + grade: async () => { + calls++ + throw new Error('must not grade') + }, + } as MatrixDependencies + + await expect( + runMatrix({ + matrixRoot: join(parent, 'matrix'), + model: 'local-model-id', + confirmation: undefined, + dependencies, + }) + ).rejects.toThrow('exactly 9 paid runs require --confirm-paid-runs=9') + expect(calls).toBe(0) +}) + +test('matrix captures and grades each committed slot exactly once', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-test-')) + const matrixRoot = join(parent, 'matrix') + const prepared: Condition[] = [] + const captured: Condition[] = [] + const graded: Condition[] = [] + const dependencies: MatrixDependencies = { + prepare: async (runRoot, condition) => { + prepared.push(condition) + return { runRoot, condition } as unknown as PreparedRun + }, + capture: async (run) => { + captured.push((run as unknown as { condition: Condition }).condition) + return 0 + }, + grade: async (_runRoot, condition) => { + graded.push(condition) + return gradable(condition, graded.length) + }, + } + + const summary = await runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + + expect(prepared).toEqual([...MATRIX_CONDITIONS]) + expect(captured).toEqual([...MATRIX_CONDITIONS]) + expect(graded).toEqual([...MATRIX_CONDITIONS]) + expect(summary.runs).toHaveLength(9) + expect(summary.runs.map((run) => run.condition)).toEqual([...MATRIX_CONDITIONS]) + expect(summary.runs[0]!.answerFailureModes).toEqual({ + schemaInvalid: true, + missingRecords: false, + unexpectedRecords: false, + duplicateRecords: false, + orderingErrors: false, + fieldMismatches: false, + }) + expect(await Bun.file(join(matrixRoot, 'summary.json')).json()).toEqual(summary) + const serialized = JSON.stringify(summary) + expect(serialized).not.toContain('synthetic-') + expect(serialized).not.toContain('local-model-id') + expect(serialized).not.toContain('session_id') + expect(serialized).not.toContain('/tmp/') +}) + +test('matrix stops without retry after a non-zero Claude exit', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-stop-')) + const matrixRoot = join(parent, 'matrix') + let preparedCount = 0 + let captureCount = 0 + let gradeCount = 0 + const dependencies: MatrixDependencies = { + prepare: async (runRoot, condition) => { + preparedCount++ + return { runRoot, condition } as unknown as PreparedRun + }, + capture: async () => { + captureCount++ + return captureCount === 2 ? 7 : 0 + }, + grade: async (_runRoot, condition) => { + gradeCount++ + return gradable(condition, gradeCount) + }, + } + + await expect( + runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + ).rejects.toThrow('Claude capture exited non-zero') + expect({ preparedCount, captureCount, gradeCount }).toEqual({ + preparedCount: 2, + captureCount: 2, + gradeCount: 1, + }) + expect(await Bun.file(join(matrixRoot, 'matrix-status.json')).json()).toEqual({ + schemaVersion: 1, + state: 'stopped', + completedRuns: 1, + stopReason: 'claude_nonzero', + }) +}) + +test('matrix records an infrastructure stop when preparation throws', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-infrastructure-')) + const matrixRoot = join(parent, 'matrix') + const dependencies: MatrixDependencies = { + prepare: async () => { + throw new Error('synthetic preparation failure') + }, + capture: async () => 0, + grade: async (_runRoot, condition) => gradable(condition, 1), + } + + await expect( + runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + ).rejects.toThrow('synthetic preparation failure') + expect(await Bun.file(join(matrixRoot, 'matrix-status.json')).json()).toEqual({ + schemaVersion: 1, + state: 'stopped', + completedRuns: 0, + stopReason: 'infrastructure_error', + }) +}) + +test.each(['capture', 'grade'] as const)( + 'matrix records an infrastructure stop when %s throws', + async (stage) => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-infrastructure-')) + const matrixRoot = join(parent, 'matrix') + const dependencies: MatrixDependencies = { + prepare: async (runRoot, condition) => ({ runRoot, condition }) as unknown as PreparedRun, + capture: async () => { + if (stage === 'capture') throw new Error('synthetic capture failure') + return 0 + }, + grade: async (_runRoot, condition) => { + if (stage === 'grade') throw new Error('synthetic grade failure') + return gradable(condition, 1) + }, + } + + await expect( + runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + ).rejects.toThrow(`synthetic ${stage} failure`) + expect(await Bun.file(join(matrixRoot, 'matrix-status.json')).json()).toEqual({ + schemaVersion: 1, + state: 'stopped', + completedRuns: 0, + stopReason: 'infrastructure_error', + }) + } +) + +test('matrix records an infrastructure stop when summary persistence throws', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-infrastructure-')) + const matrixRoot = join(parent, 'matrix') + const dependencies: MatrixDependencies = { + prepare: async (runRoot, condition) => { + await mkdir(join(matrixRoot, 'summary.json')) + return { runRoot, condition } as unknown as PreparedRun + }, + capture: async () => 0, + grade: async (_runRoot, condition) => gradable(condition, 1), + } + + await expect( + runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + ).rejects.toThrow() + expect(await Bun.file(join(matrixRoot, 'matrix-status.json')).json()).toEqual({ + schemaVersion: 1, + state: 'stopped', + completedRuns: 1, + stopReason: 'infrastructure_error', + }) +}) + +test.each(['not_gradable', 'condition_violation'] as const)( + 'matrix stops after a %s grade', + async (evidenceStatus) => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-stop-')) + const matrixRoot = join(parent, 'matrix') + let captureCount = 0 + let gradeCount = 0 + const dependencies: MatrixDependencies = { + prepare: async (runRoot, condition) => ({ runRoot, condition }) as unknown as PreparedRun, + capture: async () => { + captureCount++ + return 0 + }, + grade: async (_runRoot, condition) => { + gradeCount++ + const grade = gradable(condition, gradeCount) + return gradeCount === 3 ? { ...grade, evidenceStatus } : grade + }, + } + + await expect( + runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + ).rejects.toThrow(`matrix stopped: ${evidenceStatus}`) + expect({ captureCount, gradeCount }).toEqual({ captureCount: 3, gradeCount: 3 }) + expect(await Bun.file(join(matrixRoot, 'matrix-status.json')).json()).toEqual({ + schemaVersion: 1, + state: 'stopped', + completedRuns: 2, + stopReason: evidenceStatus, + }) + } +) + +test('matrix stops when a grade reports the wrong condition', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-condition-')) + const matrixRoot = join(parent, 'matrix') + let gradeCount = 0 + const dependencies: MatrixDependencies = { + prepare: async (runRoot, condition) => ({ runRoot, condition }) as unknown as PreparedRun, + capture: async () => 0, + grade: async (_runRoot, condition) => { + gradeCount++ + return gradable(gradeCount === 2 ? 'adoption-ablated' : condition, gradeCount) + }, + } + + await expect( + runMatrix({ + matrixRoot, + model: 'local-model-id', + confirmation: '9', + dependencies, + }) + ).rejects.toThrow('grade condition mismatch') + expect(gradeCount).toBe(2) + expect(await Bun.file(join(matrixRoot, 'matrix-status.json')).json()).toEqual({ + schemaVersion: 1, + state: 'stopped', + completedRuns: 1, + stopReason: 'not_gradable', + }) +}) + +test('matrix CLI requires the local model before creating the root', async () => { + const parent = await mkdtemp(join(tmpdir(), 'ax-continuation-matrix-cli-')) + const matrixRoot = join(parent, 'matrix') + const env = { ...process.env } + delete env.AX_BENCH_MODEL + const proc = Bun.spawnSync( + ['bun', new URL('./matrix.ts', import.meta.url).pathname, matrixRoot, '--confirm-paid-runs=9'], + { cwd: process.cwd(), env } + ) + expect(proc.exitCode).not.toBe(0) + expect(proc.stderr.toString()).toContain('AX_BENCH_MODEL is required') + expect(await Bun.file(matrixRoot).exists()).toBe(false) +}) diff --git a/bench/continuation/matrix.ts b/bench/continuation/matrix.ts new file mode 100644 index 0000000..80ab79e --- /dev/null +++ b/bench/continuation/matrix.ts @@ -0,0 +1,214 @@ +import { mkdir, rename } from 'node:fs/promises' +import { dirname, join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { gradeRunRoot, type RunGrade } from './grade-run' +import type { AdoptionGrade, ProtocolGrade } from './grade' +import { + assertExternalRunRoot, + capturePilot, + prepareRun, + type Condition, + type PreparedRun, +} from './run' + +export const MATRIX_CONDITIONS: readonly Condition[] = [ + 'conformance', + 'adoption-guided', + 'adoption-ablated', + 'conformance', + 'adoption-guided', + 'adoption-ablated', + 'conformance', + 'adoption-guided', + 'adoption-ablated', +] + +export type PublicRunSummary = { + run: number + condition: Condition + publicModel: 'Sonnet 5' + answerExact: boolean + schemaValid: boolean + answerFailureModes: { + schemaInvalid: boolean + missingRecords: boolean + unexpectedRecords: boolean + duplicateRecords: boolean + orderingErrors: boolean + fieldMismatches: boolean + } + protocolStatus: ProtocolGrade['status'] + pageCount: number + adoptedEnvelope: boolean | null + validChain: boolean | null + budgetCompliant: boolean | null + alternativeStrategy: AdoptionGrade['alternativeStrategy'] | null + turns: number + durationMs: number + totalCostUsd: number + inputTokens: number + outputTokens: number + cacheCreationInputTokens: number + cacheReadInputTokens: number +} + +export type MatrixSummary = { + schemaVersion: 1 + publicModel: 'Sonnet 5' + runs: PublicRunSummary[] +} + +type MatrixStatus = { + schemaVersion: 1 + state: 'running' | 'stopped' | 'complete' + completedRuns: number + stopReason?: 'claude_nonzero' | 'not_gradable' | 'condition_violation' | 'infrastructure_error' +} + +export type MatrixDependencies = { + prepare: (runRoot: string, condition: Condition) => Promise + capture: (prepared: PreparedRun, model: string) => Promise + grade: (runRoot: string, condition: Condition) => Promise +} + +export type MatrixOptions = { + matrixRoot: string + model: string | undefined + confirmation: string | undefined + dependencies?: MatrixDependencies +} + +const DEFAULT_DEPENDENCIES: MatrixDependencies = { + prepare: prepareRun, + capture: capturePilot, + grade: gradeRunRoot, +} + +async function writeJsonAtomic(path: string, value: unknown): Promise { + const temporary = `${path}.tmp` + await Bun.write(temporary, `${JSON.stringify(value, null, 2)}\n`) + await rename(temporary, path) +} + +function publicSummary(run: number, grade: RunGrade): PublicRunSummary { + if (!grade.answer || !grade.metrics) throw new Error('gradable run is missing answer or metrics') + const recordLevelFailuresKnown = grade.answer.schemaValid + return { + run, + condition: grade.condition, + publicModel: 'Sonnet 5', + answerExact: grade.answer.exact, + schemaValid: grade.answer.schemaValid, + answerFailureModes: { + schemaInvalid: !grade.answer.schemaValid, + missingRecords: recordLevelFailuresKnown && grade.answer.missingIds.length > 0, + unexpectedRecords: recordLevelFailuresKnown && grade.answer.unexpectedIds.length > 0, + duplicateRecords: recordLevelFailuresKnown && grade.answer.duplicateIds.length > 0, + orderingErrors: recordLevelFailuresKnown && grade.answer.outOfOrderIds.length > 0, + fieldMismatches: recordLevelFailuresKnown && grade.answer.fieldMismatchIds.length > 0, + }, + protocolStatus: grade.protocol.status, + pageCount: grade.protocol.pageCount, + adoptedEnvelope: grade.adoption?.adoptedEnvelope ?? null, + validChain: grade.adoption?.validChain ?? null, + budgetCompliant: grade.adoption?.budgetCompliant ?? null, + alternativeStrategy: grade.adoption?.alternativeStrategy ?? null, + turns: grade.metrics.turns, + durationMs: grade.metrics.durationMs, + totalCostUsd: grade.metrics.totalCostUsd, + inputTokens: grade.metrics.inputTokens, + outputTokens: grade.metrics.outputTokens, + cacheCreationInputTokens: grade.metrics.cacheCreationInputTokens, + cacheReadInputTokens: grade.metrics.cacheReadInputTokens, + } +} + +export async function runMatrix(options: MatrixOptions): Promise { + if (options.confirmation !== '9') { + throw new Error('exactly 9 paid runs require --confirm-paid-runs=9') + } + if (!options.model) throw new Error('AX_BENCH_MODEL is required') + const dependencies = options.dependencies ?? DEFAULT_DEPENDENCIES + const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '../..') + const matrixRoot = await assertExternalRunRoot(repoRoot, options.matrixRoot) + await mkdir(matrixRoot, { recursive: false }) + const summary: MatrixSummary = { schemaVersion: 1, publicModel: 'Sonnet 5', runs: [] } + const statusPath = join(matrixRoot, 'matrix-status.json') + let stopped = false + const writeStopped = async (stopReason: NonNullable) => { + await writeJsonAtomic(statusPath, { + schemaVersion: 1, + state: 'stopped', + completedRuns: summary.runs.length, + stopReason, + } satisfies MatrixStatus) + stopped = true + } + + try { + await writeJsonAtomic(statusPath, { + schemaVersion: 1, + state: 'running', + completedRuns: 0, + } satisfies MatrixStatus) + for (const [index, condition] of MATRIX_CONDITIONS.entries()) { + const runNumber = index + 1 + const runRoot = join(matrixRoot, `run-${String(runNumber).padStart(2, '0')}`) + const prepared = await dependencies.prepare(runRoot, condition) + const exitCode = await dependencies.capture(prepared, options.model) + if (exitCode !== 0) { + await writeStopped('claude_nonzero') + throw new Error('Claude capture exited non-zero') + } + const grade = await dependencies.grade(runRoot, condition) + if (grade.condition !== condition) { + await writeStopped('not_gradable') + throw new Error('grade condition mismatch') + } + if (grade.evidenceStatus !== 'gradable') { + await writeStopped(grade.evidenceStatus) + throw new Error(`matrix stopped: ${grade.evidenceStatus}`) + } + summary.runs.push(publicSummary(runNumber, grade)) + await writeJsonAtomic(join(matrixRoot, 'summary.json'), summary) + await writeJsonAtomic(statusPath, { + schemaVersion: 1, + state: 'running', + completedRuns: summary.runs.length, + } satisfies MatrixStatus) + } + await writeJsonAtomic(statusPath, { + schemaVersion: 1, + state: 'complete', + completedRuns: summary.runs.length, + } satisfies MatrixStatus) + } catch (error) { + if (!stopped) await writeStopped('infrastructure_error') + throw error + } + return summary +} + +if (import.meta.main) { + const [matrixRoot, ...flags] = process.argv.slice(2) + if (!matrixRoot) throw new Error('matrix-root is required') + const confirmation = flags + .find((flag) => flag.startsWith('--confirm-paid-runs=')) + ?.slice('--confirm-paid-runs='.length) + const summary = await runMatrix({ + matrixRoot, + model: process.env.AX_BENCH_MODEL, + confirmation, + }) + console.log( + JSON.stringify( + { + completedRuns: summary.runs.length, + matrixRoot: resolve(matrixRoot), + summaryPath: resolve(matrixRoot, 'summary.json'), + }, + null, + 2 + ) + ) +} diff --git a/bench/continuation/prompt.ts b/bench/continuation/prompt.ts new file mode 100644 index 0000000..5421a17 --- /dev/null +++ b/bench/continuation/prompt.ts @@ -0,0 +1,35 @@ +export type SkillMode = 'none' | 'guided' | 'ablated' + +const CONTINUATION_GUIDANCE_BLOCK = [ + '- For automated continuation, use `--json-envelope`. Read `data`; when', + ' `meta.state` is `more`, rerun the same command with', + ' `--offset `. Continue only while it is `more`; stop on', + ' `complete` or `past_end`; do not restart from zero or increase the budget.', +].join('\n') + +function skillBody(markdown: string): string { + const closing = markdown.indexOf('\n---', 4) + if (!markdown.startsWith('---\n') || closing === -1) { + throw new Error('Skill must contain YAML frontmatter') + } + return markdown.slice(closing + 4).replace(/^\n+/, '') +} + +export function renderContinuationPrompt(template: string, budget: number): string { + const matches = template.match(/\{\{BUDGET\}\}/g) ?? [] + if (matches.length !== 1) throw new Error('prompt must contain exactly one {{BUDGET}}') + return template.replace('{{BUDGET}}', String(budget)).trim() +} + +export function ablateContinuationGuidance(markdown: string): string { + const target = `${CONTINUATION_GUIDANCE_BLOCK}\n` + const matches = markdown.split(target).length - 1 + if (matches !== 1) throw new Error('Skill must contain exactly one continuation bullet') + return markdown.replace(target, '') +} + +export function applySkill(prompt: string, skillMarkdown: string, mode: SkillMode): string { + if (mode === 'none') return prompt + const selected = mode === 'guided' ? skillMarkdown : ablateContinuationGuidance(skillMarkdown) + return `${skillBody(selected)}\n\n${prompt}` +} diff --git a/bench/continuation/report.test.ts b/bench/continuation/report.test.ts new file mode 100644 index 0000000..3ff57b3 --- /dev/null +++ b/bench/continuation/report.test.ts @@ -0,0 +1,136 @@ +import { expect, test } from 'bun:test' +import type { MatrixSummary, PublicRunSummary } from './matrix' +import { MATRIX_CONDITIONS } from './matrix' +import { aggregateMatrix, renderResultsSection } from './report' +import type { Condition } from './run' + +function publicRun(run: number, condition: Condition): PublicRunSummary { + const adoption = condition === 'conformance' ? null : run % 2 === 0 + return { + run, + condition, + publicModel: 'Sonnet 5', + answerExact: run !== 2 && run !== 3, + schemaValid: run !== 2, + answerFailureModes: { + schemaInvalid: run === 2, + missingRecords: run === 3, + unexpectedRecords: run === 3, + duplicateRecords: false, + orderingErrors: false, + fieldMismatches: false, + }, + protocolStatus: run === 3 ? 'fail' : 'pass', + pageCount: 5, + adoptedEnvelope: adoption, + validChain: adoption, + budgetCompliant: true, + alternativeStrategy: adoption === false ? 'plain-ax' : null, + turns: run, + durationMs: run * 100, + totalCostUsd: run / 100, + inputTokens: run * 10, + outputTokens: run * 5, + cacheCreationInputTokens: run * 3, + cacheReadInputTokens: run * 7, + } +} + +function matrix(): MatrixSummary { + return { + schemaVersion: 1, + publicModel: 'Sonnet 5', + runs: MATRIX_CONDITIONS.map((condition, index) => publicRun(index + 1, condition)), + } +} + +test('aggregates exactly three runs per condition with medians', () => { + expect(aggregateMatrix(matrix())).toEqual([ + { + condition: 'conformance', + runs: 3, + exactAnswers: 3, + protocolPasses: 3, + envelopeAdoptions: null, + medianTurns: 4, + medianDurationMs: 400, + medianCostUsd: 0.04, + medianInputTokens: 40, + medianOutputTokens: 20, + }, + { + condition: 'adoption-guided', + runs: 3, + exactAnswers: 2, + protocolPasses: 3, + envelopeAdoptions: 2, + medianTurns: 5, + medianDurationMs: 500, + medianCostUsd: 0.05, + medianInputTokens: 50, + medianOutputTokens: 25, + }, + { + condition: 'adoption-ablated', + runs: 3, + exactAnswers: 2, + protocolPasses: 2, + envelopeAdoptions: 1, + medianTurns: 6, + medianDurationMs: 600, + medianCostUsd: 0.06, + medianInputTokens: 60, + medianOutputTokens: 30, + }, + ]) +}) + +test.each([ + { + name: 'wrong model label', + summary: { ...matrix(), publicModel: 'Other' }, + error: 'matrix public model must be Sonnet 5', + }, + { + name: 'wrong run count', + summary: { ...matrix(), runs: matrix().runs.slice(0, 8) }, + error: 'matrix must contain exactly 9 runs', + }, +])('rejects $name', ({ summary, error }) => { + expect(() => aggregateMatrix(summary as MatrixSummary)).toThrow(error) +}) + +test('renders a self-contained public results section without private fields', () => { + const markdown = renderResultsSection(matrix()) + expect(markdown).toContain('## JSON-envelope continuation (Sonnet 5, n=3 per condition)') + expect(markdown).toContain('### Method') + expect(markdown).toContain('### Preliminary stopped attempts') + expect(markdown).toContain( + 'Three earlier formal matrix invocations were stopped by gradability gates and excluded from the reported sample.' + ) + expect(markdown).toContain('These are adaptive, exploratory results rather') + expect(markdown).toContain('than a preregistered confirmatory evaluation.') + expect(markdown).toContain('### Per-run results') + expect(markdown).toContain('### Aggregate') + expect(markdown).toContain('### Observed failure modes') + expect(markdown).toContain( + 'Across 3 Adoption runs without envelope adoption, 3 used `plain-ax`; 1 failed protocol grading.' + ) + expect(markdown).toContain( + 'Among 2 non-exact answers, 1 was schema-invalid. Among the 1 schema-valid non-exact answer, 1 had missing records, 1 had unexpected records, 0 had duplicate records, 0 had ordering errors, and 0 had field mismatches. Categories can overlap.' + ) + expect(markdown).toContain('### Interpretation and limitations') + expect(markdown).toContain('| 9 | Adoption ablated |') + for (const forbidden of [ + 'session_id', + 'trace', + 'cwd', + 'stderr', + 'rawTrace', + 'telemetryPath', + 'private-model-route', + '/tmp/', + ]) { + expect(markdown).not.toContain(forbidden) + } +}) diff --git a/bench/continuation/report.ts b/bench/continuation/report.ts new file mode 100644 index 0000000..0e6127f --- /dev/null +++ b/bench/continuation/report.ts @@ -0,0 +1,201 @@ +import type { MatrixSummary, PublicRunSummary } from './matrix' +import type { Condition } from './run' + +export type ConditionAggregate = { + condition: Condition + runs: 3 + exactAnswers: number + protocolPasses: number + envelopeAdoptions: number | null + medianTurns: number + medianDurationMs: number + medianCostUsd: number + medianInputTokens: number + medianOutputTokens: number +} + +const CONDITION_ORDER: readonly Condition[] = ['conformance', 'adoption-guided', 'adoption-ablated'] + +const CONDITION_LABELS: Record = { + conformance: 'Conformance', + 'adoption-guided': 'Adoption guided', + 'adoption-ablated': 'Adoption ablated', +} + +function median(values: number[]): number { + return [...values].sort((left, right) => left - right)[1]! +} + +function validateMatrix(summary: MatrixSummary): void { + if (summary.publicModel !== 'Sonnet 5') { + throw new Error('matrix public model must be Sonnet 5') + } + if (summary.runs.length !== 9) throw new Error('matrix must contain exactly 9 runs') + const runNumbers = new Set(summary.runs.map((run) => run.run)) + if (runNumbers.size !== 9) throw new Error('matrix run numbers must be unique') + for (const condition of CONDITION_ORDER) { + if (summary.runs.filter((run) => run.condition === condition).length !== 3) { + throw new Error(`matrix must contain exactly 3 ${condition} runs`) + } + } +} + +export function aggregateMatrix(summary: MatrixSummary): ConditionAggregate[] { + validateMatrix(summary) + return CONDITION_ORDER.map((condition) => { + const runs = summary.runs.filter((run) => run.condition === condition) + return { + condition, + runs: 3, + exactAnswers: runs.filter((run) => run.answerExact).length, + protocolPasses: runs.filter((run) => run.protocolStatus === 'pass').length, + envelopeAdoptions: + condition === 'conformance' + ? null + : runs.filter((run) => run.adoptedEnvelope === true).length, + medianTurns: median(runs.map((run) => run.turns)), + medianDurationMs: median(runs.map((run) => run.durationMs)), + medianCostUsd: median(runs.map((run) => run.totalCostUsd)), + medianInputTokens: median(runs.map((run) => run.inputTokens)), + medianOutputTokens: median(runs.map((run) => run.outputTokens)), + } + }) +} + +function yesNo(value: boolean | null): string { + return value === null ? '—' : value ? 'yes' : 'no' +} + +function perRunRow(run: PublicRunSummary): string { + return [ + run.run, + CONDITION_LABELS[run.condition], + yesNo(run.answerExact), + run.protocolStatus, + yesNo(run.adoptedEnvelope), + run.pageCount, + run.turns, + (run.durationMs / 1000).toFixed(1), + run.inputTokens, + run.outputTokens, + run.cacheCreationInputTokens, + run.cacheReadInputTokens, + `$${run.totalCostUsd.toFixed(3)}`, + ].join(' | ') +} + +function observedFailureMode(summary: MatrixSummary): string { + const nonAdopters = summary.runs.filter( + (run) => run.condition !== 'conformance' && run.adoptedEnvelope === false + ) + if (nonAdopters.length === 0) return 'All Adoption runs adopted envelopes.' + + const strategyCounts = new Map() + for (const run of nonAdopters) { + const strategy = run.alternativeStrategy ?? 'no-recorded-alternative' + strategyCounts.set(strategy, (strategyCounts.get(strategy) ?? 0) + 1) + } + const strategies = [...strategyCounts.entries()] + .sort(([left], [right]) => left.localeCompare(right)) + .map(([strategy, count]) => `${count} used \`${strategy}\``) + .join(', ') + const protocolFailures = nonAdopters.filter((run) => run.protocolStatus === 'fail').length + + return ( + `Across ${nonAdopters.length} Adoption runs without envelope adoption, ${strategies}; ` + + `${protocolFailures} failed protocol grading.` + ) +} + +function observedAnswerFailureModes(summary: MatrixSummary): string { + const nonExact = summary.runs.filter((run) => !run.answerExact) + const schemaValidNonExact = nonExact.filter((run) => run.schemaValid) + const count = (mode: keyof PublicRunSummary['answerFailureModes']) => + schemaValidNonExact.filter((run) => run.answerFailureModes[mode]).length + const schemaInvalid = nonExact.filter((run) => run.answerFailureModes.schemaInvalid).length + const comparableAnswers = schemaValidNonExact.length + + return ( + `Among ${nonExact.length} non-exact answers, ${schemaInvalid} ` + + `${schemaInvalid === 1 ? 'was' : 'were'} schema-invalid. ` + + `Among the ${comparableAnswers} schema-valid non-exact ` + + `${comparableAnswers === 1 ? 'answer' : 'answers'}, ` + + `${count('missingRecords')} had missing records, ` + + `${count('unexpectedRecords')} had unexpected records, ` + + `${count('duplicateRecords')} had duplicate records, ` + + `${count('orderingErrors')} had ordering errors, and ` + + `${count('fieldMismatches')} had field mismatches. Categories can overlap.` + ) +} + +export function renderResultsSection(summary: MatrixSummary): string { + const aggregates = aggregateMatrix(summary) + const perRunRows = [...summary.runs] + .sort((left, right) => left.run - right.run) + .map((run) => `| ${perRunRow(run)} |`) + const aggregateRows = aggregates.map( + (aggregate) => + `| ${CONDITION_LABELS[aggregate.condition]} | ${aggregate.exactAnswers}/3 | ` + + `${aggregate.protocolPasses}/3 | ` + + `${aggregate.envelopeAdoptions === null ? '—' : `${aggregate.envelopeAdoptions}/3`} | ` + + `${aggregate.medianTurns} | ${(aggregate.medianDurationMs / 1000).toFixed(1)} | ` + + `${aggregate.medianInputTokens} | ${aggregate.medianOutputTokens} | ` + + `$${aggregate.medianCostUsd.toFixed(3)} |` + ) + + return [ + '## JSON-envelope continuation (Sonnet 5, n=3 per condition)', + '', + '### Method', + '', + 'A deterministic 120-row HTML fixture was generated from an independent fixed seed.', + 'Each run used budget 600, while ground truth remained in the parent process.', + 'Conformance explicitly taught continuation; Adoption compared the same natural task', + 'with the full Skill versus a runtime-only continuation-guidance ablation.', + 'Raw session records stayed outside the repository.', + '', + '### Preliminary stopped attempts', + '', + 'Three earlier formal matrix invocations were stopped by gradability gates and excluded from the reported sample.', + 'They exposed fixture-metadata preflights, compound or interpreter-wrapped ax commands,', + 'and terminal responses containing text outside the required JSON object.', + 'The methodology was then changed to require standalone ax calls, prohibit all non-ax', + 'fixture access, enforce a JSON-only response, and tighten access auditing before the', + 'final matrix was collected. A post-run review further replaced permissive non-ax', + 'handling with a conservative allowlist; regrading the retained runs did not change', + 'their gradability or published scores. These are adaptive, exploratory results rather', + 'than a preregistered confirmatory evaluation.', + '', + '### Per-run results', + '', + '| Run | Condition | Exact | Protocol | Envelope | Pages | Turns | Duration (s) | Input | Output | Cache create | Cache read | Cost |', + '| ---: | --- | :---: | --- | :---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |', + ...perRunRows, + '', + '### Aggregate', + '', + '| Condition | Exact | Protocol pass | Envelope adoption | Median turns | Median duration (s) | Median input | Median output | Median cost |', + '| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |', + ...aggregateRows, + '', + '### Observed failure modes', + '', + observedFailureMode(summary), + observedAnswerFailureModes(summary), + '', + '### Interpretation and limitations', + '', + 'Answer correctness, continuation-protocol correctness, and active envelope adoption', + 'are reported separately. Each condition has only three runs on one deterministic', + 'fixture and one model, so differences are descriptive rather than general claims.', + 'Claude Code event structure may also change across CLI versions.', + '', + ].join('\n') +} + +if (import.meta.main) { + const summaryPath = process.argv[2] + if (!summaryPath) throw new Error('matrix summary path is required') + const summary = (await Bun.file(summaryPath).json()) as MatrixSummary + process.stdout.write(renderResultsSection(summary)) +} diff --git a/bench/continuation/run.ts b/bench/continuation/run.ts new file mode 100644 index 0000000..dff7625 --- /dev/null +++ b/bench/continuation/run.ts @@ -0,0 +1,225 @@ +import { chmod, mkdir, realpath } from 'node:fs/promises' +import { basename, dirname, isAbsolute, join, relative, resolve, sep } from 'node:path' +import { fileURLToPath } from 'node:url' +import { + CONTINUATION_BUDGET, + CONTINUATION_SEED, + type IncidentRecord, + writeAgentFixture, +} from './fixture' +import { applySkill, renderContinuationPrompt, type SkillMode } from './prompt' + +export type Condition = 'conformance' | 'adoption-guided' | 'adoption-ablated' + +export type SafeRunManifest = { + schemaVersion: 1 + condition: Condition + publicModel: 'Sonnet 5' + seed: number + budget: number +} + +export type ProvisionedAx = { + axPath: string + realPath: string + telemetryPath: string + childEnv: Record +} + +export type PreparedRun = { + runRoot: string + agentDir: string + prompt: string + truth: IncidentRecord[] + provisioned: ProvisionedAx + rawTracePath: string + stderrPath: string +} + +export async function provisionAx(repoRoot: string, runRoot: string): Promise { + const binDir = join(runRoot, 'bin') + const telemetryDir = join(runRoot, 'telemetry') + await mkdir(binDir) + await mkdir(telemetryDir) + const realPath = join(binDir, 'ax-real') + const axPath = join(binDir, 'ax') + const telemetryPath = join(telemetryDir, 'ax.jsonl') + const build = Bun.spawnSync( + ['bun', 'build', join(repoRoot, 'src/index.ts'), '--compile', '--outfile', realPath], + { cwd: repoRoot } + ) + if (build.exitCode !== 0) throw new Error(build.stderr.toString()) + const shimSource = fileURLToPath(new URL('./ax-shim.ts', import.meta.url)) + await Bun.write(axPath, `#!/bin/sh\nexec bun "${shimSource}" "$@"\n`) + await chmod(axPath, 0o755) + const childEnv = { + ...process.env, + PATH: `${binDir}:${process.env.PATH ?? ''}`, + AX_BENCH_REAL: realPath, + AX_BENCH_TELEMETRY: telemetryPath, + } as Record + const resolved = Bun.spawnSync(['sh', '-c', 'command -v ax'], { env: childEnv }) + if (resolved.exitCode !== 0 || resolve(resolved.stdout.toString().trim()) !== resolve(axPath)) { + throw new Error('child PATH did not resolve the benchmark ax shim') + } + return { axPath, realPath, telemetryPath, childEnv } +} + +export async function preflightAx(provisioned: ProvisionedAx, fixturePath: string): Promise { + const base = [ + fixturePath, + '.incident', + '--row', + 'id=@data-id,owner=.owner,severity=.severity,status=.status', + '--json-envelope', + '--limit', + '1', + ] + const first = Bun.spawnSync(['ax', ...base], { env: provisioned.childEnv }) + if (first.exitCode !== 0) throw new Error(first.stderr.toString()) + const firstPage = JSON.parse(first.stdout.toString()) + if (firstPage.meta.returned !== 1 || firstPage.meta.next_offset !== 1) { + throw new Error('first envelope failed preflight') + } + const second = Bun.spawnSync(['ax', ...base, '--offset', '1'], { + env: provisioned.childEnv, + }) + if (second.exitCode !== 0) throw new Error(second.stderr.toString()) + const secondPage = JSON.parse(second.stdout.toString()) + if (secondPage.meta.offset !== 1 || secondPage.data[0]?.id === firstPage.data[0]?.id) { + throw new Error('second envelope overlapped preflight page') + } + await Bun.write(provisioned.telemetryPath, '') +} + +export async function assertExternalRunRoot( + repoRootInput: string, + runRootInput: string +): Promise { + const repoRoot = await realpath(repoRootInput) + const requested = resolve(runRootInput) + const parent = await realpath(dirname(requested)) + const runRoot = join(parent, basename(requested)) + const fromRepo = relative(repoRoot, runRoot) + const insideRepo = + fromRepo === '' || + (fromRepo !== '..' && !fromRepo.startsWith(`..${sep}`) && !isAbsolute(fromRepo)) + if (insideRepo) throw new Error('run-root must be outside the repository') + return runRoot +} + +function conditionFiles(condition: Condition): { + promptUrl: URL + skillMode: SkillMode +} { + if (condition === 'conformance') { + return { + promptUrl: new URL('../prompts/continuation-conformance.txt', import.meta.url), + skillMode: 'none', + } + } + return { + promptUrl: new URL('../prompts/continuation-adoption.txt', import.meta.url), + skillMode: condition === 'adoption-guided' ? 'guided' : 'ablated', + } +} + +export async function prepareRun(runRootInput: string, condition: Condition): Promise { + const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '../..') + const runRoot = await assertExternalRunRoot(repoRoot, runRootInput) + await mkdir(runRoot, { recursive: false }) + const manifest: SafeRunManifest = { + schemaVersion: 1, + condition, + publicModel: 'Sonnet 5', + seed: CONTINUATION_SEED, + budget: CONTINUATION_BUDGET, + } + await Bun.write(join(runRoot, 'manifest.json'), `${JSON.stringify(manifest, null, 2)}\n`) + const agentDir = join(runRoot, 'agent') + await mkdir(agentDir) + const truth = await writeAgentFixture(agentDir) + const { promptUrl, skillMode } = conditionFiles(condition) + const template = await Bun.file(promptUrl).text() + const skill = await Bun.file(new URL('../../skills/ax/SKILL.md', import.meta.url)).text() + const rendered = renderContinuationPrompt(template, CONTINUATION_BUDGET) + const prompt = applySkill(rendered, skill, skillMode) + const provisioned = await provisionAx(repoRoot, runRoot) + await preflightAx(provisioned, join(agentDir, 'continuation.html')) + return { + runRoot, + agentDir, + prompt, + truth, + provisioned, + rawTracePath: join(runRoot, 'stream.jsonl'), + stderrPath: join(runRoot, 'claude.stderr.log'), + } +} + +export function buildClaudeArgs(model: string, prompt: string): string[] { + return [ + 'claude', + '-p', + prompt, + '--model', + model, + '--output-format', + 'stream-json', + '--verbose', + '--allowedTools', + 'Bash,Read,Grep,Glob', + '--max-turns', + '40', + ] +} + +export async function capturePilot(prepared: PreparedRun, model: string): Promise { + const proc = Bun.spawn(buildClaudeArgs(model, prepared.prompt), { + cwd: prepared.agentDir, + env: prepared.provisioned.childEnv, + stdout: 'pipe', + stderr: 'pipe', + }) + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).arrayBuffer(), + new Response(proc.stderr).arrayBuffer(), + proc.exited, + ]) + await Promise.all([ + Bun.write(prepared.rawTracePath, new Uint8Array(stdout)), + Bun.write(prepared.stderrPath, new Uint8Array(stderr)), + ]) + return exitCode +} + +export function parseCondition(value: string | undefined): Condition { + if (value === 'conformance' || value === 'adoption-guided' || value === 'adoption-ablated') { + return value + } + throw new Error('condition must be conformance, adoption-guided, or adoption-ablated') +} + +if (import.meta.main) { + const [runRoot, conditionValue] = process.argv.slice(2) + const model = process.env.AX_BENCH_MODEL + if (!model) throw new Error('AX_BENCH_MODEL is required') + if (!runRoot) throw new Error('run-root is required') + const condition = parseCondition(conditionValue) + const prepared = await prepareRun(runRoot, condition) + const exitCode = await capturePilot(prepared, model) + console.log( + JSON.stringify( + { + condition, + exitCode, + runRoot: prepared.runRoot, + rawTracePath: prepared.rawTracePath, + telemetryPath: prepared.provisioned.telemetryPath, + }, + null, + 2 + ) + ) + process.exit(exitCode) +} diff --git a/bench/continuation/stream.test.ts b/bench/continuation/stream.test.ts new file mode 100644 index 0000000..fdd169f --- /dev/null +++ b/bench/continuation/stream.test.ts @@ -0,0 +1,186 @@ +import { expect, test } from 'bun:test' +import { normalizeClaudeStream } from './stream' + +function assistantTool(id: string, command: string): string { + return JSON.stringify({ + type: 'assistant', + message: { + content: [ + { + type: 'tool_use', + id, + name: 'Bash', + input: { command, description: 'synthetic command' }, + }, + ], + }, + }) +} + +function userResult(id: string, content: string, isError = false): string { + return JSON.stringify({ + type: 'user', + message: { + content: [{ type: 'tool_result', tool_use_id: id, content, is_error: isError }], + }, + tool_use_result: { + stdout: content, + stderr: '', + interrupted: false, + isImage: false, + noOutputExpected: false, + }, + }) +} + +function terminal(result: string): string { + return JSON.stringify({ + type: 'result', + subtype: 'success', + is_error: false, + result, + duration_ms: 1200, + num_turns: 3, + total_cost_usd: 0.125, + usage: { + input_tokens: 11, + output_tokens: 7, + cache_creation_input_tokens: 5, + cache_read_input_tokens: 13, + }, + }) +} + +test('normalizes observed assistant tool use, user result, and terminal metrics', () => { + const raw = [ + JSON.stringify({ type: 'system', subtype: 'init' }), + JSON.stringify({ type: 'system', subtype: 'thinking_tokens', estimated_tokens: 10 }), + assistantTool('tool-1', '/external/bin/ax continuation.html --json-envelope'), + userResult('tool-1', '{"data":[],"meta":{"state":"complete"}}'), + terminal('{"records":[]}'), + ].join('\n') + + expect(normalizeClaudeStream(raw)).toEqual({ + status: 'complete', + issues: [], + finalText: '{"records":[]}', + toolCalls: [ + { + ordinal: 1, + name: 'Bash', + input: { + command: '/external/bin/ax continuation.html --json-envelope', + description: 'synthetic command', + }, + resultText: '{"data":[],"meta":{"state":"complete"}}', + isError: false, + }, + ], + metrics: { + durationMs: 1200, + turns: 3, + totalCostUsd: 0.125, + inputTokens: 11, + outputTokens: 7, + cacheCreationInputTokens: 5, + cacheReadInputTokens: 13, + }, + }) +}) + +function issues(raw: string): string[] { + const normalized = normalizeClaudeStream(raw) + expect(normalized.status).toBe('not_gradable') + return normalized.issues +} + +test.each([ + ['invalid JSON', '{', 'invalid stream JSON'], + ['missing terminal', assistantTool('tool-1', 'ax --help'), 'missing terminal result'], + ['multiple terminals', [terminal('{}'), terminal('{}')].join('\n'), 'multiple terminal results'], + [ + 'failed terminal', + JSON.stringify({ + ...JSON.parse(terminal('{}')), + subtype: 'error', + is_error: true, + }), + 'terminal result was not successful', + ], + [ + 'duplicate tool id', + [ + assistantTool('tool-1', 'ax --help'), + assistantTool('tool-1', 'ax --version'), + userResult('tool-1', 'ok'), + terminal('{}'), + ].join('\n'), + 'duplicate tool use id', + ], + [ + 'unmatched tool result', + [userResult('unknown', 'no match'), terminal('{}')].join('\n'), + 'unmatched tool result', + ], + [ + 'missing tool result', + [assistantTool('tool-1', 'ax --help'), terminal('{}')].join('\n'), + 'missing tool result', + ], + [ + 'unsupported message shape', + [JSON.stringify({ type: 'assistant', message: { content: 'text' } }), terminal('{}')].join( + '\n' + ), + 'unsupported stream event shape', + ], + [ + 'missing result text', + JSON.stringify({ ...JSON.parse(terminal('{}')), result: null }), + 'terminal result text missing', + ], + [ + 'missing metrics', + JSON.stringify({ ...JSON.parse(terminal('{}')), duration_ms: '1200' }), + 'terminal metrics missing', + ], +] as const)('%s is not gradable', (_name, raw, issue) => { + expect(issues(raw)).toContain(issue) +}) + +test('reports a tool result that precedes its tool use', () => { + const raw = [ + userResult('tool-1', 'too early'), + assistantTool('tool-1', 'ax --help'), + terminal('{}'), + ].join('\n') + expect(issues(raw)).toContain('tool result preceded tool use') +}) + +test('rejects negative terminal metrics', () => { + const event = JSON.parse(terminal('{}')) + event.total_cost_usd = -1 + expect(issues(JSON.stringify(event))).toContain('terminal metrics missing') +}) + +test('ignores observed system, thinking, and text noise', () => { + const raw = [ + JSON.stringify({ type: 'system', subtype: 'thinking_tokens', estimated_tokens: 10 }), + JSON.stringify({ + type: 'assistant', + message: { + content: [ + { type: 'thinking', thinking: 'synthetic' }, + { type: 'text', text: 'synthetic' }, + ], + }, + }), + terminal('{}'), + ].join('\n') + + expect(normalizeClaudeStream(raw)).toMatchObject({ + status: 'complete', + finalText: '{}', + toolCalls: [], + }) +}) diff --git a/bench/continuation/stream.ts b/bench/continuation/stream.ts new file mode 100644 index 0000000..4b15b4b --- /dev/null +++ b/bench/continuation/stream.ts @@ -0,0 +1,200 @@ +export type NormalizedToolCall = { + ordinal: number + name: string + input: Record + resultText: string + isError: boolean +} + +export type RunMetrics = { + durationMs: number + turns: number + totalCostUsd: number + inputTokens: number + outputTokens: number + cacheCreationInputTokens: number + cacheReadInputTokens: number +} + +export type NormalizedRun = + | { + status: 'complete' + issues: [] + finalText: string + toolCalls: NormalizedToolCall[] + metrics: RunMetrics + } + | { + status: 'not_gradable' + issues: string[] + finalText: null + toolCalls: NormalizedToolCall[] + metrics: null + } + +type PendingToolCall = { + id: string + name: string + input: Record + resultText?: string + isError?: boolean +} + +function record(value: unknown): Record | null { + return value && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : null +} + +function notGradable(issues: string[], toolCalls: NormalizedToolCall[] = []): NormalizedRun { + return { + status: 'not_gradable', + issues: [...new Set(issues)], + finalText: null, + toolCalls, + metrics: null, + } +} + +function completedTools(pending: PendingToolCall[]): NormalizedToolCall[] { + return pending.flatMap((call, index) => + call.resultText === undefined || call.isError === undefined + ? [] + : [ + { + ordinal: index + 1, + name: call.name, + input: call.input, + resultText: call.resultText, + isError: call.isError, + }, + ] + ) +} + +export function normalizeClaudeStream(rawJsonl: string): NormalizedRun { + const events: Record[] = [] + for (const line of rawJsonl + .split('\n') + .map((item) => item.trim()) + .filter(Boolean)) { + try { + const event = record(JSON.parse(line)) + if (!event) return notGradable(['unsupported stream event shape']) + events.push(event) + } catch { + return notGradable(['invalid stream JSON']) + } + } + + const pending: PendingToolCall[] = [] + const byId = new Map() + const unmatchedResultIds: string[] = [] + const terminals: Record[] = [] + const issues: string[] = [] + + for (const event of events) { + if (event.type === 'system') continue + if (event.type === 'result') { + terminals.push(event) + continue + } + if (event.type !== 'assistant' && event.type !== 'user') { + issues.push('unsupported stream event shape') + continue + } + const message = record(event.message) + if (!message || !Array.isArray(message.content)) { + issues.push('unsupported stream event shape') + continue + } + for (const item of message.content) { + const block = record(item) + if (!block || (block.type !== 'tool_use' && block.type !== 'tool_result')) continue + if (block.type === 'tool_use') { + const input = record(block.input) + if (typeof block.id !== 'string' || typeof block.name !== 'string' || !input) { + issues.push('unsupported stream event shape') + continue + } + if (byId.has(block.id)) { + issues.push('duplicate tool use id') + continue + } + const call = { id: block.id, name: block.name, input } + pending.push(call) + byId.set(block.id, call) + continue + } + if ( + typeof block.tool_use_id !== 'string' || + typeof block.content !== 'string' || + typeof block.is_error !== 'boolean' + ) { + issues.push('unsupported stream event shape') + continue + } + const call = byId.get(block.tool_use_id) + if (!call) { + unmatchedResultIds.push(block.tool_use_id) + continue + } + if (call.resultText !== undefined) { + issues.push('unmatched tool result') + continue + } + call.resultText = block.content + call.isError = block.is_error + } + } + + const precededIds = new Set(unmatchedResultIds.filter((id) => byId.has(id))) + if (precededIds.size > 0) issues.push('tool result preceded tool use') + if (unmatchedResultIds.some((id) => !byId.has(id))) issues.push('unmatched tool result') + if (pending.some((call) => call.resultText === undefined && !precededIds.has(call.id))) { + issues.push('missing tool result') + } + const toolCalls = completedTools(pending) + if (terminals.length === 0) issues.push('missing terminal result') + if (terminals.length > 1) issues.push('multiple terminal results') + const terminal = terminals[0] + if (terminal && (terminal.subtype !== 'success' || terminal.is_error !== false)) { + issues.push('terminal result was not successful') + } + if (terminal && typeof terminal.result !== 'string') { + issues.push('terminal result text missing') + } + const usage = record(terminal?.usage) + const metricValues = [ + terminal?.duration_ms, + terminal?.num_turns, + terminal?.total_cost_usd, + usage?.input_tokens, + usage?.output_tokens, + usage?.cache_creation_input_tokens, + usage?.cache_read_input_tokens, + ] + if ( + terminal && + metricValues.some((value) => typeof value !== 'number' || !Number.isFinite(value) || value < 0) + ) { + issues.push('terminal metrics missing') + } + if (issues.length > 0 || !terminal || !usage) return notGradable(issues, toolCalls) + + return { + status: 'complete', + issues: [], + finalText: terminal.result as string, + toolCalls, + metrics: { + durationMs: terminal.duration_ms as number, + turns: terminal.num_turns as number, + totalCostUsd: terminal.total_cost_usd as number, + inputTokens: usage.input_tokens as number, + outputTokens: usage.output_tokens as number, + cacheCreationInputTokens: usage.cache_creation_input_tokens as number, + cacheReadInputTokens: usage.cache_read_input_tokens as number, + }, + } +} diff --git a/bench/continuation/telemetry.ts b/bench/continuation/telemetry.ts new file mode 100644 index 0000000..140d752 --- /dev/null +++ b/bench/continuation/telemetry.ts @@ -0,0 +1,42 @@ +export type AxInvocation = { + argv: string[] + cwd: string + stdout: string + stderr: string + exitCode: number + startedAtMs: number + endedAtMs: number +} + +export async function readAxTelemetry(path: string): Promise { + const file = Bun.file(path) + if (!(await file.exists())) return [] + const lines = (await file.text()).split('\n').filter(Boolean) + const parsed = lines.map((line, index) => { + let value: unknown + try { + value = JSON.parse(line) + } catch { + throw new Error(`invalid ax telemetry JSON on line ${index + 1}`) + } + const item = value as Partial + if ( + !Array.isArray(item.argv) || + item.argv.some((arg) => typeof arg !== 'string') || + typeof item.cwd !== 'string' || + typeof item.stdout !== 'string' || + typeof item.stderr !== 'string' || + typeof item.exitCode !== 'number' || + typeof item.startedAtMs !== 'number' || + typeof item.endedAtMs !== 'number' + ) { + throw new Error(`invalid ax telemetry shape on line ${index + 1}`) + } + return item as AxInvocation + }) + return parsed.sort((a, b) => a.startedAtMs - b.startedAtMs) +} + +export function hasOverlappingInvocations(items: AxInvocation[]): boolean { + return items.some((item, index) => index > 0 && item.startedAtMs < items[index - 1]!.endedAtMs) +} diff --git a/bench/prompts/continuation-adoption.txt b/bench/prompts/continuation-adoption.txt new file mode 100644 index 0000000..f242be8 --- /dev/null +++ b/bench/prompts/continuation-adoption.txt @@ -0,0 +1 @@ +continuation.html contains incident records. Use ax to extract every record as {id, owner, severity, status} in DOM order. Every extraction call must use --budget {{BUDGET}}; keep that value unchanged and never use --all or --limit. Run each ax invocation as one standalone Bash command. Do not combine it with another command or use pipes, redirections, multiline commands, heredocs, scripts, interpreters, shell functions, command substitutions, wrappers, or subprocesses. Do not use any other tool or command to access, inspect, locate, list, search, copy, transform, or read continuation.html. You may locate ax only with a command that does not mention continuation.html. Return only one JSON object {"records":[...]} containing every record in DOM order. Before responding, silently verify that the response starts with {, ends with }, and contains exactly one JSON object with no prefix, suffix, completion note, Markdown fence, or commentary. diff --git a/bench/prompts/continuation-conformance.txt b/bench/prompts/continuation-conformance.txt new file mode 100644 index 0000000..a4a8ba8 --- /dev/null +++ b/bench/prompts/continuation-conformance.txt @@ -0,0 +1 @@ +continuation.html contains incident records. Use ax with selector .incident and --row 'id=@data-id,owner=.owner,severity=.severity,status=.status'. Every extraction call must use --json-envelope and --budget {{BUDGET}}. Read only each envelope's data. While meta.state is more, repeat the same command with --offset exactly equal to meta.next_offset. Stop on complete or past_end. Never restart from zero, change the budget, use --all, or use --limit. Run each ax invocation as one standalone Bash command. Do not combine it with another command or use pipes, redirections, multiline commands, heredocs, scripts, interpreters, shell functions, command substitutions, wrappers, or subprocesses. Do not use any other tool or command to access, inspect, locate, list, search, copy, transform, or read continuation.html. You may locate ax only with a command that does not mention continuation.html. Return only one JSON object {"records":[...]} containing every record in DOM order. Before responding, silently verify that the response starts with {, ends with }, and contains exactly one JSON object with no prefix, suffix, completion note, Markdown fence, or commentary.