Rebuild the eval engine on inspect_ai and retire the legacy runner - #19

Open

rominf wants to merge 68 commits into

rominf:inspect-engine

Commits on Sep 11, 2026

Add inspect_ai engine behind --engine flag
rominf
committed
d564a66
Record what a graded run spends, and compare engines
rominf
committed
3f76a2f
Add the Claude Code verification leg
rominf
committed
fd52c3e
Count legacy model calls the same way in both commands
rominf
committed
ef3365f
Document the engine choice
rominf
committed
a97d447
Make the sandbox provider selectable, and stop it eating the skill's config
rominf
committed
ff499f7
Say in the report whether the run was isolated
rominf
committed
7b54f68
Check the model answers before starting containers
rominf
committed
991ac6d
Stop a broken sandbox looking like an idle agent
rominf
committed
4df2504
Give a containerised case a working directory
rominf
committed
6a3719d
Let the judge read the artifacts, and hear what the agent said
rominf
committed
1400d90
Stop truncation hiding the action a check turns on
rominf
committed
27660ae
Use neutral values in the header-parsing fixture
rominf
committed
98ca77f
Exercise the inspect engine in CI, on both sandboxes
rominf
committed
de64e59
Fix what CI found: platform-blind tests and a misplaced fixture root
rominf
committed
f7d5124
Do not pin a mock model to opus under CI
rominf
committed
00d1110
Stop the wiring run paying for a mock that cannot finish
rominf
committed
86283df
Say what to install when a sandbox provider will not resolve
rominf
committed
b987db7
Grade the answer the agent submitted, not the sentence introducing it
rominf
committed
24f476d
Document what podman actually needs
rominf
committed
f983879

Commits on Sep 14, 2026

Resolve a declared compose file beside machine.yml
rominf
committed
bb3a32b
Contain a skill that cannot be run to that skill
rominf
committed
b6fc197
Show the judge everything the agent said, not just the last thing
rominf
committed
8938682
Run the real CLI inside inspect's framework
rominf
committed
facfb2e
Let the benchmark compare any two engines
rominf
committed
ecf0825
Run Claude Code where the scorers look
rominf
committed
dbe49a1

Commits on Sep 15, 2026

Name the engine that was actually asked for
rominf
committed
00e5438

Commits on Sep 16, 2026

Send claude-cli to the engine it asked for
rominf
committed
8da6ee6
Launch the CLI on Windows, where it is not an executable
rominf
committed
22b214f
Give the CLI the skill it is supposed to be testing
rominf
committed
a7ada23

Commits on Sep 17, 2026

Let the shell probe survive a guest with no bash
rominf
committed
928414c
Drop the live sample buffer on Windows
rominf
committed
9015640
Make a timeout say where it got to
rominf
committed
3c3bd5e

Commits on Sep 22, 2026

Bound the model reachability probe
rominf
committed
1b7756a
Stop routing reporting a sandbox it never started
rominf
committed
1792401
Show what a sandbox declaration actually looks like
rominf
committed
07c322c

Commits on Sep 23, 2026

Drop the harness-independent engine
rominf
committed
a26a02c
Rename claude-cli to claude-code-no-sandbox
rominf
committed
30282aa
Name the no-sandbox driver after the engine it implements
rominf
committed
75e5508
Close the gaps the removal left behind
rominf
committed
8feea09
Refuse claude-code on a sandbox that shares the host filesystem
rominf
committed
93743a8
Give routing a leg on each engine that drives the real CLI
rominf
committed
514861f
Stop a guard test depending on an optional extra
rominf
committed
115d498
Make the sandbox-guard tests test the sandbox guard
rominf
committed
b695b4f
Correct what now claims routing runs on one engine
rominf
committed
b30f420
Make --case-timeout bind on the new routing legs
rominf
committed
30e060f
Bring routing termination up to legacy, and past it where it can go
rominf
committed
d4937f2
Say what each routing leg can actually stop
rominf
committed
4f4fa79
Say when the provider failed, not the skill
rominf
committed
a6e74a0
Refuse to skip a skill's hooks rather than skipping them quietly
rominf
committed
3ff376a
Treat a quiet finish as a routing result, not a failed run
rominf
committed
8db6fb7
Read the transcript too, not just the adopted messages
rominf
committed
48b9dd0

Commits on Sep 24, 2026

Count the routing budget per case, and count it the way legacy does
rominf
committed
34ee475
A case that hit a limit ran; grade it
rominf
committed
1a5ca6f
Count each turn once, not once per record it appears in
rominf
committed
bc7daf2
Fix the test helpers the de-duplication change outgrew
rominf
committed
00d7e2c
Say how much of a run the budget decided
rominf
committed
b1e0f2e
Fit the routing budget to the engine it judges
rominf
committed
9eea3ce
Calibrate the routing budget on decisions, not on totals
rominf
committed
6d976f0
Stop the host routing leg at the decision, as legacy does
rominf
committed
f2c6450
Key the turn de-duplication on what survives the bridge
rominf
committed
9b3313f

Commits on Sep 25, 2026

Give the sandboxed legs the effort they were told to use
rominf
committed
60c9475
Require an inspect-swe that accepts the effort argument
rominf
committed
9c87d2f

Commits on Sep 28, 2026

Give the sandboxed legs a whole skill, not a third of one
rominf
committed
e7978cd
Retire the legacy engine
rominf
committed
e5f37ea

Commits on Sep 29, 2026

Require inspect_ai rather than offering it as an extra
rominf
committed
97a7129
Restore _isolation_note, and render the reports in tests
rominf
committed
8c066b3
Run evals/hooks.py setup and teardown again
rominf
committed
f585f2d