feat: add explicit auto reasoning effort for openrouter/jev - #103
Conversation
|
I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.
Original prompt from Ayush
|
reasoningEffort stays required in every benchmark config. The new auto value is the default for openrouter/jev (and its variants) and maps to no reasoning object on the wire, so the router's adaptive effort selection applies. Concrete efforts are still pinned and serialized. auto is rejected by the CLI for models without adaptive effort and is never forwarded as an ori agent reasoning effort.
TL;DR
Adds an explicit
autoreasoning effort that is the default foropenrouter/jevand maps to noreasoningobject on the wire, so the router's adaptive effort selection applies while every benchmark config still carries an explicitreasoningEffort. Supersedes #102.What changed?
REASONING_EFFORTSgainsauto;PINNED_REASONING_EFFORTSis the concrete subset.reasoningEffortstays required inGenerateConfig,InferenceOverride, the search solver andJudgeConfig.defaultReasoningEffortFor(model)returnsautoforopenrouter/jev/openrouter/jev:<variant>andhighotherwise. The CLI uses it when--reasoning-effortis absent and rejects--reasoning-effort autofor any other model.reasoningRequestFor(effort)is the single wire mapping used by the Responses model, the search request builder and the judge:toOriReasoningEffort(auto)isundefined, the CLI does not deriveagentReasoningEffortfromauto(the lane's own required field must be set), and the swe-atlas / deep-swe fallbacks fail with aSolverErrorinstead of forwardingauto.Why?
Jev's decisions-router pins effort when
reasoning.effortis present and picks model + effort together when it is absent. Until now the harness always sentreasoning.effort, so Jev runs could never exercise adaptive effort, and a config withoutreasoningEffortis intentionally rejected.autokeeps the field explicit and self-describing while producing the omission the router needs.How to test
bun test src/harness/constants.test.ts src/cli/index.test.ts src/providers/responses-model.test.ts src/benchmarks/search/core/request.test.ts src/benchmarks/benchmark-config.test.tsExpected:
parseArgs(["--model","openrouter/jev"]).reasoningEffort === "auto",openai/gpt-5still defaults tohigh,--reasoning-effort autowith a non-Jev model throws, and a captured Responses request built withreasoningEffort: "auto"has noreasoningkey.Benchmark impact
Runs on
openrouter/jevthat do not pass--reasoning-effortnow let Jev choose effort instead of pinninghigh. Explicit values behave exactly as before. Other models are unaffected.Reviewer focus
autoshould be limited toADAPTIVE_REASONING_EFFORT_MODELSat the CLI only (current) or also enforced inBenchmarkRunConfigSchema.BenchmarkConfigschema in openrouter-web still uses.default("high"); it needsautoin its enum before Temporal-launched Jev runs pick this up (separate PR).Checklist
Link to Devin session: https://openrouter.devinenterprise.com/sessions/8caab1dda9764cc38dbbba486bc04687
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/8caab1dda9764cc38dbbba486bc04687?variant=devin
Requested by: @ayush-or