Skip to content

feat: add explicit auto reasoning effort for openrouter/jev - #103

Merged
ayush-or merged 3 commits into
mainfrom
devin/1790003520-jev-adaptive-effort-default
Sep 21, 2026
Merged

ayush-or merged 3 commits into
mainfrom
devin/1790003520-jev-adaptive-effort-default

Conversation

@ayush-or

@ayush-or ayush-or commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Adds an explicit auto reasoning effort that is the default for openrouter/jev and maps to no reasoning object on the wire, so the router's adaptive effort selection applies while every benchmark config still carries an explicit reasoningEffort. Supersedes #102.

What changed?

  • REASONING_EFFORTS gains auto; PINNED_REASONING_EFFORTS is the concrete subset. reasoningEffort stays required in GenerateConfig, InferenceOverride, the search solver and JudgeConfig.
  • defaultReasoningEffortFor(model) returns auto for openrouter/jev / openrouter/jev:<variant> and high otherwise. The CLI uses it when --reasoning-effort is absent and rejects --reasoning-effort auto for any other model.
  • reasoningRequestFor(effort) is the single wire mapping used by the Responses model, the search request builder and the judge:
    auto  -> (no `reasoning` property)
    other -> { effort }
  • Ori agent lanes have no adaptive mode: toOriReasoningEffort(auto) is undefined, the CLI does not derive agentReasoningEffort from auto (the lane's own required field must be set), and the swe-atlas / deep-swe fallbacks fail with a SolverError instead of forwarding auto.

Why?

Jev's decisions-router pins effort when reasoning.effort is present and picks model + effort together when it is absent. Until now the harness always sent reasoning.effort, so Jev runs could never exercise adaptive effort, and a config without reasoningEffort is intentionally rejected. auto keeps the field explicit and self-describing while producing the omission the router needs.

How to test

bun test src/harness/constants.test.ts src/cli/index.test.ts src/providers/responses-model.test.ts src/benchmarks/search/core/request.test.ts src/benchmarks/benchmark-config.test.ts

Expected: parseArgs(["--model","openrouter/jev"]).reasoningEffort === "auto", openai/gpt-5 still defaults to high, --reasoning-effort auto with a non-Jev model throws, and a captured Responses request built with reasoningEffort: "auto" has no reasoning key.

Benchmark impact

Runs on openrouter/jev that do not pass --reasoning-effort now let Jev choose effort instead of pinning high. Explicit values behave exactly as before. Other models are unaffected.

Reviewer focus

  • Whether auto should be limited to ADAPTIVE_REASONING_EFFORT_MODELS at the CLI only (current) or also enforced in BenchmarkRunConfigSchema.
  • The Temporal BenchmarkConfig schema in openrouter-web still uses .default("high"); it needs auto in its enum before Temporal-launched Jev runs pick this up (separate PR).

Checklist

  • Tests cover changed behavior
  • Public API or configuration changes are backward compatible, or the break is documented
  • Benchmark changes document dataset provenance and licensing (n/a)
  • No credentials, private results, or restricted dataset contents are included
  • Documentation is updated where needed (n/a)

Link to Devin session: https://openrouter.devinenterprise.com/sessions/8caab1dda9764cc38dbbba486bc04687
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/8caab1dda9764cc38dbbba486bc04687?variant=devin
Requested by: @ayush-or

@ayush-or
ayush-or requested a review from a team as a code owner September 21, 2026 15:13
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from Ayush

SYSTEM:
<latest_message>
Ayush Patel (U0B8L6RNMA9) [ts=1789780083.530159]: @Devin!router_benchmark, 2 epochs of gpqa diamond, openrouter/jev , no spend/acu limit
</latest_message>

=== BEGIN THREAD HISTORY (in #agents) ===
Ayush Patel (U0B8L6RNMA9) [ts=1789780083.530159]: @Devin!router_benchmark, 2 epochs of gpqa diamond, openrouter/jev , no spend/acu limit
=== END THREAD HISTORY ===
Channel ID: C07UF9XLTFF
Thread URL: https://openrouter.slack.com/archives/C07UF9XLTFF/p1789780083530159?thread_ts=1789780083.530159&amp;cid=C07UF9XLTFF

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

@playbook:playbook-67bd60e6265449979d13f557104b2ef7

reasoningEffort stays required in every benchmark config. The new auto
value is the default for openrouter/jev (and its variants) and maps to
no reasoning object on the wire, so the router's adaptive effort
selection applies. Concrete efforts are still pinned and serialized.

auto is rejected by the CLI for models without adaptive effort and is
never forwarded as an ori agent reasoning effort.
@devin-ai-integration devin-ai-integration Bot changed the title feat: default to adaptive reasoning effort for openrouter/jev feat: add explicit auto reasoning effort for openrouter/jev Sep 21, 2026
@ayush-or
ayush-or merged commit 1760092 into main Sep 21, 2026
4 checks passed
@ayush-or
ayush-or deleted the devin/1790003520-jev-adaptive-effort-default branch September 21, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant