Skip to content

fix(agentic-server): forward tools/format/think/keep_alive/num_ctx on the ollama route; select providers by name; route gpt-oss to ollama - #1980

Merged
devin-ai-integration[bot] merged 3 commits into
mainfrom
fix/ollama-route-forwarding
Nov 9, 2026
Merged

devin-ai-integration[bot] merged 3 commits into
mainfrom
fix/ollama-route-forwarding

Conversation

@pyramation

@pyramation pyramation commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A tool-calling harness could not run on an ollama provider: transformChatRequest sent only model, messages and sampling options, and transformChatResponse never returned tool_calls. Three fixes, each with a test that fails on main:

Ollama route forwards the whole request. tools, think and keep_alive pass through; response_format becomes Ollama's format (json_object → 'json', json_schema → the schema; a native format wins); num_ctx moves under options beside num_predict/temperature, with a client options object merged under them. Replayed assistant tool_calls get their arguments string parsed into the object Ollama's struct takes. The reply carries tool_calls in the OpenAI shape (id, type: 'function', arguments as a JSON string), finish_reason tool_calls/length/stop from done_reason (tool_calls only for a completed turn — a call cut off by the token budget stays length), and reasoning_content for a thinking message.

Providers are selected by name. ProviderConfig.name? (defaults to type) is what X-LLM-Provider and a name/model prefix address, falling back to type for unnamed entries — so two providers of one dialect (an in-cluster Ollama and a GPU host) are both reachable. upstreamModel strips a name/ or type/ prefix; /v1/providers lists name. An X-LLM-Provider naming no registered provider is a 400 (unknown provider 'x'; registered: …) instead of silently drifting to model-family routing or the first entry.

providers: [
  { name: 'ollama',      type: 'ollama', baseUrl: 'http://ollama:11434' },
  { name: 'host-ollama', type: 'ollama', baseUrl: 'http://host.docker.internal:11434' }
]
// X-LLM-Provider: host-ollama   or   model: 'host-ollama/qwen3:8b'

Model-family routing. gpt-oss* no longer matches the gpt* → openai rule; gpt-oss, qwen and glm route to ollama (checked before the openai rule).

Request-shape errors the dialect raises (non-JSON tool-call arguments, an unsupported content part, json_schema without a schema) are BadRequestError → 400 before any upstream call, rather than 502 Failed to reach LLM provider metered as a provider failure.

providerName, upstreamModel, the chat/embed transforms and BadRequestError are exported so a streaming lane built on this package can share the same dialect.

Not changed, by design: the nomic-embed-text embed default and the single-provider localhost:11434 fallback.

Link to Devin session: https://app.devin.ai/sessions/3b757d1a25ea436d829539dfb94b206c
Open in Devin Desktop: https://app.devin.ai/desktop/session/3b757d1a25ea436d829539dfb94b206c?variant=devin
Requested by: @pyramation

…name, gpt-oss routing

Failing on main: the ollama route drops tools, response_format/format, think,
keep_alive and num_ctx and returns no tool_calls; a second provider of one
type cannot be selected; gpt-oss routes to openai.
…_ctx on the ollama route; select providers by name; route gpt-oss to ollama

The ollama route sent only model, messages and sampling options, so a
tool-calling harness got no tools, no structured output and no tool_calls
back. It now forwards tools, response_format (as Ollama's format), think,
keep_alive and num_ctx, parses replayed tool-call arguments into objects,
and returns tool_calls, finish_reason and reasoning_content in the OpenAI
shape.

Providers are addressed by registry name (header or name/model prefix),
falling back to type for unnamed entries, so two providers of one dialect
are both reachable. gpt-oss, qwen and glm route to ollama instead of
openai or the first provider.
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@luxor-code-review

luxor-code-review Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Review complete. 🟡 3 medium

💬 Inline comments (2)

  • 🟡 Unknown X-LLM-Provider silently falls through to other routing — providers.ts:34
  • 🟡 Truncated Ollama responses report tool_calls finish reason — transforms.ts:217
📍 Findings outside the diff (1) — 🟡 1 medium — defects on lines GitHub can't attach comments to

🟡 Medium — Malformed client input surfaces as 502, not 4xx · router.ts:260–262 · unchanged line

// agentic/agentic-server/src/router.ts
260	      res.status(502).json({
261	        error: { message: 'Failed to reach LLM provider', details: err.message }
262	      });

Request-shape errors thrown by transformChatRequest — non-JSON tool-call arguments (transforms.ts:97-102), an image_url part with no url or a non-data-url (transforms.ts:63, transforms.ts:67), an unsupported content-part type (transforms.ts:72), and response_format: json_schema without a schema (transforms.ts:119-121) — are caught by the router's catch-all and returned as 502 Failed to reach LLM provider (router.ts:260-262). The upstream is never contacted, so these are client errors, but the harness sees a gateway failure attributed to the provider; metering also records them under a 502 with errorType: err.message, misattributing client mistakes to provider outages. A tool-call replay with malformed arguments therefore reads as an outage rather than a fixable request bug, defeating the loud-failure intent of the new guards.

🧹 Nitpicks (1) — 🟢 1 low
  • 🟢 Embed transforms missing from index.ts exports (index.ts:29) — The new export line in index.ts (line 29) exports providerName, transformChatRequest, transformChatResponse, and upstreamModel but omits transformEmbedRequest and transformEmbedResponse, which live in the same transforms module and are imported by router.ts (lines 22-23).

This PR extends the agentic-server gateway to route chat requests across multiple LLM providers — resolving a target provider from model prefixes, registry entries, or an X-LLM-Provider header — and translates between OpenAI and Ollama request/response shapes so Ollama-hosted models can serve OpenAI-dialect clients. Transforms cover message content flattening, tool-call round-trips, streaming chunk translation, and finish_reason mapping.

Files Change
agentic/agentic-server/src/providers.ts, src/types.ts Introduces the provider registry, model-prefix routing, and new provider/types declarations.
agentic/agentic-server/src/transforms.ts Adds OpenAI↔Ollama request/response/stream transforms, including tool-call and finish_reason mapping.
agentic/agentic-server/src/router.ts Wires provider resolution into request routing and exposes the /v1/providers listing endpoint.
agentic/agentic-server/src/index.ts Re-exports the new public API surface.
__tests__/*.test.ts, README.md Adds gateway, provider, and transform tests plus documentation of the routing model.

Reviewed commit: a753a87

@luxor-code-review luxor-code-review Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds multi-provider LLM routing (OpenAI, Ollama, Anthropic-style prefixes) to the agentic server with OpenAI↔Ollama dialect transforms, a provider-selection header, and provider registry APIs.

Key findings

  • 🟡 Malformed client input surfaces as 502, not 4xx — router.ts:260
  • 🟡 Unknown X-LLM-Provider silently falls through to other routing — providers.ts:34
  • 🟡 Truncated Ollama responses report tool_calls finish reason — transforms.ts:217

Comment on lines 34 to 37
if (requestProvider) {
const match = providers.find((p) => p.type === requestProvider);
const match = findNamed(providers, requestProvider);
if (match) return match;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 bug · medium

Unknown X-LLM-Provider silently falls through to other routing

An explicit X-LLM-Provider naming a provider not in the registry is silently ignored: findNamed misses and execution continues to model-prefix routing and eventually providers[0] (providers.ts:34-37). A client that pins a request to a specific provider — often for data-residency or cost reasons — gets its traffic routed to a different vendor (or whichever entry happens to be first) with no error, so a typo in the header becomes an undetectable misroute. An explicit routing directive that cannot be honored should fail loudly rather than degrade to a heuristic.

📋 Prompt for AI Agents

In agentic/agentic-server/src/providers.ts lines 33-37, when requestProvider is provided but findNamed(providers, requestProvider) returns undefined, make resolveProvider signal an error (e.g. throw a typed error or return a sentinel that router.ts turns into a 400 with a message like unknown provider '<name>') instead of continuing to model-prefix and family routing. Update both call sites in agentic/agentic-server/src/router.ts (lines ~104 and ~278) to catch/inspect the signal and reject the request before any upstream fetch.

message: msg || { role: 'assistant', content: '' },
finish_reason: 'stop',
message,
finish_reason: toolCalls ? 'tool_calls' : finishReason(data.done_reason),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 bug · medium

Truncated Ollama responses report tool_calls finish reason

In transformChatResponse, finish_reason: toolCalls ? 'tool_calls' : finishReason(data.done_reason) overrides the upstream done_reason whenever tool calls are present (transforms.ts:217). If Ollama ends the generation with done_reason: 'length' (context or num_predict budget exhausted) mid-tool-call, the emitted call may carry truncated arguments, yet the response asserts a complete tool-call turn. Harnesses that trust finish_reason will execute malformed or partial tool arguments instead of retrying.

📋 Prompt for AI Agents

In agentic/agentic-server/src/transforms.ts line 217, change finish_reason: toolCalls ? 'tool_calls' : finishReason(data.done_reason) to compute const reason = finishReason(data.done_reason) and emit 'tool_calls' only when tool calls are present AND reason === 'stop'; otherwise emit the truncation reason. Optionally, in finishReason (lines 275-278), log unrecognized done_reason values instead of silently coercing them to 'stop'.

…equest bodies with 400; report a truncated tool call as length

An explicit X-LLM-Provider that names no registered provider used to fall
through to model-family routing and the first entry, so a pinned request
could drift to another vendor. It is now a 400 naming the registry, in
both the registry and the single-provider mode.

Request-shape errors the ollama dialect raises (non-JSON tool-call
arguments, an unsupported content part, a json_schema format without a
schema) are BadRequestError and answered 400 before any upstream call,
instead of 502 'Failed to reach LLM provider' metered as a provider
failure.

An ollama reply with tool calls but done_reason length keeps
finish_reason length, so a harness does not execute half an argument
object as a completed tool turn.
@devin-ai-integration
devin-ai-integration Bot merged commit 9fdc538 into main Nov 9, 2026
19 checks passed
@devin-ai-integration
devin-ai-integration Bot deleted the fix/ollama-route-forwarding branch November 9, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant