Repository navigation
fix(agentic-server): forward tools/format/think/keep_alive/num_ctx on the ollama route; select providers by name; route gpt-oss to ollama - #1980
Conversation
…name, gpt-oss routing Failing on main: the ollama route drops tools, response_format/format, think, keep_alive and num_ctx and returns no tool_calls; a second provider of one type cannot be selected; gpt-oss routes to openai.
…_ctx on the ollama route; select providers by name; route gpt-oss to ollama The ollama route sent only model, messages and sampling options, so a tool-calling harness got no tools, no structured output and no tool_calls back. It now forwards tools, response_format (as Ollama's format), think, keep_alive and num_ctx, parses replayed tool-call arguments into objects, and returns tool_calls, finish_reason and reasoning_content in the OpenAI shape. Providers are addressed by registry name (header or name/model prefix), falling back to type for unnamed entries, so two providers of one dialect are both reachable. gpt-oss, qwen and glm route to ollama instead of openai or the first provider.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
Review complete. 🟡 3 medium 💬 Inline comments (2)
📍 Findings outside the diff (1) — 🟡 1 medium — defects on lines GitHub can't attach comments to🟡 Medium — Malformed client input surfaces as 502, not 4xx · router.ts:260–262 · // agentic/agentic-server/src/router.ts
260 res.status(502).json({
261 error: { message: 'Failed to reach LLM provider', details: err.message }
262 });Request-shape errors thrown by 🧹 Nitpicks (1) — 🟢 1 low
This PR extends the agentic-server gateway to route chat requests across multiple LLM providers — resolving a target provider from model prefixes, registry entries, or an
Reviewed commit: a753a87 |
There was a problem hiding this comment.
Adds multi-provider LLM routing (OpenAI, Ollama, Anthropic-style prefixes) to the agentic server with OpenAI↔Ollama dialect transforms, a provider-selection header, and provider registry APIs.
Key findings
- 🟡 Malformed client input surfaces as 502, not 4xx — router.ts:260
- 🟡 Unknown
X-LLM-Providersilently falls through to other routing — providers.ts:34 - 🟡 Truncated Ollama responses report
tool_callsfinish reason — transforms.ts:217
| if (requestProvider) { | ||
| const match = providers.find((p) => p.type === requestProvider); | ||
| const match = findNamed(providers, requestProvider); | ||
| if (match) return match; | ||
| } |
There was a problem hiding this comment.
🟡 bug · medium
Unknown X-LLM-Provider silently falls through to other routing
An explicit X-LLM-Provider naming a provider not in the registry is silently ignored: findNamed misses and execution continues to model-prefix routing and eventually providers[0] (providers.ts:34-37). A client that pins a request to a specific provider — often for data-residency or cost reasons — gets its traffic routed to a different vendor (or whichever entry happens to be first) with no error, so a typo in the header becomes an undetectable misroute. An explicit routing directive that cannot be honored should fail loudly rather than degrade to a heuristic.
📋 Prompt for AI Agents
In agentic/agentic-server/src/providers.ts lines 33-37, when requestProvider is provided but findNamed(providers, requestProvider) returns undefined, make resolveProvider signal an error (e.g. throw a typed error or return a sentinel that router.ts turns into a 400 with a message like unknown provider '<name>') instead of continuing to model-prefix and family routing. Update both call sites in agentic/agentic-server/src/router.ts (lines ~104 and ~278) to catch/inspect the signal and reject the request before any upstream fetch.
| message: msg || { role: 'assistant', content: '' }, | ||
| finish_reason: 'stop', | ||
| message, | ||
| finish_reason: toolCalls ? 'tool_calls' : finishReason(data.done_reason), |
There was a problem hiding this comment.
🟡 bug · medium
Truncated Ollama responses report tool_calls finish reason
In transformChatResponse, finish_reason: toolCalls ? 'tool_calls' : finishReason(data.done_reason) overrides the upstream done_reason whenever tool calls are present (transforms.ts:217). If Ollama ends the generation with done_reason: 'length' (context or num_predict budget exhausted) mid-tool-call, the emitted call may carry truncated arguments, yet the response asserts a complete tool-call turn. Harnesses that trust finish_reason will execute malformed or partial tool arguments instead of retrying.
📋 Prompt for AI Agents
In agentic/agentic-server/src/transforms.ts line 217, change finish_reason: toolCalls ? 'tool_calls' : finishReason(data.done_reason) to compute const reason = finishReason(data.done_reason) and emit 'tool_calls' only when tool calls are present AND reason === 'stop'; otherwise emit the truncation reason. Optionally, in finishReason (lines 275-278), log unrecognized done_reason values instead of silently coercing them to 'stop'.
…equest bodies with 400; report a truncated tool call as length An explicit X-LLM-Provider that names no registered provider used to fall through to model-family routing and the first entry, so a pinned request could drift to another vendor. It is now a 400 naming the registry, in both the registry and the single-provider mode. Request-shape errors the ollama dialect raises (non-JSON tool-call arguments, an unsupported content part, a json_schema format without a schema) are BadRequestError and answered 400 before any upstream call, instead of 502 'Failed to reach LLM provider' metered as a provider failure. An ollama reply with tool calls but done_reason length keeps finish_reason length, so a harness does not execute half an argument object as a completed tool turn.
Summary
A tool-calling harness could not run on an
ollamaprovider:transformChatRequestsent onlymodel,messagesand sampling options, andtransformChatResponsenever returnedtool_calls. Three fixes, each with a test that fails onmain:Ollama route forwards the whole request.
tools,thinkandkeep_alivepass through;response_formatbecomes Ollama'sformat(json_object→'json',json_schema→ the schema; a nativeformatwins);num_ctxmoves underoptionsbesidenum_predict/temperature, with a clientoptionsobject merged under them. Replayed assistanttool_callsget theirargumentsstring parsed into the object Ollama's struct takes. The reply carriestool_callsin the OpenAI shape (id,type: 'function',argumentsas a JSON string),finish_reasontool_calls/length/stopfromdone_reason(tool_callsonly for a completed turn — a call cut off by the token budget stayslength), andreasoning_contentfor athinkingmessage.Providers are selected by name.
ProviderConfig.name?(defaults totype) is whatX-LLM-Providerand aname/modelprefix address, falling back totypefor unnamed entries — so two providers of one dialect (an in-cluster Ollama and a GPU host) are both reachable.upstreamModelstrips aname/ortype/prefix;/v1/providerslistsname. AnX-LLM-Providernaming no registered provider is a400(unknown provider 'x'; registered: …) instead of silently drifting to model-family routing or the first entry.Model-family routing.
gpt-oss*no longer matches thegpt*→ openai rule;gpt-oss,qwenandglmroute to ollama (checked before the openai rule).Request-shape errors the dialect raises (non-JSON tool-call
arguments, an unsupported content part,json_schemawithout a schema) areBadRequestError→400before any upstream call, rather than502 Failed to reach LLM providermetered as a provider failure.providerName,upstreamModel, the chat/embed transforms andBadRequestErrorare exported so a streaming lane built on this package can share the same dialect.Not changed, by design: the
nomic-embed-textembed default and the single-providerlocalhost:11434fallback.Link to Devin session: https://app.devin.ai/sessions/3b757d1a25ea436d829539dfb94b206c
Open in Devin Desktop: https://app.devin.ai/desktop/session/3b757d1a25ea436d829539dfb94b206c?variant=devin
Requested by: @pyramation