Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions agentic/agentic-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,26 @@ app.listen(3001, () => {

`POST /v1/chat/completions` with `stream: true` is relayed as a `text/event-stream`: the gateway forwards the provider's frames verbatim, so a client receives tokens as they are produced. It adds `stream_options: { include_usage: true }` unless the caller set `stream_options` itself, and meters the usage the final frame carries. Streaming is available for OpenAI-compatible providers; a provider whose stream would need translation (`ollama`, `anthropic`) is rejected `501` rather than answered with a non-streaming body.

## Providers and routing

`providers` is a registry; each entry has a `type` (the dialect: `openai`, `anthropic`, `ollama`) and an optional `name` it is addressed by, which defaults to the type. Two entries of one dialect — an in-cluster Ollama and one on a GPU host — are both reachable by name:

```typescript
createAgenticServer({
providers: [
{ name: 'ollama', type: 'ollama', baseUrl: 'http://ollama:11434' },
{ name: 'host-ollama', type: 'ollama', baseUrl: 'http://host.docker.internal:11434' },
{ type: 'openai', baseUrl: 'https://api.openai.com', apiKey: process.env.OPENAI_API_KEY }
]
});
```

A request picks its provider, in this order: the `X-LLM-Provider` header (a name, or a type for an entry with no name); a `name/model` prefix such as `host-ollama/qwen3:8b` (stripped before the call goes upstream); a known model family (`claude*` → anthropic; `gpt-oss*`, `llama*`, `mistral*`, `gemma*`, `nomic*`, `qwen*`, `glm*` → ollama; `gpt*`, `o1*`, `o3*` → openai); otherwise the first entry.

### The Ollama dialect

A request to an `ollama` provider is rewritten for `/api/chat`: content parts are flattened to the string Ollama takes; `max_tokens` becomes `options.num_predict` and `num_ctx` becomes `options.num_ctx` (a client `options` object is merged under them); `tools`, `think` and `keep_alive` are forwarded as-is; `response_format` becomes `format` (`json_object` → `'json'`, `json_schema` → the schema; a native `format` is passed through). The reply comes back in the OpenAI shape, including `tool_calls` (arguments as a JSON string, `finish_reason: 'tool_calls'`) and `reasoning_content` for a `thinking` message, so a tool-calling harness can run on an Ollama-served model.

## Identity & tenancy

Requests carry tenant identity via headers: `X-Database-Id` (**required** — requests without it are rejected `400`), `X-Entity-Id`, and `X-Actor-Id`. Routing to a specific provider can be forced with `X-LLM-Provider`.
Expand Down
90 changes: 90 additions & 0 deletions agentic/agentic-server/__tests__/gateway.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,20 @@ beforeAll(async () => {
});
return;
}
if (req.body?.model === 'tool-user') {
res.json({
message: {
role: 'assistant',
content: '',
tool_calls: [{ function: { name: 'get_weather', arguments: { city: 'Paris' } } }]
},
done: true,
done_reason: 'stop',
prompt_eval_count: 12,
eval_count: 8
});
return;
}
res.json({
message: { role: 'assistant', content: 'Mock Ollama response' },
prompt_eval_count: 12,
Expand Down Expand Up @@ -223,6 +237,81 @@ describe('POST /v1/chat/completions', () => {
]);
});

it('carries a tool-calling turn to ollama and back in the OpenAI shape', async () => {
const tool = {
type: 'function',
function: { name: 'get_weather', parameters: { type: 'object', properties: { city: { type: 'string' } } } }
};
const res = await fetch(`http://localhost:${agenticPort}/v1/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Database-Id': 'db-tools-1' },
body: JSON.stringify({
model: 'tool-user',
messages: [{ role: 'user', content: [{ type: 'text', text: 'weather in Paris?' }] }],
tools: [tool],
response_format: { type: 'json_object' },
think: false,
keep_alive: '10m',
num_ctx: 16384
})
});

expect(res.status).toBe(200);
expect(llmRequests[0].body).toMatchObject({
tools: [tool],
format: 'json',
think: false,
keep_alive: '10m',
options: { num_ctx: 16384 }
});
const data = (await res.json()) as any;
expect(data.choices[0].finish_reason).toBe('tool_calls');
expect(data.choices[0].message.tool_calls[0]).toMatchObject({
type: 'function',
function: { name: 'get_weather', arguments: '{"city":"Paris"}' }
});
});

it('rejects (400) an X-LLM-Provider that is not registered rather than routing elsewhere', async () => {
const res = await fetch(`http://localhost:${agenticPort}/v1/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Database-Id': 'db-pin-1', 'X-LLM-Provider': 'gpu-ollama' },
body: JSON.stringify({ model: 'llama3', messages: [{ role: 'user', content: 'hi' }] })
});

expect(res.status).toBe(400);
const data = (await res.json()) as any;
expect(data.error.message).toBe("unknown provider 'gpu-ollama'; registered: ollama");
await new Promise((r) => setTimeout(r, 50));
expect(llmRequests).toHaveLength(0);
expect(sinkEntries).toHaveLength(0);
});

it('rejects (400) a request the ollama dialect cannot express, without calling upstream', async () => {
const res = await fetch(`http://localhost:${agenticPort}/v1/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-Database-Id': 'db-bad-1' },
body: JSON.stringify({
model: 'llama3',
messages: [
{ role: 'user', content: 'weather?' },
{
role: 'assistant',
content: '',
tool_calls: [{ id: 'call_1', type: 'function', function: { name: 'get_weather', arguments: '{not json' } }]
}
]
})
});

expect(res.status).toBe(400);
const data = (await res.json()) as any;
expect(data.error.message).toMatch(/^ollama: tool call 'get_weather' carries arguments that are not JSON/);
await new Promise((r) => setTimeout(r, 50));
expect(llmRequests).toHaveLength(0);
expect(sinkEntries).toHaveLength(0);
});

it('names the upstream reason in the error message the harness reads', async () => {
const res = await fetch(`http://localhost:${agenticPort}/v1/chat/completions`, {
method: 'POST',
Expand Down Expand Up @@ -419,5 +508,6 @@ describe('GET /v1/providers', () => {
const data = (await res.json()) as any;
expect(data.providers).toHaveLength(1);
expect(data.providers[0].type).toBe('ollama');
expect(data.providers[0].name).toBe('ollama');
});
});
79 changes: 79 additions & 0 deletions agentic/agentic-server/__tests__/providers.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
/**
* Provider selection over a registry.
*
* A deployment may run several providers of one dialect (two Ollama hosts: one
* in-cluster, one on a GPU box), so an entry is addressed by its registry
* `name`, not by its `type`. Model-name routing must also not mistake an
* open-weights model for a vendor's hosted one.
*/

import { BadRequestError } from '../src/errors';
import { resolveProvider } from '../src/providers';
import type { AgenticServerOptions } from '../src/types';

const twoOllamas: AgenticServerOptions = {
providers: [
{ name: 'openai', type: 'openai', baseUrl: 'https://api.openai.test', apiKey: 'sk-oai' },
{ name: 'ollama', type: 'ollama', baseUrl: 'http://ollama.svc:11434' },
{ name: 'host-ollama', type: 'ollama', baseUrl: 'http://host.docker.internal:11434' }
]
};

describe('resolveProvider by registry name', () => {
it('selects the second provider of a type when X-LLM-Provider names it', () => {
const provider = resolveProvider(twoOllamas, 'qwen3:8b', 'host-ollama');
expect(provider.baseUrl).toBe('http://host.docker.internal:11434');
});

it('selects a provider by a `name/model` prefix and strips the prefix', () => {
const provider = resolveProvider(twoOllamas, 'host-ollama/qwen3:8b');
expect(provider.baseUrl).toBe('http://host.docker.internal:11434');
expect(provider.defaultModel).toBe('qwen3:8b');
});

it('still answers to the type when an entry carries no name', () => {
const options: AgenticServerOptions = {
providers: [
{ type: 'openai', baseUrl: 'https://api.openai.test', apiKey: 'sk-oai' },
{ type: 'ollama', baseUrl: 'http://ollama.svc:11434' }
]
};
expect(resolveProvider(options, 'anything', 'ollama').baseUrl).toBe('http://ollama.svc:11434');
expect(resolveProvider(options, 'ollama/qwen3').baseUrl).toBe('http://ollama.svc:11434');
});

it('rejects an X-LLM-Provider that names no registered provider instead of routing elsewhere', () => {
expect(() => resolveProvider(twoOllamas, 'qwen3:8b', 'host-olama')).toThrow(BadRequestError);
expect(() => resolveProvider(twoOllamas, 'qwen3:8b', 'host-olama')).toThrow(
"unknown provider 'host-olama'; registered: openai, ollama, host-ollama"
);
});
});

describe('resolveProvider by model name', () => {
const openaiAndOllama: AgenticServerOptions = {
providers: [
{ type: 'openai', baseUrl: 'https://api.openai.test', apiKey: 'sk-oai' },
{ type: 'ollama', baseUrl: 'http://ollama.svc:11434' }
]
};

it('routes gpt-oss to ollama, not to openai', () => {
expect(resolveProvider(openaiAndOllama, 'gpt-oss:20b').type).toBe('ollama');
});

it('still routes gpt-4o to openai', () => {
expect(resolveProvider(openaiAndOllama, 'gpt-4o').type).toBe('openai');
});

it('routes qwen and glm to ollama', () => {
const ollamaLast: AgenticServerOptions = {
providers: [
{ type: 'anthropic', baseUrl: 'https://api.anthropic.test', apiKey: 'sk-ant' },
{ type: 'ollama', baseUrl: 'http://ollama.svc:11434' }
]
};
expect(resolveProvider(ollamaLast, 'qwen3:8b').type).toBe('ollama');
expect(resolveProvider(ollamaLast, 'glm-4.6').type).toBe('ollama');
});
});
170 changes: 170 additions & 0 deletions agentic/agentic-server/__tests__/transforms.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
/**
* The Ollama dialect of a chat request and response.
*
* A tool-calling harness (pi) sends `tools`, a `response_format`, and reads
* `tool_calls` back in the OpenAI shape; a model served by Ollama needs `think`,
* `keep_alive` and `num_ctx` to be reachable at all. Each must cross the
* gateway, or `ollama/…` can serve chat but no agent.
*/

import { transformChatRequest, transformChatResponse } from '../src/transforms';
import type { ResolvedProvider } from '../src/types';

const ollama: ResolvedProvider = { type: 'ollama', baseUrl: 'http://ollama.test:11434' };

const weatherTool = {
type: 'function',
function: {
name: 'get_weather',
description: 'Weather for a city',
parameters: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'] }
}
};

describe('transformChatRequest for ollama', () => {
it('forwards tools and tool_choice', () => {
const body = transformChatRequest(ollama, {
model: 'qwen3:8b',
messages: [{ role: 'user', content: 'weather in Paris?' }],
tools: [weatherTool],
tool_choice: 'auto'
});
expect(body.tools).toEqual([weatherTool]);
});

it('turns response_format json_object into format json', () => {
const body = transformChatRequest(ollama, {
model: 'qwen3:8b',
messages: [],
response_format: { type: 'json_object' }
});
expect(body.format).toBe('json');
});

it('turns response_format json_schema into the schema as format', () => {
const schema = { type: 'object', properties: { intent: { type: 'string' } } };
const body = transformChatRequest(ollama, {
model: 'qwen3:8b',
messages: [],
response_format: { type: 'json_schema', json_schema: { name: 'route', schema } }
});
expect(body.format).toEqual(schema);
});

it('passes an ollama-native format through', () => {
const body = transformChatRequest(ollama, { model: 'qwen3:8b', messages: [], format: 'json' });
expect(body.format).toBe('json');
});

it('forwards think and keep_alive, and num_ctx under options', () => {
const body = transformChatRequest(ollama, {
model: 'qwen3:8b',
messages: [],
think: true,
keep_alive: '30m',
num_ctx: 32768,
temperature: 0.1,
max_tokens: 512
});
expect(body.think).toBe(true);
expect(body.keep_alive).toBe('30m');
expect(body.options).toEqual({ temperature: 0.1, num_predict: 512, num_ctx: 32768 });
});

it('omits what the caller did not send', () => {
const body = transformChatRequest(ollama, { model: 'qwen3:8b', messages: [] });
expect(body).toEqual({ model: 'qwen3:8b', messages: [], stream: false });
});

it('parses the arguments string of an earlier assistant tool call into the object ollama takes', () => {
const body = transformChatRequest(ollama, {
model: 'qwen3:8b',
messages: [
{ role: 'user', content: 'weather in Paris?' },
{
role: 'assistant',
content: '',
tool_calls: [
{ id: 'call_1', type: 'function', function: { name: 'get_weather', arguments: '{"city":"Paris"}' } }
]
},
{ role: 'tool', tool_call_id: 'call_1', content: '{"temp_c":18}' }
]
});
const messages = body.messages as Array<Record<string, any>>;
expect(messages[1].tool_calls[0].function.arguments).toEqual({ city: 'Paris' });
expect(messages[2]).toMatchObject({ role: 'tool', content: '{"temp_c":18}' });
});

it('strips a `name/` prefix that names the provider, not only a `type/` one', () => {
const named: ResolvedProvider = { name: 'host-ollama', type: 'ollama', baseUrl: 'http://host:11434' };
expect(transformChatRequest(named, { model: 'host-ollama/qwen3:8b', messages: [] }).model).toBe('qwen3:8b');
expect(transformChatRequest(named, { model: 'ollama/qwen3:8b', messages: [] }).model).toBe('qwen3:8b');
});
});

describe('transformChatResponse for ollama', () => {
it('returns tool calls in the OpenAI shape with finish_reason tool_calls', () => {
const { body } = transformChatResponse(
{
message: {
role: 'assistant',
content: '',
tool_calls: [{ function: { name: 'get_weather', arguments: { city: 'Paris' } } }]
},
done: true,
done_reason: 'stop',
prompt_eval_count: 20,
eval_count: 10
},
ollama
);
const choice = (body.choices as any[])[0];
expect(choice.finish_reason).toBe('tool_calls');
expect(choice.message.tool_calls).toHaveLength(1);
expect(choice.message.tool_calls[0]).toMatchObject({
type: 'function',
function: { name: 'get_weather', arguments: '{"city":"Paris"}' }
});
expect(typeof choice.message.tool_calls[0].id).toBe('string');
expect(choice.message.tool_calls[0].id).not.toBe('');
});

it('reports a tool call cut off by the token budget as length, not as a completed tool_calls turn', () => {
const { body } = transformChatResponse(
{
message: {
role: 'assistant',
content: '',
tool_calls: [{ function: { name: 'get_weather', arguments: { city: 'Par' } } }]
},
done: true,
done_reason: 'length'
},
ollama
);
const choice = (body.choices as any[])[0];
expect(choice.finish_reason).toBe('length');
expect(choice.message.tool_calls).toHaveLength(1);
});

it('reports a truncated answer as finish_reason length', () => {
const { body } = transformChatResponse(
{ message: { role: 'assistant', content: 'partial' }, done: true, done_reason: 'length' },
ollama
);
expect((body.choices as any[])[0].finish_reason).toBe('length');
});

it('surfaces thinking as reasoning_content beside the answer', () => {
const { body } = transformChatResponse(
{ message: { role: 'assistant', content: 'yes', thinking: 'the user asked…' }, done: true, done_reason: 'stop' },
ollama
);
expect((body.choices as any[])[0].message).toEqual({
role: 'assistant',
content: 'yes',
reasoning_content: 'the user asked…'
});
});
});
Loading
Loading