Ready-made clients: Bruno (26 requests, .bru files) and the
Postman collection. A machine-readable OpenAPI 3
document is served at /v3/api-docs (Swagger UI at
/swagger-ui.html when GATEWAY_SWAGGER_UI=true). Everything below is the same
contract in prose.
Base URL: http://localhost:8080 (override with PORT).
Two auth schemes:
| Surface | Header | Secret |
|---|---|---|
/v1/** |
Authorization: Bearer sk-gw-... |
client key from /admin/keys |
/admin/** |
X-Admin-Token: ... |
GATEWAY_ADMIN_TOKEN |
All bodies are JSON. Streaming responses are text/event-stream.
A ready-to-import Postman collection lives in
path.postman_collection.json with
path.postman_environment.json.
Every failure uses the OpenAI envelope:
{"error": {"message": "The model 'nope' does not exist.", "type": "invalid_request_error", "code": "model_not_found"}}| Code | HTTP | Meaning |
|---|---|---|
invalid_api_key |
401 | no Authorization header, unknown key, or disabled key — deliberately one code, so the response cannot be used to test whether a key exists |
invalid_admin_token |
401 | bad X-Admin-Token |
forbidden |
403 | admin API disabled (no token configured) |
model_not_found |
404 | no enabled route for that model id |
route_not_found / provider_not_found / key_not_found |
404 | unknown admin target |
invalid_request / invalid_json |
400 | body validation |
insufficient_credits |
402 | balance cannot cover the call |
rate_limit_exceeded |
429 | per-key RPM limit (has Retry-After) |
provider_not_configured |
502 | provider row has no credential |
provider_auth_error |
502 | upstream rejected the gateway's key |
upstream_invalid_request |
400 | upstream rejected the client's body |
upstream_timeout |
504 | connect/read/idle timeout |
upstream_error |
502 | upstream 5xx or unmapped failure |
all_providers_failed |
502 | every route in the chain failed |
circuit_open |
503 | provider parked by the circuit breaker |
concurrency_limit_exceeded |
429 | too many in-flight requests for this key |
payload_too_large |
413 | request body above GATEWAY_MAX_REQUEST_BYTES |
unsupported_parameter |
422 | parameter has no equivalent for this provider |
model_not_allowed |
403 | key's allowed_models does not cover the model |
idempotency_conflict |
409 | same Idempotency-Key, different body |
idempotency_in_progress |
409 | duplicate while the first attempt is running |
shutting_down |
503 | instance is draining; retry elsewhere |
unsupported_endpoint |
501 | endpoint not available for this provider type |
route_exists / provider_exists |
409 | duplicate admin create |
OpenAI chat-completions body. The gateway adds three optional routing fields:
| Field | Type | Meaning |
|---|---|---|
models |
string[] | fallback model ids, tried after model |
provider.order |
string[] | provider slug preference, e.g. ["groq","openai"] |
provider.allow_fallbacks |
bool | false pins the first route; upstream errors surface |
Everything the provider understands is forwarded untouched; model is rewritten to
the route's upstream id. Gateway-only fields are stripped before forwarding.
Non-streaming
curl -s http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi"}]}'{
"id": "chatcmpl-...",
"object": "chat.completion",
"created": 1700000000,
"model": "gpt-4o-mini",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "Hi"}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 11, "completion_tokens": 7, "total_tokens": 18,
"prompt_tokens_details": {"cached_tokens": 5}}
}Response headers:
| Header | Example | Notes |
|---|---|---|
X-Gateway-Provider |
groq |
provider that actually served it |
X-Gateway-Upstream-Model |
llama-3.3-70b-versatile |
upstream id used |
X-Gateway-Cost-Usd |
0.00000548 |
8-decimal itemised total |
X-Gateway-Latency-Ms |
412 |
end-to-end |
X-Idempotent-Replay |
true |
present only when replaying a stored response |
X-Path-Cache |
hit / miss / off |
response cache outcome for this call |
Response cache
Opt-in per route (cache_enabled, cache_ttl_seconds). A request qualifies when it
sets temperature: 0 explicitly and uses no tools. A hit replays the stored body
(streams replay as the same SSE sequence), costs nothing, and counts as savings in
/v1/usage. Cache keys are scoped to the client key.
curl -s http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"gpt-4o-mini","temperature":0,"messages":[{"role":"user","content":"Hi"}]}' \
-D - | grep -i x-path-cacheIdempotency
Send an Idempotency-Key header (up to 255 characters) to make a request safe to retry:
curl -s http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-H "Idempotency-Key: 7f2c-orders-2026-11-17-001" \
-d '{...}'| Situation | Result |
|---|---|
| First request | runs normally, response stored for GATEWAY_IDEMPOTENCY_TTL |
| Repeat, same body | stored response replayed, X-Idempotent-Replay: true, no upstream call, no charge |
| Repeat, different body | 409 idempotency_conflict |
| Repeat while first is still running | 409 idempotency_in_progress |
Streaming responses are captured and replayed as the same SSE sequence. A response
larger than GATEWAY_IDEMPOTENCY_MAX_BYTES is not stored; a retry will call the
provider again.
Streaming
curl -N http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"claude-3.5-haiku","stream":true,
"stream_options":{"include_usage":true},
"messages":[{"role":"user","content":"Say hi"}]}': open
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hi"}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":19,"completion_tokens":7,"total_tokens":26}}
data: [DONE]
stream_options.include_usageadds the usage-only chunk before[DONE].- A failure after the first chunk arrives as an error object then
[DONE]; the HTTP status is already 200, so clients must inspect the payload. - Fallback happens only before the first chunk.
Tool calls are passed through for OpenAI-shaped providers and translated to
tool_use / tool_result blocks for Anthropic.
End-user attribution
{"model":"gpt-4o-mini","end_user":"customer-8821",
"messages":[{"role":"user","content":"..."}]}end_user is stored on the usage row and indexed with the key and timestamp, so a key
with end_user_daily_limit_usd can cap each of its customers separately and /v1/usage
can report end_users[] back to the caller. user is accepted as an alias because that
is the field OpenAI clients already send; end_user wins when both are present.
At most 128 characters.
Free, authenticated, and bounded exactly like chat: the model allowlist, drain state,
concurrency limiter and RPM limiter all apply. Returns prompt_tokens, estimated
(true when the chars/4 heuristic was used rather than a real encoding) and a fits block
with the route's context window and your max_output_tokens. Tool definitions and
tool-result messages are counted, since that is where estimates are usually wrong.
OpenAI-shaped embeddings. Only OPENAI-type providers are supported; an Anthropic route
returns 501 unsupported_endpoint.
curl -s http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"text-embedding-3-small","input":"hello"}'Metered on input tokens only, with the same headers as chat.
Public when gateway.public-models=true (default).
{
"object": "list",
"data": [{
"id": "auto-fast", "object": "model", "created": 1700000000, "owned_by": "openai",
"context_length": 128000,
"providers": ["openai", "groq"],
"pricing": {"prompt_per_million": 0.15, "completion_per_million": 0.60},
"routing": [
{"provider": "openai", "upstream_model": "gpt-4o-mini", "priority": 1,
"input_price_per_million": 0.15, "output_price_per_million": 0.60}
]
}]
}One catalogue entry. Dots and slashes in the id are supported. 404 model_not_found
when unknown.
Per-key totals for the window (30s, 15m, 24h, 7d, 4w or ISO-8601).
{
"object": "usage", "window": "PT24H", "requests": 18,
"prompt_tokens": 213, "completion_tokens": 117, "total_tokens": 330,
"cached_input_tokens": 69, "cache_hit_ratio": 0.3239,
"cached_savings_usd": 0.0000195, "reasoning_tokens": 0,
"cost_usd": 0.000378,
"cost_breakdown": {"input": 0.0000858, "cached_input": 0.0000192,
"cache_write": 0.000012, "output": 0.0002622},
"credit_limit_usd": null, "used_usd": 0.000378, "remaining_usd": null,
"recent": [{"id": 42, "at": "...", "model": "gpt-4o-mini", "provider": "openai",
"cached_input_tokens": 5, "ttft_ms": 12, "tokens_per_second": 61.5,
"attempts": 1, "usage_estimated": false}]
}{"object":"credits","key_id":1,"key_name":"demo-local","currency":"USD",
"credit_limit_usd":25.00,"used_usd":0.000378,"remaining_usd":24.999622,"unlimited":false}All admin calls need X-Admin-Token. When GATEWAY_ADMIN_TOKEN is empty the whole
surface returns 403.
| Method | Path | Purpose |
|---|---|---|
| GET | /admin/providers |
list with credential source/hint |
| POST | /admin/providers |
create |
| PATCH | /admin/providers/{id} |
update |
| DELETE | /admin/providers/{id} |
delete (cascades routes) |
{
"slug": "together",
"display_name": "Together",
"type": "OPENAI",
"base_url": "https://api.together.xyz/v1",
"api_key": "optional; omit to use GATEWAY_UPSTREAM_TOGETHER_KEY",
"enabled": true
}type is OPENAI (chat-completions) or ANTHROPIC (Messages API). Response adds
credential_source (database / config / env), credential_configured,
credential_hint (last 4 only), route_count.
| Method | Path | Purpose |
|---|---|---|
| GET | /admin/routes?model= |
list (all enabled, or one public id) |
| POST | /admin/routes |
create |
| PATCH | /admin/routes/{id} |
update |
| DELETE | /admin/routes/{id} |
delete |
{
"public_model_id": "fast",
"provider_id": 1,
"upstream_model_id": "gpt-4o-mini",
"input_price_per_million": 0.15,
"cached_input_price_per_million": 0.075,
"output_price_per_million": 0.60,
"cache_write_price_per_million": null,
"priority": 1,
"context_length": 128000,
"supports_streaming": true,
"supports_tools": true,
"enabled": true,
"description": "cheap default"
}Lower priority runs first. Cached/write prices fall back to the input price when
omitted.
| Method | Path | Purpose |
|---|---|---|
| GET | /admin/keys |
list (masked) |
| POST | /admin/keys |
create, returns the plaintext key once |
| PATCH | /admin/keys/{id} |
rename, change limits, enable/disable |
| POST | /admin/keys/{id}/revoke |
disable permanently |
| POST | /admin/keys/{id}/topup |
{"amount_usd": 10} |
{"name": "team-a", "credit_limit_usd": 25, "rpm_limit": 120}{"id": 4, "name": "team-a", "masked_key": "sk-gw-AbCd...Wxyz",
"key": "sk-gw-...", "warning": "Store this key now. It is not shown again.",
"credit_limit_usd": 25, "used_usd": 0, "rpm_limit": 120, "enabled": true}POST /admin/models/sync — calls each provider's /models, filters non-chat ids,
upserts routes at priority 100. Known ids get prices from a built-in table; unknown
ids land at 0.00 and are flagged in description.
{"created": 4, "updated": 0, "skipped_providers": ["groq: no credential"], "errors": []}| Method | Path | Purpose |
|---|---|---|
| GET | /admin/stats?window=24h |
totals, percentiles, cache, providers/models |
| GET | /admin/stats?window=24h&bucket=1h |
the same, plus a bucketed time series |
| GET | /admin/usage?window=24h&limit=100&keyId= |
raw usage rows |
| POST | /admin/digest/test?date=2026-11-19 |
send yesterday's summary now and return its text |
| GET | /admin/usage/export?from=2026-01-01&to=2026-02-01 |
CSV for an explicit date slice (archive before a purge) |
{
"requests": 21, "total_tokens": 384, "cost_usd": 0.000393,
"latency_ms": {"p50": 2, "p95": 44, "p99": 44},
"ttft_ms": {"p50": 1, "p95": 5},
"tokens_per_second": {"p50": 7000.0},
"cache": {"prompt_tokens": 246, "cached_input_tokens": 84,
"hit_ratio": 0.3415, "savings_usd": 0.00002061, "semantic_hits": 3},
"fallback_requests": 3,
"estimated_usage_requests": 0,
"by_provider": {"openai": {"requests": 12, "cost_usd": 0.0002}},
"by_model": [{"model": "gpt-4o-mini", "requests": 12, "tokens": 180, "cost_usd": 0.0002}]
}bucket is optional and accepts 1m, 5m, 1h or 1d; anything else is a 400 that
names the accepted values. When present the response also carries:
{
"bucket": "1h",
"series": [
{"from": "2026-11-17T10:00:00Z", "requests": 412, "cost_usd": 0.71,
"failures": 3, "p95_latency_ms": 812}
]
}Every bucket in the window is present, zeroes included, and the window is clamped to at
most 500 points. The series is one query over usage_records for the whole window,
bucketed in Java (AdminReportingService), not one query per point; p95_latency_ms
covers successful calls only. Omitting bucket leaves the response exactly as before.
On a route whose provider type is GEMINI or VERTEX, the chat body accepts two
Gemini-native fields in addition to the OpenAI shape:
{"model": "gemini-2.5-pro",
"messages": [{"role": "user", "content": "..."}],
"thinkingConfig": {"thinkingBudget": 2048, "includeThoughts": true},
"safetySettings": [{"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_ONLY_HIGH"}]}reasoning_effort (minimal, low, medium, high) is accepted as a shortcut for a
thinking budget. Thinking tokens are returned in usage.completion_tokens (and broken out
as completion_tokens_details.reasoning_tokens) because Gemini bills them as output. A
blocked request returns 400 content_blocked with the safety category; a parameter the
adapter cannot honour returns 422 unsupported_parameter.
Each usage row records:
| Field | Meaning |
|---|---|
prompt_tokens |
total prompt, including cache read and cache write |
completion_tokens |
output tokens, including reasoning |
cached_input_tokens |
subset of prompt served from cache |
cache_write_tokens |
Anthropic cache writes (subset of prompt) |
reasoning_tokens |
reasoning/thinking tokens |
input_cost_usd, cached_input_cost_usd, cache_write_cost_usd, output_cost_usd |
itemised |
cost_usd |
total, 8 decimals |
cached_savings_usd |
what cached tokens would have cost at the input rate |
cache_tier |
exact or semantic when a cache served the call, null otherwise; how /admin/stats counts semantic hits |
ttft_ms |
time to first streamed token |
tokens_per_second |
generation throughput (post-TTFT window) |
upstream_request_id |
provider request id, for support |
finish_reason |
stop, length, tool_calls, ... |
attempts |
1 = first route worked; >1 = fallback happened |
fallback_from |
provider that failed first |
usage_estimated |
counts were inferred, not reported |
Billing rule: billableInput = prompt_tokens − cached_input_tokens − cache_write_tokens.