Skip to content

Latest commit

 

History

History
437 lines (349 loc) · 16.3 KB

API.md

File metadata and controls

437 lines (349 loc) · 16.3 KB

API reference

Ready-made clients: Bruno (26 requests, .bru files) and the Postman collection. A machine-readable OpenAPI 3 document is served at /v3/api-docs (Swagger UI at /swagger-ui.html when GATEWAY_SWAGGER_UI=true). Everything below is the same contract in prose.

Base URL: http://localhost:8080 (override with PORT).

Two auth schemes:

Surface Header Secret
/v1/** Authorization: Bearer sk-gw-... client key from /admin/keys
/admin/** X-Admin-Token: ... GATEWAY_ADMIN_TOKEN

All bodies are JSON. Streaming responses are text/event-stream.

A ready-to-import Postman collection lives in path.postman_collection.json with path.postman_environment.json.


Errors

Every failure uses the OpenAI envelope:

{"error": {"message": "The model 'nope' does not exist.", "type": "invalid_request_error", "code": "model_not_found"}}
Code HTTP Meaning
invalid_api_key 401 no Authorization header, unknown key, or disabled key — deliberately one code, so the response cannot be used to test whether a key exists
invalid_admin_token 401 bad X-Admin-Token
forbidden 403 admin API disabled (no token configured)
model_not_found 404 no enabled route for that model id
route_not_found / provider_not_found / key_not_found 404 unknown admin target
invalid_request / invalid_json 400 body validation
insufficient_credits 402 balance cannot cover the call
rate_limit_exceeded 429 per-key RPM limit (has Retry-After)
provider_not_configured 502 provider row has no credential
provider_auth_error 502 upstream rejected the gateway's key
upstream_invalid_request 400 upstream rejected the client's body
upstream_timeout 504 connect/read/idle timeout
upstream_error 502 upstream 5xx or unmapped failure
all_providers_failed 502 every route in the chain failed
circuit_open 503 provider parked by the circuit breaker
concurrency_limit_exceeded 429 too many in-flight requests for this key
payload_too_large 413 request body above GATEWAY_MAX_REQUEST_BYTES
unsupported_parameter 422 parameter has no equivalent for this provider
model_not_allowed 403 key's allowed_models does not cover the model
idempotency_conflict 409 same Idempotency-Key, different body
idempotency_in_progress 409 duplicate while the first attempt is running
shutting_down 503 instance is draining; retry elsewhere
unsupported_endpoint 501 endpoint not available for this provider type
route_exists / provider_exists 409 duplicate admin create

Public API

POST /v1/chat/completions

OpenAI chat-completions body. The gateway adds three optional routing fields:

Field Type Meaning
models string[] fallback model ids, tried after model
provider.order string[] provider slug preference, e.g. ["groq","openai"]
provider.allow_fallbacks bool false pins the first route; upstream errors surface

Everything the provider understands is forwarded untouched; model is rewritten to the route's upstream id. Gateway-only fields are stripped before forwarding.

Non-streaming

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi"}]}'
{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "gpt-4o-mini",
  "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hi"}, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 11, "completion_tokens": 7, "total_tokens": 18,
            "prompt_tokens_details": {"cached_tokens": 5}}
}

Response headers:

Header Example Notes
X-Gateway-Provider groq provider that actually served it
X-Gateway-Upstream-Model llama-3.3-70b-versatile upstream id used
X-Gateway-Cost-Usd 0.00000548 8-decimal itemised total
X-Gateway-Latency-Ms 412 end-to-end
X-Idempotent-Replay true present only when replaying a stored response
X-Path-Cache hit / miss / off response cache outcome for this call

Response cache

Opt-in per route (cache_enabled, cache_ttl_seconds). A request qualifies when it sets temperature: 0 explicitly and uses no tools. A hit replays the stored body (streams replay as the same SSE sequence), costs nothing, and counts as savings in /v1/usage. Cache keys are scoped to the client key.

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"gpt-4o-mini","temperature":0,"messages":[{"role":"user","content":"Hi"}]}' \
  -D - | grep -i x-path-cache

Idempotency

Send an Idempotency-Key header (up to 255 characters) to make a request safe to retry:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -H "Idempotency-Key: 7f2c-orders-2026-11-17-001" \
  -d '{...}'
Situation Result
First request runs normally, response stored for GATEWAY_IDEMPOTENCY_TTL
Repeat, same body stored response replayed, X-Idempotent-Replay: true, no upstream call, no charge
Repeat, different body 409 idempotency_conflict
Repeat while first is still running 409 idempotency_in_progress

Streaming responses are captured and replayed as the same SSE sequence. A response larger than GATEWAY_IDEMPOTENCY_MAX_BYTES is not stored; a retry will call the provider again.

Streaming

curl -N http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"claude-3.5-haiku","stream":true,
       "stream_options":{"include_usage":true},
       "messages":[{"role":"user","content":"Say hi"}]}'
: open

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hi"}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":19,"completion_tokens":7,"total_tokens":26}}
data: [DONE]
  • stream_options.include_usage adds the usage-only chunk before [DONE].
  • A failure after the first chunk arrives as an error object then [DONE]; the HTTP status is already 200, so clients must inspect the payload.
  • Fallback happens only before the first chunk.

Tool calls are passed through for OpenAI-shaped providers and translated to tool_use / tool_result blocks for Anthropic.

End-user attribution

{"model":"gpt-4o-mini","end_user":"customer-8821",
 "messages":[{"role":"user","content":"..."}]}

end_user is stored on the usage row and indexed with the key and timestamp, so a key with end_user_daily_limit_usd can cap each of its customers separately and /v1/usage can report end_users[] back to the caller. user is accepted as an alias because that is the field OpenAI clients already send; end_user wins when both are present. At most 128 characters.

POST /v1/tokens/count

Free, authenticated, and bounded exactly like chat: the model allowlist, drain state, concurrency limiter and RPM limiter all apply. Returns prompt_tokens, estimated (true when the chars/4 heuristic was used rather than a real encoding) and a fits block with the route's context window and your max_output_tokens. Tool definitions and tool-result messages are counted, since that is where estimates are usually wrong.

POST /v1/embeddings

OpenAI-shaped embeddings. Only OPENAI-type providers are supported; an Anthropic route returns 501 unsupported_endpoint.

curl -s http://localhost:8080/v1/embeddings \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"text-embedding-3-small","input":"hello"}'

Metered on input tokens only, with the same headers as chat.

GET /v1/models

Public when gateway.public-models=true (default).

{
  "object": "list",
  "data": [{
    "id": "auto-fast", "object": "model", "created": 1700000000, "owned_by": "openai",
    "context_length": 128000,
    "providers": ["openai", "groq"],
    "pricing": {"prompt_per_million": 0.15, "completion_per_million": 0.60},
    "routing": [
      {"provider": "openai", "upstream_model": "gpt-4o-mini", "priority": 1,
       "input_price_per_million": 0.15, "output_price_per_million": 0.60}
    ]
  }]
}

GET /v1/models/{id}

One catalogue entry. Dots and slashes in the id are supported. 404 model_not_found when unknown.

GET /v1/usage?window=24h

Per-key totals for the window (30s, 15m, 24h, 7d, 4w or ISO-8601).

{
  "object": "usage", "window": "PT24H", "requests": 18,
  "prompt_tokens": 213, "completion_tokens": 117, "total_tokens": 330,
  "cached_input_tokens": 69, "cache_hit_ratio": 0.3239,
  "cached_savings_usd": 0.0000195, "reasoning_tokens": 0,
  "cost_usd": 0.000378,
  "cost_breakdown": {"input": 0.0000858, "cached_input": 0.0000192,
                     "cache_write": 0.000012, "output": 0.0002622},
  "credit_limit_usd": null, "used_usd": 0.000378, "remaining_usd": null,
  "recent": [{"id": 42, "at": "...", "model": "gpt-4o-mini", "provider": "openai",
              "cached_input_tokens": 5, "ttft_ms": 12, "tokens_per_second": 61.5,
              "attempts": 1, "usage_estimated": false}]
}

GET /v1/credits

{"object":"credits","key_id":1,"key_name":"demo-local","currency":"USD",
 "credit_limit_usd":25.00,"used_usd":0.000378,"remaining_usd":24.999622,"unlimited":false}

Admin API

All admin calls need X-Admin-Token. When GATEWAY_ADMIN_TOKEN is empty the whole surface returns 403.

Providers

Method Path Purpose
GET /admin/providers list with credential source/hint
POST /admin/providers create
PATCH /admin/providers/{id} update
DELETE /admin/providers/{id} delete (cascades routes)
{
  "slug": "together",
  "display_name": "Together",
  "type": "OPENAI",
  "base_url": "https://api.together.xyz/v1",
  "api_key": "optional; omit to use GATEWAY_UPSTREAM_TOGETHER_KEY",
  "enabled": true
}

type is OPENAI (chat-completions) or ANTHROPIC (Messages API). Response adds credential_source (database / config / env), credential_configured, credential_hint (last 4 only), route_count.

Routes

Method Path Purpose
GET /admin/routes?model= list (all enabled, or one public id)
POST /admin/routes create
PATCH /admin/routes/{id} update
DELETE /admin/routes/{id} delete
{
  "public_model_id": "fast",
  "provider_id": 1,
  "upstream_model_id": "gpt-4o-mini",
  "input_price_per_million": 0.15,
  "cached_input_price_per_million": 0.075,
  "output_price_per_million": 0.60,
  "cache_write_price_per_million": null,
  "priority": 1,
  "context_length": 128000,
  "supports_streaming": true,
  "supports_tools": true,
  "enabled": true,
  "description": "cheap default"
}

Lower priority runs first. Cached/write prices fall back to the input price when omitted.

Client keys

Method Path Purpose
GET /admin/keys list (masked)
POST /admin/keys create, returns the plaintext key once
PATCH /admin/keys/{id} rename, change limits, enable/disable
POST /admin/keys/{id}/revoke disable permanently
POST /admin/keys/{id}/topup {"amount_usd": 10}
{"name": "team-a", "credit_limit_usd": 25, "rpm_limit": 120}
{"id": 4, "name": "team-a", "masked_key": "sk-gw-AbCd...Wxyz",
 "key": "sk-gw-...", "warning": "Store this key now. It is not shown again.",
 "credit_limit_usd": 25, "used_usd": 0, "rpm_limit": 120, "enabled": true}

Model sync

POST /admin/models/sync — calls each provider's /models, filters non-chat ids, upserts routes at priority 100. Known ids get prices from a built-in table; unknown ids land at 0.00 and are flagged in description.

{"created": 4, "updated": 0, "skipped_providers": ["groq: no credential"], "errors": []}

Reporting

Method Path Purpose
GET /admin/stats?window=24h totals, percentiles, cache, providers/models
GET /admin/stats?window=24h&bucket=1h the same, plus a bucketed time series
GET /admin/usage?window=24h&limit=100&keyId= raw usage rows
POST /admin/digest/test?date=2026-11-19 send yesterday's summary now and return its text
GET /admin/usage/export?from=2026-01-01&to=2026-02-01 CSV for an explicit date slice (archive before a purge)
{
  "requests": 21, "total_tokens": 384, "cost_usd": 0.000393,
  "latency_ms": {"p50": 2, "p95": 44, "p99": 44},
  "ttft_ms": {"p50": 1, "p95": 5},
  "tokens_per_second": {"p50": 7000.0},
  "cache": {"prompt_tokens": 246, "cached_input_tokens": 84,
            "hit_ratio": 0.3415, "savings_usd": 0.00002061, "semantic_hits": 3},
  "fallback_requests": 3,
  "estimated_usage_requests": 0,
  "by_provider": {"openai": {"requests": 12, "cost_usd": 0.0002}},
  "by_model": [{"model": "gpt-4o-mini", "requests": 12, "tokens": 180, "cost_usd": 0.0002}]
}

bucket is optional and accepts 1m, 5m, 1h or 1d; anything else is a 400 that names the accepted values. When present the response also carries:

{
  "bucket": "1h",
  "series": [
    {"from": "2026-11-17T10:00:00Z", "requests": 412, "cost_usd": 0.71,
     "failures": 3, "p95_latency_ms": 812}
  ]
}

Every bucket in the window is present, zeroes included, and the window is clamped to at most 500 points. The series is one query over usage_records for the whole window, bucketed in Java (AdminReportingService), not one query per point; p95_latency_ms covers successful calls only. Omitting bucket leaves the response exactly as before.


Gemini extensions

On a route whose provider type is GEMINI or VERTEX, the chat body accepts two Gemini-native fields in addition to the OpenAI shape:

{"model": "gemini-2.5-pro",
 "messages": [{"role": "user", "content": "..."}],
 "thinkingConfig": {"thinkingBudget": 2048, "includeThoughts": true},
 "safetySettings": [{"category": "HARM_CATEGORY_HATE_SPEECH",
                     "threshold": "BLOCK_ONLY_HIGH"}]}

reasoning_effort (minimal, low, medium, high) is accepted as a shortcut for a thinking budget. Thinking tokens are returned in usage.completion_tokens (and broken out as completion_tokens_details.reasoning_tokens) because Gemini bills them as output. A blocked request returns 400 content_blocked with the safety category; a parameter the adapter cannot honour returns 422 unsupported_parameter.

Metering fields

Each usage row records:

Field Meaning
prompt_tokens total prompt, including cache read and cache write
completion_tokens output tokens, including reasoning
cached_input_tokens subset of prompt served from cache
cache_write_tokens Anthropic cache writes (subset of prompt)
reasoning_tokens reasoning/thinking tokens
input_cost_usd, cached_input_cost_usd, cache_write_cost_usd, output_cost_usd itemised
cost_usd total, 8 decimals
cached_savings_usd what cached tokens would have cost at the input rate
cache_tier exact or semantic when a cache served the call, null otherwise; how /admin/stats counts semantic hits
ttft_ms time to first streamed token
tokens_per_second generation throughput (post-TTFT window)
upstream_request_id provider request id, for support
finish_reason stop, length, tool_calls, ...
attempts 1 = first route worked; >1 = fallback happened
fallback_from provider that failed first
usage_estimated counts were inferred, not reported

Billing rule: billableInput = prompt_tokens − cached_input_tokens − cache_write_tokens.