feat(providers): preserve prompt cache affinity

.env.template

@@ -397,14 +397,15 @@
 # Underscores in the suffix become hyphens in the provider name.
 #
 # Give ONE provider several API keys with <PROVIDER>_API_KEY_<n>, numbered from 2.
-# Requests then rotate across the keys round robin, which lifts the per-key rate
-# limit. Works for any provider that authenticates with an API key.
-# Caveat: rotation defeats provider prompt caching, because providers scope the
-# cache to the key that filled it. Rotate to raise rate limits, not to cut cost.
+# Identified sessions stay on one key by default to preserve provider prompt-cache
+# affinity; sessionless traffic still rotates. Set
+# <PROVIDER>[_SUFFIX]_SESSION_STICKY_KEYS=false to rotate every request instead.
+# Works for any provider that authenticates with an API key.
 
 # OpenAI
 # OPENAI_API_KEY=sk-...
 # OPENAI_API_KEY_2=sk-...
+# OPENAI_SESSION_STICKY_KEYS=false
 # OPENAI_BASE_URL=https://api.openai.com/v1
 
 # Anthropic

CLAUDE.md

@@ -135,7 +135,7 @@ Full reference: `.env.template` and `config/config.yaml`
 - **Resilience:** Configured via `config/config.yaml` - global `resilience.retry.*` and `resilience.circuit_breaker.*` defaults with optional per-provider overrides under `providers.<name>.resilience.retry.*` and `providers.<name>.resilience.circuit_breaker.*`. Retry defaults: `max_retries` (3), `initial_backoff` (1s), `max_backoff` (30s), `backoff_factor` (2.0), `jitter_factor` (0.1). Circuit breaker defaults: `failure_threshold` (5), `success_threshold` (2), `timeout` (30s). Breaker state is per-process and exported as the `gomodel_circuit_breaker_state` gauge when metrics are enabled. The dashboard's provider status also folds in real-traffic request health: each provider's runtime row carries `request_health` (last observed breaker state plus per-model request/error counts over a 10-minute sliding window; a model with ≥3 errors and a ≥50% error rate is flagged). An open breaker turns the provider card's status pill to "Circuit Open" (unhealthy), a half-open breaker or a flagged model degrades an otherwise healthy provider ("Recovering"/"Degraded"), and the expanded card details list recent per-model traffic with the latest error. Signals only worsen the discovery-based status, never improve it; tracking is in-memory per instance, and providers with no recent requests show discovery-based status only.
 - **Metrics:** `METRICS_ENABLED` (false), `METRICS_ENDPOINT` (/metrics)
 - **Guardrails:** Definitions are persisted in the `guardrail_definitions` store and managed via the admin API/dashboard; `config/config.yaml` entries are validated and upserted into that store at startup (a seed, not the source of truth). `GUARDRAILS_ENABLED` env var gates the feature.
-- **Provider API key rotation:** Any API-key provider accepts several keys: `<PROVIDER>[_SUFFIX]_API_KEY_<n>` env vars (numbered from 2; `_1` is accepted as a synonym for the unsuffixed key) or `providers.<name>.api_keys` in `config.yaml` (merged after `api_key`, de-duplicated, unresolved `${...}` entries dropped; env replaces the whole YAML list). Two or more keys turn on round-robin rotation, drawn per outbound HTTP request — including retries, so a 429'd request retries under the next key. Realtime websocket sessions pick a key per session. Counters are in-memory per instance. The trailing number names a key, not a provider: `OPENAI_API_KEY_2` is key 2 of `openai`, while `OPENAI_REGION_2_API_KEY` is the sole key of provider `openai-region-2`. **Rotation defeats provider prompt caching** (providers scope the cache to the key that filled it); use it to lift per-key rate limits, not to save cost. Keyless (Ollama, vLLM) and non-API-key providers (Vertex, Bedrock) are unaffected.
+- **Provider API key rotation:** Any API-key provider accepts several keys: `<PROVIDER>[_SUFFIX]_API_KEY_<n>` env vars (numbered from 2; `_1` is accepted as a synonym for the unsuffixed key) or `providers.<name>.api_keys` in `config.yaml` (merged after `api_key`, de-duplicated, unresolved `${...}` entries dropped; env replaces the whole YAML list). Identified sessions deterministically stay on one key by default, preserving provider prompt-cache affinity while spreading different sessions across the configured keys; sessionless requests remain round robin. Set `providers.<name>.session_sticky_keys: false` or untick **Session-sticky API keys** in the provider editor for strict per-request round robin. Realtime sessions use the same affinity. The trailing number names a key, not a provider: `OPENAI_API_KEY_2` is key 2 of `openai`, while `OPENAI_REGION_2_API_KEY` is the sole key of provider `openai-region-2`. Keyless (Ollama, vLLM) and non-API-key providers (Vertex, Bedrock) are unaffected.
 - **Provider credentials without env vars:** Every provider below can instead be configured from the admin dashboard's Providers page (or `/admin/provider-credentials` GET/PUT/DELETE), persisted to the `provider_credentials` store — the same declarative-shadows-store precedence as MCP servers: a provider name declared via env vars/`config.yaml` is read-only in the dashboard (`managed: true`), and a store row upsert/delete hot-registers or unregisters the provider into the live registry immediately, no restart. `GOMODEL` boots fine with zero providers configured (empty catalog) so this is a complete alternative to env-var credentials, not just a supplement. API keys (`api_keys`, an ordered rotation list, same semantics as `providers.<name>.api_keys`) and service-account secrets are redacted as `***********` on read; an upsert echoing any all-asterisk mask of at least three characters at a position preserves the stored value there (rejected if that position was never set). Disabling a row (`enabled: false`) unregisters it from routing without deleting the stored credentials. `GET /admin/provider-credentials/types` lists every constructible provider type with the credential form it accepts (`fields[]` of `name`/`required`/`advanced`/`options`, plus `default_base_url`), derived from each provider's `DiscoveryConfig` — the dashboard renders only those fields, so an OpenAI-type provider asks for an API key while a Vertex one asks for project/location/service account and no key at all. Upserts are validated against that form (and against Google's project-or-base-URL and service-account rules) *before* anything is persisted, so an unusable credential is rejected with a 400 naming the offending field in `error.param` rather than stored as a broken row.
 - **Providers:** `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `ANTHROPIC_DEFAULT_MAX_TOKENS` (optional default `max_tokens` for Anthropic-translated requests that omit it; default 4096), `GEMINI_API_KEY`, `USE_GOOGLE_GEMINI_NATIVE_API` (true by default; false uses Gemini's OpenAI-compatible chat API), `XAI_API_KEY`, `GROQ_API_KEY`, `FIREWORKS_API_KEY`, `FIREWORKS_BASE_URL` (optional Fireworks AI endpoint override; default `https://api.fireworks.ai/inference/v1`), `META_API_KEY`, `META_BASE_URL` (optional Meta Model API endpoint override; default `https://api.meta.ai/v1`; Muse Spark models, e.g. `muse-spark-1.1`), `OPENROUTER_API_KEY`, `OPENROUTER_SITE_URL`/`OPENROUTER_APP_NAME` (optional OpenRouter attribution headers), `ZAI_API_KEY`, `ZAI_BASE_URL` (optional Z.ai endpoint override), `MINIMAX_API_KEY`, `MINIMAX_BASE_URL` (optional MiniMax endpoint override), `XIAOMI_API_KEY`, `XIAOMI_BASE_URL` (optional Xiaomi MiMo endpoint override), `OPENCODE_GO_API_KEY`, `OPENCODE_GO_BASE_URL` (optional OpenCode Go/Zen endpoint override; default `https://opencode.ai/zen/go/v1`), `OPENCODE_GO_MESSAGES_MODELS` (optional comma-separated model IDs routed to the Anthropic-native `/messages` endpoint instead of `/chat/completions`; default `qwen3.7-max`), `BAILIAN_API_KEY`, `BAILIAN_BASE_URL` (optional Bailian base URL for region switching; default `https://dashscope.aliyuncs.com/compatible-mode/v1`), `AZURE_API_KEY`, `AZURE_BASE_URL` (Azure OpenAI deployment base URL), `AZURE_API_VERSION` (optional Azure API version), `ORACLE_API_KEY` (Oracle API key), `ORACLE_BASE_URL` (Oracle OpenAI-compatible base URL), `BEDROCK_BASE_URL` (Bedrock Runtime region or endpoint), `BEDROCK_MANTLE_API_KEY`, `BEDROCK_MANTLE_BASE_URL` (Mantle region or endpoint), `BEDROCK_MANTLE_API_MODE` (`auto`, `openai`, or `standard`), `<PROVIDER>[_SUFFIX]_MODELS` (comma-separated configured model list for any provider type), `OLLAMA_BASE_URL`, `VLLM_BASE_URL`, `VLLM_API_KEY` (optional upstream vLLM bearer token)
 - **Provider model metadata:** `providers.<name>.models` accepts either model IDs (strings) or `{id, metadata}` objects. When `metadata` is supplied (`display_name`, `context_window`, `max_output_tokens`, `modes`, `capabilities`, `pricing`, …) it is merged onto the remote ai-model-list entry during enrichment, with operator values winning per-field. Primary use case: advertising context windows, capabilities, and pricing for local models (Ollama) and other custom endpoints whose IDs are not in the upstream registry.

config/config.example.yaml

@@ -314,13 +314,14 @@ providers:
   openai:
     type: openai
     api_key: "sk-..."
-    # Several keys for one provider rotate round robin, raising the per-key rate
-    # limit. Equivalent env vars: OPENAI_API_KEY_2, OPENAI_API_KEY_3, ...
-    # Caveat: rotation defeats provider prompt caching, since providers scope the
-    # cache to the key that filled it.
+    # Identified sessions stay on one key by default, preserving provider prompt
+    # cache affinity while still spreading different sessions across keys.
+    # Equivalent env vars: OPENAI_API_KEY_2, OPENAI_API_KEY_3, ...
     # api_keys:
     #   - "${OPENAI_API_KEY_2}"
     #   - "${OPENAI_API_KEY_3}"
+    # Set false for strict per-request round robin (usually worse for prompt caching).
+    # session_sticky_keys: false
     # Per-provider resilience overrides (optional).
     # Only specified fields override the global defaults above.
     # resilience:

config/providers.go

@@ -7,9 +7,13 @@ type RawProviderConfig struct {
 	Type   string `yaml:"type"`
 	APIKey string `yaml:"api_key"`
 	// APIKeys lists additional API keys for this provider. When more than one
-	// key is resolved (counting APIKey), requests rotate across them round
-	// robin. Set it via `api_keys:` or the `<PROVIDER>_API_KEY_<n>` env vars.
-	APIKeys                  []string             `yaml:"api_keys"`
+	// key is resolved (counting APIKey), identified sessions stay on one key by
+	// default while sessionless requests rotate round robin. Set it via
+	// `api_keys:` or the `<PROVIDER>_API_KEY_<n>` env vars.
+	APIKeys []string `yaml:"api_keys"`
+	// SessionStickyKeys defaults to true. Set false to restore round-robin key
+	// selection for every request, including requests carrying a session ID.
+	SessionStickyKeys        *bool                `yaml:"session_sticky_keys"`
 	BaseURL                  string               `yaml:"base_url"`
 	APIVersion               string               `yaml:"api_version"`
 	Backend                  string               `yaml:"backend"`

docs/adr/0007-anthropic-messages-ingress.md

@@ -112,8 +112,6 @@ end to end, including request-validation and upstream errors.
 ### What is explicitly not implemented (v1)
 
 - **`/v1/messages/batches`** (Messages Batches API) — deferred; batches has its own pipeline.
-- **`cache_control` breakpoints** — dropped during the canonical hop; prompt-caching cost
-  benefits are not preserved when translating through `core.ChatRequest`.
 - **Extended-thinking signatures and `thinking` blocks on input messages** — dropped; the
   canonical chat type has no first-class field for them.
 - **Server/built-in tools** (web search, code execution, etc.) — a `tools[]` entry
@@ -144,10 +142,10 @@ end to end, including request-validation and upstream errors.
 ### Negative / Mitigations
 
 - **Lossy round-trip to the Anthropic provider.** A `/v1/messages` request routed *to* the
-  Anthropic provider is translated Anthropic → `core.ChatRequest` → Anthropic; provider-only
-  features (`cache_control`, thinking signatures, server tools) do not survive the canonical
-  hop. Mitigation: clients needing byte-exact Anthropic fidelity (including prompt-cache
-  breakpoints) can still use the `/p/anthropic/v1/messages` passthrough. A future optimization
+  Anthropic provider is translated Anthropic → `core.ChatRequest` → Anthropic. Prompt-cache
+  controls survive on supported request, system/content, custom-tool, and tool-history
+  locations, but thinking signatures and server tools do not. Mitigation: clients needing
+  byte-exact Anthropic fidelity can still use the `/p/anthropic/v1/messages` passthrough. A future optimization
   could add an Anthropic → Anthropic fast path that skips the canonical hop.
 - **count_tokens is an estimate**, typically within ~10–25% of a tokenizer-exact count, and is
   not model-specific. Mitigation: documented clearly; adequate for budgeting/UX sizing, not for

docs/advanced/anthropic-messages-api.mdx

@@ -115,8 +115,8 @@ is recorded under the `/v1/messages` endpoint so it can be filtered in the dashb
 `/v1/messages` translates through GoModel's canonical chat type. Anthropic-specific
 features that have no canonical equivalent are not preserved end to end:
 
-- **`cache_control` breakpoints** are dropped — prompt-caching cost benefits are not
-  carried through the canonical hop.
+- **`cache_control`** is preserved on the request, system/content blocks,
+  custom tools, and tool-use/tool-result history when routed to Anthropic.
 - **Extended-thinking signatures** and `thinking` blocks on input messages are dropped.
 - **Server/built-in tools** (web search, code execution, …) are rejected with a clear
   `400`; only custom tools (`type` absent or `"custom"`) translate.
@@ -137,7 +137,7 @@ features that have no canonical equivalent are not preserved end to end:
   streaming `message_start` event; the authoritative counts arrive in the final
   `message_delta` event, which SDK accumulators prefer.
 
-For byte-exact Anthropic fidelity (including prompt-cache breakpoints), use the
+For byte-exact Anthropic fidelity beyond the supported cache controls, use the
 `/p/anthropic/v1/messages` passthrough route instead.
 
 See [ADR-0007](https://github.com/ENTERPILOT/GoModel/blob/main/docs/adr/0007-anthropic-messages-ingress.md)

docs/dev/2026-07-17_anthropic-sdk-compat-findings.md

@@ -23,7 +23,7 @@ Models exercised (cheap/free tiers): `openai/gpt-4o-mini`, `gemini/gemini-2.5-fl
 | F3 message_start usage=0 | **Fixed (best effort)** — `message_start` now carries the chars/4 heuristic estimate; authoritative usage still lands in `message_delta`, which SDK accumulators prefer. |
 | F4 models list shape | **Fixed** — `GET /v1/models` renders the Anthropic list shape (`type`, `display_name`, `created_at`, `has_more`/`first_id`/`last_id`) when the request carries the `anthropic-version` header Anthropic SDKs always send. |
 | F5 non-canonical 404 | **Fixed** — unknown routes return the canonical error envelope, Anthropic-shaped for Anthropic-dialect callers (`e.RouteNotFound` + `handleRouteNotFound`). Messages batches API itself stays unimplemented, now documented. |
-| F6 cache_control dropped | **Documented** (`docs/advanced/anthropic-messages-api.mdx`) — real propagation needs cache-breakpoint representation in the canonical type; use `/p/anthropic` for prompt caching. |
+| F6 cache_control dropped | **Fixed 2026-07-31** — request, system/content block, custom-tool, tool-use, and tool-result controls survive the canonical hop to Anthropic. |
 | F7 heuristic count_tokens | **Documented** — passthrough gives exact counts. |
 | F8 unsigned thinking blocks | **Documented** — by design; replay against the gateway works. |
 
@@ -101,14 +101,14 @@ works perfectly (10 models, `claude-sonnet-5` first).
 (b) unknown-route 404s under `/v1/` could use the canonical error envelope so SDK
 clients raise a clean typed error.
 
-### F6 — `cache_control` accepted but silently dropped · limitation
+### F6 — `cache_control` accepted but silently dropped · fixed 2026-07-31
 
 `cache_control` markers on system/content blocks are tolerated (no 400 — good), but the
 translation flattens system prompts to plain strings and drops the markers, so **prompt
 caching never activates**, even when the request routes to the Anthropic provider.
 Usage responses show no cache fields. Fine for correctness, costs money for heavy users.
-Worth documenting; propagating breakpoints on the anthropic-provider path would be the
-real fix.
+The canonical representation now retains these controls in metadata and the Anthropic
+provider restores them at every supported wire location.
 
 ### F7 — `count_tokens` is heuristic · documented, keep an eye on it
 

docs/features/session-keeping.mdx

@@ -10,6 +10,9 @@ uses it in two places:
 - **Sticky load balancing** — a load-balanced virtual model routes every
   request of a session to the target that served it first, which keeps provider
   prompt caches warm and model behavior consistent mid-conversation.
+- **Sticky API-key selection** — when one provider has several keys, every
+  request in a detected session uses the same key, preserving provider
+  prompt-cache affinity.
 - **Threaded audit logs** — the dashboard's Audit Logs page groups a session's
   requests into one thread: the latest request as the row, with an expander
   that unfolds the older requests beneath it.
@@ -84,6 +87,13 @@ Pins are in-memory per instance (like rate-limit counters): after a restart or
 on another replica, the next request of a session simply re-pins. Idle
 sessions expire after 6 hours.
 
+API-key affinity is deterministic rather than stored, so the same ordered key
+set selects the same key across replicas and restarts. Disable it per provider
+with `session_sticky_keys: false`,
+`<PROVIDER>[_SUFFIX]_SESSION_STICKY_KEYS=false`, or the dashboard's
+**Session-sticky API keys**
+checkbox. Requests with no detected session remain round robin.
+
 ## Threaded audit logs
 
 Audit entries record the session id (`session_id`), and the Audit Logs page