Summary
When a provider is configured with a static model list (providers.<name>.models) and models.configured_provider_models_mode: allowlist, that provider — and every model under it — can never become unhealthy based on observed failures. Repeated 429 Too Many Requests (quota exhausted, upstream rate limit, etc.) neither trigger configured failover rules nor trip any request-time health signal. The 429 is passed straight back to the client even when a working fallback model is configured and even when the failure is sustained for a long period.
In short, allowlist mode makes the provider permanently "healthy" from GoModel's perspective, regardless of what is happening at request time.
Real-world scenario
Using the Kimi provider: Kimi's upstream /models endpoint does not list embedding models, so the only way to expose them through GoModel is to declare them statically.
models:
configured_provider_models_mode: allowlist
providers:
kimi:
base_url: https://api.moonshot.ai/v1
api_key: ${KIMI_API_KEY}
models:
- kimi-k2-0711-preview
- moonshot-v1-8k
- kimi-embedding-001 # only reachable via static list
failover:
rules:
kimi-embedding-001:
- openai/text-embedding-3-smallObserved behavior: when Kimi starts returning sustained 429s (e.g. usage quota exhausted), every request to kimi-embedding-001 keeps returning 429 to the client. The configured fallback openai/text-embedding-3-small is never attempted, and the provider never flips unhealthy, so the situation never self-heals.
Expected behavior: after the primary model returns 429, GoModel should attempt the configured failover chain; and a provider that consistently fails at request time should degrade in health independently of how its model inventory was discovered.
Root-cause analysis
Two cooperating gaps:
1. Static inventory is unconditionally marked healthy (and stays that way)
internal/providers/registry_init.go — in allowlist mode the upstream ListModels call is skipped entirely and the inventory is synthesized from the configured list (registry_init.go:400-419, configured_models.go:46-74). registry_init.go:213-216 then sets fetchedInventory.runtimeUpdates[*].lastModelFetchSuccessAt, which the status classifier reads as "healthy" (see TestClassifyProviderStatus_HealthyForAllowlistInventory in internal/admin/handler_providers_test.go:10-16).
Health, in the current model, is derived from model-fetch success — not from request success. In allowlist mode the model fetch is a local no-op that always "succeeds", so the health signal is decoupled from reality by construction.
2. Request-time safety nets don't cover 429, and the failover chain can be silently emptied
- Circuit breaker (
internal/llmclient/client.go:302-307, :330-333): shouldTripCircuitBreaker explicitly returns false for 429, and RecordFailure is skipped for 429 unless half-open. By design, repeated 429s never trip the breaker.
- Failover selector resolution (
internal/failover/resolver.go:328-349 resolveSelector): every failover candidate is validated against registry.GetModel(model), and the registry is allowlist-filtered (internal/app/app.go:602 wires the resolver with the already-filtered registry). Any failover target not present in the statically-listed inventory is silently dropped from the chain (manualSelectorsFor, resolver.go:154-172). With an empty chain, tryFailoverResponse short-circuits at internal/gateway/failover.go:69 (len(failovers) == 0) and the primary 429 is returned untouched — even though ShouldAttemptFailover (failover.go:262-307) correctly treats 429 as failover-eligible.
Net effect: 429 says "fail over", but the chain is empty and nothing ever marks the primary unhealthy.
Reproduction
- Configure a provider with a static
models: list and configured_provider_models_mode: allowlist.
- Add a failover rule from one of those static models to a model on another provider.
- Have the primary return 429 (exhaust quota, or point
base_url at a mock that always 429s — tests/e2e/failover_test.go:58-160 already has the harness; TestRateLimitFullySaturatedAliasReturns429_E2E is close but does not cover static-models mode).
- Observe: client gets raw 429, fallback never attempted, provider stays "healthy" in admin status endpoints indefinitely.
Proposed direction
Reframe health measurement around observed requests rather than inventory discovery:
- Primary health signal = request outcomes. Provider / model health should be driven primarily by recent request outcomes (successes vs. sustained failures including 4xx/5xx, with 429 included by default or opt-in). Configurable thresholds should determine when a sustained failure pattern flips a model (or the provider's view of that model) to unhealthy.
- Model-fetch health check used only as a fallback. When there are no request outcomes in the measurement window, fall back to inventory-fetch health. Static-list inventories naturally land in this branch and can keep their current "always healthy" default — that part is fine and is not the bug.
- Known failure overrides the static default. When the request-time path does observe failures, it should override the always-healthy default and degrade the model/provider's health. The system should never ignore a known, recent failure just because the inventory came from a static list.
- 429 in the breaker / failover decision. A 429 is a strong signal that the current route is saturated or the upstream is rate-limiting; repeated 429s should contribute to the request-time health signal and the per-model route gate, not be silently dropped before the failover chain.
- Failover selector resolution must not silently empty the chain. If a configured failover target is dropped because it is missing from the source provider's allowlist-filtered inventory, log at warn level and/or surface it in the dashboard. Silently producing an empty chain from a user-configured rule is the worst failure mode here.
- Configurable. A new flag (e.g.
failover.treat_429_as_failure: true and/or a health.request_failure_threshold for 4xx-class errors) so users keep opt-out control.
Acceptance criteria
Environment
- GoModel @
7c16a72 (main)
- Kimi provider (Moonshot API), embedding model reachable only via static
models: list
Summary
When a provider is configured with a static model list (
providers.<name>.models) andmodels.configured_provider_models_mode: allowlist, that provider — and every model under it — can never become unhealthy based on observed failures. Repeated429 Too Many Requests(quota exhausted, upstream rate limit, etc.) neither trigger configured failover rules nor trip any request-time health signal. The 429 is passed straight back to the client even when a working fallback model is configured and even when the failure is sustained for a long period.In short, allowlist mode makes the provider permanently "healthy" from GoModel's perspective, regardless of what is happening at request time.
Real-world scenario
Using the Kimi provider: Kimi's upstream
/modelsendpoint does not list embedding models, so the only way to expose them through GoModel is to declare them statically.Observed behavior: when Kimi starts returning sustained
429s (e.g. usage quota exhausted), every request tokimi-embedding-001keeps returning 429 to the client. The configured fallbackopenai/text-embedding-3-smallis never attempted, and the provider never flips unhealthy, so the situation never self-heals.Expected behavior: after the primary model returns 429, GoModel should attempt the configured failover chain; and a provider that consistently fails at request time should degrade in health independently of how its model inventory was discovered.
Root-cause analysis
Two cooperating gaps:
1. Static inventory is unconditionally marked healthy (and stays that way)
internal/providers/registry_init.go— in allowlist mode the upstreamListModelscall is skipped entirely and the inventory is synthesized from the configured list (registry_init.go:400-419,configured_models.go:46-74).registry_init.go:213-216then setsfetchedInventory.runtimeUpdates[*].lastModelFetchSuccessAt, which the status classifier reads as "healthy" (seeTestClassifyProviderStatus_HealthyForAllowlistInventoryininternal/admin/handler_providers_test.go:10-16).Health, in the current model, is derived from model-fetch success — not from request success. In allowlist mode the model fetch is a local no-op that always "succeeds", so the health signal is decoupled from reality by construction.
2. Request-time safety nets don't cover 429, and the failover chain can be silently emptied
internal/llmclient/client.go:302-307,:330-333):shouldTripCircuitBreakerexplicitly returnsfalsefor 429, andRecordFailureis skipped for 429 unless half-open. By design, repeated 429s never trip the breaker.internal/failover/resolver.go:328-349resolveSelector): every failover candidate is validated againstregistry.GetModel(model), and the registry is allowlist-filtered (internal/app/app.go:602wires the resolver with the already-filtered registry). Any failover target not present in the statically-listed inventory is silently dropped from the chain (manualSelectorsFor,resolver.go:154-172). With an empty chain,tryFailoverResponseshort-circuits atinternal/gateway/failover.go:69(len(failovers) == 0) and the primary 429 is returned untouched — even thoughShouldAttemptFailover(failover.go:262-307) correctly treats 429 as failover-eligible.Net effect: 429 says "fail over", but the chain is empty and nothing ever marks the primary unhealthy.
Reproduction
models:list andconfigured_provider_models_mode: allowlist.base_urlat a mock that always 429s —tests/e2e/failover_test.go:58-160already has the harness;TestRateLimitFullySaturatedAliasReturns429_E2Eis close but does not cover static-models mode).Proposed direction
Reframe health measurement around observed requests rather than inventory discovery:
failover.treat_429_as_failure: trueand/or ahealth.request_failure_thresholdfor 4xx-class errors) so users keep opt-out control.Acceptance criteria
tests/e2e/failover_test.go).docs/advanced/configuration.mdxalongsideconfigured_provider_models_mode.Environment
7c16a72(main)models:list