[fylorn]

Summary

An upstream that rate-limits (429) no longer turns into errors for every client request. The gateway pauses that upstream, queues requests for it, and resends them once it recovers; streaming clients keep their early 200 and keep-alives while they wait, so a well-behaved agent only sees a slower first token. Upstreams can also carry their own usage limits (providers[].limits), so the gateway can stay under a relay's "N tokens per M minutes" rule before it ever answers 429.

Behaviour

• One wait deadline per request: failover.wait_secs (default 120 s, 0–600) replaces slot_wait_secs. Key windows, concurrency slots, rate-limit pauses and upstream usage windows all spend from it.
• Upstream 429 (not "quota used up"): the whole upstream pauses until retry-after-ms / Retry-After / the body's delay, else backoff 5, 10, 20, 40, then 60 s (reset on success). Hints under 1 s (under 1 ms for retry-after-ms) count as missing; absurd values are capped by rate_limit_max_pause_secs. A 429 is no longer a failure: no breaker, no success-rate sample.
• Queueing: a request first tries any candidate it can send to now; otherwise it waits on the paused / full upstreams (one FIFO queue per upstream) and sends to whichever recovers first. When a pause ends only the queue head is sent, as a probe; the rest follow once it passes (non-429 answer, judged at first content for held streams) or after a 10 s grace. Nothing is admitted after the request's deadline.
• Single candidate: waited for, not bypassed (the bypass stays for the breaker only).
• Sticky conversations wait for their own upstream when it can recover in time, and let go otherwise; skipped load-balance members are not charged.
• Upstream usage limits use the key-limit format plus every ({ per: minute, every: 5, tokens: 10000000 }); every works for key limits too. Minute/hour windows full → wait or switch; a used-up day/week/month → skip, never wait. Requests count per hop sent and not answered 429; tokens and cost settle from the serving hop; calendar periods are rebuilt from request records on restart.
• Final 429 (deadline reached without sending): Retry-After / retry-after-ms for the earliest recovery and x-should-retry: true; when every candidate has used up its period, Retry-After is the earliest reset and x-should-retry: false.
• Count-tokens requests never queue; they are estimated locally while the upstream is unavailable. WebSocket turns follow the same rules.

API (protocol 51)

• ThrottleKind, Throttle; ProviderView.throttle, ProviderView.limits, ProviderInput.limits
• Event::ThrottleChanged, Event::RequestQueued, Event::RequestDequeued
• ServeSkip: throttled, window_full, period_used_up; AttemptView.pause_ms
• KeyLimitView / KeyLimitInput.every; FailoverView.wait_secs (was slot_wait_secs)
• New message codes: config.key_limit_every*, config.upstream_limit_*, gw.key_limit.*_every, gw.upstream.{paused,window_full,period_used_up}, gw.wait.{rate_limited,window_full,used_up,mixed}

Upgrade notes

• failover.slot_wait_secs is gone with no alias: a config that sets it starts in safe mode, and the repair removes the key (its value is not carried over).
• The default wait goes from 30 s to 120 s.

Testing

cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass locally (3932 tests), including new end-to-end tests for the queue, probe, backoff, deadline headers, usage windows, renames, sleep-safe pauses and WebSocket turns.

🤖 Generated with Claude Code