feat(gateway): rate limits, retries, breakers and fallback for Bud voice deployments

[dittops]

Summary

A Bud voice deployment's Rate limiting and Resilience settings (FRD-022, bud-runtime specs/022-gateway-rate-limit-resilience/) are now enforced by WaaV. It uses the same limiter as budgateway: BudEcosystem/resil, pinned by rev with the redis-0-27 adapter.

• Policy from voice_table: bud-auth parses rate_limits, max_concurrent, retry_config and fallback_models leniently (a bad policy block drops the policy, never the endpoint). The contract fixture is byte-identical to budapp's.
• Limits hold cluster-wide: each replica admits from credit reserved in Redis, with no Redis call per request. A 429 is JSON with Retry-After and X-RateLimit-* headers; max_concurrent rejections return concurrency_limit_exceeded.
• /v1/audio/speech and /v1/audio/transcriptions:
  • Retryable vendor failures are retried.
  • The request fails over along the deployment's fallback chain; each fallback has its own limits.
  • Two-tier circuit breakers: one per deployment, and one per vendor that opens only when ≥ 2 deployments fail. A vendor 429 carrying Retry-After opens the breaker for that long.
  • Caller errors (400/404/413/422) are neither retried nor counted.
• Response headers: x-bud-endpoint-id (the deployment that served), x-bud-fallback, x-bud-voice-substituted.
• Spans: bud.voice.served_endpoint_id, bud.voice.fallback_from, bud.voice.retry_count, bud.rate_limit.outcome. Cost is recorded at the served deployment's price.
• Built on the voice analytics work (feat(gateway): voice analytics spans, vendor call spans and per-call cost; Groq uploads stop waiting 45 s #8, fix(gateway,bud-auth): serve published voice deployments to customer keys; close the raw endpoint-id bypass #9):
  • Every retry and fallback hop runs in the turn's vendor scope.
  • Failures stay classified: SynthesisError gains Vendor{status, retry_after} and Saturated.
  • Admission refusals open no turn.
• Hardening:
  • Per-vendor TTS concurrency is a bounded wait (WAAV_TTS_MAX_CONCURRENT_PER_VENDOR, default 64; it was a hard-coded 4 behind an unbounded queue), returning 503 when saturated.
  • Connection slots release on drop, fixing the realtime and /ws per-IP slot leak. A per-IP cap of 0 means off.
  • /ready answers 503 while draining.
• New env vars: WAAV_RATE_LIMIT_LAST_MILE (sync | local), WAAV_RATE_LIMIT_ON_STORE_UNAVAILABLE (deny | allow), WAAV_TTS_MAX_CONCURRENT_PER_VENDOR.

Dependencies and merge order

1. docs: describe resil as a general-purpose library, not Bud's gateways resil#1 first, with a merge commit (this branch pins its head 9be62f18ac968cb98e4ccc621c42cd73e960f9ba).
2. This PR.
3. BudEcosystem/bud-runtime (budapp publishes the policy; budadmin shows the panels): the companion PR.

An older WaaV ignores the new voice_table fields, and this WaaV treats their absence as "no policy", so either side can deploy first. Only the budadmin panels would be inert until this image ships.

No secret, IAM or Dapr changes. deny.toml allows the resil git source.

Testing

cargo fmt --all --check
cargo test --release --lib --features dag-routing,turn-ensemble,noise-filter,openapi
cargo test --release --manifest-path ../bud-auth/Cargo.toml
cargo test --release --test voice_span_contract --features dag-routing,turn-ensemble,noise-filter,openapi
cargo clippy --release --all-targets --features dag-routing,turn-ensemble,noise-filter,openapi -- -D warnings
python3 gateway/scripts/deployment_policy_e2e.py   # two replicas + Valkey + mock vendors

• Library tests: 7013 pass. 4 core::onnx tests need libonnxruntime.so, which isn't in the builder image; this change doesn't touch them.
• bud-auth: 171 tests pass. Span contract: passes.
• clippy: nothing new in touched files. main already has 60 findings on rust 1.96 (mostly MutexGuard held across await in tests).
• Local e2e: 37/37 across two replicas, covering:
  • cluster-wide limits, live change, max_concurrent
  • retry on 503, no retry on 400, failover on 401
  • vendor 429 → 429 with Retry-After; breaker OpenFor
  • tenant isolation, the vendor-tier breaker, fallback limits
  • transcription retry and fallback, the realtime slot release

Live on pde-ditto (image dittops/waav:resil-1, 2026-09-27):

• Rate limits (STT and TTS): exact limit, 429 codes and headers, live change, the off switch, max_concurrent.
• Deepgram made unreachable from the WaaV pod: every request to the Deepgram deployments was served by the ElevenLabs scribe-v2 fallback (x-bud-fallback: true), and the deployment breaker opened after 5 failures.
• ClickHouse: VoiceTurnFact rows carry the served deployment, fallback source and rate-limit outcome.

🤖 Generated with Claude Code

[dittops]

Companion PR: BudEcosystem/bud-runtime#3058 (budapp publishes the policy; budadmin shows the panels). Depends on BudEcosystem/resil#1.