feat(gateway): realtime voice sessions on Bud deployments — /v1/realtime relay + translate engine, /ws legs (FRD-023) - #12
Merged
Conversation
…ve-session reach checks
FRD-023 WP-RT2.3 and Q-7.
- voice_table config.realtime parses into RealtimeSettings {session_type, defaults, limits,
policy}, leniently: a malformed block or field is dropped with a warning, never the endpoint.
RealtimePolicy's accessors apply the secure defaults (no MCP tools, no stored prompts).
- VoicePricing.rates: a realtime token price is its per-modality rates; a token price without
rates is refused (served unpriced) rather than priced at zero. Keys are the closed
REALTIME_RATE_KEYS set.
- Principal.expires_at carries a JWT caller's verified exp (client-secret lifetime cap).
- BudPlane::hash_reaches / subject_reaches / subject_aliases: the revalidation a live realtime
session and an ek_bud_ parent make, by snapshot hash or JWT subject.
- user_projects:{sub} events evict that subject's cached grants and project_models:* events
clear them all, so JWT revocation reaches live sessions without waiting OIDC_AUTHZ_TTL_SECS.
- tests/fixtures/realtime_voice_entry.json is byte-identical to budapp's publisher fixture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3 RT0)
Under the Bud control plane every vendor credential comes from the deployment (voice_table).
Four socket paths still reached for the PROCESS's keys, latent only because the Bud chart sets
none; the first operator to add one to "make realtime work" would have exposed it:
- X-1 /ws conversation_config: a client-chosen base_url with api_key omitted sent the
platform's OPENAI_API_KEY to that host. base_url, api_key, reasoning_base_url and
reasoning_api_key are now refused in Bud mode, and LlmClient never falls back to the
environment (neither ${VAR} nor the default env var) while the process is in Bud mode.
- X-2 /ws DAG: an inline dag_config.definition is refused, and a DAG node's own credential
(literal or ${VAR}) is refused in Bud mode.
- X-3 native /realtime: a config message is refused with deployment_required; Bud deployments
are served by /v1/realtime?model=<deployment>.
- X-4 /ws STT/TTS legs: a provider-only leg no longer falls back to config.get_api_key.
- F-2: a second native /realtime config is refused instead of replacing the provider without
disconnect().
process_in_bud_mode() is set once when BudMode starts, so a code path with no AppState in reach
cannot forget the rule. Two existing tests asserted the X-4 fallback; they now assert its refusal
(TC-SEC-06). Every guard (TC-SEC-01..06, 08) was seen failing with its check removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
try_acquire_connection held its DashMap entry (the shard's WRITE lock) while the "Connection acquired" debug! evaluated ip_connection_count(), which takes the same shard's READ lock. The field is evaluated only when DEBUG is enabled, so any deployment run with debug logging froze the worker thread of the first WebSocket upgrade forever (FRD-022 connection slots). Found by FRD-023's relay tests, whose span capture enables DEBUG. The entry is dropped before logging, and the count comes from the fetch_add result. Regression test acquires and releases a slot under a DEBUG subscriber with a 10 s deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me GA (FRD-023 RT2, RT3, RT5) wss://gateway/v1/realtime?model=<deployment> now speaks OpenAI Realtime GA, so the OpenAI SDKs, the Agents SDKs, LiveKit and Pipecat connect by changing only the base URL and the key. WaaV's native protocol stays on /realtime. - Handshake (RT2.1): credentials from Authorization: Bearer, the api-key header, or the openai-insecure-api-key.<cred> subprotocol (the server selects `realtime`, never echoes the credential); ?token=, OpenAI-Beta and call_id refused; model required; no permessage-deflate. Pre-upgrade failures are HTTP in OpenAI's error envelope. - Resolution and admission (RT2.2): the deployment resolves through the caller's allowlist as on REST; FRD-022 admission is taken once and its concurrency slot held for the session; an open breaker refuses before connecting. - Upstream (RT2.4): URL, auth header and model only from voice_table — OpenAI wss://api.openai.com/v1/realtime (or api_base, SSRF-validated), Azure GA wss://<resource>/openai/v1/realtime with api-key and no api-version; transcription deployments connect with intent=transcription. Connect deadline 10 s. - Relay and policy (RT2.5): two-stage parse, audio frames forwarded byte-for-byte; session.model and tracing stripped; MCP tools, stored prompts, image input, instruction overrides, session type changes and off-list transcription models refused per the deployment's policy with event_not_allowed (session continues); max_output_tokens clamped; vendor rate_limits.updated dropped; session.model rewritten to the deployment name. The deployment's defaults are sent first and client frames held until the vendor applies them (5 s). A client that stops reading is closed 1011 client_too_slow after 5 s — audio is never dropped silently. - Lifecycle (RT2.6): pings both legs every 20 s (3 missed → 1011), idle and maximum-length limits with a 60 s warning, revalidation every 30 s and before each response.create (key revoked, JWT subject removed from the project, endpoint unpublished → session_revoked + 1008), drain → server_shutdown + 1012. JWT expiry alone does not end a started session. - Metering (RT3.1): a voice.turn per response.done, per input transcription and per 60 s duration segment under a minute/second price — each the ROOT of its own trace with a link to the session's voice.session span (VoiceTurnFact coalesces on TraceId). Per-modality token cost with cached tokens subtracted from their class; a component without a rate is named in bud.voice.unpriced_components, never priced at zero. New attributes are declared in voice_turn_span!/voice_session_span! and match budmetrics' contract byte for byte. - Client secrets (RT5.1): POST /v1/realtime/client_secrets mints a stateless ek_bud_ secret sealed with XChaCha20-Poly1305 (claims opaque to the holder), lifetime capped by a JWT parent's exp, parent = the full snapshot hash; connect revalidates the parent. Keys from WAAV_CLIENT_SECRET_KEYS (first seals, all open); a bad key fails startup; none → 501. - Metrics (RT2.7): waav_realtime_sessions_active/_total, relay latency, policy refusals, unknown client events. Tests: lib units per module and tests/openai_realtime_relay.rs (39 cases against a mock GA vendor: TC-HS, TC-UP, TC-EVT, TC-LIFE, TC-MET, TC-EK, TC-SEC-07). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two let-else blocks become ? and to_vendor becomes send_to_vendor (a to_* method taking &mut self trips wrong_self_convention under -D warnings). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under the Bud control plane a /ws session's legs are deployments, resolved through the caller's own allowlist, exactly as the REST routes resolve them. - STT/TTS legs (WP-RT6.1): stt_config.model / tts_config.model name a transcription / text-to-speech deployment; vendor, model, credential, api_base, voice and settings come from voice_table (request > deployment). provider is optional and ignored. Refused by name: provider-only legs (deployment_required), unreachable names (model_not_found), upload-only STT (unsupported_deployment), deployments that cannot reach their vendor (deployment_misconfigured, e.g. AWS without its key pair), and a client's own api_key (client_key_not_accepted). Client extras are replaced by the deployment's: Groq and Azure Speech read a destination from them. - One admission per leg deployment, held for the session; a capped leg closes 1013 and releases the other leg (MessageRoute::CloseWith). - Voice agent LLM leg (WP-RT6.2): conversation_config.model names a Bud chat deployment reached through WAAV_LLM_BASE_URL with the caller's credential, read per call; an auth message on an authenticated session refreshes it (same API key or user only); an expired credential fails the turn with auth_expired and the session stays open. Conversation turns are attributed. - DAG templates (WP-RT6.3): TTS nodes bind to deployments (admitted, metered, revalidated); LLM/translate nodes go to the Bud gateway as the caller; realtime nodes are refused until RT7. - stt.streaming (WP-RT6.4) is modelled in bud-auth and applied on /ws only. - Metering and revocation (WP-RT6.5): a voice.turn per vendor final (audio seconds since the previous one, the remainder at close) and per speak (characters), fully attributed; revalidation every 30 s closes 1008. Tests: TC-WS-01..13 (lib + conversation_loop); every guard seen red with its check removed. Live: 21/21 on pde-ditto against Deepgram, ElevenLabs and a chat deployment through budgateway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rrent defaults (FRD-023 F-1, F-3, F-4) F-1: OpenAIRealtimeModel was a closed enum whose parser turned every id it did not list into `gpt-realtime`, so gpt-realtime-1.5, -2.1 and -2.1-mini silently became a model that shuts down on 2027-01-20. It is now a string carried verbatim; only an empty id takes the default (TC-XL-08). F-3: the native `/realtime` session update dropped tools and turn detection, and the scaffold's update_session replaced the whole config, so the key and the server-set endpoint override were lost and the next reconnect dialled without a key. The update now carries both and merges. F-4: Gemini Live defaulted to gemini-2.0-flash-live-001 (shut down 2025-12-09) and Nova Sonic to v1 (EOL 2026-09-14): now gemini-3.8-live and amazon.nova-2-sonic-v1:0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…7.3) `grok` joins the relay vendors: upstream wss://api.x.ai/v1/realtime?model= with the deployment's key as Bearer; an api_base is SSRF-validated like OpenAI's. xAI opens its session with `conversation.created` and never sends `session.created`, so that event now starts the session (the deployment defaults are applied once) and the client is given a GA `session.created`. Cumulative `…input_audio_transcription.updated` events are forwarded verbatim, and nothing waits on `rate_limits.updated` (TC-XL-06). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…velope The per-IP and global connection limits answered a plain-text 429/503 with no Retry-After. On /v1/realtime the client is an OpenAI SDK, which reads error.code and backs off on the header, so those refusals now use the relay's own envelope (rate_limit_exceeded / server_at_capacity, Retry-After: 1); native routes keep their text body and gain the header. Found by the TC-COMPAT run under concurrent load (FRD-023 §5.2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on plan, sign Nova with deployment keys (FRD-023 RT7.0-RT7.2) The S2S scaffold gains what the translate engine needs: - S2sEvent::Usage (per-modality tokens or seconds; cumulative or delta), ItemAdded, ItemDone and GoAway, and a raw event tap (BaseRealtime::on_event) that sees every normalized event in wire order. - Planned reconnects: a vendor goAway, or a connection cap (Nova Sonic's 8 minutes, replaced 30 s early), replaces the connection at the next turn boundary with no backoff and without counting as a failure; the resumption handle is carried. The reconnect is announced only once the session is ready again. - Per-protocol input sample rates (Gemini, Nova, ElevenLabs 16 kHz; Hume its configured rate). - Gemini: usageMetadata becomes one cumulative Usage, first in its message's events; goAway carries its deadline. - Nova 2 Sonic: usageEvent becomes one delta Usage (a totals-only report is billed as the difference, per connection); the assistant audio block is an item; history is replayed as text blocks on a new stream. - The Bedrock factory can sign with a deployment's static key pair and nothing else (no environment, shared config or instance identity), and in Bud mode refuses to dial without one. The stream now opens in the background: the SDK's send() returns only at the first output event, which Nova sends only after input, so awaiting it in connect deadlocked until the dial timeout. - bud-auth keeps provider_params.region for vendor nova_sonic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ova 2 Sonic and per-minute agents (FRD-023 RT7.0, RT7.1, RT7.2, RT7.4) A deployment whose vendor has no OpenAI Realtime GA surface (CONTRACTS C7: gemini, nova_sonic, deepgram_voice_agent, elevenlabs_convai, hume_evi) is served by WaaV's native provider behind a GA facade (handlers/openai_realtime/facade.rs). Everything around the session stays the relay's: authentication and ek_bud_ secrets, resolution, the admission held for the session, revalidation, idle and maximum-length limits, pings, the 1012 drain, the session span and the client-event policy. - The vendor leg comes from voice_table only: the credential (Nova: the key pair in credential_parts and provider_params.region, refused without either), the model, and an api_base that is converted to ws(s) and SSRF-validated. A provider that cannot be built from the entry is refused before the upgrade (502 deployment_misconfigured). The vendors serve speech-to-speech sessions only. - The vendor setup is deferred to the client's first session.update, so the one opening message these vendors accept carries the client's voice, instructions, tools and turn detection over the deployment's defaults; afterwards a change to any of them is refused with event_not_allowed naming the field (the same value again is accepted). - GA in: session.update, input_audio_buffer.append (24 kHz resampled to the vendor's rate)/commit/clear, conversation.item.create (user text, function_call_output), response.create/cancel. Refused by name: conversation.item.truncate/retrieve/delete, output_audio_buffer.clear, audio and image content, MCP tools, stored prompts, out-of-band responses, unknown events. GA out: response.created ... audio and transcript deltas (resampled to 24 kHz; WAV chunks unwrapped) ... function calls ... response.done with a usage the gateway computed. - Metering: a vendor usage report is the usage of the response it belongs to and exactly one voice.turn; a report outside a response is billed alone; a response cut off by close keeps its usage. Per-minute vendors bill duration segments from the moment the vendor connection opens, through the relay's segment clock (now a shared SegmentClock). - A vendor that reconnects (goAway, the Nova cap) is invisible to the client: vendor-bound work waits out the gap. Tests (in-process mocks of each wire protocol, including a Bedrock event-stream connector): TC-XL-01...05 and 07, plus xAI priced per minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the /v1/realtime connection-limit envelope fix (548618c) onto RT7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-01) WaaV never set TCP_NODELAY on the sockets it accepts. A realtime client streams 20 ms frames and reads small events back; with Nagle on, each relayed write waited for the client's next frame to acknowledge the previous one, adding a frame interval to every frame: TC-PERF-01 measured p50 19.5 ms / p99 20.4 ms added against a 5 ms budget. The vendor sockets already set it. server::nodelay_listener (axum::serve) and server::tls_server (axum-server's NoDelayAcceptor under rustls) now serve every route; main.rs uses both. Tests: server::accepted_sockets_set_tcp_nodelay; the TC-PERF-01 harness (tests/openai_realtime_perf.rs, #[ignore]) now serves through the production listener: p50 0.37 ms / p99 0.70 ms added (debug build). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration price A Deepgram Voice Agent deployment published with no price held a 72 s session on pde-ditto and left no usage record: the facade only segmented a per-minute vendor when the deployment carried a minute/second price. The vendor bills by time whatever Bud charges, so per-minute vendors always meter duration segments; a segment with no minute/second price is recorded with its billed seconds, no cost, and unpriced_components = "duration" (never $0). Tests: realtime_cost an_unpriced_duration_segment_names_what_is_unpriced; realtime_translate tc_xl_07_an_unpriced_per_minute_vendor_still_meters_its_time (red with the old condition restored). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings TCP_NODELAY on accepted client sockets (6cd3850) onto RT7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ver VAD
Every translate vendor detects turns itself unless told otherwise, but the
facade echoed an unset turn_detection as null, which a GA client reads as
push-to-talk: the playground showed a manual Send beside a Deepgram Voice
Agent session that answers on its own. Unset now reports {type: server_vad};
an explicit null still reports off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te finals Deepgram's Voice Agent (ConversationText) and Hume's EVI (assistant_message) send an answer as one finished sentence per message, and Nova Sonic as separate blocks, with no whitespace between them. The facade forwarded each verbatim as a response.output_audio_transcript.delta, so a GA client that appends deltas showed "…culture.Iconic landmarks…", and the transcript in response.output_audio_transcript.done read the same (seen live on a Deepgram Voice Agent deployment). A new segment after earlier text in the same response now gets one space, unless either side is already whitespace or the script does not space sentences (Chinese, Japanese). A streamed chunk continuing a segment stays verbatim, so Gemini's word chunks are unchanged. Also gives the ignored perf test the bedrock_http_client field the RT7 merge added, so --all-targets compiles again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bud stores turn detection off as {"type":"none"}. The relay forwarded it as-is
in the defaults session.update, and OpenAI GA refuses the type ("Invalid
value: 'none'. Supported values are: 'server_vad' and 'semantic_vad'", seen
live on gpt-realtime-2.1-mini) -- which drops the whole update, so a
deployment set to "None" silently lost its voice, instructions and every other
default and ran on server VAD. GA turns detection off with null; send that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…out, openapi regenerated CI on #12: - bud_legs::bind_dag used crate::dag unconditionally, so every build without the dag-routing feature (the default matrix entry, server boot, VAD accuracy, openapi drift) failed to compile. It and its DAG-template tests are gated on dag-routing, as its caller already was. - The /ws leg tests read bud-auth's git-ignored fixture key, which a clean checkout does not have (26 failures). test_support now generates a key pair once per test binary and encrypts the test credential to it the way budapp does (RSA-OAEP-SHA-256, hex). - docs/openapi.yaml regenerated for RT6's /ws config (provider and base_url optional under the Bud control plane). Default features: 6865 lib tests pass; production features: all but the four ONNX tests that need the runtime library (green in CI); openapi drift passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
FRD-023 (realtime voice sessions) on the WaaV side. Spec, contracts and live-test record:
specs/023-realtime-voice-sessions/in bud-runtime.voice_table); a process*_API_KEYis never used for a Bud session./v1/realtime— OpenAI Realtime GA on Bud deployments (RT2/RT3/RT5).session.update.voice.turnandvoice.sessionspans.ek_bud_client secrets are sealed with XChaCha20-Poly1305./wson Bud deployments (RT6): the STT, TTS and LLM legs address Bud deployments with the caller's credential (the LLM leg goes through budgateway), and DAG templates bind to deployments.null. OpenAI rejects{"type":"none"}, which dropped the whole defaults update.Companion PRs: BudEcosystem/bud-runtime#3078 (budapp, budmetrics, frontends, chart, spec), BudEcosystem/bud-connect#42 (catalog).
Linked issues
None filed; tracked by FRD-023.
Migration steps
None in WaaV. Deploy order: bud-runtime's budapp release N (enum readers) → this WaaV → budapp N+1 (writers) → bud-connect's catalog PR.
Testing
Live on pde-ditto (image
waav:rt023-9):/ws24/24;Against a real OpenAI
gpt-realtime-2.1-minideployment, every deployment setting was verified (26/26): OpenAI echoes each value and the behaviour follows. Policy and limits were verified 11/11.Known pre-existing failures, unrelated to this PR:
test_openai_in_supported_realtime_providersstill expectsspeechmatics;Callouts
WAAV_CLIENT_SECRET_KEYS(kid:base64(32 bytes)pairs, first seals). It comes from the chart's SOPS-managed Secret; unset meansPOST /v1/realtime/client_secretsanswers 501.voice/turn_detectionat the top ofsessionand call the languagelanguage_hint, while the relay sends GA placement.🤖 Generated with Claude Code