diff --git a/chatkit.mdx b/chatkit.mdx index fa4cdb4..4a630c2 100644 --- a/chatkit.mdx +++ b/chatkit.mdx @@ -617,6 +617,66 @@ export const POST = chatKitEndpoint({ ChatKit preserves the messages inside a session. [Memory](/memory) stores selected knowledge that should be available across sessions, channels, or agents. Use both when an agent needs conversational continuity and longer-lived organizational context. +## Configure realtime voice + +Voice settings belong to your Tilde agent. In agent registration, select a voice +profile and its models, voice, and maximum conversation duration. You can change +these settings in the agent editor later. + +- **Text agent with speech**: Tilde transcribes incoming audio, invokes your normal + `chatKitEndpoint` callback, and speaks its streamed text response. +- **Telnyx Conversation Relay**: Telnyx handles recognition and speech synthesis + for incoming phone calls. Tilde receives caller text, invokes your normal + callback, and streams its response text to Telnyx. This profile is phone-only. +- **OpenAI Realtime**: the realtime model generates spoken responses directly. + Tilde records the final transcripts without invoking your text endpoint again. + +Your text callback receives `context.audio` for transcribed speech turns and +`context.telnyx` for Telnyx calls. Provider facts are supplied in the signed +request. A subsequent typed message does not become a speech turn merely because +its session previously contained a call. + +Each call creates a normal ChatKit session. Browser media admission is one-time, +expires after five minutes, and remains subject to current session membership. +Call duration is limited by the configured maximum. Call recordings are not +retained by this initial realtime implementation. + +For a manual test, use the SDK repository's `examples/realtime-voice` example. It +registers three agents. Its microphone page supports the two OpenAI modes; a +dedicated relay endpoint handles phone calls. To receive calls, configure a +**Telnyx Voice** chat provider with an existing encrypted Telnyx Voice credential, +Voice API application, phone number, and default agent. This creates a real +ChatKit channel that owns the caller's participant route. + +Choose **self-managed webhook setup** to copy Tilde's returned webhook URL into +Telnyx yourself. Choose **managed webhook setup** to let Tilde update the existing +application's webhook URL using your credential. Managed setup does not provision +or fund a Telnyx account, purchase a number, or assign numbers to applications. +Use a dedicated test application for the manual example. The Tilde API must be publicly +reachable through HTTPS and WSS. Hookdeck webhook replay alone cannot carry live +bidirectional calls. + +For relay, select `telnyx_relay`, transcription model `deepgram/nova-3`, voice +`Telnyx.Ultra.Callie`, language `en-US`, and interruption enabled. The Telnyx +route uses its own credential; relay does not require an OpenAI speech key. +Your callback continues to use its own text model and tools. The SDK example +accepts `TELNYX_AGENT_MODE=telnyx_relay` and keeps the browser demos available. + +Partial relay transcripts do not start agent turns. When speech interrupts a +response, Tilde retains generated text and separately records the spoken prefix +reported by the carrier. The SDK annotates both text and UI history so the next +turn can distinguish generated words from that reported prefix. Caller ID does +not authorize access to a Tilde human's personal tools. + +The [Telnyx Conversation Relay guide](https://developers.telnyx.com/docs/voice/programmable-voice/conversation-relay) +describes the text WebSocket protocol used between Telnyx and Tilde. + +The initial browser implementation streams audio through the Rust API. Direct +browser-to-provider WebRTC and direct SIP routing are separate transport options +and are not implied by the OpenAI Realtime profile. Native Realtime uses its +configured instructions and does not inherit endpoint tools. Browser voice +identifies its caller but does not yet establish personal-tool federation. + ## Change resources through native tools Agents use native Tilde API/MCP operations under their existing permissions. diff --git a/llms/chatkit.md b/llms/chatkit.md index eee68f1..00ae0df 100644 --- a/llms/chatkit.md +++ b/llms/chatkit.md @@ -191,6 +191,63 @@ Use `tilde_list_signal_provider_instances` and `tilde_list_signal_rules` before In application code, handle typed GitHub, Slack, Sentry, and Firecrawl metadata as shown in the [human ChatKit guide](https://trytilde.ai/docs/chatkit). `onUnprocessed` runs once per unprocessed message; later conversions reuse its cached result. +## Agent-owned realtime audio + +Use the selected tenant host and explicit `team_id` for these REST operations: + +1. `GET /api/v1/chatkit/audio/profiles` returns supported profile defaults and + server-authored fields. Render these descriptors rather than generating + provider-specific setup instructions in frontend code. +2. Register an HTTP agent with optional `audio` configuration, or use + `PUT /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio` with `{ "audio": + }`. Set `audio` to null on the PUT route to disable voice. +3. Configuration fields are `mode` (`pipeline`, `realtime`, or `telnyx_relay`), `credential_id` + (optional), `stt_model`, `tts_model`, `realtime_model`, `voice`, `instructions`, + `language` (default `en-US`), `interruptible` (default true), and + `max_duration_seconds` (10–1800). The OpenAI Audio credential source is + `chatkit_openai_audio`; omitting it uses the server OpenAI key for OpenAI modes. + Relay uses `stt_model: "deepgram/nova-3"`, `voice: "Telnyx.Ultra.Callie"`, + and null `credential_id`; its phone route owns the Telnyx credential. +4. `POST /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio/sessions` creates a + normal browser session for OpenAI modes and returns `audio_session`, `websocket_path`, and a one-time + token. Connect with WebSocket subprotocols `chatkit-audio` and `token.`. + This endpoint rejects `telnyx_relay`; relay starts from an incoming call. + Send mono signed PCM16 little-endian audio at 24 kHz as base64 `audio` frames. +5. `PUT /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio/telnyx` accepts + `credential_id` (source `chatkit_telnyx_voice`), `public_key`, `phone_number`, + `connection_id`, and public HTTPS `media_base_url`. It returns `route` and + `webhook_url`; successful setup also returns the assigned + `route.channel_inbox_id`. Use the webhook URL in the dedicated Telnyx application. +6. The generic channel catalog entry is `chatkit.chat_channel.telnyx_voice`, + provider `chatkit.channel.telnyx_voice`. Use auth method + `chatkit.channel.telnyx_voice.auth.self_managed` for a returned URL or + `chatkit.channel.telnyx_voice.auth.managed` for Tilde to update the existing + Voice API application's webhook. Both use your existing encrypted + `chatkit_telnyx_voice` credential and existing number/application. Pass the + same five setup fields and the normal default agent selection. Managed setup + updates webhook configuration; it does not buy, assign, or fund numbers. + Calls use the resulting channel as their participant origin. + +Pipeline mode invokes the existing callback only when a user speech turn is +ready. Rust synthesizes the response. Relay invokes the same signed callback +with `context.audio.mode = "telnyx_relay"`; Telnyx transcribes and synthesizes, +while Tilde exchanges text frames with the carrier. Realtime mode owns spoken generation; +transcript observations must not trigger another model turn or external send. +`context.audio` and `context.telnyx` come from typed, server-authored speech +provenance rather than client message metadata. Both persisted text and UI +messages can carry `speech`. Interrupted generated speech retains its original +text with `interrupted: true`, optional `played_audio_ms`, and optional +`reported_spoken_text` supplied by the carrier. Preserve the reported prefix +separately; do not rewrite it as the complete generated response. SDK history +conversion adds the corresponding annotation before the original content. + +The manual browser/carrier example is `examples/realtime-voice` in `trytilde/dispatch` +(the `@trytilde/sdk` packages). It never buys phone numbers or changes existing carrier routing. Agent +settings and credential setup references are portable; live connections and +media tokens are not exported. Configure Telnyx number/application bindings +again in the destination installation. Native mode does not inherit endpoint +tools, and browser voice does not establish personal-tool federation. + ## Change resources through native tools Agents use native Tilde API/MCP operations under their existing permissions.