From 8ac62c79dbd3f43dd3a2d0f25d0a3ed18573e444 Mon Sep 17 00:00:00 2001 From: Daniel Blignaut Date: Mon, 7 Sep 2026 21:24:17 +0200 Subject: [PATCH 1/3] docs(chatkit): explain agent voice and Telnyx setup --- chatkit.mdx | 36 ++++++++++++++++++++++++ docs/updates/pending/realtime-voice.md | 38 ++++++++++++++++++++++++++ llms/chatkit.md | 37 +++++++++++++++++++++++++ 3 files changed, 111 insertions(+) create mode 100644 docs/updates/pending/realtime-voice.md diff --git a/chatkit.mdx b/chatkit.mdx index b43a6ca..4490aa0 100644 --- a/chatkit.mdx +++ b/chatkit.mdx @@ -613,3 +613,39 @@ export const POST = chatKitEndpoint({ ## ChatKit and memory ChatKit preserves the messages inside a session. [Memory](/memory) stores selected knowledge that should be available across sessions, channels, or agents. Use both when an agent needs conversational continuity and longer-lived organizational context. + + +## Configure realtime voice + +Voice settings belong to your Tilde agent. In agent registration, select a voice +profile and its models, voice, and maximum conversation duration. You can change +these settings in the agent editor later. + +- **Text agent with speech**: Tilde transcribes incoming audio, invokes your normal + `chatKitEndpoint` callback, and speaks its streamed text response. +- **OpenAI Realtime**: the realtime model generates spoken responses directly. + Tilde records the final transcripts without invoking your text endpoint again. + +Your text callback receives `context.audio` for transcribed speech turns and +`context.telnyx` for Telnyx calls. Provider facts are supplied in the signed +request. A subsequent typed message does not become a speech turn merely because +its session previously contained a call. + +Each call creates a normal ChatKit session. Browser media admission is one-time, +expires after five minutes, and remains subject to current session membership. +Call duration is limited by the configured maximum. Call recordings are not +retained by this initial realtime implementation. + +For a manual test, use the SDK repository's `examples/realtime-voice` example. It +registers one agent for each mode and supplies a microphone page. To receive +phone calls, create a **Telnyx Voice** managed credential, bind a dedicated test +Voice API application and number to the agent, and configure the application's +webhook URL with the URL returned by Tilde. The Tilde API must be publicly +reachable through HTTPS and WSS. Hookdeck webhook replay alone cannot carry live +bidirectional audio. + +The initial browser implementation streams audio through the Rust API. Direct +browser-to-provider WebRTC and direct SIP routing are separate transport options +and are not implied by the OpenAI Realtime profile. Native Realtime uses its +configured instructions and does not inherit endpoint tools. Browser voice +identifies its caller but does not yet establish personal-tool federation. diff --git a/docs/updates/pending/realtime-voice.md b/docs/updates/pending/realtime-voice.md new file mode 100644 index 0000000..7bb4f25 --- /dev/null +++ b/docs/updates/pending/realtime-voice.md @@ -0,0 +1,38 @@ +# Realtime voice documentation + +## Intent of the change + +Explain agent-owned speech configuration, the existing per-turn endpoint callback, +and the manual browser/Telnyx voice workflow for the matching API and SDK release. + +## Architecture changes + +ADR review: no new decision in this repository. This guide documents the API's +ADR 0023: speech orchestration belongs to Tilde and calls use normal ChatKit sessions. + +```mermaid +flowchart LR + Caller[Browser or Telnyx caller] <--> Rust[Tilde audio runtime] + Rust <--> Speech[OpenAI speech services] + Rust --> Endpoint[Normal text endpoint in pipeline mode] + Rust --> History[ChatKit transcript history] +``` + +## Summarized changes + +- Human guide: agent voice settings, pipeline/native modes, callback context, + browser admission, Telnyx setup, and explicit transport/tool limitations. +- Agent guide: exact REST paths, configuration fields, media format, managed + credential sources, and portable versus installation-specific configuration. +- Validation: Mintlify build, broken links, and accessibility checks pass. +- Live OpenAI inference was tested in the API worktree. A live Telnyx call and + browser microphone interaction remain manual checks; no deployment is claimed. + +## Critical to apply + +yes + +Publish after the matching API deployment and SDK release. Operators need public +HTTPS/WSS, an OpenAI server key or managed credential, and a dedicated Telnyx +application/number binding. Native endpoint tools, browser personal-tool federation, +direct WebRTC/SIP, ElevenLabs, voice notes, and retained audio are outside this slice. diff --git a/llms/chatkit.md b/llms/chatkit.md index e4f36f3..8cae5e0 100644 --- a/llms/chatkit.md +++ b/llms/chatkit.md @@ -189,3 +189,40 @@ Provider/rule visibility controls discovery and delivery reads. Ownership contro Use `tilde_list_signal_provider_instances` and `tilde_list_signal_rules` before updating or deleting resources. Their mutation functions are `tilde_update_signal_provider`, `tilde_delete_signal_provider`, `tilde_update_signal_rule`, and `tilde_delete_signal_rule`. In application code, handle typed GitHub, Slack, Sentry, and Firecrawl metadata as shown in the [human ChatKit guide](https://trytilde.ai/docs/chatkit). `onUnprocessed` runs once per unprocessed message; later conversions reuse its cached result. + + +## Agent-owned realtime audio + +Use the selected tenant host and explicit `team_id` for these REST operations: + +1. `GET /api/v1/chatkit/audio/profiles` returns supported profile defaults and + server-authored fields. Render these descriptors rather than generating + provider-specific setup instructions in frontend code. +2. Register an HTTP agent with optional `audio` configuration, or use + `PUT /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio` with `{ "audio": + }`. Set `audio` to null on the PUT route to disable voice. +3. Configuration fields are `mode` (`pipeline` or `realtime`), `credential_id` + (optional), `stt_model`, `tts_model`, `realtime_model`, `voice`, `instructions`, + and `max_duration_seconds` (10–1800). The OpenAI Audio credential source is + `chatkit_openai_audio`; omitting the credential uses the server OpenAI key. +4. `POST /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio/sessions` creates a + normal session and returns `audio_session`, `websocket_path`, and a one-time + token. Connect with WebSocket subprotocols `chatkit-audio` and `token.`. + Send mono signed PCM16 little-endian audio at 24 kHz as base64 `audio` frames. +5. `PUT /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio/telnyx` accepts + `credential_id` (source `chatkit_telnyx_voice`), `public_key`, `phone_number`, + `connection_id`, and public HTTPS `media_base_url`. It returns `route` and + `webhook_url`; use that exact webhook URL in the dedicated Telnyx application. + +Pipeline mode invokes the existing callback only when a user speech turn is +ready. Rust synthesizes the response. Realtime mode owns spoken generation; +transcript observations must not trigger another model turn or external send. +`context.audio` and `context.telnyx` come from typed, server-authored speech +provenance rather than client message metadata. + +The manual browser/carrier example is `examples/realtime-voice` in the Harness +SDK. It never buys phone numbers or changes existing carrier routing. Agent +settings and credential setup references are portable; live connections and +media tokens are not exported. Configure Telnyx number/application bindings +again in the destination installation. Native mode does not inherit endpoint +tools, and browser voice does not establish personal-tool federation. From aa66565e4fef033a4bbefba3b7ede1998e37cf41 Mon Sep 17 00:00:00 2001 From: Daniel Blignaut Date: Mon, 7 Sep 2026 21:40:30 +0200 Subject: [PATCH 2/3] docs(chatkit): link voice PRs and record release prerequisites --- docs/updates/{pending/realtime-voice.md => 37.md} | 9 ++++++++- llms/chatkit.md | 4 ++-- 2 files changed, 10 insertions(+), 3 deletions(-) rename docs/updates/{pending/realtime-voice.md => 37.md} (82%) diff --git a/docs/updates/pending/realtime-voice.md b/docs/updates/37.md similarity index 82% rename from docs/updates/pending/realtime-voice.md rename to docs/updates/37.md index 7bb4f25..07a479d 100644 --- a/docs/updates/pending/realtime-voice.md +++ b/docs/updates/37.md @@ -1,4 +1,6 @@ -# Realtime voice documentation +# PR 37: Realtime voice documentation + +https://github.com/trytilde/docs/pull/37 ## Intent of the change @@ -36,3 +38,8 @@ Publish after the matching API deployment and SDK release. Operators need public HTTPS/WSS, an OpenAI server key or managed credential, and a dedicated Telnyx application/number binding. Native endpoint tools, browser personal-tool federation, direct WebRTC/SIP, ElevenLabs, voice notes, and retained audio are outside this slice. + +Related API: https://github.com/trytilde/api/pull/277. +Related SDK: https://github.com/trytilde/dispatch/pull/155. +The API needs current-main conflict resolution; the SDK needs a complete contract +refresh from that integrated API head before release. diff --git a/llms/chatkit.md b/llms/chatkit.md index 8cae5e0..d77da35 100644 --- a/llms/chatkit.md +++ b/llms/chatkit.md @@ -220,8 +220,8 @@ transcript observations must not trigger another model turn or external send. `context.audio` and `context.telnyx` come from typed, server-authored speech provenance rather than client message metadata. -The manual browser/carrier example is `examples/realtime-voice` in the Harness -SDK. It never buys phone numbers or changes existing carrier routing. Agent +The manual browser/carrier example is `examples/realtime-voice` in `trytilde/dispatch` +(the `@trytilde/sdk` packages). It never buys phone numbers or changes existing carrier routing. Agent settings and credential setup references are portable; live connections and media tokens are not exported. Configure Telnyx number/application bindings again in the destination installation. Native mode does not inherit endpoint From 40a1886d94784d50ecc1d2f7ea75226f5f92cb10 Mon Sep 17 00:00:00 2001 From: Daniel Blignaut Date: Mon, 7 Sep 2026 22:54:49 +0200 Subject: [PATCH 3/3] docs(chatkit): explain Telnyx Conversation Relay setup --- chatkit.mdx | 35 ++++++++++++++++++++++++++++++----- docs/updates/37.md | 26 +++++++++++++++++--------- llms/chatkit.md | 35 ++++++++++++++++++++++++++++------- 3 files changed, 75 insertions(+), 21 deletions(-) diff --git a/chatkit.mdx b/chatkit.mdx index 4490aa0..3999a86 100644 --- a/chatkit.mdx +++ b/chatkit.mdx @@ -623,6 +623,9 @@ these settings in the agent editor later. - **Text agent with speech**: Tilde transcribes incoming audio, invokes your normal `chatKitEndpoint` callback, and speaks its streamed text response. +- **Telnyx Conversation Relay**: Telnyx handles recognition and speech synthesis + for incoming phone calls. Tilde receives caller text, invokes your normal + callback, and streams its response text to Telnyx. This profile is phone-only. - **OpenAI Realtime**: the realtime model generates spoken responses directly. Tilde records the final transcripts without invoking your text endpoint again. @@ -637,12 +640,34 @@ Call duration is limited by the configured maximum. Call recordings are not retained by this initial realtime implementation. For a manual test, use the SDK repository's `examples/realtime-voice` example. It -registers one agent for each mode and supplies a microphone page. To receive -phone calls, create a **Telnyx Voice** managed credential, bind a dedicated test -Voice API application and number to the agent, and configure the application's -webhook URL with the URL returned by Tilde. The Tilde API must be publicly +registers three agents. Its microphone page supports the two OpenAI modes; a +dedicated relay endpoint handles phone calls. To receive calls, configure a +**Telnyx Voice** chat provider with an existing encrypted Telnyx Voice credential, +Voice API application, phone number, and default agent. This creates a real +ChatKit channel that owns the caller's participant route. + +Choose **self-managed webhook setup** to copy Tilde's returned webhook URL into +Telnyx yourself. Choose **managed webhook setup** to let Tilde update the existing +application's webhook URL using your credential. Managed setup does not provision +or fund a Telnyx account, purchase a number, or assign numbers to applications. +Use a dedicated test application for the manual example. The Tilde API must be publicly reachable through HTTPS and WSS. Hookdeck webhook replay alone cannot carry live -bidirectional audio. +bidirectional calls. + +For relay, select `telnyx_relay`, transcription model `deepgram/nova-3`, voice +`Telnyx.Ultra.Callie`, language `en-US`, and interruption enabled. The Telnyx +route uses its own credential; relay does not require an OpenAI speech key. +Your callback continues to use its own text model and tools. The SDK example +accepts `TELNYX_AGENT_MODE=telnyx_relay` and keeps the browser demos available. + +Partial relay transcripts do not start agent turns. When speech interrupts a +response, Tilde retains generated text and separately records the spoken prefix +reported by the carrier. The SDK annotates both text and UI history so the next +turn can distinguish generated words from that reported prefix. Caller ID does +not authorize access to a Tilde human's personal tools. + +The [Telnyx Conversation Relay guide](https://developers.telnyx.com/docs/voice/programmable-voice/conversation-relay) +describes the text WebSocket protocol used between Telnyx and Tilde. The initial browser implementation streams audio through the Rust API. Direct browser-to-provider WebRTC and direct SIP routing are separate transport options diff --git a/docs/updates/37.md b/docs/updates/37.md index 07a479d..53191a1 100644 --- a/docs/updates/37.md +++ b/docs/updates/37.md @@ -5,16 +5,19 @@ https://github.com/trytilde/docs/pull/37 ## Intent of the change Explain agent-owned speech configuration, the existing per-turn endpoint callback, -and the manual browser/Telnyx voice workflow for the matching API and SDK release. +and the manual browser/Telnyx media and Conversation Relay workflows for the matching API and SDK release. ## Architecture changes ADR review: no new decision in this repository. This guide documents the API's -ADR 0023: speech orchestration belongs to Tilde and calls use normal ChatKit sessions. +ADR 0025: speech orchestration belongs to Tilde and calls use normal ChatKit sessions. ```mermaid flowchart LR - Caller[Browser or Telnyx caller] <--> Rust[Tilde audio runtime] + Caller[Browser or Telnyx media caller] <--> Rust[Tilde audio runtime] + Phone[Phone caller] <--> Relay[Telnyx speech recognition and synthesis] + Relay <-->|Text frames| Adapter[Tilde relay adapter] + Adapter --> Endpoint Rust <--> Speech[OpenAI speech services] Rust --> Endpoint[Normal text endpoint in pipeline mode] Rust --> History[ChatKit transcript history] @@ -22,11 +25,15 @@ flowchart LR ## Summarized changes -- Human guide: agent voice settings, pipeline/native modes, callback context, +- Human guide: agent voice settings, OpenAI pipeline/native and Telnyx relay modes, callback context, browser admission, Telnyx setup, and explicit transport/tool limitations. - Agent guide: exact REST paths, configuration fields, media format, managed credential sources, and portable versus installation-specific configuration. -- Validation: Mintlify build, broken links, and accessibility checks pass. +- Relay revision adds a real Telnyx Voice channel, self-managed or managed + webhook setup on an existing customer application, phone-only language/interruption settings, and + separate generated-text versus carrier-reported spoken-prefix semantics. +- Validation: relay-revision `mint validate`, `mint broken-links`, and `mint a11y` + pass. No live Conversation Relay or carrier call was performed. - Live OpenAI inference was tested in the API worktree. A live Telnyx call and browser microphone interaction remain manual checks; no deployment is claimed. @@ -35,11 +42,12 @@ flowchart LR yes Publish after the matching API deployment and SDK release. Operators need public -HTTPS/WSS, an OpenAI server key or managed credential, and a dedicated Telnyx -application/number binding. Native endpoint tools, browser personal-tool federation, +HTTPS/WSS, an OpenAI server key or managed credential for OpenAI modes, and a dedicated Telnyx +application/number binding. Relay uses the carrier speech service and the +endpoint's own text model; it needs no OpenAI speech key. Native endpoint tools, browser personal-tool federation, direct WebRTC/SIP, ElevenLabs, voice notes, and retained audio are outside this slice. Related API: https://github.com/trytilde/api/pull/277. Related SDK: https://github.com/trytilde/dispatch/pull/155. -The API needs current-main conflict resolution; the SDK needs a complete contract -refresh from that integrated API head before release. +API main is integrated and Rust validation is ongoing. SDK generation must retain +the optional wallet surface already exposed by its generated client before release. diff --git a/llms/chatkit.md b/llms/chatkit.md index d77da35..2fe4e72 100644 --- a/llms/chatkit.md +++ b/llms/chatkit.md @@ -201,24 +201,45 @@ Use the selected tenant host and explicit `team_id` for these REST operations: 2. Register an HTTP agent with optional `audio` configuration, or use `PUT /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio` with `{ "audio": }`. Set `audio` to null on the PUT route to disable voice. -3. Configuration fields are `mode` (`pipeline` or `realtime`), `credential_id` +3. Configuration fields are `mode` (`pipeline`, `realtime`, or `telnyx_relay`), `credential_id` (optional), `stt_model`, `tts_model`, `realtime_model`, `voice`, `instructions`, - and `max_duration_seconds` (10–1800). The OpenAI Audio credential source is - `chatkit_openai_audio`; omitting the credential uses the server OpenAI key. + `language` (default `en-US`), `interruptible` (default true), and + `max_duration_seconds` (10–1800). The OpenAI Audio credential source is + `chatkit_openai_audio`; omitting it uses the server OpenAI key for OpenAI modes. + Relay uses `stt_model: "deepgram/nova-3"`, `voice: "Telnyx.Ultra.Callie"`, + and null `credential_id`; its phone route owns the Telnyx credential. 4. `POST /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio/sessions` creates a - normal session and returns `audio_session`, `websocket_path`, and a one-time + normal browser session for OpenAI modes and returns `audio_session`, `websocket_path`, and a one-time token. Connect with WebSocket subprotocols `chatkit-audio` and `token.`. + This endpoint rejects `telnyx_relay`; relay starts from an incoming call. Send mono signed PCM16 little-endian audio at 24 kHz as base64 `audio` frames. 5. `PUT /api/v1/team/{team_id}/chatkit/agents/{agent_id}/audio/telnyx` accepts `credential_id` (source `chatkit_telnyx_voice`), `public_key`, `phone_number`, `connection_id`, and public HTTPS `media_base_url`. It returns `route` and - `webhook_url`; use that exact webhook URL in the dedicated Telnyx application. + `webhook_url`; successful setup also returns the assigned + `route.channel_inbox_id`. Use the webhook URL in the dedicated Telnyx application. +6. The generic channel catalog entry is `chatkit.chat_channel.telnyx_voice`, + provider `chatkit.channel.telnyx_voice`. Use auth method + `chatkit.channel.telnyx_voice.auth.self_managed` for a returned URL or + `chatkit.channel.telnyx_voice.auth.managed` for Tilde to update the existing + Voice API application's webhook. Both use your existing encrypted + `chatkit_telnyx_voice` credential and existing number/application. Pass the + same five setup fields and the normal default agent selection. Managed setup + updates webhook configuration; it does not buy, assign, or fund numbers. + Calls use the resulting channel as their participant origin. Pipeline mode invokes the existing callback only when a user speech turn is -ready. Rust synthesizes the response. Realtime mode owns spoken generation; +ready. Rust synthesizes the response. Relay invokes the same signed callback +with `context.audio.mode = "telnyx_relay"`; Telnyx transcribes and synthesizes, +while Tilde exchanges text frames with the carrier. Realtime mode owns spoken generation; transcript observations must not trigger another model turn or external send. `context.audio` and `context.telnyx` come from typed, server-authored speech -provenance rather than client message metadata. +provenance rather than client message metadata. Both persisted text and UI +messages can carry `speech`. Interrupted generated speech retains its original +text with `interrupted: true`, optional `played_audio_ms`, and optional +`reported_spoken_text` supplied by the carrier. Preserve the reported prefix +separately; do not rewrite it as the complete generated response. SDK history +conversion adds the corresponding annotation before the original content. The manual browser/carrier example is `examples/realtime-voice` in `trytilde/dispatch` (the `@trytilde/sdk` packages). It never buys phone numbers or changes existing carrier routing. Agent