feat(sdk): add Telnyx relay voice context and examples

.changeset/chatkit-realtime-voice.md

@@ -0,0 +1,7 @@
+---
+"@trytilde/api-client": minor
+"@trytilde/sdk": minor
+"@trytilde/sdk-vercel-ai-node": minor
+---
+
+Add agent audio configuration, voice session admission, Telnyx Conversation Relay and media routing, and typed speech context for `chatKitEndpoint` callbacks. Preserve carrier-reported interrupted speech separately from generated text and UI history.

.gitignore

@@ -25,3 +25,6 @@ apps/mobile/android/
 apps/mobile/ios/
 .vercel
 .claude/worktrees/
+
+# Local voice example registrations contain one-time endpoint credentials.
+examples/realtime-voice/.agents.local.json

docs/adrs/0030-tilde-sdk-and-cli-ownership.md

@@ -69,3 +69,9 @@ flowchart LR
 - 2026-08-31: OpenCode and Gemini CLI receive matching dedicated adapters and
   native fail-open audit installation, completing ChatKit audit support across
   every coding harness configured by `openbot plugin`.
+
+- 2026-09-07T20:04:46Z: The public SDK exposes `telnyx_relay` alongside the OpenAI voice
+  modes. The API/carrier owns speech conversion; the SDK validates signed
+  callback context and annotates interrupted text/UI history with the distinct
+  carrier-reported spoken prefix. This adds no OpenBot provider or browser relay
+  runtime.

docs/updates/155.md

@@ -0,0 +1,72 @@
+# PR 155: ChatKit realtime voice SDK
+
+https://github.com/trytilde/dispatch/pull/155
+
+## Intent of the change
+
+Expose the companion Tilde voice API through the current public SDK packages and
+provide a manually runnable browser/Telnyx agent example, including carrier-owned
+recognition and synthesis through Telnyx Conversation Relay.
+
+## Architecture changes
+
+ADR review: ADR 0030 records SDK ownership and now documents the relay adapter
+contract. The API's ADR 0027 governs agent-owned speech configuration. Typed signed speech
+context is validated before it reaches the ordinary endpoint callback.
+
+```mermaid
+flowchart LR
+  Example[Manual voice example] --> SDK[Public SDK audio wrappers]
+  SDK --> API[Tilde audio API]
+  API --> Callback[Signed pipeline or relay text callback]
+  Carrier[Telnyx speech services] <-->|Text relay frames| API
+  Callback --> Context[context.audio and context.telnyx]
+  API --> Native[Native Realtime without duplicate callback]
+```
+
+## Summarized changes
+
+- SDK: agent audio registration/configuration, retrieval, browser admission, and
+  Telnyx route binding using camelCase inputs and the API's typed wire fields.
+- Node adapter: verified speech context, current-user-turn context selection, and
+  interrupted generated-speech annotations during history conversion.
+- Relay: third mode, language and interruptible settings, phone-only admission,
+  and carrier-reported spoken-prefix annotations for both text and UI history.
+- Example: three agents, current package names and AI SDK 7, explicit agentLoop
+  response mode, local microphone/player, and carrier setup instructions.
+- Package READMEs and minor Changeset for both public SDK packages.
+- No application client, provider composition, protobuf, or fork configuration changes.
+- Generated contracts were refreshed from the full wallet-enabled API after
+  integrating current provider and direct-workflow changes in both repositories.
+  The schema contains 732 operations, adds seven voice operations, removes none,
+  and preserves all 22 existing wallet/payment operations.
+
+Validation: `pnpm openbot sdk refresh` passes all SDK package builds and 327 tests.
+The current core SDK and Node adapter separately pass their typechecks and 313
+focused tests. The manual example typechecks. Scoped lint previously passed with
+existing warnings in unchanged webhook tests. API proof includes the actual
+Telnyx text/control runtime against HTTP/Postgres and live OpenAI pipeline/native
+inference; it does not establish a real Telnyx call. Public docs validation,
+broken links, accessibility, and publication-boundary checks pass. Full Dispatch
+application checks/builds and real carrier/browser interactions were not run for
+this isolated SDK contribution.
+
+## Critical to apply
+
+yes
+
+Deploy API PR 277 with its migrations and voice routes, then release these SDK
+packages and publish docs PR 37. The generated client and hand-authored audio
+wrappers both include the voice operations. Existing endpoints need the updated
+Node adapter to recognize `telnyx_relay` context.
+
+Speech configuration is portable agent intent; credentials stay server-side and
+example registration secrets are ignored. Media tokens are ephemeral. Telnyx
+routing is installation-specific. Its real ChatKit channel supports self-managed
+or managed webhook setup with existing customer credentials/application/number;
+managed setup does not purchase or fund carrier resources. Native endpoint-tool bridging, browser
+personal-tool federation, direct WebRTC/SIP, ElevenLabs, recordings, and voice
+notes are outside this slice.
+
+Related: https://github.com/trytilde/api/pull/277 and
+https://github.com/trytilde/docs/pull/37.

examples/realtime-voice/.env.example

@@ -0,0 +1,18 @@
+TILDE_BASE_URL=https://your-org.api.trytilde.ai
+TILDE_ORG_ID=your-org-id
+TILDE_TEAM_ID=your-team-id
+# Human-owned key used only by the localhost setup/test UI.
+TILDE_API_KEY=
+# Reachable by the Tilde API. Local development can use a private network address.
+AGENT_ENDPOINT_ORIGIN=http://127.0.0.1:31247
+PORT=31247
+OPENAI_API_KEY=
+TEXT_MODEL=gpt-4.1-mini
+
+# Optional dedicated test phone route. Relay is phone-only; leave empty for browser demos.
+TELNYX_CREDENTIAL_ID=
+TELNYX_PUBLIC_KEY=
+TELNYX_PHONE_NUMBER=
+TELNYX_CONNECTION_ID=
+TELNYX_AGENT_MODE=telnyx_relay
+VOICE_MEDIA_BASE_URL=

examples/realtime-voice/README.md

@@ -0,0 +1,132 @@
+# ChatKit voice example
+
+Three registered agents demonstrate the supported voice paths:
+
+- **Pipeline:** Rust streams speech to OpenAI transcription, invokes the normal
+  `chatKitEndpoint` callback once per response turn, then streams the callback's
+  text through OpenAI TTS.
+- **Telnyx relay:** Telnyx Conversation Relay handles transcription and speech
+  synthesis. Tilde exchanges text with Telnyx and invokes the normal signed
+  callback at `/agent/telnyx_relay`. This mode accepts phone calls only.
+- **Realtime:** Rust maintains an OpenAI Realtime conversation. The model
+  generates speech directly; its transcript does not invoke the text callback.
+
+All create ordinary ChatKit sessions. Pipeline and relay callbacks receive typed
+`context.audio` and, for phone calls, `context.telnyx`. Audio models and voice
+settings are registered on the Tilde agent, not on the callback.
+
+## Run locally
+
+Requires the matching `codex/chatkit-realtime-voice` API and SDK changes, Node 24+
+and pnpm. Use a Tilde API reachable from the browser and a callback URL reachable
+from that API. The example binds to loopback and is intended for local testing.
+
+From the Dispatch repository root:
+
+```sh
+pnpm install --frozen-lockfile
+pnpm --filter @trytilde/api-client build
+pnpm --filter @trytilde/sdk build
+pnpm --filter @trytilde/sdk-vercel-ai-node build
+cd examples/realtime-voice
+pnpm install --ignore-workspace
+cp .env.example .env.local
+```
+
+Set `TILDE_BASE_URL`, `TILDE_ORG_ID`, `TILDE_TEAM_ID`, and a **human-owned**
+`TILDE_API_KEY` in `.env.local`. Browser media admission rechecks that human's
+current session membership. Set `AGENT_ENDPOINT_ORIGIN` to a URL the Rust API can
+reach. When both run on this machine, `http://127.0.0.1:31247` works; a hosted API
+requires an authenticated development tunnel or deployed endpoint.
+
+Set `OPENAI_API_KEY` on the **Rust API** for transcription, synthesis and native
+Realtime. Alternatively bind a managed `chatkit_openai_audio` credential to each
+agent. Set `OPENAI_API_KEY` in this example for its ordinary text model too.
+
+```sh
+pnpm setup
+pnpm dev
+```
+
+Setup registers three new agents and saves their endpoint credentials in ignored
+`.agents.local.json` with mode 0600. It does not print those secrets. Re-running
+setup creates new demo agents; delete old demo agents through Tilde when done.
+
+Open `http://localhost:31247`, select a mode, click **Start conversation**, and
+allow microphone access. Local HTTPS certificates must be trusted by both the
+browser and Node (`NODE_EXTRA_CA_CERTS` can name your local root certificate).
+
+Test:
+
+1. Ask “What is two plus two?” Hear a short spoken answer.
+2. Interrupt while the agent speaks. Queued audio should stop.
+3. End the conversation. Microphone tracks and upstream audio connection close.
+4. Open the displayed ChatKit session ID in Tilde. Final transcripts are stored.
+5. In pipeline mode, server output shows one normal callback per user turn and
+   `context.audio.mode = pipeline`. Native mode makes no speech-triggered HTTP
+   callback. Transcripts describe generated speech; exact word-level playback
+   alignment is not guaranteed after interruption.
+
+The sample uses a mono 24 kHz PCM WebSocket to the Rust API. Native Realtime also
+uses the Rust media runtime here; this example does not negotiate direct
+browser-to-OpenAI WebRTC. Audio recordings are not retained by this first slice. Native Realtime uses its
+configured instructions; it does not inherit tools or prompts from the text
+callback. Agent state exports carry speech settings and credential setup references;
+configure Telnyx number/application routes again in the destination installation.
+
+## Attach a Telnyx test number
+
+Use a dedicated test Voice API application. Do not repoint a production number.
+
+The Telnyx Voice chat provider (`chatkit.channel.telnyx_voice`) uses your existing
+credential, application, and number and creates a ChatKit channel for incoming
+call participants. Generic provider setup offers self-managed webhook setup
+(copy the returned URL) and managed webhook setup (Tilde updates your existing
+application's webhook). Neither creates a Telnyx account, funds service, buys a
+number, or assigns numbers. This example uses `configureTelnyx()` and the manual
+webhook step below.
+
+1. In Tilde managed credentials, create a **Telnyx Voice** credential
+   (`chatkit_telnyx_voice`) containing your Telnyx `api_key`. Copy its ID.
+2. In Telnyx, create a Voice API application and select a test number. Copy the
+   application/connection ID and account Ed25519 public key.
+3. Add these optional settings to `.env.local` before `pnpm setup`:
+
+```dotenv
+TELNYX_CREDENTIAL_ID=your-tilde-managed-credential-id
+TELNYX_PUBLIC_KEY=your-base64-ed25519-public-key
+TELNYX_PHONE_NUMBER=+12025550100
+TELNYX_CONNECTION_ID=your-telnyx-voice-application-id
+TELNYX_AGENT_MODE=telnyx_relay
+VOICE_MEDIA_BASE_URL=https://your-public-tilde-api-origin
+```
+
+4. Setup prints the exact webhook URL. Put it in the test application's **Webhook
+   URL**, select POST, and associate the number with that application.
+5. Call your test number manually. Tilde verifies the webhook, checks the number
+   and application, answers, and connects a one-time authenticated Conversation
+   Relay socket. The relay profile uses `deepgram/nova-3`, `Telnyx.Ultra.Callie`,
+   `language: "en-US"`, and `interruptible: true`. It needs no OpenAI speech
+   credential; the callback still uses its configured text model.
+6. Interrupt a spoken response. The next callback's converted history marks the
+   interrupted response and distinguishes the carrier-reported spoken prefix
+   from generated text. Ending the call stops the relay and pending turn.
+7. To test native speech, bind the route to the realtime demo agent with
+   `client.chatkit.audio.configureTelnyx()` and update the application's webhook
+   to the returned URL. Do not run both routes for the same application at once.
+
+The API's public HTTPS origin must also accept WSS upgrades. Hookdeck can capture
+and replay lifecycle webhooks but cannot replace the live media WebSocket.
+No ngrok process is started by this example. Relay uses text WebSocket frames;
+Telnyx owns its audio path. Select `TELNYX_AGENT_MODE=pipeline` or `realtime` to
+exercise the existing media adapter, which receives PCMU at 8 kHz and converts
+24 kHz generated PCM for playback. The browser microphone page keeps these two
+modes; it cannot start a `telnyx_relay` session.
+
+See [Telnyx Conversation Relay](https://developers.telnyx.com/docs/voice/programmable-voice/conversation-relay)
+for the carrier protocol. Partial prompts do not invoke the callback; final
+prompts do. Caller ID is carrier context, not a verified Tilde human identity.
+
+For WhatsApp calls, first enable WhatsApp Business Calling in Telnyx and use the
+actual calling application's connection ID. It reaches the same media adapter.
+This example does not initiate outbound calls or buy/assign phone numbers.

examples/realtime-voice/index.html

@@ -0,0 +1,67 @@
+<!doctype html>
+<html lang="en"><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
+<title>ChatKit voice test</title>
+<style>body{font:17px system-ui;max-width:720px;margin:60px auto;padding:24px;color:#183027;background:#f4f7f3}button,select{font:inherit;padding:12px;margin:8px 8px 8px 0}#transcript{background:white;border:1px solid #cdd9ce;border-radius:12px;padding:20px;min-height:160px}small{color:#496354}p{line-height:1.5}</style>
+<h1>ChatKit voice test</h1>
+<p>Speak, hear the answer, then interrupt mid-sentence. Both modes store their transcript in a normal ChatKit session.</p>
+<label for="mode">Agent</label>
+<select id="mode"><option value="pipeline">Text callback + STT/TTS</option><option value="realtime">OpenAI Realtime</option></select>
+<div><button id="start">Start conversation</button><button id="stop" disabled>End conversation</button></div>
+<p id="status" role="status">Ready</p><small id="session"></small><div id="transcript" aria-live="polite"></div>
+<script type="module">
+const start=document.querySelector('#start'),stop=document.querySelector('#stop'),status=document.querySelector('#status');
+let ws,context,microphone,worklet,nextPlay=0,generation=0,ready=false;
+const sources=new Set(),transcripts=new Map();
+function clearAudio(){generation++;for(const source of sources){try{source.stop();}catch{}}sources.clear();nextPlay=context?.currentTime??0;}
+async function cleanup(){ready=false;clearAudio();microphone?.getTracks().forEach(t=>t.stop());worklet?.disconnect();if(context&&context.state!=='closed')await context.close();start.disabled=false;stop.disabled=true;document.querySelector('#mode').disabled=false;}
+function send(value){if(ws?.readyState===WebSocket.OPEN)ws.send(JSON.stringify(value));}
+start.onclick=async()=>{
+  start.disabled=true;document.querySelector('#mode').disabled=true;status.textContent='Connecting…';document.querySelector('#transcript').replaceChildren();transcripts.clear();
+  try{
+    context=new AudioContext({sampleRate:24000});await context.resume();
+    microphone=await navigator.mediaDevices.getUserMedia({audio:{channelCount:1,echoCancellation:true,noiseSuppression:true}});
+    const response=await fetch(`/start?mode=${encodeURIComponent(document.querySelector('#mode').value)}`,{method:'POST'});
+    if(!response.ok)throw Error(await response.text());
+    const connection=await response.json();document.querySelector('#session').textContent=`ChatKit session: ${connection.sessionId}`;
+    ws=new WebSocket(connection.websocketUrl,['chatkit-audio',`token.${connection.token}`]);
+    ws.onclose=()=>{status.textContent='Conversation ended';void cleanup();};
+    ws.onerror=()=>{status.textContent='Connection failed';};
+    ws.onmessage=({data})=>{
+      const event=JSON.parse(data);
+      if(event.type==='ready'){ready=true;stop.disabled=false;status.textContent='Listening — speak now';}
+      if(event.type==='clear'){clearAudio();status.textContent='Listening';}
+      if(event.type==='error'){status.textContent=event.message;}
+      if(event.type==='transcript'){
+        let line=transcripts.get(event.utterance_id);
+        if(!line){line=document.createElement('p');transcripts.set(event.utterance_id,line);document.querySelector('#transcript').append(line);}
+        line.textContent=`${event.role}: ${event.text}`;
+      }
+      if(event.type==='speech_interrupted'){const line=transcripts.get(event.utterance_id);if(line)line.textContent+=' [Interrupted; generated transcript may include unplayed words]';}
+      if(event.type==='audio'){
+        const raw=atob(event.audio),samples=new Float32Array(raw.length/2);
+        for(let i=0;i<samples.length;i++){let value=raw.charCodeAt(i*2)|(raw.charCodeAt(i*2+1)<<8);if(value>=32768)value-=65536;samples[i]=value/32768;}
+        const buffer=context.createBuffer(1,samples.length,24000);buffer.copyToChannel(samples,0);
+        const source=context.createBufferSource();source.buffer=buffer;source.connect(context.destination);
+        const currentGeneration=generation;sources.add(source);source.onended=()=>{
+          sources.delete(source);
+          if(currentGeneration===generation&&event.utterance_id)send({type:'playback',utterance_id:event.utterance_id,audio_end_ms:event.audio_end_ms});
+        };
+        nextPlay=Math.max(nextPlay,context.currentTime+0.025);source.start(nextPlay);nextPlay+=buffer.duration;status.textContent='Agent speaking — you can interrupt';
+      }
+      if(event.type==='ended'){ws.close();}
+    };
+    const code=`class Capture extends AudioWorkletProcessor { constructor(){super();this.data=[];} process(inputs){const input=inputs[0]?.[0];if(input){this.data.push(...input);if(this.data.length>=480){this.port.postMessage(Float32Array.from(this.data));this.data=[];}}return true;} } registerProcessor('capture',Capture);`;
+    const workletUrl=URL.createObjectURL(new Blob([code],{type:'text/javascript'}));await context.audioWorklet.addModule(workletUrl);URL.revokeObjectURL(workletUrl);
+    worklet=new AudioWorkletNode(context,'capture');
+    worklet.port.onmessage=({data})=>{
+      if(!ready||ws.bufferedAmount>256000)return;
+      const pcm=new Uint8Array(data.length*2),view=new DataView(pcm.buffer);
+      for(let i=0;i<data.length;i++)view.setInt16(i*2,Math.round(Math.max(-1,Math.min(1,data[i]))*32767),true);
+      send({type:'audio',audio:btoa(String.fromCharCode(...pcm))});
+    };
+    context.createMediaStreamSource(microphone).connect(worklet);worklet.connect(context.destination);
+  }catch(error){status.textContent=error.message;ws?.close();await cleanup();}
+};
+stop.onclick=()=>{send({type:'stop'});ws?.close();void cleanup();};
+window.addEventListener('pagehide',()=>{ws?.close();microphone?.getTracks().forEach(t=>t.stop());});
+</script></html>

examples/realtime-voice/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "chatkit-realtime-voice-example",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "setup": "tsx --env-file=.env.local setup.ts",
+    "dev": "tsx --env-file=.env.local server.ts",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@trytilde/sdk": "link:../../packages/sdk",
+    "@trytilde/sdk-vercel-ai-node": "link:../../packages/sdk-vercel-ai-node",
+    "@ai-sdk/openai": "4.0.60",
+    "ai": "7.0.64"
+  },
+  "devDependencies": {
+    "@types/node": "^24.3.0",
+    "tsx": "^4.20.6",
+    "typescript": "^5.9.3"
+  }
+}