diff --git a/docs/internals/voice-call.md b/docs/internals/voice-call.md new file mode 100644 index 0000000000000..4e1ba228dc674 --- /dev/null +++ b/docs/internals/voice-call.md @@ -0,0 +1,553 @@ +# Voice call (plan) + +Status: implementation plan for the fork, not yet built. PAD's decisions of +2026-10-21/22 are the source; the v1 scope cut and the gpt-live-1.1 findings +override older notes. When the feature lands, rewrite this page into the +durable decisions and traps and drop the phase list. + +A voice call lets the user talk to esveo code. A realtime voice model +(OpenAI's GPT-Live, on the user's ChatGPT subscription) holds the +conversation, hands real work to the thread that is open on screen, and +reads the result back. It is separate from [voice input](voice-input.md), +which only dictates into the composer. + +## Goal and non-goals + +Goal for v1: + +- Start a call from the open thread on desktop (Electron) or web. Everything + the voice delegates goes to the thread on screen: it steers the active run, + or starts a normal turn when the thread is idle. +- The backend is the thread's own provider and model. The voice is only the + front end; no extra model hop sits between them. +- Results, questions and approvals of that thread come back as speech. +- The call survives thread switches (the target follows the screen), pauses + itself when it is quiet because open time is billed, and wakes on speech + or on a result. +- A voice panel with both transcripts and a card per delegated message, a + global call bar, and past calls stored on the server. +- Behind a feature flag, off by default. + +Non-goals for v1: + +- Routing: no "send this to thread 2", no new threads by voice, no queueing, + no promote-to-steer, no request header format. See [Later](#later). +- An overview of all threads for the voice. It sees only the open thread. +- Mobile, multi-device handover, other voice vendors, sound effects. +- The classic Realtime protocol with function tools (works in the spike, but + its quota pool and terms are unclear). + +## Relation to upstream + +Upstream PR [pingdotgg/t3code#14865](https://github.com/pingdotgg/t3code/pull/14865) +(open, "Codex voice conversations") shares the browser-audio / server-SDP split +and has desktop microphone permission code worth copying. It differs in the +parts that matter here: + +- It runs the call through the Codex app-server (`thread/realtime/start`), so + every handoff starts a Codex turn and only Codex threads can be the backend. + We call the realtime endpoint directly and dispatch into any provider. +- Its conversation lives as long as one client stream and is not persisted. + We pause, resume across views, and keep call history. + +If upstream merges it, rebase onto its desktop permission and client-runtime +peer pieces instead of keeping our own copies. + +## Architecture + +```mermaid +flowchart LR + subgraph Client["Browser / Electron renderer"] + Mic[Mic + speaker] + Peer[RTCPeerConnection
audio + oai-events DC] + UI[Call button, call bar,
voice panel] + end + subgraph Server["esveo code server"] + RPC[voiceCall.* RPCs] + Svc[VoiceCallService] + Auth[VoiceCallAuth
Codex ChatGPT token] + Side[Sideband WS client] + TM[ThreadManagementService
sendToThread] + Orch[Orchestrator V2
domain events] + DB[(fork_voice_call_* tables)] + end + OAI[(chatgpt.com realtime/calls
api.openai.com/v1/live/id)] + + Mic --> Peer + Peer <-- "audio (WebRTC, direct)" --> OAI + UI <-- "SDP offer/answer, state, transcript" --> RPC + RPC --> Svc + Svc --> Auth + Svc -- "POST offer + session" --> OAI + Side <-- "events, context appends" --> OAI + Svc --- Side + Svc -- "delegation text" --> TM --> Orch + Orch -- "run / turn item / request events" --> Svc + Svc --> DB +``` + +- Audio never touches the esveo code socket. The client owns the mic, the + speaker and the peer; audio flows between the client and OpenAI, so remote + and tunnelled clients cost the same as local ones. +- The access token never reaches the client. The server POSTs the client's + SDP offer and returns the answer. +- The server is the only writer of realtime events and reads them from the + sideband. The client keeps the `oai-events` data channel open because the + session needs it, but sends nothing on it and ignores what arrives, so no + event is sent or handled twice. + +## Protocol + +Everything here comes from PAD's two local spikes (`/private/tmp/voice-spike/`, +not in the repo; 2026-10-22, codex-cli 0.163) and from Codex `main` @ `94350a9`. It is an +undocumented endpoint; re-check it when Codex changes its realtime code. + +**Creating a call** + +- `POST https://chatgpt.com/backend-api/codex/realtime/calls?intent=quicksilver&architecture=avas` +- Headers: `Authorization: Bearer `, `ChatGPT-Account-ID`, + `originator: codex_cli_rs`, `openai-alpha: quicksilver=v2`, + `User-Agent: codex_cli_rs/ …`, `Content-Type: application/json`. +- Body `{ sdp, session }` with `session`: + - `model: "gpt-live-1-codex"`. The string selects nothing; the account + decides the revision (1.1 on Plus and up). Never send `gpt-live-1.1-codex`. + - `audio.output.voice`: one of the 25 v3 voices. An unknown voice ends the + session with `error {code:"forbidden", message:"Voice session access denied."}`. + - `delegation: { type: "client" }`. No `tools`: the v3 session rejects them + and dies. + - `instructions`: front-end prompt, language line, persona, recap (see + [Prompts](#prompts)). +- Response: `201`, the SDP answer as text, `Location: …/rtc_`. The call id + is the last path segment. The POST took about 1.2 s. + +**Sideband** + +- `wss://api.openai.com/v1/live/{call_id}` with the same bearer and account + headers. It carries the same event stream as the data channel and accepts + our events. `session.update` with `delegation:{type:"client"}` works there + and returns `session.updated`. +- Effect's `Socket.makeWebSocket` cannot send request headers, so the server + adds `ws` as a direct dependency (it is already in the tree through + `@effect/platform-node-shared`; `@types/ws` is in the catalog) or wraps it + in a custom `Socket.WebSocketConstructor`. + +**Events we read** + +| Event | Use | +|---|---| +| `session.started {session:{id, status, expires_at}}` | segment is live; `expires_at` is about 2 h out | +| `input_transcript.added {item.text, start_ms, end_ms}` | user transcript deltas, silence timer | +| `output_transcript.added` | voice transcript deltas, silence timer | +| `turn.created` / `turn.done {turn:{id, role, transcript}}` | speaking gate; persisted transcript on `turn.done` | +| `delegation.created {item:{id, content[].text}, handoff_id, user_bidi_turn_id}` | the request to dispatch | +| `delegation.context.appended`, `session.context.appended` | delivery acks | +| `session.usage.updated {usage.audio_duration_ms, usage_limit{status, reset_seconds}}` | quota, every 15 s | +| `session.closed`, `error` | segment ended; `forbidden` means a bad voice | + +`session.input_audio.append` and `session.output_audio.delta` also arrive and +are dropped unread. No event names the model revision. + +**Events we send (sideband only)** + +| Event | Use | +|---|---| +| `delegation.context.append {delegation_item_id, channel:"speakable", content:[{type:"input_text", text}]}` | the answer to one delegation | +| `session.context.append {channel:"speakable", content:[…]}` | unprompted events: results, questions, approvals | +| `session.close` | pause, hang-up, expiry, failed connect | + +Appends are limited to 500 tokens each. Like Codex, cap a result at about +1,000 tokens (`REALTIME_ASSISTANT_OUTPUT_TOKEN_BUDGET`) and send it in +500-byte chunks. + +**Proven:** a delegation held for 180 s is still spoken when answered; an +unsolicited `session.context.append` is spoken; the data channel alone works +end to end. **Not tested:** holds over 180 s, several open delegations at +once, barge-in during injected speech, a non-spoken context channel +(`session.thinking.append`, `initial_items`). Phase 2 tests the open ones. + +## Server modules + +Everything new lives in `apps/server/src/voiceCall/` (the name avoids the +existing `voiceInput/` dictation module), with the contract in +`packages/contracts/src/voiceCall.ts`. + +- **`VoiceCallAuth.ts`** returns `{ accessToken, accountId }` for the Codex + provider instance (v1: the default `codex` instance). + - *Existing mode* (PAD's setup: no `providerInstances.codex`, so the default + mode, with `~/.codex/auth.json`): read `/auth.json` on every + mint (resolve the home as + [`threadRoots.ts`](../../apps/server/src/forkInstructions/threadRoots.ts) + does). `tokens.access_token` and `tokens.account_id` are what the spike + used. **Never write the file**: the Codex CLI owns it and a concurrent + write would race its refresh. If the JWT `exp` is near, ask Codex to + refresh (to verify in phase 1: the app-server account read with token + refresh); if that is not possible, fail with "renew the Codex login". + - *Managed mode*: `makeCodexChatGptAuth(...).access` in + [`CodexChatGptAuth.ts`](../../apps/server/src/provider/CodexChatGptAuth.ts) + already refreshes under the shared session lock. The account id is not a + field there; read it from the id token's ChatGPT claim. Its token is + scoped to `api.openai.com/v1` with `chatgpt.tokens.use.direct`; verify + that chatgpt.com accepts it. + - An API-key Codex login cannot make this call; report it as unavailable. +- **`RealtimeCallClient.ts`**: the POST (via `HttpClient`, as in + [`cliproxyApi.ts`](../../apps/server/src/usage/cliproxyApi.ts)), the sideband + socket, typed decode of the events above, `send`, `close`. No call logic. +- **`VoiceCallSegment.ts`**: one realtime session. Owns the speaking gate, + the injection queue, the silence timer and usage tracking. +- **`VoiceCallService.ts`**: one call per environment. Owns the target + thread, open delegations, the segment lifecycle (connect, pause, resume, + expiry, retry), persistence and the change stream. A second client that + connects takes the call over; the first gets a `takenOver` state and closes + its peer. +- **`VoiceCallReadBack.ts`**: subscribes to `streamDomainEvents` (pattern of + [`DelegatedTaskQuestionReactor.ts`](../../apps/server/src/orchestration-v2/DelegatedTaskQuestionReactor.ts) + with `forkParked`) and turns run, turn-item and runtime-request events into + answers and injections. +- **`voiceCallPrompt.ts`**: front-end instructions, backend hint, recap. + +**RPCs** (`VOICE_CALL_WS_METHODS`, all `AuthOrchestrationOperateScope`): + +```ts +voiceCall.subscribe: {} -> stream VoiceCallEvent // snapshot, then deltas +voiceCall.connect: { offerSdp, target: ThreadRef | null, retryOf?: SegmentId } + -> { answerSdp, segmentId } // start, resume and retry +voiceCall.setTarget: { target: ThreadRef | null } -> void +voiceCall.pause: {} -> void // manual "lazy hang-up" +voiceCall.hangUp: {} -> void +voiceCall.listCalls / voiceCall.readCall // history, phase 4 +``` + +The subscription sends deltas, not full snapshots, so a long call does not +resend its whole transcript on every word. When the owning client's +subscription ends while a segment is live, the server closes the segment +at once, so a dropped client cannot leave a billed session open. + +Wiring follows the existing fork RPCs (commit `d1de6178f0` lists the files): +`forkWsRpc.ts` (`ForkWsRpcGroup`, `FORK_WS_RPC_TAGS`), contracts `index.ts`, +`forkHandlers` in `ws.ts`, `RPC_REQUIRED_SCOPES` in +`auth/RpcAuthorization.ts`, `observability/forkRpcAggregates.ts`, the layer in +`server.ts`, and **`EnvironmentSubscriptionRpcTag` in +`packages/client-runtime/src/rpc/client.ts`**, without which the client types +the stream as unary. [`ForkNotes.ts`](../../apps/server/src/forkNotes/ForkNotes.ts) +is the template for the service, its table and its PubSub stream. + +### The steer path + +On `delegation.created`: + +1. No target (draft, settings page, no thread open) or an archived thread: + answer the delegation at once with one sentence saying so. +2. Otherwise call `ThreadManagementService.sendToThread` with `mode: "auto"`. + It already is the v1 rule: it steers a steerable run and otherwise starts + one. Ids follow + [`ScheduledTaskService`](../../apps/server/src/scheduledTasks/ScheduledTaskService.ts): + `CommandId` `voice-call::`, `MessageId` + `voice-call-message::`, so a retried event cannot + dispatch twice. `createdBy: "user"`, `creationSource: "server"`. +3. Record the delegation `{ delegationItemId, segmentId, threadId, messageId, + runId, status: open }` and push a card to the panel. + +The voice has already said a filler ("On it.") and stays conversational; the +answer follows when the run settles. A dispatch failure is answered at once +with a short reason. + +**Marking the message as voice.** Do not add `"voice"` to `creationSource`; +it is an upstream `Schema.Literals` and every sync would conflict. Use a +context record instead: the text starts with a reference +`[Voice](t3-context://voice-call/)` and `context.records` holds +`{ kind: "voice-call", payload: { hint } }`. `UnknownContextRecord` accepts +the custom kind, and +[`projectComposerContextForProvider`](../../packages/shared/src/composerContextReferences.ts) +puts the payload into the `` envelope of the provider prompt for +every provider. The reference renders as a chip in the user bubble, which +doubles as the "via voice" badge; check how the web renders an unknown kind +and add a small fork chip if it looks wrong. + +## Read-back + +- **Answer to a delegation.** When a run linked to open delegations reaches a + terminal status, take its last assistant message, cap it at about 1,000 + tokens (cut at a sentence; replace code blocks with "code is in the + thread"), prefix the thread title, and send it with + `delegation.context.append` on the newest open delegation of that run. + Older open delegations of the same run (steered twice) are marked + superseded and get nothing. A failed or interrupted run answers with one + sentence and the error. +- **Unprompted events** in the thread that is the target *when the event + fires*: a run that settles with no open delegation, a new user-input + question, a new approval request. They go out as `session.context.append` + ("Update from : …"). A run that answered a delegation never also + triggers an unprompted update. +- **Speaking gate.** Injections wait in a queue until no user turn is open + (every `turn.created role:user` has its `turn.done`), no assistant turn is + open, and the last input transcript delta is at least 1 s old. Answers to + delegations pass the same gate. Codex has no such gate and relies on the + model; we add it because PAD wants events held until he is done talking. +- **Segment gone.** A delegation id belongs to its segment. If that segment + has closed (pause, expiry), the answer goes out as `session.context.append` + on the next segment. While the call is auto-paused, a result wakes it (see + [Pause](#pause-resume-expiry-and-quota)); while manually paused, results + queue and are spoken after resume. +- **Server restart.** Open delegations are persisted. On start, each one is + checked against its run in `ProjectionStoreV2`; a run that settled while + the server was down is delivered as an unprompted result on the next + segment. `streamDomainEvents` is live-tail only, so there is no replay to + rely on. + +### Questions and approvals (decision for PAD) + +With no routing, "yes, allow it" would become a steer and the request would +stay pending. v1 needs exactly one rule for this. Recommendation: + +- While the target thread has exactly one pending `user_input` request, the + next delegation is its free-text answer: `runtime-request.respond` with + `answers` (the path MCP `t3_pending_request_respond` uses), not a steer. +- Approvals: PAD runs with full access, so they are rare. The voice announces + them and the front-end prompt tells it to delegate exactly `approve` or + `decline` as the answer. A delegation that normalises to one of those words + while one approval is pending is answered with `decision: accept|decline`; + anything else is a normal steer. Unlike MCP, this server path may send + `decision`, so the rule stays this narrow. +- More than one pending request: announce how many, answer nothing by voice, + point to the screen. + +Alternatives: approvals only announced, never answered by voice; or a +request header format (moved to Later). + +## Prompts + +`voiceCallPrompt.ts` builds three texts; `prompt.py` in Bloo's `bloo-voice` +plugin is the reference for tone and for the doze notes. + +- **Front-end instructions** (POST `instructions`): you are the voice of + esveo code talking to the user; chat on your own, but hand every request + about code, files, the thread or its agent to the client as a cleaned-up + instruction in your own words; the client sends it to the thread open on + screen; say a short filler and stay responsive while it runs; read results + back short and without Markdown; when told about a question or approval, + ask the user and pass the answer on (exact words for approvals); do not + invent results. Plus the language line (`auto`: answer in the user's + language; `de`: always German, from the first sentence), the persona text, + the current target thread's title, the time, and on resume a recap. +- **Backend hint** (the `voice-call` context record), modelled on Codex's + `realtime_start.md`: this message comes from a live voice call, written by + the voice assistant from speech, so it may contain recognition errors; your + final answer is read aloud by that assistant; keep it short and + action-oriented; leave long output, code and lists in the thread and say + that they are there. +- **Recap** for a new segment: the last 8 transcript turns (Bloo's + `RECAP_TURNS`) and any open delegations, as part of `instructions`. That is + the one carry-over mechanism we control and know works; `initial_items` and + a silent context channel are unverified. + +## Client modules + +New folder `apps/web/src/components/voiceCall/`. Mobile is out of scope, so +the controller stays in the web app instead of `packages/client-runtime`. + +- **`voiceCallState.ts`**: atoms built with + `createEnvironmentRpcSubscriptionAtomFamily` and `createEnvironmentRpcCommand`, + copied from [`forkNotesState.ts`](../../apps/web/src/components/forkNotes/forkNotesState.ts). +- **`voiceCallPeer.ts`**: `getUserMedia`, the peer, the `oai-events` channel, + remote audio playback. Needs a secure context, like dictation. +- **`useVoiceCallController.ts`**: connect, retry, pause and wake (see + below), takeover, hang-up. +- **`useVoiceCallTarget.ts`**: the thread on screen, sent with `setTarget` on + change. In split view that is the focused pane (`activeLeafId` in + `splitThreadStore`, `"route"` meaning the route thread); otherwise + `useThreadRouteRef()`. A draft has no target. +- **`VoiceCallButton.tsx`**: start, and stop while live. Mounted with one + `// Fork:` line in the `panelToggleControls` span of `ChatView.tsx`, next to + `ThreadFindHeaderButton`. Also a command palette action. +- **`VoiceCallBar.tsx`**: a small fixed pill on every route: state + (connecting, live, paused, auto-paused, error), minutes, quota warning, + pause/resume, hang up, open panel. Mounted in the coordinator row of + `RootRouteView` in `routes/__root.tsx`. Popout windows do not get it. +- **`VoiceCallPanel.tsx`**: a `voice-call` right-panel surface registered + through `useForkRightPanelSurfaces` (model: the Instructions tab, commit + `f0373c7ffc`). The right-panel store is per thread, so the panel opens on + the focused thread and follows to the new thread when the target changes + while it is open. The content reads the call store, so it is the same + everywhere. It shows the user and voice transcripts and a card per + delegation (thread, text, status, answer excerpt), plus the past calls list. +- **`voiceCallJump.ts`**: clicking a card navigates to + `/$environmentId/$threadId` with the message id in history state; a small + hook mounted in `MessagesTimeline` finds the row by `data-message-id`, + pages in earlier history like `useAssistantCitationTarget`, and calls + `scrollToIndex`. Citations stay as they are; they only target assistant + messages. +- **`VoiceCallSettingRows.tsx`**: see [Settings](#settings-and-feature-flag). + +**Desktop.** Dictation already records in the desktop app (see "Dictation" in +`docs/fork/setup.md`), so check first whether WebRTC needs more. If it does, +copy `MainWindowPermissions.ts` from #14865 into a fork file mounted in +`DesktopWindow.ts`, and add `NSMicrophoneUsageDescription` in all three +places: `scripts/build-desktop-artifact.ts` (`extendInfo`), +`apps/desktop/scripts/electron-launcher.mjs`, and `scripts/fork-app.sh`, +which re-signs ad hoc and so needs its own plist patch. + +## Persistence + +Plain fork tables, created in `ensureForkSchema` +([`forkSchema.ts`](../../apps/server/src/threadOrchestration/forkSchema.ts)), +not orchestration events: a call is not thread state. + +- `fork_voice_calls`: id, started/ended at, state, voice, last target, + total audio ms. +- `fork_voice_call_segments`: id, call id, realtime call id, started/ended + at, end reason (pause, auto-pause, expiry, retry, hang-up, error), + audio ms, expires at. +- `fork_voice_call_entries`: id, call id, segment id, sequence, kind + (`user`, `voice`, `delegation`, `injection`), text, thread id, message id, + run id, delegation status, timestamps. + +Transcript turns are written on `turn.done`, never per delta. Live deltas only +go over the stream. Delegations are written when they arrive, because the +read-back needs them after a restart. + +## Settings and feature flag + +Server settings, because the server builds the session. They follow the +`enable*` fork pattern in `packages/contracts/src/settings.ts` (commit +`57befeadb1` has the touch list): a field with a decoding default plus the +matching optional key in `ServerSettingsPatch`. + +- `enableVoiceCall: boolean`, default `false`. Off: the RPCs fail with + `VoiceCallDisabledError`, the button, bar and panel are hidden. +- `voiceCallVoice`: one of the 25 v3 voices (a constant list in the contract: + juniper, maple, spruce, ember, vale, breeze, arbor, sol, cove, aube, + bubbie, haetsal, hanul, himari, leher, miko, neer, porto, rindo, rio, + rivage, selva, sonna, tinta, viola). Default `sonna` (German worked; + PAD may prefer `miko`). Only the list is offered, because an off-list voice + kills the session. +- `voiceCallLanguage: "auto" | "de"`, default `"auto"`. +- `voiceCallPersona: string`, default empty, capped in length. + +Rows sit in Settings next to `VoiceInputSettingRow`, use +`esveoSearchableSetting`, and their search items set `fork: true`. The voice, +language and persona rows show only when the flag is on. The subscription +reports availability, so the UI can explain why a call cannot start: flag +off, no Codex instance, Codex logged in with an API key, login expired. + +## Pause, resume, expiry and quota + +`audio_duration_ms` is wall-clock session time, so an idle open session uses +quota. Pausing therefore closes the realtime session; the call itself stays. + +- **Auto-pause.** The server pauses after 30 s without any transcript delta + in either direction, when no turn is open and the voice is not speaking. + An open delegation does not hold the session open; its result wakes it. + The server sends `session.close`, persists the segment, and tells the + client, which closes its peer but keeps the mic. +- **Wake on speech.** While auto-paused the client runs a level gate (Bloo: + peak about 500/32767 for 300 ms of speech) and keeps a 12 s ring buffer. + On speech it connects a new segment and replays the buffer first, so the + model hears the whole sentence that woke it. Bloo drops a replay shorter + than 1.5 s, because the model makes things up from a cough. **Trap:** + Discord sends no packets in silence, so Bloo's backlog drains by itself; a + browser mic produces frames all the time, so a replayed buffer never + catches up. The input track (an `AudioWorklet` into a + `MediaStreamAudioDestinationNode`) must skip quiet frames while a backlog + remains. +- **Wake on a result.** The server asks the owning client to connect; the + queued answer is spoken once the new segment has started. With no client + subscribed, it stays queued. +- **Manual pause ("lazy hang-up").** Closes the session and releases the mic. + No auto-wake. Results queue and are spoken after resume. Resume works from + the call bar on any route of the same app; no cross-device resume. +- **Expiry.** `expires_at` is about 2 h. From 5 minutes before, the next quiet + moment (the speaking gate) triggers a close and an immediate new segment + with a recap; at 1 minute before it is forced. +- **Retry.** The data channel sometimes never opens (one of 17 v3 calls in + the 1.1 spike; worse on the classic route). If the peer is not connected + and the channel not open within 15 s, or the sideband fails to connect, the + client calls `connect` with `retryOf`; the server closes the old call and + mints a new one. Three attempts, then an error state. Each attempt is + logged. +- **Quota.** `session.usage.updated` gives segment minutes; the bar shows this + call's minutes and the last 24 h from the segments table. The plan limit is + not known reliably (help-center figures conflict), so show minutes used, + not minutes left. When `usage_limit.status` is not null, show it and + `reset_seconds` as a warning. + +## Phases + +Each phase is one reviewable PR on the feature branch, behind the flag. + +1. **Server call and steer path.** Contract and RPC wiring, `VoiceCallAuth` + (existing mode first), `RealtimeCallClient` with the sideband, + `VoiceCallService` with `connect` / `setTarget` / `hangUp` / `subscribe`, + the steer path with the voice context record, the front-end prompt, the + flag and voice/language/persona settings (no rows yet), connect retry. A + bare `VoiceCallButton` behind the flag (start and stop, transcript in the + browser console) is the end-to-end check; no panel, bar or pause yet. + Desktop mic only if the check shows it is needed. +2. **Read-back.** `VoiceCallReadBack`, delegation answers with cap and + chunking, unprompted events for the target, the speaking gate, + questions and approvals per PAD's decision, the delegations table and + restart re-arm. Settles the untested protocol points listed above. +3. **Pause, resume, expiry, quota.** Auto-pause, client level gate and + replay buffer, wake on result, manual pause, recap, 2 h rollover, usage + tracking. +4. **UI.** Voice panel with transcripts and cards, jump to message, call + bar, command palette action, settings rows, history tables and + `listCalls` / `readCall`. Rewrite this page into the durable internals + doc, add a user doc section and the changelog entry. + +## Testing + +- Unit tests with a fake `RealtimeCallClient` that replays recorded spike + event sequences (`/private/tmp/voice-spike/logs/a1_v3_dc.txt`, + `c1_v3_ws_hold60.txt`, `l11_v_marin.txt`): delegation to dispatch, ids + stable on duplicates, no-target and archived answers, auth failures. +- Read-back with the orchestrator test harness: drain + `OrchestrationEffectWorkerV2.drain` or await the persisted run event, never + sleep. Cover steer twice then one answer, unprompted versus delegated + dedupe, the speaking gate holding an injection until `turn.done`, segment + gone, restart re-arm. +- Pause logic with a test clock: silence timer, expiry window, retry limit. +- `VoiceCallAuth`: existing mode never writes `auth.json`; expired token + path; managed mode account id from the id token. +- Manual, per phase: a real call on PAD's login in a worktree dev server + seeded with `vp run migrate-dev-db`, never against `~/.t3`. The spike + scripts (`t_a_direct.py`, `t_live11.py`, `cleanup_calls.py`) stay the + reference for protocol questions; close every test call. +- Run only the touched specs and a scoped typecheck, per `FORK.md`. + +## Risks + +- **Undocumented endpoint.** `/backend-api/codex/realtime/calls` can change + or close at any time. We send `originator: codex_cli_rs` and a Codex user + agent, which is a gray area under OpenAI's terms for a third-party app on a + subscription login. `X-OAI-Attestation` is optional today and could become + mandatory, which would end this path. Keep the code behind + `RealtimeCallClient` so a switch to the metered `gpt-live-1` API (API key, + per minute) touches one module. +- **No model signal.** No event reports the model revision. 1.1 is inferred + from the account plan and from 1.1-only voices rendering. If voices and + model ever decouple, we cannot tell. +- **Idle billing.** Open time counts, whether anyone talks or not. A bug in + pause, takeover or disconnect handling burns the user's voice allowance; + the server must close a segment whenever its owning client goes away. +- **Token handling.** Existing mode depends on the Codex CLI keeping + `auth.json` fresh. The token must never be logged or sent to the client. +- **Unclear quota pool.** Whether voice minutes draw on the ChatGPT voice + allowance or the Codex limits is unknown. +- **Recognition errors.** Without routing, every delegation steers the thread + on screen. A misheard request lands in whatever thread PAD is looking at. +- **Untested protocol points** listed under [Protocol](#protocol). + +## Later + +- Routing: send to another thread (and switch the UI there), open a new + thread with project and model inference, queue instead of steer, promote + the last queued message to a steer. Options: a fixed request header the + voice writes and the server parses, a cheap router model as fallback, or + the classic Realtime protocol with real function tools once its quota and + terms are clear. +- An overview of all active and recent threads for the voice. +- A silent context channel for target-change notes, once one is verified. +- Mobile (react-native-webrtc, as in #14865). +- A choice of Codex instance when there are several. +- A health check that starts a 1.1-only voice to confirm the revision. +- Keybinding for start/stop and mute.