feat: add local realtime voice agent stack by cvsz · Pull Request #144 · cvsz/z-platform

cvsz · 2026-08-01T22:26:53Z

Summary

Adds a production-oriented local realtime voice vertical slice to Z Platform:

apps/zvoice: browser voice UI with AudioWorklet microphone capture, 16 kHz PCM16 streaming, live transcripts, audio playback, cancellation, and barge-in.
services/voice-gateway: service-authenticated ticket issuance, short-lived HMAC one-time WebSocket tickets, replay rejection, concurrency admission, raw WebSocket tunneling, health, metrics, and redacted structured logs.
services/voice-agent: pinned Hugging Face speech-to-speech runtime using Faster Whisper STT, the OpenAI-compatible chat-completions backend, and Qwen3-TTS.
compose.voice.yml: optional Ollama, llama.cpp, and vLLM profiles while keeping services/ai-gateway as the sole LLM policy boundary.
Architecture, operations runbook, environment template, and smoke checks.

Architecture

Browser -> apps/zvoice -> services/voice-gateway -> services/voice-agent
                                                  -> services/ai-gateway
                                                     -> Ollama / llama.cpp / vLLM

Security properties

Browser never receives Z_PLATFORM_SERVICE_TOKEN or provider keys.
WebSocket access uses a 10–300 second signed ticket transported through Sec-WebSocket-Protocol.
Tickets are single-use and reject tampering, expiry, and replay.
Speech runtime has no published host port.
Published application/gateway/runtime-provider ports bind to loopback by default.
External production exposure remains gated on reviewed identity, TLS, Redis-backed replay state, privacy/retention controls, load tests, and human release approval.

Validation completed

node --check for gateway, ZVoice server, browser client, and AudioWorklet.
Node tests: 6 passed, 0 failed.
- ticket signing, expiry, single-use, tamper rejection, endpoint authentication, browser-safe response contract;
- ZVoice health secret non-disclosure and identity/service-token proxy boundary.
Shell syntax checks for runtime entrypoint and smoke script.
Python bytecode compilation for the health check.
JSON and YAML parsing for root package configuration and Compose overlay.
Git comparison: branch is 21 commits ahead and 0 behind main.

Local run — Ollama

cp configs/voice-agent.env.example .env.voice
# Set Z_PLATFORM_SERVICE_TOKEN and VOICE_TICKET_SECRET with independent random values.

docker compose --env-file .env.voice \
  -f compose.yml -f compose.voice.yml \
  --profile voice-ollama up -d --build

docker compose --env-file .env.voice \
  -f compose.yml -f compose.voice.yml \
  exec ollama ollama pull qwen3:8b

Open http://127.0.0.1:3022.

Remaining release checks

This PR is intentionally draft because the current execution environment could not perform Docker image builds, model downloads, GPU compatibility validation, or an end-to-end browser microphone/speaker test. Those checks are documented in docs/operations/voice-agent.md and must pass on the target host before merge or external exposure.

The first implementation stores consumed ticket nonces in memory and is therefore single-replica. Move replay/admission state to Redis before horizontal scaling.

gemini-code-assist · 2026-08-01T22:26:56Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

ecc-tools · 2026-08-01T22:27:04Z

ECC bundle files are already tracked in this repository. Skipping generation of another bundle PR.

+
+export function parseBearer(header) {
+  if (typeof header !== "string") return null;
+  const match = /^Bearer\s+(.+)$/i.exec(header.trim());


gemini-code-assist · 2026-08-02T01:52:22Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eb6429a594

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-02T02:04:14Z

+  const tenantId = String(request.headers["x-tenant-id"] || "").trim();
+  const subjectId = String(
+    request.headers["x-subject-id"] || request.headers["cf-access-authenticated-user-email"] || "",
+  ).trim();


Derive ticket claims from verified identity

When ZVoice is exposed behind the documented identity proxy, a same-origin caller can supply X-Tenant-Id and X-Subject-Id directly; these browser-settable values are accepted even with anonymous access disabled, and X-Subject-Id takes precedence over the Cloudflare-authenticated identity. The server then uses its service token to mint a valid ticket for the attacker-selected tenant and subject, so these claims must instead come from verified middleware or proxy headers that clients cannot override.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:14Z

+      socket.setTimeout(0);
+      upstreamSocket.setTimeout(0);


Enforce a finite WebSocket session timeout

When a client opens a valid WebSocket and then remains idle or an upstream connection stalls, both sockets explicitly have timeouts disabled and the admission slot is released only on close/error. Such clients can therefore occupy every VOICE_MAX_SESSIONS slot indefinitely, causing all later sessions to receive 429 responses; enforce an idle or maximum-session deadline that closes both sockets and add timeout-path coverage.

AGENTS.md reference: AGENTS.md:L35-L38

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:14Z

+    const session = await requestVoiceSession();
+    if (modelField) modelField.value = session.model || modelField.value;
+    await startCapture();
+
+    socket = new WebSocket(session.websocket_url, [`zticket.${session.ticket}`]);


Request the ticket after microphone setup

On first use, the browser requests the short-lived ticket before awaiting the microphone permission prompt and AudioWorklet setup. If the user takes longer than the configured ticket TTL—60 seconds by default—the subsequent WebSocket handshake is rejected as expired, even though the user just approved access; complete capture setup before requesting the ticket, or refresh it immediately before opening the socket.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:14Z

  "scripts": {
    "build": "pnpm -r --if-present build",
-    "test": "pnpm -r --workspace-concurrency=1 --if-present test && node --test scripts/test/*.test.mjs",
+    "test": "pnpm -r --workspace-concurrency=1 --if-present test && node --test scripts/test/*.test.mjs apps/zvoice/test/*.test.mjs services/voice-gateway/test/*.test.mjs",


Run the new voice tests in CI

The added root test command is not invoked by either checked workflow: .github/workflows/ci.yml and .github/workflows/validate.yml run npm test only for fixed workspace lists, while neither apps/zvoice nor services/voice-gateway has a package.json or appears in those lists. Consequently the ticket-authentication and browser-boundary tests added by this commit never run in GitHub Actions, allowing regressions in these security controls to pass the release gate; add explicit workflow coverage or package manifests that the workflow executes.

AGENTS.md reference: AGENTS.md:L41-L43

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:14Z

+function serializeUpgradeRequest(request, target, claims, forwardedProtocols) {
+  const headers = { ...request.headers };
+  delete headers["proxy-connection"];
+  delete headers["content-length"];
+  delete headers["sec-websocket-protocol"];


Allowlist headers forwarded to the voice agent

When the gateway is deployed behind Cloudflare Access or another cookie-based identity proxy, cloning the entire upgrade request forwards browser cookies, Access assertions, authorization headers, and proxy identity metadata into the less-trusted speech runtime. None of these credentials is needed after the gateway verifies the signed ticket, so construct an allowlist containing only the WebSocket handshake headers and the three synthesized Z Platform identity fields.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:14Z

+  const url = new URL(request.url || "/", "http://voice-gateway.local");
+  return {
+    ticket: url.searchParams.get("ticket"),
+    forwardedProtocols: protocols,


Remove the query-string ticket fallback

When a client uses the fallback ?ticket=... transport, reverse proxies and access-log middleware commonly record the full WebSocket URL, placing a still-valid tenant-bearing credential in logs where another reader can race the intended client. The browser already uses the documented Sec-WebSocket-Protocol transport, so reject query-string tickets rather than silently accepting this leak-prone alternative.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:14Z

+RUN python -m pip install --no-cache-dir --upgrade pip setuptools wheel \
+    && python -m pip install --no-cache-dir \
+       "qwentts-cpp-python==${QWENTTS_CPP_VERSION}" \
+       --find-links "${QWENTTS_WHEEL_INDEX}" \
+    && python -m pip install --no-cache-dir \
+       "speech-to-speech[faster-whisper]==${SPEECH_TO_SPEECH_VERSION}"


Include voice-agent dependencies in audit and SBOM gates

The new Python runtime installs its production dependencies only through Dockerfile pip install commands, with no requirements or lock manifest. The checked dependency workflow inventories only package.json files, while the Python audit job covers only apps/zaicoder/backend, so speech-to-speech, qwentts-cpp-python, and their dynamically resolved transitive dependencies are absent from dependency-policy and repository SBOM verification; record a reproducible dependency manifest and include it in the audit/SBOM gates.

AGENTS.md reference: AGENTS.md:L41-L43

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:15Z

+## Status
+
+Initial production-oriented vertical slice. External traffic remains disabled by default; published ports bind to loopback.


Add voice services to canonical production controls

Although this document declares a new production-oriented application and two deployable services, the checked docs/operations/production-master.md, docs/operations/staging-readiness.md, and docs/requirements/master-requirements.md contain no voice service, requirement, readiness, rollback, or release-gate entries. The migration manifest and architecture index likewise do not register the new boundary, so the repository's canonical production review cannot track the conditions listed only in this standalone document; update those control documents before treating the slice as releaseable.

AGENTS.md reference: AGENTS.md:L45-L53

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-02T02:04:15Z

+  const ticketProtocol = protocols.find((value) => value.startsWith("zticket."));
+  if (ticketProtocol) {
+    return {
+      ticket: ticketProtocol.slice("zticket.".length),
+      forwardedProtocols: protocols.filter((value) => !value.startsWith("zticket.")),
+    };


Complete the ticket subprotocol handshake

The browser opens the socket with only zticket.<ticket> as its requested WebSocket subprotocol, but the gateway removes that value before forwarding the upgrade, so a compliant upstream returns no Sec-WebSocket-Protocol selection. Browsers then reject the otherwise successful 101 response because none of their requested protocols was negotiated, making the documented ZVoice connection fail before any realtime events can flow; terminate or rewrite the handshake so the gateway acknowledges an accepted non-secret protocol without exposing the ticket upstream.

Useful? React with 👍 / 👎.

cvsz added 21 commits August 2, 2026 05:21

feat(zvoice): add application container

6878f5f

docs(zvoice): add application overview

99441dc

feat(voice-agent): add socket health check

9907e1d

feat(voice-gateway): add gateway container

95080bb

test(voice): add HTTP smoke script

73848da

feat(voice-agent): add speech runtime container

618399a

feat(voice-agent): add configurable runtime entrypoint

b348f7c

config(voice): add local runtime example

c7fbec4

test(zvoice): cover secure session boundary

9347366

test(voice-gateway): cover ticket security contract

ace7f04

feat(voice): add compose runtime profiles

b0b7bd0

docs(voice): add architecture design

7ce75f3

docs(voice): add operations runbook

2a502fe

feat(zvoice): add secure voice session server

28854d9

feat(zvoice): add realtime voice interface

bd60199

style(zvoice): add responsive voice UI

9f56dbc

feat(zvoice): add microphone worklet

3e57814

feat(zvoice): add realtime audio client

fae5246

feat(voice-gateway): add secure realtime tunnel

c6da8bd

docs: document local realtime voice stack

3986634

test: include voice stack tests

eb6429a

github-advanced-security AI found potential problems Aug 1, 2026

View reviewed changes

Comment thread services/voice-gateway/index.mjs

export function parseBearer(header) {

if (typeof header !== "string") return null;

const match = /^Bearer\s+(.+)$/i.exec(header.trim());

cvsz marked this pull request as ready for review August 2, 2026 01:52

cvsz merged commit 6e8c507 into main Aug 2, 2026
13 of 14 checks passed

chatgpt-codex-connector Bot reviewed Aug 2, 2026

View reviewed changes

		## Status

		Initial production-oriented vertical slice. External traffic remains disabled by default; published ports bind to loopback.

Conversation

cvsz commented Aug 1, 2026

Summary

Architecture

Security properties

Validation completed

Local run — Ollama

Remaining release checks

Uh oh!

gemini-code-assist Bot commented Aug 1, 2026

Uh oh!

ecc-tools Bot commented Aug 1, 2026

Uh oh!

gemini-code-assist Bot commented Aug 2, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants