From 1fc4ea6f507f3fa643d15dba3e1e83804d94da6d Mon Sep 17 00:00:00 2001 From: Jonathan Locala Date: Wed, 15 Jul 2026 12:46:22 -0400 Subject: [PATCH 1/3] docs(hey-ozwell): flagship Overview rewrite + autodocs across the voice stories The Overview predated most of what shipped; other components get rich auto-generated docs pages via autodocs, which the Hey Ozwell stories never enabled. This brings the voice docs in line with the rest of the library: - HeyOzwell.mdx rewritten as the flagship overview: quick start (drop-in + headless hook), the activation flow as the user experiences it, a full options table, Visit Scribe, a "what runs where" architecture table (AudioWorklet / main thread / Web Worker / network), models-caching-hosting table with the runtime overrides, backend integration seams (onSend / ozwellConfig / Ollama sentinel / proxy note), and a troubleshooting section (mic permission, first-load download, strict-CSP worker-src, doctor-gate tuning, benign ORT warnings). - tags: ['autodocs'] on the six product-facing story files (+ component on Demo and Hands-Free Chat) so each surface gets the standard generated docs page with props tables from the existing JSDoc. The Speaker Verify dev diagnostic intentionally stays autodocs-free. Docs-only; no component code changed. Verified: storybook serves all 7 docs pages, story tests 11/11, prettier/tsc/eslint clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../AI/HeyOzwell/HandsFreeChat.stories.tsx | 2 + src/components/AI/HeyOzwell/HeyOzwell.mdx | 244 ++++++++++-------- .../AI/HeyOzwell/HeyOzwell.stories.tsx | 2 + .../AI/HeyOzwell/VisitScribe.stories.tsx | 1 + .../AI/HeyOzwell/VoiceManager.stories.tsx | 1 + .../AI/HeyOzwell/VoiceSetup.stories.tsx | 1 + .../HeyOzwell/WakeWord/WakeWord.stories.tsx | 1 + 7 files changed, 138 insertions(+), 114 deletions(-) diff --git a/src/components/AI/HeyOzwell/HandsFreeChat.stories.tsx b/src/components/AI/HeyOzwell/HandsFreeChat.stories.tsx index 1a91c17d0..f93f8846e 100644 --- a/src/components/AI/HeyOzwell/HandsFreeChat.stories.tsx +++ b/src/components/AI/HeyOzwell/HandsFreeChat.stories.tsx @@ -10,6 +10,8 @@ import { suggestedActions } from '../storyData'; const meta: Meta = { title: 'Product/Feature Modules/AI/Hey Ozwell/Hands-Free Chat', + component: HandsFreeChat, + tags: ['autodocs'], parameters: { layout: 'fullscreen', docs: { diff --git a/src/components/AI/HeyOzwell/HeyOzwell.mdx b/src/components/AI/HeyOzwell/HeyOzwell.mdx index 6ab2079e8..21af732f0 100644 --- a/src/components/AI/HeyOzwell/HeyOzwell.mdx +++ b/src/components/AI/HeyOzwell/HeyOzwell.mdx @@ -5,55 +5,47 @@ import * as HeyOzwellStories from './HeyOzwell.stories'; # Hey Ozwell -Hey Ozwell is the **voice entry point** for the assistant. Everything runs -**on-device** — wake-word detection, speaker verification, dictation, and -ambient-visit scribing (diarization) all happen in the browser, and **audio -never leaves the page**. +Hey Ozwell is the **voice entry point** for the assistant: wake word, speaker +verification, dictation, hands-free chat, and ambient visit scribing — all +running **on-device, in the browser**. **Audio never leaves the page.** The +only network dependency is the assistant's text reply (any OpenAI-compatible +endpoint), which makes the stack PHI-safe by default. -This page documents how the UX works (issue #287). The controls below are live — -click the octopus. +## Quick start + +```tsx +import { HeyOzwell } from '@mieweb/ui'; + +// Drop the octopus into your header — that's the whole integration. +; +``` + +Click the octopus to turn Ozwell on, say **"hey ozwell"** to dictate, and +**"ozwell I'm done"** to transcribe + send. For custom layouts, use the +headless hook and compose the parts yourself: + +```tsx +import { useHeyOzwell, HeyOzwellToggle } from '@mieweb/ui'; + +const oz = useHeyOzwell({ autoDictateOnWake: true, requireDoctor: true }); +// oz.toggleProps → , oz.chatProps → , plus +// send/startDictation/stopDictation for fully custom surfaces. +``` ## The flow -1. **The Ozwell octopus lives in the top bar.** When Hey Ozwell is **off**, it's - gray and muted. -2. **Click it to activate.** The octopus turns full colour and **pulses with the - room volume** — the same mic-reactive glow as the Voice Setup enrollment screen. -3. **While active, the floating chat button appears** in the bottom-right corner. - This is the library's existing `FloatingAIChat` — the normal `AIChat`. -4. **From there it behaves like Hands-Free Chat:** say **"hey ozwell"** to start - dictating, **"ozwell I'm done"** to transcribe on-device and send the message - to the assistant. (You can also open the chat and type.) -5. **Turn Ozwell off** and the floating button disappears. - -So the header toggle gates the floating chat button: no floating button unless -Ozwell is active in the header. - -### Options - -- **Auto-dictate on wake** (Controls panel) — ON, "hey ozwell" opens the chat *and* - starts dictating (hands-free). OFF, "hey ozwell" just **opens the chat and waits**, - for experimenting with more specific wake-word directions instead of always going - into transcription. -- **Close chat on "I'm done"** (Controls panel) — after "ozwell I'm done" sends, also - close the chat popup, or leave it open to read the reply. -- **Transcription** (Controls panel) — **browser** runs Whisper **on-device, in a Web Worker** - (audio never leaves the page — the PHI-safe default; the worker keeps the heavy model load and - inference off the main thread, so the wake word stays responsive while it loads); **server** POSTs - the audio to the OpenAI-compatible `/v1/audio/transcriptions` endpoint (audio leaves the browser; - falls back to on-device if it fails). An experiment toggle for comparing the two. -- **Require doctor** (Controls panel) — the speaker-verify gate: only the **enrolled clinician's - voice** triggers the wake word and dictation (an invisible WHO + WHAT check on every fire). Off, - anyone's "hey ozwell" works. -- **Conversation mode** (Controls panel) — diarize the dictated clip on send so the assistant gets - **who-said-what** ("Dr. Jane: … / Patient: …") instead of a flat transcript. -- **Live caption** (Controls panel) — show a rough real-time transcript in the composer as you - speak; the clean final text still re-transcribes the whole clip on send. -- **Review before send** (Controls panel) — drop the finished transcript into the composer to - **glance/edit before it sends**, instead of auto-firing. -- **Ozwell settings** — **right-click or long-press** the octopus to open a settings - menu that links back to Voice enrollment (set up / re-enroll / add a condition) and - the wake-word / speaker-verify diagnostics. +1. **The octopus lives in the top bar.** Off, it sits gray and muted. +2. **Click to activate.** A ring fills while the wake detector starts; it + completes **green the moment the mic is live** — ring done means "you can + talk now". The heavier transcription model keeps loading on a thin outer + arc, but it never gates you: audio buffers and the transcript catches up. +3. **Say "hey ozwell".** The chat opens and (with auto-dictate) starts + recording. The octopus pulses with the room volume. +4. **Say "ozwell I'm done".** The clip is trimmed (so the stop phrase never + reaches the transcript), transcribed on-device, and sent. The reply streams + into the floating chat. +5. **Turn Ozwell off** and the mic is released — gray octopus means the mic is + genuinely off. @@ -63,82 +55,106 @@ Off (gray) → active and quiet → active and hearing speech: -## How it's wired - -`HeyOzwellToggle` is **presentational** — it never opens the microphone itself. -That matters because there is **one shared mic**: the wake-word detector opens -`getUserMedia` once and everything else reads from it via `getStream()` (a second -`getUserMedia` would silence the detector). So the host: +## Options -- gates the detector on the toggle with `useWakeWord({ enabled: active })`, -- feeds the live room volume as `level` (0..1) from an analyser on the detector's - **shared stream** — the same pulse source as Voice Setup, -- on `onWake`, dictates ("hey ozwell" → record) and sends ("ozwell I'm done" → - transcribe + send), and renders the floating chat **only while active**. +All options are props on `` / `useHeyOzwell` (live in the Demo's +Controls panel): -The dictation recorder attaches to the detector's **same** `getStream()` — never a -second `getUserMedia`, which would silence the listener. This is the Hands-Free -Chat wiring, driving the library `FloatingAIChat`: +| Option | What it does | +| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `autoDictateOnWake` | ON: "hey ozwell" opens the chat **and** starts dictating. OFF: it just opens the chat and waits. | +| `requireDoctor` | The speaker-verify gate: every wake is checked invisibly against the **enrolled clinician's voiceprint** (WHO) _and_ a phrase-print (WHAT) — anyone else is ignored. Open until a voice is enrolled. | +| `transcription` | `'browser'` (default) runs Whisper on-device in a **Web Worker**; `'server'` POSTs audio to an OpenAI-compatible `/v1/audio/transcriptions` endpoint (audio leaves the browser — falls back to on-device on failure). | +| `conversationMode` | On "done", **diarize** the clip and send who-said-what ("Dr. Jane: … / Patient: …") instead of a flat transcript. | +| `liveTranscript` | Show a rough real-time caption in the composer while dictating; the final send still re-transcribes the full clip. | +| `reviewBeforeSend` | Drop the finished transcript into the composer to glance/edit before sending, instead of auto-firing. | +| `closeChatOnDone` | Close the chat popup after the dictated message sends. | +| `onSend` | Route sends to your own backend/message store instead of the built-in client. | +| `assetBase` | Override where the wake/speaker models load from. | -```tsx -const wake = useWakeWord({ - enabled: active, - onWake: (name) => { - if (name === 'hey-ozwell' && phase === 'listening') { setChatOpen(true); startDictation(); } - else if (name === "ozwell-i'm-done" && phase === 'dictating') { stopDictation(); } // → transcribe + send - }, -}); - -// pulse from the SAME mic the detector opened — no second getUserMedia -const level = useRoomLevel(wake.getStream, wake.ready); - -function startDictation() { - const rec = new MediaRecorder(wake.getStream()); // the listener's own stream - rec.ondataavailable = (e) => chunks.push(e.data); - rec.start(); -} -function stopDictation() { - rec.onstop = async () => send(stripStopPhrase(await transcribeBlob(new Blob(chunks)))); - rec.stop(); -} - -// header + the existing library floating chat, gated on `active`: - -{active && ( - -)} -``` - -Transcription runs in a **Web Worker** (see `whisperTranscribe.ts`), so loading the ~1.3 GB Whisper -model never blocks the main thread where the wake detector does its own on-device inference — "hey -ozwell" keeps responding while the model loads. Same model and output as before; the public API is -unchanged, so every surface (dictation, live caption, Visit Scribe, diarization) gets it for free. +Right-click (or long-press) the octopus for **Ozwell settings** — voice +enrollment and the live "Models & versions" readout. ## Visit Scribe (ambient scribe) -A second surface, for **multi-person encounters** — no wake word, no dictation turns. Just record the -room, stop, and get an **on-device, speaker-labeled transcript** (Doctor / Patient / Nurse). -Attribution comes from `useDiarization`: Whisper timestamps + TitaNet speaker embeddings clustered -into turns, anchored to enrolled voiceprints, with optional LLM role inference. You can **rename, -merge, and reassign speakers** afterward, and optionally show a rough **live transcript** while -recording. Same guarantee as everything else here — audio never leaves the page. +The second surface, for **multi-person encounters** — no wake word, no turns. +Record the room, stop, and get an **on-device speaker-labeled transcript** +(Doctor / Patient / Nurse): Whisper timestamps + TitaNet speaker embeddings → +clustering → anchored to enrolled voiceprints, with optional LLM role +inference. Rename, merge, and reassign speakers afterward; optionally show a +rough live transcript while recording. Same guarantee — audio never leaves +the page. See _Visit Scribe_ in this section. + +## How it works — what runs where + +One shared mic: the wake detector opens `getUserMedia` once and every other +consumer (dictation recorder, volume meter, verification buffer) reads from +`getStream()` — a second `getUserMedia` would silence the detector. + +| Stage | Tech | Where it runs | +| -------------------------------------------------------------------- | ---------------------------------------------- | -------------- | +| Mic capture + 16 kHz resample | AudioWorklet | Audio thread | +| Wake word ("hey ozwell" / "ozwell i'm done") | ONNX (onnxruntime-web, ~6 MB models) | Main thread | +| Speaker verification (WHO gate, AS-norm) | sherpa-onnx + TitaNet (~50 MB) | Main thread | +| Transcription (Whisper turbo ~1.3 GB, WebGPU; base.en WASM fallback) | transformers.js | **Web Worker** | +| Diarization clustering | Whisper timestamps + TitaNet embeddings | Worker + main | +| Assistant reply | OpenAI-compatible `/v1/chat/completions` (SSE) | Network | + +Transcription lives in a Web Worker so downloading/compiling the 1.3 GB model +never starves the main-thread wake detector — **"hey ozwell" responds while +Whisper is still loading**, and the buffered audio catches up. + +### Models, caching, hosting + +| Asset | Cache | Runtime override | +| ---------------------------- | --------------------------------------- | ------------------------------------------ | +| Wake models (~6 MB) | OPFS (pre-fetched on mount) | `assetBase` prop / `window.__ozwellAssets` | +| Speaker runtime (~50 MB) | Cache API (service worker) | `window.__ozwellAssets` | +| Whisper weights | Cache API (streamed by transformers.js) | `window.__ozwellWhisperHost` | +| transformers.js ESM (pinned) | browser cache | `window.__ozwellTransformersUrl` | + +Voiceprints persist in **IndexedDB**; `localStorage` holds config only. Every +host is config-swappable — no source edits to repoint models. + +## Backend integration + +The component is backend-agnostic. Two seams: + +- **`onSend` prop** — the hook hands you the final text; call whatever backend + you like and render the reply yourself. +- **`ozwellConfig`** — leave `onSend` off and the built-in client streams from + any OpenAI-compatible endpoint: `localStorage.ozwellConfig` / + `window.__ozwell` with `{ apiKey, baseURL, model, system, temperature }`. + The sentinel `apiKey: 'ollama'` targets a local Ollama server (no auth + header). With no key configured, the demo falls back to a canned reply — no + secret is ever committed. + +> ⚠️ A browser-held Bearer key is visible to anyone with the page. For public +> deploys, proxy the key server-side and point `baseURL` at the proxy. See +> `src/components/AI/OZWELL-BACKEND.md`. + +## Troubleshooting + +- **Octopus ring never completes** — mic permission denied or no input + device; the toggle surfaces "Microphone unavailable" after ~8 s. +- **First activation is slow** — the Whisper download (up to ~1.3 GB) happens + once, then it's served from the Cache API. The wake word works during the + download; only transcription waits. +- **Strict CSP hosts** — the worker needs `worker-src blob:`, plus the model + CDN allowances (or self-host everything via the overrides above). +- **"hey ozwell" ignored with the doctor gate on** — re-enroll in conditions + matching real use (same mic/room); the WHO gate is tunable via + `window.SpeakerVerify` thresholds. +- **Console `[W:onnxruntime …]` messages during model load** — harmless ONNX + runtime warnings about CPU-assigned graph nodes, logged at error level. ## Related -- **Voice Setup** — one-time speaker enrollment; the octopus-pulse pattern reused here. -- **Wake Word** — the on-device "hey ozwell" detector that drives the pulse and wake. -- **Hands-Free Chat** — the voice conversation flow. -- **Visit Scribe** — the ambient multi-person scribe described above. - -All live under *Product / Feature Modules / AI* in this sidebar. - -### Integration & backend +- **Voice Setup** — one-time speaker enrollment (the octopus-pulse pattern). +- **Voice Manager** — manage / rename / remove enrolled voices. +- **Wake Word** — the on-device detector primitive. +- **Hands-Free Chat** — the voice conversation composition. +- **Visit Scribe** — the ambient multi-person scribe. -- **Backend wiring** (`OZWELL-BACKEND.md`) — the assistant's reply is an OpenAI-compatible - `POST /v1/chat/completions` (ozwellai-api by default, or local Ollama via the `apiKey: 'ollama'` - sentinel). A host overrides it with the **`onSend`** prop or an `ozwellConfig` **`baseURL`** — the - component is backend-agnostic. Only the chat reply (and the opt-in server transcription) touch a - network; wake, speaker verify, Whisper, and diarization are all on-device. -- **Model hosting** (`MODEL-HOSTING.md`) — wake / speaker / Whisper models are fetched at runtime - from an asset host so the repo stays code-only; every host is runtime-overridable via config - (`window.__ozwellAssets`, `window.__ozwellWhisperHost`, `window.__ozwellTransformersUrl`). +Deeper docs in the repo: `MODEL-HOSTING.md` (asset hosting + cutover), +`OZWELL-BACKEND.md` (chat wiring), `DIARIZATION.md` (speaker attribution). diff --git a/src/components/AI/HeyOzwell/HeyOzwell.stories.tsx b/src/components/AI/HeyOzwell/HeyOzwell.stories.tsx index a028e0de5..3d4dc6c5b 100644 --- a/src/components/AI/HeyOzwell/HeyOzwell.stories.tsx +++ b/src/components/AI/HeyOzwell/HeyOzwell.stories.tsx @@ -21,6 +21,8 @@ import { suggestedActions } from '../storyData'; const meta: Meta = { title: 'Product/Feature Modules/AI/Hey Ozwell/Demo', + component: HeyOzwell, + tags: ['autodocs'], parameters: { layout: 'fullscreen', docs: { diff --git a/src/components/AI/HeyOzwell/VisitScribe.stories.tsx b/src/components/AI/HeyOzwell/VisitScribe.stories.tsx index 98c510828..5bde4736a 100644 --- a/src/components/AI/HeyOzwell/VisitScribe.stories.tsx +++ b/src/components/AI/HeyOzwell/VisitScribe.stories.tsx @@ -10,6 +10,7 @@ import { VisitScribe } from './VisitScribe'; const meta: Meta = { title: 'Product/Feature Modules/AI/Hey Ozwell/Visit Scribe', component: VisitScribe, + tags: ['autodocs'], parameters: { layout: 'fullscreen', docs: { diff --git a/src/components/AI/HeyOzwell/VoiceManager.stories.tsx b/src/components/AI/HeyOzwell/VoiceManager.stories.tsx index 60879f5d7..5627df4b1 100644 --- a/src/components/AI/HeyOzwell/VoiceManager.stories.tsx +++ b/src/components/AI/HeyOzwell/VoiceManager.stories.tsx @@ -9,6 +9,7 @@ import { VoiceManager } from './VoiceManager'; const meta: Meta = { title: 'Product/Feature Modules/AI/Hey Ozwell/Voice Manager', component: VoiceManager, + tags: ['autodocs'], parameters: { layout: 'fullscreen', docs: { diff --git a/src/components/AI/HeyOzwell/VoiceSetup.stories.tsx b/src/components/AI/HeyOzwell/VoiceSetup.stories.tsx index e0d6ddf01..3cd32708f 100644 --- a/src/components/AI/HeyOzwell/VoiceSetup.stories.tsx +++ b/src/components/AI/HeyOzwell/VoiceSetup.stories.tsx @@ -8,6 +8,7 @@ import { VoiceSetup } from './VoiceSetup'; const meta: Meta = { title: 'Product/Feature Modules/AI/Hey Ozwell/Voice Setup', component: VoiceSetup, + tags: ['autodocs'], parameters: { layout: 'fullscreen', docs: { diff --git a/src/components/AI/HeyOzwell/WakeWord/WakeWord.stories.tsx b/src/components/AI/HeyOzwell/WakeWord/WakeWord.stories.tsx index cbd5220b0..333d6617a 100644 --- a/src/components/AI/HeyOzwell/WakeWord/WakeWord.stories.tsx +++ b/src/components/AI/HeyOzwell/WakeWord/WakeWord.stories.tsx @@ -15,6 +15,7 @@ import { useWakeWord } from './useWakeWord'; const meta: Meta = { title: 'Product/Feature Modules/AI/Hey Ozwell/Wake Word', + tags: ['autodocs'], parameters: { layout: 'centered', docs: { From c648bf7e87adf262fce0b73a3c9c30e2f0c3f096 Mon Sep 17 00:00:00 2001 From: Jonathan Locala Date: Wed, 15 Jul 2026 12:59:31 -0400 Subject: [PATCH 2/3] docs(hey-ozwell): lists instead of pipe tables (no remark-gfm in this storybook), drop mid-sentence em dashes Pipe tables render as raw text in this Storybook's MDX (remark-gfm is not configured), so the options / architecture / hosting tables are now bulleted lists that render everywhere. Also reworded to remove mid-sentence em dashes. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/components/AI/HeyOzwell/HeyOzwell.mdx | 155 ++++++++++++---------- 1 file changed, 85 insertions(+), 70 deletions(-) diff --git a/src/components/AI/HeyOzwell/HeyOzwell.mdx b/src/components/AI/HeyOzwell/HeyOzwell.mdx index 21af732f0..7fae44f69 100644 --- a/src/components/AI/HeyOzwell/HeyOzwell.mdx +++ b/src/components/AI/HeyOzwell/HeyOzwell.mdx @@ -6,7 +6,7 @@ import * as HeyOzwellStories from './HeyOzwell.stories'; # Hey Ozwell Hey Ozwell is the **voice entry point** for the assistant: wake word, speaker -verification, dictation, hands-free chat, and ambient visit scribing — all +verification, dictation, hands-free chat, and ambient visit scribing, all running **on-device, in the browser**. **Audio never leaves the page.** The only network dependency is the assistant's text reply (any OpenAI-compatible endpoint), which makes the stack PHI-safe by default. @@ -16,12 +16,12 @@ endpoint), which makes the stack PHI-safe by default. ```tsx import { HeyOzwell } from '@mieweb/ui'; -// Drop the octopus into your header — that's the whole integration. +// Drop the octopus into your header. That's the whole integration. ; ``` Click the octopus to turn Ozwell on, say **"hey ozwell"** to dictate, and -**"ozwell I'm done"** to transcribe + send. For custom layouts, use the +**"ozwell I'm done"** to transcribe and send. For custom layouts, use the headless hook and compose the parts yourself: ```tsx @@ -35,17 +35,17 @@ const oz = useHeyOzwell({ autoDictateOnWake: true, requireDoctor: true }); ## The flow 1. **The octopus lives in the top bar.** Off, it sits gray and muted. -2. **Click to activate.** A ring fills while the wake detector starts; it - completes **green the moment the mic is live** — ring done means "you can +2. **Click to activate.** A ring fills while the wake detector starts, and it + completes **green the moment the mic is live**: ring done means "you can talk now". The heavier transcription model keeps loading on a thin outer - arc, but it never gates you: audio buffers and the transcript catches up. + arc, but it never gates you. Audio buffers and the transcript catches up. 3. **Say "hey ozwell".** The chat opens and (with auto-dictate) starts recording. The octopus pulses with the room volume. -4. **Say "ozwell I'm done".** The clip is trimmed (so the stop phrase never - reaches the transcript), transcribed on-device, and sent. The reply streams +4. **Say "ozwell I'm done".** The clip is trimmed so the stop phrase never + reaches the transcript, transcribed on-device, and sent. The reply streams into the floating chat. -5. **Turn Ozwell off** and the mic is released — gray octopus means the mic is - genuinely off. +5. **Turn Ozwell off** and the mic is released. A gray octopus means the mic + is genuinely off. @@ -60,74 +60,89 @@ Off (gray) → active and quiet → active and hearing speech: All options are props on `` / `useHeyOzwell` (live in the Demo's Controls panel): -| Option | What it does | -| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `autoDictateOnWake` | ON: "hey ozwell" opens the chat **and** starts dictating. OFF: it just opens the chat and waits. | -| `requireDoctor` | The speaker-verify gate: every wake is checked invisibly against the **enrolled clinician's voiceprint** (WHO) _and_ a phrase-print (WHAT) — anyone else is ignored. Open until a voice is enrolled. | -| `transcription` | `'browser'` (default) runs Whisper on-device in a **Web Worker**; `'server'` POSTs audio to an OpenAI-compatible `/v1/audio/transcriptions` endpoint (audio leaves the browser — falls back to on-device on failure). | -| `conversationMode` | On "done", **diarize** the clip and send who-said-what ("Dr. Jane: … / Patient: …") instead of a flat transcript. | -| `liveTranscript` | Show a rough real-time caption in the composer while dictating; the final send still re-transcribes the full clip. | -| `reviewBeforeSend` | Drop the finished transcript into the composer to glance/edit before sending, instead of auto-firing. | -| `closeChatOnDone` | Close the chat popup after the dictated message sends. | -| `onSend` | Route sends to your own backend/message store instead of the built-in client. | -| `assetBase` | Override where the wake/speaker models load from. | - -Right-click (or long-press) the octopus for **Ozwell settings** — voice +- **`autoDictateOnWake`**: ON, "hey ozwell" opens the chat **and** starts + dictating. OFF, it just opens the chat and waits. +- **`requireDoctor`**: the speaker-verify gate. Every wake is checked + invisibly against the **enrolled clinician's voiceprint** (WHO) and a + phrase-print (WHAT); anyone else is ignored. Stays open until a voice is + enrolled. +- **`transcription`**: `'browser'` (default) runs Whisper on-device in a + **Web Worker**. `'server'` POSTs audio to an OpenAI-compatible + `/v1/audio/transcriptions` endpoint (audio leaves the browser; falls back + to on-device on failure). +- **`conversationMode`**: on "done", **diarize** the clip and send + who-said-what ("Dr. Jane: … / Patient: …") instead of a flat transcript. +- **`liveTranscript`**: show a rough real-time caption in the composer while + dictating. The final send still re-transcribes the full clip. +- **`reviewBeforeSend`**: drop the finished transcript into the composer to + glance and edit before sending, instead of auto-firing. +- **`closeChatOnDone`**: close the chat popup after the dictated message + sends. +- **`onSend`**: route sends to your own backend or message store instead of + the built-in client. +- **`assetBase`**: override where the wake/speaker models load from. + +Right-click (or long-press) the octopus for **Ozwell settings**: voice enrollment and the live "Models & versions" readout. ## Visit Scribe (ambient scribe) -The second surface, for **multi-person encounters** — no wake word, no turns. -Record the room, stop, and get an **on-device speaker-labeled transcript** -(Doctor / Patient / Nurse): Whisper timestamps + TitaNet speaker embeddings → -clustering → anchored to enrolled voiceprints, with optional LLM role -inference. Rename, merge, and reassign speakers afterward; optionally show a -rough live transcript while recording. Same guarantee — audio never leaves -the page. See _Visit Scribe_ in this section. +The second surface, for **multi-person encounters**. No wake word, no turns: +record the room, stop, and get an **on-device speaker-labeled transcript** +(Doctor / Patient / Nurse). Under the hood it's Whisper timestamps plus +TitaNet speaker embeddings, clustered into turns and anchored to enrolled +voiceprints, with optional LLM role inference. Rename, merge, and reassign +speakers afterward, and optionally show a rough live transcript while +recording. Same guarantee: audio never leaves the page. See _Visit Scribe_ +in this section. -## How it works — what runs where +## How it works: what runs where One shared mic: the wake detector opens `getUserMedia` once and every other consumer (dictation recorder, volume meter, verification buffer) reads from -`getStream()` — a second `getUserMedia` would silence the detector. - -| Stage | Tech | Where it runs | -| -------------------------------------------------------------------- | ---------------------------------------------- | -------------- | -| Mic capture + 16 kHz resample | AudioWorklet | Audio thread | -| Wake word ("hey ozwell" / "ozwell i'm done") | ONNX (onnxruntime-web, ~6 MB models) | Main thread | -| Speaker verification (WHO gate, AS-norm) | sherpa-onnx + TitaNet (~50 MB) | Main thread | -| Transcription (Whisper turbo ~1.3 GB, WebGPU; base.en WASM fallback) | transformers.js | **Web Worker** | -| Diarization clustering | Whisper timestamps + TitaNet embeddings | Worker + main | -| Assistant reply | OpenAI-compatible `/v1/chat/completions` (SSE) | Network | - -Transcription lives in a Web Worker so downloading/compiling the 1.3 GB model -never starves the main-thread wake detector — **"hey ozwell" responds while -Whisper is still loading**, and the buffered audio catches up. +`getStream()`. A second `getUserMedia` would silence the detector. + +- **Mic capture + 16 kHz resample**: AudioWorklet, on the audio thread. +- **Wake word** ("hey ozwell" / "ozwell i'm done"): ONNX via onnxruntime-web + (~6 MB of models), on the main thread. +- **Speaker verification** (the WHO gate, AS-norm scoring): sherpa-onnx + + TitaNet (~50 MB), on the main thread. +- **Transcription**: Whisper turbo (~1.3 GB, WebGPU; base.en WASM fallback) + via transformers.js, in a **Web Worker**. +- **Diarization**: Whisper timestamps + TitaNet embeddings, worker + main. +- **Assistant reply**: OpenAI-compatible `/v1/chat/completions` (SSE + streaming). This is the only network stage. + +Transcription lives in a Web Worker so downloading and compiling the 1.3 GB +model never starves the main-thread wake detector. **"Hey ozwell" responds +while Whisper is still loading**, and the buffered audio catches up. ### Models, caching, hosting -| Asset | Cache | Runtime override | -| ---------------------------- | --------------------------------------- | ------------------------------------------ | -| Wake models (~6 MB) | OPFS (pre-fetched on mount) | `assetBase` prop / `window.__ozwellAssets` | -| Speaker runtime (~50 MB) | Cache API (service worker) | `window.__ozwellAssets` | -| Whisper weights | Cache API (streamed by transformers.js) | `window.__ozwellWhisperHost` | -| transformers.js ESM (pinned) | browser cache | `window.__ozwellTransformersUrl` | +- **Wake models** (~6 MB): cached in OPFS, pre-fetched on mount. Override + with the `assetBase` prop or `window.__ozwellAssets`. +- **Speaker runtime** (~50 MB): cached via the Cache API (service worker). + Override with `window.__ozwellAssets`. +- **Whisper weights**: streamed into the Cache API by transformers.js. + Override with `window.__ozwellWhisperHost`. +- **transformers.js ESM** (pinned version): browser cache. Override with + `window.__ozwellTransformersUrl`. Voiceprints persist in **IndexedDB**; `localStorage` holds config only. Every -host is config-swappable — no source edits to repoint models. +host is config-swappable, so repointing models requires no source edits. ## Backend integration The component is backend-agnostic. Two seams: -- **`onSend` prop** — the hook hands you the final text; call whatever backend +- **`onSend` prop**: the hook hands you the final text. Call whatever backend you like and render the reply yourself. -- **`ozwellConfig`** — leave `onSend` off and the built-in client streams from - any OpenAI-compatible endpoint: `localStorage.ozwellConfig` / - `window.__ozwell` with `{ apiKey, baseURL, model, system, temperature }`. +- **`ozwellConfig`**: leave `onSend` off and the built-in client streams from + any OpenAI-compatible endpoint. Configure via `localStorage.ozwellConfig` + or `window.__ozwell` with `{ apiKey, baseURL, model, system, temperature }`. The sentinel `apiKey: 'ollama'` targets a local Ollama server (no auth - header). With no key configured, the demo falls back to a canned reply — no - secret is ever committed. + header). With no key configured, the demo falls back to a canned reply, so + no secret is ever committed. > ⚠️ A browser-held Bearer key is visible to anyone with the page. For public > deploys, proxy the key server-side and point `baseURL` at the proxy. See @@ -135,26 +150,26 @@ The component is backend-agnostic. Two seams: ## Troubleshooting -- **Octopus ring never completes** — mic permission denied or no input - device; the toggle surfaces "Microphone unavailable" after ~8 s. -- **First activation is slow** — the Whisper download (up to ~1.3 GB) happens +- **Octopus ring never completes**: mic permission denied or no input + device. The toggle surfaces "Microphone unavailable" after ~8 s. +- **First activation is slow**: the Whisper download (up to ~1.3 GB) happens once, then it's served from the Cache API. The wake word works during the download; only transcription waits. -- **Strict CSP hosts** — the worker needs `worker-src blob:`, plus the model +- **Strict CSP hosts**: the worker needs `worker-src blob:`, plus the model CDN allowances (or self-host everything via the overrides above). -- **"hey ozwell" ignored with the doctor gate on** — re-enroll in conditions - matching real use (same mic/room); the WHO gate is tunable via +- **"hey ozwell" ignored with the doctor gate on**: re-enroll in conditions + matching real use (same mic, same room). The WHO gate is tunable via `window.SpeakerVerify` thresholds. -- **Console `[W:onnxruntime …]` messages during model load** — harmless ONNX +- **Console `[W:onnxruntime …]` messages during model load**: harmless ONNX runtime warnings about CPU-assigned graph nodes, logged at error level. ## Related -- **Voice Setup** — one-time speaker enrollment (the octopus-pulse pattern). -- **Voice Manager** — manage / rename / remove enrolled voices. -- **Wake Word** — the on-device detector primitive. -- **Hands-Free Chat** — the voice conversation composition. -- **Visit Scribe** — the ambient multi-person scribe. +- **Voice Setup**: one-time speaker enrollment (the octopus-pulse pattern). +- **Voice Manager**: manage, rename, and remove enrolled voices. +- **Wake Word**: the on-device detector primitive. +- **Hands-Free Chat**: the voice conversation composition. +- **Visit Scribe**: the ambient multi-person scribe. Deeper docs in the repo: `MODEL-HOSTING.md` (asset hosting + cutover), `OZWELL-BACKEND.md` (chat wiring), `DIARIZATION.md` (speaker attribution). From c12fc9ba728e1a1d71f42cab46d78ec53611c5f0 Mon Sep 17 00:00:00 2001 From: Jonathan Locala Date: Thu, 16 Jul 2026 12:32:38 -0400 Subject: [PATCH 3/3] docs(hey-ozwell): embed the 80-second video walkthrough in the Overview First video embed in the library's docs: an 80s walkthrough of activation, dictation during model load, on-device transcription, the voice stop phrase, Visit Scribe diarization, and the one-line integration. Plain YouTube link included as a fallback for environments that block the iframe. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/components/AI/HeyOzwell/HeyOzwell.mdx | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/components/AI/HeyOzwell/HeyOzwell.mdx b/src/components/AI/HeyOzwell/HeyOzwell.mdx index 7fae44f69..493838e2a 100644 --- a/src/components/AI/HeyOzwell/HeyOzwell.mdx +++ b/src/components/AI/HeyOzwell/HeyOzwell.mdx @@ -11,6 +11,22 @@ running **on-device, in the browser**. **Audio never leaves the page.** The only network dependency is the assistant's text reply (any OpenAI-compatible endpoint), which makes the stack PHI-safe by default. +**80-second walkthrough** of everything on this page: + +