Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/components/AI/HeyOzwell/HandsFreeChat.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ import { suggestedActions } from '../storyData';

const meta: Meta = {
title: 'Product/Feature Modules/AI/Hey Ozwell/Hands-Free Chat',
component: HandsFreeChat,
tags: ['autodocs'],
parameters: {
layout: 'fullscreen',
docs: {
Expand Down
281 changes: 164 additions & 117 deletions src/components/AI/HeyOzwell/HeyOzwell.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,55 +5,63 @@ import * as HeyOzwellStories from './HeyOzwell.stories';

# Hey Ozwell

Hey Ozwell is the **voice entry point** for the assistant. Everything runs
**on-device** — wake-word detection, speaker verification, dictation, and
ambient-visit scribing (diarization) all happen in the browser, and **audio
never leaves the page**.
Hey Ozwell is the **voice entry point** for the assistant: wake word, speaker
verification, dictation, hands-free chat, and ambient visit scribing, all
running **on-device, in the browser**. **Audio never leaves the page.** The
only network dependency is the assistant's text reply (any OpenAI-compatible
endpoint), which makes the stack PHI-safe by default.
Comment on lines +8 to +12

**80-second walkthrough** of everything on this page:

<iframe
width="315"
height="560"
src="https://www.youtube.com/embed/hQBGofjXqmg"
title="Hey Ozwell Demo"
frameBorder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowFullScreen
style={{ borderRadius: 12, maxWidth: '100%' }}
/>
Comment on lines +16 to +25

If the embed doesn't load in your environment:
[watch on YouTube](https://www.youtube.com/shorts/hQBGofjXqmg).

## Quick start

This page documents how the UX works (issue #287). The controls below are live —
click the octopus.
```tsx
import { HeyOzwell } from '@mieweb/ui';

// Drop the octopus into your header. That's the whole integration.
<HeyOzwell chatProps={{ userName: 'Dr. Jane' }} />;
```

Click the octopus to turn Ozwell on, say **"hey ozwell"** to dictate, and
**"ozwell I'm done"** to transcribe and send. For custom layouts, use the
headless hook and compose the parts yourself:

```tsx
import { useHeyOzwell, HeyOzwellToggle } from '@mieweb/ui';

const oz = useHeyOzwell({ autoDictateOnWake: true, requireDoctor: true });
// oz.toggleProps → <HeyOzwellToggle>, oz.chatProps → <FloatingAIChat>, plus
// send/startDictation/stopDictation for fully custom surfaces.
```

## The flow

1. **The Ozwell octopus lives in the top bar.** When Hey Ozwell is **off**, it's
gray and muted.
2. **Click it to activate.** The octopus turns full colour and **pulses with the
room volume** — the same mic-reactive glow as the Voice Setup enrollment screen.
3. **While active, the floating chat button appears** in the bottom-right corner.
This is the library's existing `FloatingAIChat` — the normal `AIChat`.
4. **From there it behaves like Hands-Free Chat:** say **"hey ozwell"** to start
dictating, **"ozwell I'm done"** to transcribe on-device and send the message
to the assistant. (You can also open the chat and type.)
5. **Turn Ozwell off** and the floating button disappears.

So the header toggle gates the floating chat button: no floating button unless
Ozwell is active in the header.

### Options

- **Auto-dictate on wake** (Controls panel) — ON, "hey ozwell" opens the chat *and*
starts dictating (hands-free). OFF, "hey ozwell" just **opens the chat and waits**,
for experimenting with more specific wake-word directions instead of always going
into transcription.
- **Close chat on "I'm done"** (Controls panel) — after "ozwell I'm done" sends, also
close the chat popup, or leave it open to read the reply.
- **Transcription** (Controls panel) — **browser** runs Whisper **on-device, in a Web Worker**
(audio never leaves the page — the PHI-safe default; the worker keeps the heavy model load and
inference off the main thread, so the wake word stays responsive while it loads); **server** POSTs
the audio to the OpenAI-compatible `/v1/audio/transcriptions` endpoint (audio leaves the browser;
falls back to on-device if it fails). An experiment toggle for comparing the two.
- **Require doctor** (Controls panel) — the speaker-verify gate: only the **enrolled clinician's
voice** triggers the wake word and dictation (an invisible WHO + WHAT check on every fire). Off,
anyone's "hey ozwell" works.
- **Conversation mode** (Controls panel) — diarize the dictated clip on send so the assistant gets
**who-said-what** ("Dr. Jane: … / Patient: …") instead of a flat transcript.
- **Live caption** (Controls panel) — show a rough real-time transcript in the composer as you
speak; the clean final text still re-transcribes the whole clip on send.
- **Review before send** (Controls panel) — drop the finished transcript into the composer to
**glance/edit before it sends**, instead of auto-firing.
- **Ozwell settings** — **right-click or long-press** the octopus to open a settings
menu that links back to Voice enrollment (set up / re-enroll / add a condition) and
the wake-word / speaker-verify diagnostics.
1. **The octopus lives in the top bar.** Off, it sits gray and muted.
2. **Click to activate.** A ring fills while the wake detector starts, and it
completes **green the moment the mic is live**: ring done means "you can
talk now". The heavier transcription model keeps loading on a thin outer
arc, but it never gates you. Audio buffers and the transcript catches up.
3. **Say "hey ozwell".** The chat opens and (with auto-dictate) starts
recording. The octopus pulses with the room volume.
4. **Say "ozwell I'm done".** The clip is trimmed so the stop phrase never
reaches the transcript, transcribed on-device, and sent. The reply streams
into the floating chat.
5. **Turn Ozwell off** and the mic is released. A gray octopus means the mic
is genuinely off.

<Canvas of={HeyOzwellStories.Interactive} />

Expand All @@ -63,82 +71,121 @@ Off (gray) → active and quiet → active and hearing speech:

<Canvas of={HeyOzwellStories.ToggleStates} />

## How it's wired

`HeyOzwellToggle` is **presentational** — it never opens the microphone itself.
That matters because there is **one shared mic**: the wake-word detector opens
`getUserMedia` once and everything else reads from it via `getStream()` (a second
`getUserMedia` would silence the detector). So the host:

- gates the detector on the toggle with `useWakeWord({ enabled: active })`,
- feeds the live room volume as `level` (0..1) from an analyser on the detector's
**shared stream** — the same pulse source as Voice Setup,
- on `onWake`, dictates ("hey ozwell" → record) and sends ("ozwell I'm done" →
transcribe + send), and renders the floating chat **only while active**.

The dictation recorder attaches to the detector's **same** `getStream()` — never a
second `getUserMedia`, which would silence the listener. This is the Hands-Free
Chat wiring, driving the library `FloatingAIChat`:

```tsx
const wake = useWakeWord({
enabled: active,
onWake: (name) => {
if (name === 'hey-ozwell' && phase === 'listening') { setChatOpen(true); startDictation(); }
else if (name === "ozwell-i'm-done" && phase === 'dictating') { stopDictation(); } // → transcribe + send
},
});

// pulse from the SAME mic the detector opened — no second getUserMedia
const level = useRoomLevel(wake.getStream, wake.ready);

function startDictation() {
const rec = new MediaRecorder(wake.getStream()); // the listener's own stream
rec.ondataavailable = (e) => chunks.push(e.data);
rec.start();
}
function stopDictation() {
rec.onstop = async () => send(stripStopPhrase(await transcribeBlob(new Blob(chunks))));
rec.stop();
}

// header + the existing library floating chat, gated on `active`:
<HeyOzwellToggle active={active} level={level} onToggle={setActive} />
{active && (
<FloatingAIChat open={chatOpen} onOpenChange={setChatOpen} messages={messages} onSendMessage={send} />
)}
```

Transcription runs in a **Web Worker** (see `whisperTranscribe.ts`), so loading the ~1.3 GB Whisper
model never blocks the main thread where the wake detector does its own on-device inference — "hey
ozwell" keeps responding while the model loads. Same model and output as before; the public API is
unchanged, so every surface (dictation, live caption, Visit Scribe, diarization) gets it for free.
## Options

All options are props on `<HeyOzwell>` / `useHeyOzwell` (live in the Demo's
Controls panel):

- **`autoDictateOnWake`**: ON, "hey ozwell" opens the chat **and** starts
dictating. OFF, it just opens the chat and waits.
- **`requireDoctor`**: the speaker-verify gate. Every wake is checked
invisibly against the **enrolled clinician's voiceprint** (WHO) and a
phrase-print (WHAT); anyone else is ignored. Stays open until a voice is
enrolled.
- **`transcription`**: `'browser'` (default) runs Whisper on-device in a
**Web Worker**. `'server'` POSTs audio to an OpenAI-compatible
`/v1/audio/transcriptions` endpoint (audio leaves the browser; falls back
to on-device on failure).
- **`conversationMode`**: on "done", **diarize** the clip and send
who-said-what ("Dr. Jane: … / Patient: …") instead of a flat transcript.
- **`liveTranscript`**: show a rough real-time caption in the composer while
dictating. The final send still re-transcribes the full clip.
- **`reviewBeforeSend`**: drop the finished transcript into the composer to
glance and edit before sending, instead of auto-firing.
- **`closeChatOnDone`**: close the chat popup after the dictated message
sends.
- **`onSend`**: route sends to your own backend or message store instead of
the built-in client.
- **`assetBase`**: override where the wake/speaker models load from.

Right-click (or long-press) the octopus for **Ozwell settings**: voice
enrollment and the live "Models & versions" readout.

## Visit Scribe (ambient scribe)

A second surface, for **multi-person encounters** — no wake word, no dictation turns. Just record the
room, stop, and get an **on-device, speaker-labeled transcript** (Doctor / Patient / Nurse).
Attribution comes from `useDiarization`: Whisper timestamps + TitaNet speaker embeddings clustered
into turns, anchored to enrolled voiceprints, with optional LLM role inference. You can **rename,
merge, and reassign speakers** afterward, and optionally show a rough **live transcript** while
recording. Same guarantee as everything else here — audio never leaves the page.
The second surface, for **multi-person encounters**. No wake word, no turns:
record the room, stop, and get an **on-device speaker-labeled transcript**
(Doctor / Patient / Nurse). Under the hood it's Whisper timestamps plus
TitaNet speaker embeddings, clustered into turns and anchored to enrolled
voiceprints, with optional LLM role inference. Rename, merge, and reassign
speakers afterward, and optionally show a rough live transcript while
recording. Same guarantee: audio never leaves the page. See _Visit Scribe_
in this section.

## How it works: what runs where

One shared mic: the wake detector opens `getUserMedia` once and every other
consumer (dictation recorder, volume meter, verification buffer) reads from
`getStream()`. A second `getUserMedia` would silence the detector.

- **Mic capture + 16 kHz resample**: AudioWorklet, on the audio thread.
- **Wake word** ("hey ozwell" / "ozwell i'm done"): ONNX via onnxruntime-web
(~6 MB of models), on the main thread.
- **Speaker verification** (the WHO gate, AS-norm scoring): sherpa-onnx +
TitaNet (~50 MB), on the main thread.
- **Transcription**: Whisper turbo (~1.3 GB, WebGPU; base.en WASM fallback)
via transformers.js, in a **Web Worker**.
- **Diarization**: Whisper timestamps + TitaNet embeddings, worker + main.
- **Assistant reply**: OpenAI-compatible `/v1/chat/completions` (SSE
streaming). This is the only network stage.

Transcription lives in a Web Worker so downloading and compiling the 1.3 GB
model never starves the main-thread wake detector. **"Hey ozwell" responds
while Whisper is still loading**, and the buffered audio catches up.

### Models, caching, hosting

- **Wake models** (~6 MB): cached in OPFS, pre-fetched on mount. Override
with the `assetBase` prop or `window.__ozwellAssets`.
- **Speaker runtime** (~50 MB): cached via the Cache API (service worker).
Override with `window.__ozwellAssets`.
- **Whisper weights**: streamed into the Cache API by transformers.js.
Override with `window.__ozwellWhisperHost`.
- **transformers.js ESM** (pinned version): browser cache. Override with
`window.__ozwellTransformersUrl`.

Voiceprints persist in **IndexedDB**; `localStorage` holds config only. Every
host is config-swappable, so repointing models requires no source edits.

## Backend integration

The component is backend-agnostic. Two seams:

- **`onSend` prop**: the hook hands you the final text. Call whatever backend
you like and render the reply yourself.
- **`ozwellConfig`**: leave `onSend` off and the built-in client streams from
any OpenAI-compatible endpoint. Configure via `localStorage.ozwellConfig`
or `window.__ozwell` with `{ apiKey, baseURL, model, system, temperature }`.
The sentinel `apiKey: 'ollama'` targets a local Ollama server (no auth
header). With no key configured, the demo falls back to a canned reply, so
no secret is ever committed.

> ⚠️ A browser-held Bearer key is visible to anyone with the page. For public
> deploys, proxy the key server-side and point `baseURL` at the proxy. See
> `src/components/AI/OZWELL-BACKEND.md`.

## Troubleshooting

- **Octopus ring never completes**: mic permission denied or no input
device. The toggle surfaces "Microphone unavailable" after ~8 s.
- **First activation is slow**: the Whisper download (up to ~1.3 GB) happens
once, then it's served from the Cache API. The wake word works during the
download; only transcription waits.
- **Strict CSP hosts**: the worker needs `worker-src blob:`, plus the model
CDN allowances (or self-host everything via the overrides above).
- **"hey ozwell" ignored with the doctor gate on**: re-enroll in conditions
matching real use (same mic, same room). The WHO gate is tunable via
`window.SpeakerVerify` thresholds.
- **Console `[W:onnxruntime …]` messages during model load**: harmless ONNX
runtime warnings about CPU-assigned graph nodes, logged at error level.

## Related

- **Voice Setup** — one-time speaker enrollment; the octopus-pulse pattern reused here.
- **Wake Word** — the on-device "hey ozwell" detector that drives the pulse and wake.
- **Hands-Free Chat** — the voice conversation flow.
- **Visit Scribe** — the ambient multi-person scribe described above.

All live under *Product / Feature Modules / AI* in this sidebar.

### Integration & backend
- **Voice Setup**: one-time speaker enrollment (the octopus-pulse pattern).
- **Voice Manager**: manage, rename, and remove enrolled voices.
- **Wake Word**: the on-device detector primitive.
- **Hands-Free Chat**: the voice conversation composition.
- **Visit Scribe**: the ambient multi-person scribe.

- **Backend wiring** (`OZWELL-BACKEND.md`) — the assistant's reply is an OpenAI-compatible
`POST /v1/chat/completions` (ozwellai-api by default, or local Ollama via the `apiKey: 'ollama'`
sentinel). A host overrides it with the **`onSend`** prop or an `ozwellConfig` **`baseURL`** — the
component is backend-agnostic. Only the chat reply (and the opt-in server transcription) touch a
network; wake, speaker verify, Whisper, and diarization are all on-device.
- **Model hosting** (`MODEL-HOSTING.md`) — wake / speaker / Whisper models are fetched at runtime
from an asset host so the repo stays code-only; every host is runtime-overridable via config
(`window.__ozwellAssets`, `window.__ozwellWhisperHost`, `window.__ozwellTransformersUrl`).
Deeper docs in the repo: `MODEL-HOSTING.md` (asset hosting + cutover),
`OZWELL-BACKEND.md` (chat wiring), `DIARIZATION.md` (speaker attribution).
2 changes: 2 additions & 0 deletions src/components/AI/HeyOzwell/HeyOzwell.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import { suggestedActions } from '../storyData';

const meta: Meta = {
title: 'Product/Feature Modules/AI/Hey Ozwell/Demo',
component: HeyOzwell,
tags: ['autodocs'],
parameters: {
layout: 'fullscreen',
docs: {
Expand Down
1 change: 1 addition & 0 deletions src/components/AI/HeyOzwell/VisitScribe.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { VisitScribe } from './VisitScribe';
const meta: Meta<typeof VisitScribe> = {
title: 'Product/Feature Modules/AI/Hey Ozwell/Visit Scribe',
component: VisitScribe,
tags: ['autodocs'],
parameters: {
layout: 'fullscreen',
docs: {
Expand Down
1 change: 1 addition & 0 deletions src/components/AI/HeyOzwell/VoiceManager.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { VoiceManager } from './VoiceManager';
const meta: Meta<typeof VoiceManager> = {
title: 'Product/Feature Modules/AI/Hey Ozwell/Voice Manager',
component: VoiceManager,
tags: ['autodocs'],
parameters: {
layout: 'fullscreen',
docs: {
Expand Down
1 change: 1 addition & 0 deletions src/components/AI/HeyOzwell/VoiceSetup.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { VoiceSetup } from './VoiceSetup';
const meta: Meta<typeof VoiceSetup> = {
title: 'Product/Feature Modules/AI/Hey Ozwell/Voice Setup',
component: VoiceSetup,
tags: ['autodocs'],
parameters: {
layout: 'fullscreen',
docs: {
Expand Down
1 change: 1 addition & 0 deletions src/components/AI/HeyOzwell/WakeWord/WakeWord.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import { useWakeWord } from './useWakeWord';

const meta: Meta = {
title: 'Product/Feature Modules/AI/Hey Ozwell/Wake Word',
tags: ['autodocs'],
parameters: {
Comment on lines 16 to 19
layout: 'centered',
docs: {
Expand Down
Loading