diff --git a/.gitignore b/.gitignore index 0fbcf34..c928c4a 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,5 @@ coverage docs_out *.tsbuildinfo +.worktrees +.superpowers diff --git a/openspec/changes/add-audio-processing-strategies/.openspec.yaml b/openspec/changes/add-audio-processing-strategies/.openspec.yaml new file mode 100644 index 0000000..9e5b8a1 --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-23 diff --git a/openspec/changes/add-audio-processing-strategies/design.md b/openspec/changes/add-audio-processing-strategies/design.md new file mode 100644 index 0000000..c5fe31d --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/design.md @@ -0,0 +1,158 @@ +## Context + +`@bengreenier/react-user-media` acquires streams (`useMedia`), observes audio tracks, plays (`AudioPlayer`), and records (`useMediaRecorder`). There is no signal-processing path on main. Draft PR #6 explored hand-rolled workers + WebCodecs; this change does not revive it. + +Consumers need different primitives for different jobs. This design specs **three strategies** under one shared family so callers pick by workload, with common types and lifecycle conventions. + +Library constraints: React 19 peer, tsup CJS+ESM, Vitest browser tests, secure context, ESM-first module URLs for worklets/workers. + +## Goals / Non-Goals + +**Goals:** + +- Shared contracts: `AudioFrame`, `AudioProcessOptions`, `AudioProcessResult`, discriminated idle/loading/ready/error, session invalidation, no default track-stop +- Strategy A — **AudioWorklet realtime**: graph-native DSP on the audio thread +- Strategy B — **Dedicated Worker + Comlink**: async buffer jobs via `expose`/`wrap`/`transfer`/`proxy` +- Strategy C — **Worklet + Comlink RPC**: `process()` stays sync; Comlink models port control/results +- Clear selection guidance; phased ship order; tree-shake-friendly subpath exports where practical +- **Forward compatibility:** architecture MUST NOT preclude a future change adding video processing and/or WebCodecs on this library (especially via the worker + Comlink job path) + +**Non-Goals (deferred, not forbidden):** + +- One strategy pretending to cover all workloads +- Shipping video strategies or WebCodecs APIs in *this* change +- Replacing MediaRecorder +- CJS-default worker/worklet factories without overrides +- Heavy DSP suite beyond a v1 level-meter proving each path + +## Selection guide + +``` + Need Use + ───────────────────────────────────────────────────── + Live mic meters / FX / VAD-lite audio-worklet + Offline / heavy / buffer jobs audio-worker (Comlink) + Live DSP + typed configure/sub audio-worklet-rpc +``` + +## Decisions + +### One family, three capabilities + +**Choice:** Three OpenSpec capabilities (`audio-worklet`, `audio-worker`, `audio-worklet-rpc`) sharing types and lifecycle rules. React surface: + +- `useMediaAudioProcessor(media, { strategy: "worklet" | "worklet-rpc", ... })` for stream-oriented paths +- `useAudioWorker(options?)` for buffer/job-oriented path (`strategy: "worker"` implicit) +- Optional unified overload later; not required for v1 + +**Rationale:** Lets consumers choose tradeoffs without learning three unrelated products; keeps specs separable for phased delivery. + +**Alternatives considered:** Single capability with only worklet (too narrow); worker-only + Comlink (rejects realtime); three unrelated hook families (DX tax). + +### Shared types and ownership + +**Choice:** Public shared types for frames/results/options. Media-capture owns `MediaStream` lifecycle. Processors MUST NOT call `getUserMedia` and MUST NOT stop tracks on unmount by default. Hooks follow boolean discriminants + generation/session ids (ignore late results after restart). + +**Rationale:** Matches existing library conventions (`useMedia`, `useMediaRecorder`). + +### Forward compatibility: video and WebCodecs + +**Choice:** This change ships **audio-only** APIs, but treats video/WebCodecs as a planned extension surface—not a closed door: + +- Keep lifecycle/session/ownership rules **media-kind agnostic** (idle/loading/ready/error, no default track-stop, ESM module overrides) so a future `video-*` capability can mirror them +- Prefer namespaced audio types (`AudioFrame`, `useAudioWorker`, `audio-worklet` subpath) over sealing a single global `MediaProcessor` that only fits PCM—so video can add `VideoFrame` / WebCodecs types beside them without a breaking rename +- Treat the **Comlink Dedicated Worker** path as the primary future home for WebCodecs (`AudioEncoder`/`VideoEncoder`, transferable `AudioData`/`VideoFrame`): same `configure` / process / `subscribe` / `dispose` + `transfer` shape +- Do **not** hard-wire worker internals to “Float32 PCM forever” in a way that blocks transferring other frame types later (keep frame handling behind typed process methods / narrow modules) +- AudioWorklet remains audio-specific (correct); video realtime would be a separate future primitive (e.g. `MediaStreamTrackProcessor` / transform streams), not forced through AudioWorklet + +**Rationale:** Priority is audio now; consumers and the library should still be able to grow into video/WebCodecs without throwing away the processing family. + +**Alternatives considered:** Listing video/WebCodecs as absolute non-goals—over-constrains the product; implementing WebCodecs in this change—out of priority/scope. + +### Strategy A — AudioWorklet realtime + +**Choice:** + +``` + MediaStream → MediaStreamAudioSourceNode → AudioWorkletNode → silent Gain / destination +``` + +Processor implements sync `process(inputs, outputs, parameters)`. v1 built-in: RMS/peak (and optional passthrough to outputs). Metrics to main via `port.postMessage` at a throttled rate (e.g. ≤60 Hz), not every quantum. Hook handles `audioWorklet.addModule`, context create/resume (user-gesture aware), graph connect/teardown. + +**Rationale:** Correct clock for live stream DSP; avoids main-thread frame pull. + +**Alternatives considered:** AnalyserNode-only on main — simpler but not an extensible processing strategy; Dedicated Worker as primary — wrong clock. + +### Strategy B — Dedicated Worker + Comlink + +**Choice:** Depend on [`comlink`](https://www.npmjs.com/package/comlink) (GoogleChromeLabs). Worker `Comlink.expose`s: + +```ts +interface AudioWorkerApi { + configure(options: AudioProcessOptions): Promise; + processFrame(frame: AudioFrame): Promise; + subscribe(onResult: (r: AudioProcessResult) => void): Promise; + dispose(): Promise; +} +``` + +Hot path uses `Comlink.transfer` for `ArrayBuffer`s. Main `Comlink.wrap`; teardown `releaseProxy` + `terminate`. Optional thin bridge can pull PCM from a stream for convenience, but docs steer live realtime use to worklet. + +**Rationale:** Best DX for async jobs; transfer avoids structured-clone copies. + +### Strategy C — Worklet + Comlink on port + +**Choice:** Same realtime `process()` as strategy A. Main thread `Comlink.wrap(audioWorkletNode.port)`; worklet side `Comlink.expose(controlApi, port)` for `configure` / `subscribe` / `dispose`. **No awaits inside `process()`** — configure mutates processor fields; results are queued/posted from process or a timer coalesced on the worklet side without blocking the quantum. + +**Rationale:** Keeps audio-thread correctness while giving Comlink-shaped control plane for consumers who want it. + +**Alternatives considered:** Hand-typed port protocol only (strategy A) — enough for many apps; forcing Comlink for all worklet users — unnecessary dependency weight. + +### Packaging + +**Choice:** + +- ESM subpaths for worklet module(s) and worker script(s), e.g. `@bengreenier/react-user-media/audio-worklet`, `.../audio-worker` +- `addModule(url)` / `new Worker(url, { type: "module" })` via `import.meta.url` factories +- Overrides: `workletModuleUrl`, `createWorker`, etc. for tests/bundlers +- `comlink` as a dependency of worker and worklet-rpc entry paths; worklet-only consumers should not need to load Comlink if split correctly +- Document CJS limitation + +**Rationale:** Dual publish makes default URL workers/worklets unreliable under `require`. + +### Implementation phases + +1. Shared types + **audio-worklet** (level meter + hook) +2. **audio-worker** (Comlink job API + hook) +3. **audio-worklet-rpc** (Comlink on port; reuse worklet DSP) + +**Rationale:** Delivers the common live-mic case first; each phase is independently demoable/testable. + +### Contrast with PR #6 / prior change + +**Choice:** Do not reuse PR #6’s hand-rolled message enums. Supersede planning change `add-audio-worker-comlink` (worker-only). WebCodecs ideas from that exploration remain valid for a **future** change on the Comlink worker path—not in this delivery. + +## Risks / Trade-offs + +- **[Risk] Three strategies increase API/doc surface** → Mitigation: selection guide, shared types, phased ship, subpath exports +- **[Risk] Worklet `process()` glitches if too heavy** → Mitigation: keep v1 metering cheap; document budget; push heavy work to worker strategy +- **[Risk] Comlink on worklet port misused inside process** → Mitigation: spec forbids async in process; code review + tests +- **[Risk] Module URL / COOP packaging pain** → Mitigation: overrides + ESM docs; avoid SharedArrayBuffer requirement for v1 +- **[Risk] AudioContext suspended until gesture** → Mitigation: loading/error states; resume helper documented +- **[Risk] Audio-first naming paints a corner against video** → Mitigation: namespaced audio modules/types; shared lifecycle conventions; worker transfer path kept frame-type-pluggable +- **[Trade-off] Spec all three before all are implemented** → Accepted; tasks are phased with checkboxes +- **[Trade-off] Defer WebCodecs/video** → Accepted for priority; extensibility decision above is mandatory + +## Migration Plan + +- Additive APIs only; no breaking changes to existing hooks +- Minor version bump when first strategy ships; subsequent strategies additive +- Remove superseded OpenSpec change `add-audio-worker-comlink` from `openspec/changes/` +- Rollback: remove new exports/deps; no data migration + +## Open Questions + +- Exact public names: `useMediaAudioProcessor` vs `useAudioWorklet` / `useAudioWorkletRpc` as separate exports (lean: strategy param on stream hook + dedicated `useAudioWorker`) +- Whether worklet-rpc is a separate npm subpath or a `strategy: "worklet-rpc"` flag (lean: strategy flag reusing worklet module + thin Comlink adapter) +- Default metric post rate and whether results are per-channel or mixed (lean: mixed summary + optional per-channel in result type) +- When to open a follow-up change for WebCodecs/video (after worker path ships vs. parallel spike) diff --git a/openspec/changes/add-audio-processing-strategies/proposal.md b/openspec/changes/add-audio-processing-strategies/proposal.md new file mode 100644 index 0000000..d1301fe --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/proposal.md @@ -0,0 +1,42 @@ +## Why + +The library captures media but offers no off-main-thread processing path. Different workloads need different browser primitives: **AudioWorklet** for realtime graph DSP, a **Dedicated Worker + Comlink** for async buffer jobs, and **Comlink on the worklet port** when consumers want typed configure/subscribe without hand-rolled messages. Speccing all three as one family lets consumers pick the right tradeoff. This change prioritizes **audio**, but the patterns (lifecycle, Comlink jobs, transferables, ESM module packaging) MUST leave the door open for later **video / WebCodecs** work on the same library. + +## What Changes + +- Add a shared media-processing type/lifecycle family for audio first (frame/result/options, idle/loading/ready/error, stream ownership stays in media-capture), structured so video/WebCodecs can reuse the same conventions later +- Add **AudioWorklet realtime** strategy: `MediaStream` → Web Audio graph → `AudioWorkletProcessor.process()` +- Add **Dedicated Worker + Comlink** strategy: `expose`/`wrap`/`transfer` job API for PCM buffers (natural on-ramp for future WebCodecs encode/decode jobs) +- Add **Worklet + Comlink RPC** strategy: realtime `process()` plus Comlink on `AudioWorkletNode.port` for control/results +- Ship ESM-first module URLs for worklet and worker scripts; allow `create*` / URL overrides +- Phased implementation: worklet → worker → worklet-rpc; shared types first +- Examples + browser tests per strategy; selection guide in docs + +## Non-goals (this change only) + +- Implementing video processors, WebCodecs encode/decode APIs, or video strategies in this change (deferred; must remain possible) +- Replacing `useMediaRecorder` or owning `getUserMedia` / track teardown by default +- SharedWorker, ServiceWorker +- Reviving draft PR #6’s hand-rolled message protocol wholesale (WebCodecs direction may return in a future change on the Comlink worker path) +- Guaranteeing default factories under CJS `require` without overrides +- Shipping a full DSP plugin marketplace (v1: level meter + extensibility hooks) + +## Capabilities + +### New Capabilities + +- `audio-worklet`: Realtime AudioWorklet processing of live `MediaStream` audio in the Web Audio graph, with React lifecycle hooks and port-based metrics. +- `audio-worker`: Comlink-backed Dedicated Worker API and hooks for async PCM frame jobs (`processFrame` + transferables). +- `audio-worklet-rpc`: Comlink RPC layered on `AudioWorkletNode.port` for typed configure/subscribe while DSP remains in `process()`. + +### Modified Capabilities + +- (none — `openspec/specs/` is empty; baseline changes are documentation-only) + +## Impact + +- Package: `packages/react-user-media` — shared audio types, worklet + worker modules, hooks, `comlink` dependency, multi-entry packaging/exports +- Examples: demos for at least worklet metering; optional worker/rpc demos +- Tests: Vitest browser coverage per strategy; shared lifecycle contracts +- Docs: strategy selection guide; ESM module URL / override notes +- Supersedes planning-only change `add-audio-worker-comlink` (worker-only intent) diff --git a/openspec/changes/add-audio-processing-strategies/specs/audio-worker/spec.md b/openspec/changes/add-audio-processing-strategies/specs/audio-worker/spec.md new file mode 100644 index 0000000..565a66b --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/specs/audio-worker/spec.md @@ -0,0 +1,80 @@ +## ADDED Requirements + +### Requirement: Comlink-exposed worker API +The library SHALL ship a Dedicated Worker module that exposes an audio job API via `Comlink.expose`. The main thread SHALL obtain a typed proxy with `Comlink.wrap`. The shared TypeScript contract SHALL include at least `configure`, `processFrame`, and `dispose`. Proxy method results SHALL be awaitable. + +#### Scenario: Wrap and configure +- **WHEN** the consumer creates the library audio worker and wraps it with Comlink +- **THEN** awaiting `configure` with valid options resolves and subsequent `processFrame` calls are accepted + +#### Scenario: Dispose fails closed +- **WHEN** the consumer awaits `dispose` on the proxy +- **THEN** further `processFrame` calls reject or otherwise fail closed and worker-owned state is cleared + +### Requirement: Transferable PCM frames +`processFrame` SHALL accept PCM frame data movable with `Comlink.transfer` so `ArrayBuffer` ownership transfers to the worker on the hot path. After a successful transfer of a buffer used as an argument, that buffer MUST NOT remain usable on the calling thread (detached / zero-length semantics). + +#### Scenario: Process transferred frame +- **WHEN** the consumer calls `processFrame` with channel data wrapped in `Comlink.transfer` +- **THEN** the worker returns an `AudioProcessResult` +- **AND** the transferred `ArrayBuffer` is detached on the caller side + +### Requirement: Built-in level analysis +After configuration, `processFrame` SHALL support at least RMS and peak computation over the provided frame, returned as `AudioProcessResult` using the shared type family. + +#### Scenario: RMS and peak for a known frame +- **WHEN** the consumer processes a frame of known constant amplitude +- **THEN** the result’s peak reflects that amplitude and RMS is within an accepted numeric tolerance + +### Requirement: Optional subscription via Comlink.proxy +The worker API MAY expose `subscribe(onResult)` where `onResult` is passed with `Comlink.proxy`. Unsubscribe or `dispose` MUST stop further callbacks for that session. + +#### Scenario: Proxied callback receives results +- **WHEN** the consumer subscribes with a proxied callback and processes frames +- **THEN** the callback is invoked with `AudioProcessResult` values +- **WHEN** the consumer disposes the API +- **THEN** no further subscription callbacks are delivered for that session + +### Requirement: `useAudioWorker` lifecycle hook +The library SHALL export `useAudioWorker` that manages worker creation and Comlink wrap/release with idle/loading/ready/error state. Unmount and stop MUST `releaseProxy` and `worker.terminate()`. A session/generation id MUST ignore late results after restart. + +#### Scenario: Ready after start +- **WHEN** the consumer starts the hook and the worker initializes successfully +- **THEN** state is ready and a usable Comlink proxy API is available + +#### Scenario: Cleanup on unmount +- **WHEN** the component unmounts while a worker is active +- **THEN** the proxy is released and the worker is terminated + +#### Scenario: Ignores late results after restart +- **WHEN** the consumer restarts the worker and a previous instance later delivers a result +- **THEN** the new session’s ready-state data is not overwritten by the stale result + +#### Scenario: Init failure +- **WHEN** worker construction or Comlink setup fails +- **THEN** state is error, ready is false, and no leaked worker remains owned by the hook + +### Requirement: Stream bridge does not own capture +If a convenience bridge feeds `MediaStream` PCM into the worker, it MUST NOT call `getUserMedia` and MUST NOT stop tracks on the provided stream by default when the bridge stops. Docs SHOULD steer realtime live-stream DSP to the worklet strategy. + +#### Scenario: Bridge stop leaves tracks running +- **WHEN** a stream-fed worker bridge stops +- **THEN** tracks on the provided `MediaStream` remain in their prior `readyState` + +### Requirement: ESM worker packaging and overrides +The package SHALL provide an ESM-consumable worker factory or URL. Consumers MUST be able to override worker creation for tests and bundlers. Default CJS `require` without an override MAY be unsupported and MUST be documented. + +#### Scenario: Custom createWorker in tests +- **WHEN** the consumer supplies a `createWorker` override returning a Comlink-compatible endpoint +- **THEN** `useAudioWorker` can reach ready state without the production worker script + +### Requirement: Extensible job surface for future media kinds +The worker + Comlink lifecycle (`configure`, frame processing, optional `subscribe`, `dispose`, transferable payloads, hook session cleanup) SHALL be structured so a future library change can add video and/or WebCodecs job APIs without replacing that lifecycle. This change MUST NOT ship video or WebCodecs APIs; audio PCM remains the only implemented frame kind. + +#### Scenario: Audio-only in this change +- **WHEN** a consumer uses the shipped worker API from this change +- **THEN** only audio PCM processing APIs are available (no WebCodecs encode/decode or video frame processors yet) + +#### Scenario: Lifecycle shape stays reusable +- **WHEN** maintainers review the public worker control surface +- **THEN** it still follows configure / process / subscribe / dispose with transfer-friendly payloads suitable for later non-PCM frame types diff --git a/openspec/changes/add-audio-processing-strategies/specs/audio-worklet-rpc/spec.md b/openspec/changes/add-audio-processing-strategies/specs/audio-worklet-rpc/spec.md new file mode 100644 index 0000000..0defbdf --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/specs/audio-worklet-rpc/spec.md @@ -0,0 +1,48 @@ +## ADDED Requirements + +### Requirement: Realtime DSP remains in process +The worklet-rpc strategy SHALL use an AudioWorklet graph equivalent to the realtime worklet strategy. Audio sample processing MUST occur in synchronous `AudioWorkletProcessor.process`. Comlink calls MUST NOT be awaited inside `process`. + +#### Scenario: Process stays synchronous under RPC +- **WHEN** worklet-rpc is running and Comlink configure/subscribe are in use +- **THEN** audio quanta continue to be handled by synchronous `process` without awaiting Comlink + +### Requirement: Comlink on AudioWorkletNode port +The main thread SHALL obtain a typed control proxy via `Comlink.wrap` on the `AudioWorkletNode`’s `MessagePort` (or an equivalent port endpoint documented by the library). The worklet side SHALL `Comlink.expose` a control API on that port that includes at least `configure` and `dispose`. Optional `subscribe` SHALL accept callbacks via `Comlink.proxy`. + +#### Scenario: Configure through Comlink proxy +- **WHEN** the consumer awaits `configure` on the worklet-rpc proxy with valid options +- **THEN** subsequent processing uses the configured options without rebuilding a Dedicated Worker + +#### Scenario: Dispose stops RPC session +- **WHEN** the consumer awaits `dispose` on the control proxy +- **THEN** further configure/subscribe operations fail closed for that session and subscription callbacks stop + +### Requirement: Shared result type for subscriptions +Subscription or polled results exposed through the Comlink control plane SHALL use the shared `AudioProcessResult` type (including RMS/peak for the default level processor). + +#### Scenario: Subscribe receives levels +- **WHEN** the consumer subscribes with a proxied callback and audio is flowing +- **THEN** the callback receives `AudioProcessResult` values with level fields + +### Requirement: React strategy surface +The library SHALL expose worklet-rpc through the stream-oriented processor hook via an explicit strategy (e.g. `strategy: "worklet-rpc"`) or an equivalently documented dedicated export. Lifecycle SHALL match the worklet strategy: idle/loading/ready/error, session invalidation, no default track stop, teardown of graph and Comlink proxy on stop/unmount. + +#### Scenario: Ready with worklet-rpc strategy +- **WHEN** the consumer starts the stream processor with worklet-rpc strategy and a valid stream +- **THEN** state becomes ready and both realtime processing and Comlink control are available + +#### Scenario: Cleanup releases proxy and graph +- **WHEN** the component unmounts during an active worklet-rpc session +- **THEN** the Comlink proxy is released and the audio graph is torn down + +#### Scenario: Does not stop capture tracks +- **WHEN** worklet-rpc stops or unmounts +- **THEN** tracks on the provided `MediaStream` remain in their prior `readyState` + +### Requirement: Depends on worklet module packaging +Worklet-rpc SHALL reuse the same ESM worklet module packaging rules as the realtime worklet capability (URL/factory + overrides). Comlink MUST be loaded for this strategy; worklet-only consumers MUST remain able to use the non-rpc worklet path without requiring Comlink if packaging splits allow. + +#### Scenario: Override module URL still works +- **WHEN** the consumer overrides the worklet module URL under worklet-rpc strategy +- **THEN** the hook can reach ready state using that module with Comlink control attached to the node port diff --git a/openspec/changes/add-audio-processing-strategies/specs/audio-worklet/spec.md b/openspec/changes/add-audio-processing-strategies/specs/audio-worklet/spec.md new file mode 100644 index 0000000..b9d6823 --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/specs/audio-worklet/spec.md @@ -0,0 +1,55 @@ +## ADDED Requirements + +### Requirement: Realtime graph wiring +The library SHALL provide an AudioWorklet-based path that connects a consumer-supplied `MediaStream` into a Web Audio graph: `MediaStreamAudioSourceNode` → `AudioWorkletNode` → a non-audible sink or explicit destination as documented. The library MUST load the processor via `audioContext.audioWorklet.addModule` before constructing the node. This path MUST NOT call `getUserMedia` / `getDisplayMedia`. + +#### Scenario: Graph connects for a live stream +- **WHEN** the consumer starts the worklet processor with a `MediaStream` that has an audio track +- **THEN** an `AudioWorkletNode` is created after successful `addModule` and the source is connected through that node + +### Requirement: Synchronous process callback +Custom processors shipped by the library SHALL implement `AudioWorkletProcessor.process` as a synchronous, realtime-safe callback. `process` MUST NOT perform Comlink awaits or other asynchronous RPC. Returning `true` MUST keep the processor alive while the session is active. + +#### Scenario: Process runs without async RPC +- **WHEN** audio quanta flow through the worklet node +- **THEN** `process` completes synchronously and continues to be invoked while the session remains active + +### Requirement: Built-in level analysis +The default library worklet processor SHALL compute at least RMS and peak levels from input frames and expose them to the main thread as `AudioProcessResult` values (shared type family). + +#### Scenario: Levels from live input +- **WHEN** the worklet processes a live stream with audible audio +- **THEN** the main-thread consumer eventually observes a non-null level result with RMS and peak fields + +### Requirement: Throttled port metrics +Level or analysis results SHALL be delivered to the main thread via `MessagePort` at a throttled rate suitable for UI updates. The implementation MUST NOT require posting a full result on every audio quantum. + +#### Scenario: Metrics arrive for UI +- **WHEN** the processor is running and producing levels +- **THEN** the main thread receives periodic `AudioProcessResult` messages without one mandatory message per quantum + +### Requirement: React hook lifecycle +The library SHALL export a React hook for the worklet strategy that uses discriminated idle/loading/ready/error state. Start SHALL create/resume context and build the graph; stop and unmount SHALL disconnect nodes, release ports, and close or otherwise dispose the context as documented. A session/generation id MUST ignore late port messages after restart. Tracks on the provided `MediaStream` MUST NOT be stopped by default on stop/unmount. + +#### Scenario: Ready after successful start +- **WHEN** the consumer starts the worklet hook with a valid stream and module URL +- **THEN** state becomes ready and level results can flow + +#### Scenario: Cleanup on unmount +- **WHEN** the component unmounts while the worklet session is active +- **THEN** the graph is torn down and no further state updates are applied from that session + +#### Scenario: Does not stop capture tracks +- **WHEN** the worklet hook stops or unmounts +- **THEN** tracks on the provided `MediaStream` remain in their prior `readyState` + +#### Scenario: Module or context failure +- **WHEN** `addModule` or graph setup fails +- **THEN** state is error, ready is false, and resources from the failed attempt are not left owned by the hook + +### Requirement: ESM module URL and overrides +The package SHALL provide an ESM-consumable URL or factory for the worklet processor module. Consumers MUST be able to override the module URL for tests and alternate bundlers. Default CJS `require` usage without an override MAY be unsupported and MUST be documented. + +#### Scenario: Custom module URL in tests +- **WHEN** the consumer supplies a custom worklet module URL compatible with `addModule` +- **THEN** the hook can reach ready state using that module diff --git a/openspec/changes/add-audio-processing-strategies/tasks.md b/openspec/changes/add-audio-processing-strategies/tasks.md new file mode 100644 index 0000000..4ede51b --- /dev/null +++ b/openspec/changes/add-audio-processing-strategies/tasks.md @@ -0,0 +1,37 @@ +## 1. Shared foundation + +- [x] 1.1 Define shared public types: `AudioFrame`, `AudioProcessOptions`, `AudioProcessResult`, and strategy/lifecycle unions (audio-namespaced; do not seal a single PCM-only global processor type) +- [x] 1.2 Document strategy selection guide in README (worklet vs worker vs worklet-rpc); note video/WebCodecs as future extensions on the worker/Comlink path +- [x] 1.3 Add packaging stubs for ESM subpaths and override options (`workletModuleUrl`, `createWorker`) +- [x] 1.4 Design-review checkpoint: worker job surface (`configure` / process / `subscribe` / `dispose` + transfer) remains plausible for future `VideoFrame` / WebCodecs jobs without a rewrite + +## 2. Phase 1 — AudioWorklet realtime + +- [x] 2.1 Implement level-meter `AudioWorkletProcessor` with sync `process` and throttled port metrics +- [x] 2.2 Build ESM worklet module emit + `addModule` factory +- [x] 2.3 Implement stream hook for `strategy: "worklet"` (graph wire-up, context resume, teardown, session ids) +- [x] 2.4 Tests: module failure → error; cleanup; tracks not stopped; levels from synthetic/live audio +- [x] 2.5 Examples demo: live level meter via worklet + +## 3. Phase 2 — Dedicated Worker + Comlink + +- [x] 3.1 Add `comlink` dependency; implement worker `Comlink.expose` API (`configure` / `processFrame` / `subscribe` / `dispose`) +- [x] 3.2 Support `Comlink.transfer` on PCM buffers; built-in RMS/peak +- [x] 3.3 Implement `useAudioWorker` with releaseProxy + terminate + overrides +- [x] 3.4 Optional stream→PCM bridge that does not stop tracks; docs point realtime to worklet +- [x] 3.5 Tests: transfer detachment; lifecycle/restart; subscribe+proxy; createWorker override +- [x] 3.6 Optional examples demo for buffer/job path + +## 4. Phase 3 — Worklet + Comlink RPC + +- [x] 4.1 Expose Comlink control API on `AudioWorkletNode.port` without awaits inside `process` +- [x] 4.2 Wire `strategy: "worklet-rpc"` (or dedicated export) reusing worklet DSP + shared lifecycle +- [x] 4.3 Ensure worklet-only path can avoid loading Comlink when packaged as separate entry +- [x] 4.4 Tests: configure/subscribe via proxy; process stays sync; cleanup releases proxy+graph; tracks not stopped +- [x] 4.5 Optional examples demo for worklet-rpc configure/subscribe + +## 5. Verification + +- [x] 5.1 Run package tests and lint; fix regressions +- [x] 5.2 Validate change with `openspec validate add-audio-processing-strategies` +- [x] 5.3 Remove superseded change `add-audio-worker-comlink` from `openspec/changes/` if still present diff --git a/packages/examples/src/AudioLevelMeter.tsx b/packages/examples/src/AudioLevelMeter.tsx new file mode 100644 index 0000000..51a6bf6 --- /dev/null +++ b/packages/examples/src/AudioLevelMeter.tsx @@ -0,0 +1,33 @@ +import { useMediaAudioProcessor } from "@bengreenier/react-user-media"; + +const workletModuleUrl = new URL( + "../../react-user-media/src/audio-worklet/level-meter.ts", + import.meta.url, +); + +export function AudioLevelMeter({ media }: { media: MediaStream }) { + const processor = useMediaAudioProcessor({ + strategy: "worklet", + workletModuleUrl, + }); + + return ( +
+

Live audio level

+ {!processor.isReady && ( + + )} + {processor.isLoading &&

Starting meter...

} + {processor.isError &&

{processor.error.message}

} + {processor.isReady && ( + <> +

RMS: {processor.result?.rms.toFixed(3) ?? "0.000"}

+

Peak: {processor.result?.peak.toFixed(3) ?? "0.000"}

+ + + )} +
+ ); +} diff --git a/packages/examples/src/WebcamPreview.tsx b/packages/examples/src/WebcamPreview.tsx index dd1b28c..e840734 100644 --- a/packages/examples/src/WebcamPreview.tsx +++ b/packages/examples/src/WebcamPreview.tsx @@ -4,6 +4,7 @@ import { useMediaTracks, VideoPlayer, } from "@bengreenier/react-user-media"; +import { AudioLevelMeter } from "./AudioLevelMeter"; export function WebcamPreview() { const { request, isError, error, isLoading, isReady, media } = @@ -40,6 +41,7 @@ export function WebcamPreview() { {isReady && ( <> +
    {tracks.map((track) => (
  • diff --git a/packages/react-user-media/README.md b/packages/react-user-media/README.md index b284bf0..2925268 100644 --- a/packages/react-user-media/README.md +++ b/packages/react-user-media/README.md @@ -25,6 +25,81 @@ A collection of hooks and components for easier access to [`getUserMedia`](https - `useMediaVideoDevices()` - `useMediaRecorder()` +- `useMediaAudioProcessor({ strategy: "worklet" })` +- `useAudioWorker()` — asynchronous PCM buffer jobs in a Dedicated Worker + +## Audio processing strategies + +Choose the browser primitive that matches the workload: + +| Need | Strategy | +| --- | --- | +| Live microphone meters, lightweight effects, or VAD-lite | `audio-worklet` (`useMediaAudioProcessor`) | +| Offline, heavy, or buffer-oriented jobs | `audio-worker` (`useAudioWorker` + Comlink) | +| Live DSP with an RPC-shaped control plane | `audio-worklet-rpc` (`useMediaAudioWorkletRpc` via `@bengreenier/react-user-media/audio-worklet-rpc`) | + +### Worklet (realtime) + +The worklet hook receives a caller-owned `MediaStream`; it creates a +`MediaStreamAudioSourceNode` → `AudioWorkletNode` → silent gain graph and +never stops the stream's tracks when stopped or unmounted. + +`@bengreenier/react-user-media/audio-worklet` provides +`getLevelMeterWorkletModuleUrl()` and `addLevelMeterWorklet()` for ESM +consumers. Pass `workletModuleUrl` to `useMediaAudioProcessor` when testing or +when a bundler needs an explicit URL. The default module-relative URL is not +guaranteed under CJS `require`. + +### Worker (async jobs) + +Use `useAudioWorker()` for asynchronous or offline PCM jobs. Readiness changes +after asynchronous worker configuration, so call the ready Comlink proxy's +`processFrame` from an `isReady`-dependent effect or event handler. Wrap +`Float32Array` channel data in `Comlink.transfer` to avoid copying buffers. It +reports RMS and peak levels. + +```tsx +import * as Comlink from "comlink"; +import { useEffect } from "react"; +import { useAudioWorker } from "@bengreenier/react-user-media"; + +function LevelAnalyzer() { + const { api, isReady, start } = useAudioWorker(); + + useEffect(() => { + start(); + }, [start]); + + useEffect(() => { + if (!isReady || !api) { + return; + } + + const samples = new Float32Array([0.5, -0.5]); + void api.processFrame( + Comlink.transfer( + { channelData: [samples], sampleRate: 48_000 }, + [samples.buffer], + ), + ); + }, [api, isReady]); +} +``` + +`@bengreenier/react-user-media/audio-worker` provides `createAudioWorker()`. +CommonJS consumers and test runners can pass `createWorker` to `useAudioWorker` +instead. `useAudioWorker` never acquires capture or stops tracks. + +The worker surface uses `configure` / process / `subscribe` / `dispose` so a +future WebCodecs / `VideoFrame` job API can share the same lifecycle without +replacing these audio APIs. + +### Worklet RPC + +Import `useMediaAudioWorkletRpc` from +`@bengreenier/react-user-media/audio-worklet-rpc` (not the package root) so +default importers do not pull Comlink. Realtime DSP stays in synchronous +`process()`; configure/subscribe use Comlink on the worklet `MessagePort`. ## Components diff --git a/packages/react-user-media/package.json b/packages/react-user-media/package.json index 7ed99db..36e86f6 100644 --- a/packages/react-user-media/package.json +++ b/packages/react-user-media/package.json @@ -8,6 +8,20 @@ "import": "./dist/index.js", "require": "./dist/index.cjs" }, + "./audio-worklet": { + "types": "./dist/types/audio-worklet/index.d.ts", + "import": "./dist/audio-worklet/index.js", + "require": "./dist/audio-worklet/index.cjs" + }, + "./audio-worker": { + "types": "./dist/types/audio/audio-worker.d.ts", + "import": "./dist/audio/audio-worker.js" + }, + "./audio-worklet-rpc": { + "types": "./dist/types/audio/worklet-rpc/index.d.ts", + "import": "./dist/audio/worklet-rpc/index.js", + "require": "./dist/audio/worklet-rpc/index.cjs" + }, "./package.json": "./package.json" }, "publishConfig": { @@ -28,7 +42,7 @@ } ], "scripts": { - "build": "tsc -p ./tsconfig.types.json --emitDeclarationOnly && tsup --tsconfig ./tsconfig.lib.json --format cjs,esm src/index.ts", + "build": "tsc -p ./tsconfig.types.json --emitDeclarationOnly && tsup --tsconfig ./tsconfig.lib.json --format cjs,esm --splitting false src/index.ts src/audio-worklet/index.ts src/audio-worklet/level-meter.ts src/audio/audio-worker.ts src/audio/worker/audio-worker.ts src/audio/worklet-rpc/index.ts src/audio/worklet-rpc/processor.ts", "doc": "typedoc --plugin typedoc-github-theme --tsconfig ./tsconfig.lib.json --readme ./README.md --basePath ./src --out ./docs_out --entryPoints ./src/index.ts", "test": "vitest run --coverage", "test-visible": "vitest run --coverage --browser.headless false", @@ -56,5 +70,8 @@ "files": [ "dist", "README.md" - ] + ], + "dependencies": { + "comlink": "^4.4.2" + } } diff --git a/packages/react-user-media/src/audio-worklet/index.ts b/packages/react-user-media/src/audio-worklet/index.ts new file mode 100644 index 0000000..eaa9473 --- /dev/null +++ b/packages/react-user-media/src/audio-worklet/index.ts @@ -0,0 +1,19 @@ +/** + * Returns the ESM module URL for the built-in level-meter AudioWorklet. + * + * CJS consumers should provide `workletModuleUrl`, because module-relative + * worklet URLs are only guaranteed for ESM consumers. + */ +export function getLevelMeterWorkletModuleUrl(): URL { + return new URL(/* @vite-ignore */ "./level-meter.js", import.meta.url); +} + +/** + * Loads the built-in level-meter processor into an AudioContext. + */ +export function addLevelMeterWorklet( + audioContext: AudioContext, + moduleUrl: string | URL = getLevelMeterWorkletModuleUrl(), +): Promise { + return audioContext.audioWorklet.addModule(moduleUrl); +} diff --git a/packages/react-user-media/src/audio-worklet/level-meter.spec.ts b/packages/react-user-media/src/audio-worklet/level-meter.spec.ts new file mode 100644 index 0000000..9b5d4c8 --- /dev/null +++ b/packages/react-user-media/src/audio-worklet/level-meter.spec.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { calculateAudioLevels } from "./levels"; + +test("calculates mixed RMS and peak from input channels", () => { + const result = calculateAudioLevels([ + [new Float32Array([0, 1, -0.5])], + [new Float32Array([0.5, -1])], + ]); + + expect(result.rms).toBeCloseTo(Math.sqrt(2.5 / 5)); + expect(result.peak).toBe(1); +}); + +test("reports silent levels when the worklet has no input frames", () => { + expect(calculateAudioLevels([])).toEqual({ rms: 0, peak: 0 }); +}); diff --git a/packages/react-user-media/src/audio-worklet/level-meter.ts b/packages/react-user-media/src/audio-worklet/level-meter.ts new file mode 100644 index 0000000..a4b33ed --- /dev/null +++ b/packages/react-user-media/src/audio-worklet/level-meter.ts @@ -0,0 +1,35 @@ +import type { AudioProcessResult } from "../audio"; +import { calculateAudioLevels } from "./levels"; + +const DEFAULT_METRIC_INTERVAL_SECONDS = 1 / 30; + +declare abstract class AudioWorkletProcessor { + readonly port: MessagePort; +} + +declare function registerProcessor( + name: string, + processorCtor: new () => AudioWorkletProcessor, +): void; + +declare const currentTime: number; + +class LevelMeterProcessor extends AudioWorkletProcessor { + #lastMetricTime = -Infinity; + + process(inputs: Float32Array[][]): boolean { + if (currentTime - this.#lastMetricTime >= DEFAULT_METRIC_INTERVAL_SECONDS) { + this.#lastMetricTime = currentTime; + const levels = calculateAudioLevels(inputs); + const result: AudioProcessResult = { + ...levels, + timestamp: currentTime, + }; + this.port.postMessage(result); + } + + return true; + } +} + +registerProcessor("react-user-media-level-meter", LevelMeterProcessor); diff --git a/packages/react-user-media/src/audio-worklet/levels.ts b/packages/react-user-media/src/audio-worklet/levels.ts new file mode 100644 index 0000000..4d8e9dd --- /dev/null +++ b/packages/react-user-media/src/audio-worklet/levels.ts @@ -0,0 +1,28 @@ +/** + * Calculates a mixed RMS and peak value for the input channels in a render + * quantum. Keeping this pure also makes synthetic frame testing possible. + */ +export function calculateAudioLevels(inputs: readonly Float32Array[][]): { + rms: number; + peak: number; +} { + let sampleCount = 0; + let sumOfSquares = 0; + let peak = 0; + + for (const input of inputs) { + for (const channel of input) { + for (const sample of channel) { + const magnitude = Math.abs(sample); + sumOfSquares += sample * sample; + peak = Math.max(peak, magnitude); + sampleCount += 1; + } + } + } + + return { + rms: sampleCount === 0 ? 0 : Math.sqrt(sumOfSquares / sampleCount), + peak, + }; +} diff --git a/packages/react-user-media/src/audio.ts b/packages/react-user-media/src/audio.ts new file mode 100644 index 0000000..2ac9143 --- /dev/null +++ b/packages/react-user-media/src/audio.ts @@ -0,0 +1,99 @@ +/** + * A PCM audio frame for processor strategies that operate on buffered audio. + * Planar `channelData` is the transfer-friendly shape for worker jobs. + * Future worker strategies can add video/WebCodecs frame types alongside this + * audio-specific contract without changing the lifecycle surface. + */ +export interface AudioFrame { + readonly channelData: Float32Array[]; + readonly sampleRate: number; + readonly timestamp?: number; +} + +/** + * Common processing options for audio strategies. + */ +export interface AudioProcessOptions { + readonly metricIntervalMs?: number; + /** Required by the Dedicated Worker configure path. */ + readonly sampleRate?: number; +} + +/** + * A summary emitted by an audio processor. + */ +export interface AudioProcessResult { + readonly rms: number; + readonly peak: number; + readonly timestamp: number; + readonly frameLength?: number; +} + +/** + * Available audio processing strategies. + */ +export type AudioProcessingStrategy = "worklet" | "worker" | "worklet-rpc"; + +interface AudioProcessorStateBase { + readonly isLoading: boolean; + readonly isReady: boolean; + readonly isError: boolean; + readonly error: Error | null; + readonly result: AudioProcessResult | null; + start(media: MediaStream): void; + stop(): void; +} + +export interface AudioProcessorIdleState extends AudioProcessorStateBase { + readonly isLoading: false; + readonly isReady: false; + readonly isError: false; + readonly error: null; + readonly result: null; +} + +export interface AudioProcessorLoadingState extends AudioProcessorStateBase { + readonly isLoading: true; + readonly isReady: false; + readonly isError: false; + readonly error: null; + readonly result: null; +} + +export interface AudioProcessorReadyState extends AudioProcessorStateBase { + readonly isLoading: false; + readonly isReady: true; + readonly isError: false; + readonly error: null; +} + +export interface AudioProcessorErrorState extends AudioProcessorStateBase { + readonly isLoading: false; + readonly isReady: false; + readonly isError: true; + readonly error: Error; + readonly result: null; +} + +export type AudioProcessorState = + | AudioProcessorIdleState + | AudioProcessorLoadingState + | AudioProcessorReadyState + | AudioProcessorErrorState; + +export interface MediaAudioProcessorOptions extends AudioProcessOptions { + /** + * This phase supports the graph-native AudioWorklet strategy. + */ + readonly strategy: "worklet"; + /** + * An ESM URL passed to `audioWorklet.addModule`. Supply this when a bundler + * cannot resolve the package's default ESM worklet module URL. + */ + readonly workletModuleUrl?: string | URL; + /** + * Creates the owned AudioContext. This injection point supports tests and + * applications with specialized browser audio context configuration. + */ + readonly createAudioContext?: () => AudioContext; +} diff --git a/packages/react-user-media/src/audio/audio-worker.ts b/packages/react-user-media/src/audio/audio-worker.ts new file mode 100644 index 0000000..85286eb --- /dev/null +++ b/packages/react-user-media/src/audio/audio-worker.ts @@ -0,0 +1,11 @@ +/** + * Creates the library's ESM Dedicated Worker for asynchronous PCM jobs. + * + * CommonJS consumers should pass a `createWorker` override to + * {@link useAudioWorker}, because module-worker URLs are ESM-only. + */ +export function createAudioWorker(): Worker { + return new Worker(new URL("./worker/audio-worker.js", import.meta.url), { + type: "module", + }); +} diff --git a/packages/react-user-media/src/audio/types.ts b/packages/react-user-media/src/audio/types.ts new file mode 100644 index 0000000..c16bd45 --- /dev/null +++ b/packages/react-user-media/src/audio/types.ts @@ -0,0 +1,28 @@ +export type { + AudioFrame, + AudioProcessOptions, + AudioProcessResult, +} from "../audio"; + +import type { + AudioFrame, + AudioProcessOptions, + AudioProcessResult, +} from "../audio"; + +export type AudioProcessSubscriber = ( + result: AudioProcessResult, +) => void | Promise; + +/** + * Audio-specific job surface exposed from the Dedicated Worker. + * + * Its configure/process/subscribe/dispose lifecycle intentionally mirrors the + * transferable job shape future media-specific worker APIs can use. + */ +export interface AudioWorkerApi { + configure(options: AudioProcessOptions): Promise; + processFrame(frame: AudioFrame): Promise; + subscribe(onResult: AudioProcessSubscriber): Promise; + dispose(): Promise; +} diff --git a/packages/react-user-media/src/audio/use-audio-worker.ts b/packages/react-user-media/src/audio/use-audio-worker.ts new file mode 100644 index 0000000..9839c76 --- /dev/null +++ b/packages/react-user-media/src/audio/use-audio-worker.ts @@ -0,0 +1,199 @@ +import * as Comlink from "comlink"; +import { useCallback, useEffect, useRef, useState } from "react"; +import { createAudioWorker } from "./audio-worker"; +import type { AudioProcessOptions, AudioWorkerApi } from "./types"; + +export type AudioWorkerProxy = Comlink.Remote; + +export interface UseAudioWorkerOptions { + createWorker?: () => Worker; + processOptions?: AudioProcessOptions; +} + +interface AudioWorkerStateBase { + api: AudioWorkerProxy | null; + error: Error | null; + isError: boolean; + isIdle: boolean; + isLoading: boolean; + isReady: boolean; + start(): void; + stop(): void; +} + +interface AudioWorkerIdleState extends AudioWorkerStateBase { + api: null; + error: null; + isError: false; + isIdle: true; + isLoading: false; + isReady: false; +} + +interface AudioWorkerLoadingState extends AudioWorkerStateBase { + api: null; + error: null; + isError: false; + isIdle: false; + isLoading: true; + isReady: false; +} + +interface AudioWorkerReadyState extends AudioWorkerStateBase { + api: AudioWorkerProxy; + error: null; + isError: false; + isIdle: false; + isLoading: false; + isReady: true; +} + +interface AudioWorkerErrorState extends AudioWorkerStateBase { + api: null; + error: Error; + isError: true; + isIdle: false; + isLoading: false; + isReady: false; +} + +export type AudioWorkerState = + | AudioWorkerIdleState + | AudioWorkerLoadingState + | AudioWorkerReadyState + | AudioWorkerErrorState; + +const idleState = { + api: null, + error: null, + isError: false, + isIdle: true, + isLoading: false, + isReady: false, +} as const; + +/** + * Manages an audio-job worker and its Comlink proxy. + * + * This hook does not acquire media or stop tracks. Use it for asynchronous + * buffer jobs; use an AudioWorklet for live stream DSP. + */ +export function useAudioWorker( + options: UseAudioWorkerOptions = {}, +): AudioWorkerState { + const workerRef = useRef(null); + const proxyRef = useRef(null); + const sessionIdRef = useRef(0); + const [state, setState] = + useState>(idleState); + + const releaseCurrentWorker = useCallback(function releaseCurrentWorker() { + sessionIdRef.current += 1; + + const proxy = proxyRef.current; + proxyRef.current = null; + if (proxy) { + void proxy.dispose().catch(() => undefined); + proxy[Comlink.releaseProxy](); + } + + const worker = workerRef.current; + workerRef.current = null; + worker?.terminate(); + }, []); + + const stop = useCallback( + function stopAudioWorker() { + releaseCurrentWorker(); + setState(idleState); + }, + [releaseCurrentWorker], + ); + + const start = useCallback( + function startAudioWorker() { + releaseCurrentWorker(); + const sessionId = sessionIdRef.current; + setState({ + api: null, + error: null, + isError: false, + isIdle: false, + isLoading: true, + isReady: false, + }); + + let worker: Worker; + let proxy: AudioWorkerProxy; + try { + worker = (options.createWorker ?? createAudioWorker)(); + proxy = Comlink.wrap(worker); + } catch (error) { + if (sessionIdRef.current === sessionId) { + setState({ + api: null, + error: toError(error), + isError: true, + isIdle: false, + isLoading: false, + isReady: false, + }); + } + return; + } + + workerRef.current = worker; + proxyRef.current = proxy; + + void proxy + .configure(options.processOptions ?? { sampleRate: 48_000 }) + .then(() => { + if (sessionIdRef.current !== sessionId) { + return; + } + + setState({ + api: proxy, + error: null, + isError: false, + isIdle: false, + isLoading: false, + isReady: true, + }); + }) + .catch((error: unknown) => { + if (sessionIdRef.current !== sessionId) { + return; + } + + releaseCurrentWorker(); + setState({ + api: null, + error: toError(error), + isError: true, + isIdle: false, + isLoading: false, + isReady: false, + }); + }); + }, + [options.createWorker, options.processOptions, releaseCurrentWorker], + ); + + useEffect( + function cleanupAudioWorkerOnUnmount() { + return releaseCurrentWorker; + }, + [releaseCurrentWorker], + ); + + return { + ...state, + start, + stop, + } as AudioWorkerState; +} + +function toError(error: unknown): Error { + return error instanceof Error ? error : new Error(String(error)); +} diff --git a/packages/react-user-media/src/audio/worker/audio-worker-api.ts b/packages/react-user-media/src/audio/worker/audio-worker-api.ts new file mode 100644 index 0000000..c9d4235 --- /dev/null +++ b/packages/react-user-media/src/audio/worker/audio-worker-api.ts @@ -0,0 +1,106 @@ +import type { + AudioFrame, + AudioProcessOptions, + AudioProcessResult, + AudioProcessSubscriber, + AudioWorkerApi, +} from "../types"; + +export interface AudioWorkerApiOverrides { + configure?: (options: AudioProcessOptions) => void | Promise; +} + +/** + * Creates the audio-only job API exposed by the Dedicated Worker. + * + * Keeping PCM analysis behind this narrow API allows future media job kinds to + * reuse the lifecycle and transfer model without changing this contract. + */ +export function createAudioWorkerApi( + overrides: AudioWorkerApiOverrides = {}, +): AudioWorkerApi { + let isDisposed = false; + let options: AudioProcessOptions | null = null; + const subscribers = new Set(); + + function assertActive() { + if (isDisposed) { + throw new Error("Audio worker has been disposed"); + } + } + + return { + async configure(nextOptions) { + assertActive(); + + if ( + nextOptions.sampleRate === undefined || + !Number.isFinite(nextOptions.sampleRate) || + nextOptions.sampleRate <= 0 + ) { + throw new Error("Audio worker requires a positive sampleRate"); + } + + await overrides.configure?.(nextOptions); + assertActive(); + options = nextOptions; + }, + + async processFrame(frame) { + assertActive(); + + if (options === null || options.sampleRate === undefined) { + throw new Error( + "Audio worker must be configured before processing frames", + ); + } + + if (frame.sampleRate !== options.sampleRate) { + throw new Error( + "Audio frame sampleRate does not match worker configuration", + ); + } + + const result = calculateLevels(frame); + + for (const subscriber of subscribers) { + await subscriber(result); + } + + return result; + }, + + async subscribe(onResult) { + assertActive(); + subscribers.add(onResult); + }, + + async dispose() { + isDisposed = true; + options = null; + subscribers.clear(); + }, + }; +} + +function calculateLevels(frame: AudioFrame): AudioProcessResult { + let peak = 0; + let sumOfSquares = 0; + let sampleCount = 0; + + for (const channel of frame.channelData) { + for (const sample of channel) { + const amplitude = Math.abs(sample); + peak = Math.max(peak, amplitude); + sumOfSquares += sample * sample; + sampleCount += 1; + } + } + + return { + frameLength: frame.channelData[0]?.length ?? 0, + peak, + rms: sampleCount === 0 ? 0 : Math.sqrt(sumOfSquares / sampleCount), + timestamp: frame.timestamp ?? 0, + }; +} diff --git a/packages/react-user-media/src/audio/worker/audio-worker.spec.tsx b/packages/react-user-media/src/audio/worker/audio-worker.spec.tsx new file mode 100644 index 0000000..0039459 --- /dev/null +++ b/packages/react-user-media/src/audio/worker/audio-worker.spec.tsx @@ -0,0 +1,253 @@ +import "@testing-library/jest-dom"; +import { act, cleanup, render, screen, waitFor } from "@testing-library/react"; +import * as Comlink from "comlink"; +import { useEffect } from "react"; +import { afterEach, expect, test, vi } from "vitest"; +import type { AudioWorkerApi } from "../types"; +import { useAudioWorker } from "../use-audio-worker"; +import { createAudioWorkerApi } from "./audio-worker-api"; + +afterEach(() => { + cleanup(); + vi.restoreAllMocks(); +}); + +function createEndpoint(api: AudioWorkerApi) { + const channel = new MessageChannel(); + Comlink.expose(api, channel.port1); + const terminate = vi.fn(() => channel.port2.close()); + + return { + worker: Object.assign(channel.port2, { terminate }) as unknown as Worker, + terminate, + }; +} + +function WorkerTestComponent({ + createWorker, + onStateChange, +}: { + createWorker: () => Worker; + onStateChange?: (state: string) => void; +}) { + const { api, error, isError, isIdle, isLoading, isReady, start, stop } = + useAudioWorker({ createWorker }); + + const state = isReady + ? "ready" + : isLoading + ? "loading" + : isError + ? "error" + : "idle"; + + useEffect(() => { + onStateChange?.(state); + }, [onStateChange, state]); + + return ( + <> + + +

    {state}

    +

    {api ? "available" : "none"}

    +

    {error?.message ?? "none"}

    +

    {String(isIdle)}

    + + ); +} + +test("delivers results to a Comlink-proxied subscriber", async () => { + const { worker } = createEndpoint(createAudioWorkerApi()); + const api = Comlink.wrap(worker); + const onResult = vi.fn(); + + await api.configure({ sampleRate: 48_000 }); + await api.subscribe(Comlink.proxy(onResult)); + + const result = await api.processFrame({ + channelData: [new Float32Array([0.5, -0.5])], + sampleRate: 48_000, + }); + + expect(result).toMatchObject({ peak: 0.5, rms: 0.5, frameLength: 2 }); + await waitFor(() => { + expect(onResult).toHaveBeenCalledWith(result); + }); + + await api.dispose(); + await expect( + api.processFrame({ + channelData: [new Float32Array([0.5])], + sampleRate: 48_000, + }), + ).rejects.toThrow("disposed"); + expect(onResult).toHaveBeenCalledOnce(); + + api[Comlink.releaseProxy](); + worker.terminate(); +}); + +test("fails closed after dispose", async () => { + const api = createAudioWorkerApi(); + await api.configure({ sampleRate: 48_000 }); + await api.dispose(); + + await expect( + api.processFrame({ + channelData: [new Float32Array([0.5])], + sampleRate: 48_000, + }), + ).rejects.toThrow("disposed"); +}); + +test("processes a transferred PCM buffer through a Comlink proxy", async () => { + const { worker } = createEndpoint(createAudioWorkerApi()); + const api = Comlink.wrap(worker); + const samples = new Float32Array([0.25, -0.25]); + const frame = { + channelData: [samples], + sampleRate: 48_000, + }; + + await api.configure({ sampleRate: 48_000 }); + const result = await api.processFrame( + Comlink.transfer(frame, [samples.buffer]), + ); + + expect(result).toMatchObject({ peak: 0.25, rms: 0.25, frameLength: 2 }); + expect(samples.byteLength).toBe(0); + + api[Comlink.releaseProxy](); + worker.terminate(); +}); + +test("releases the proxy and terminates a custom worker on stop", async () => { + const { worker, terminate } = createEndpoint(createAudioWorkerApi()); + + render( worker} />); + + act(() => { + screen.getByText("Start").click(); + }); + + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("ready"); + expect(screen.getByTestId("api")).toHaveTextContent("available"); + }); + + act(() => { + screen.getByText("Stop").click(); + }); + + expect(terminate).toHaveBeenCalledOnce(); + expect(screen.getByTestId("state")).toHaveTextContent("idle"); + expect(screen.getByTestId("api")).toHaveTextContent("none"); +}); + +test("releases and terminates an active worker on unmount", async () => { + let resolveConfigure!: () => void; + const configure = new Promise((resolve) => { + resolveConfigure = resolve; + }); + const finalizer = vi.fn(); + const api = Object.assign( + createAudioWorkerApi({ configure: () => configure }), + { + [Comlink.finalizer]: finalizer, + }, + ); + const { worker, terminate } = createEndpoint(api); + const onStateChange = vi.fn(); + + const { unmount } = render( + worker} + onStateChange={onStateChange} + />, + ); + + act(() => { + screen.getByText("Start").click(); + }); + + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("loading"); + }); + + unmount(); + + await waitFor(() => { + expect(finalizer).toHaveBeenCalledOnce(); + }); + expect(terminate).toHaveBeenCalledOnce(); + + const stateUpdatesBeforeLateConfigure = onStateChange.mock.calls.length; + await act(async () => { + resolveConfigure(); + await configure; + }); + + expect(onStateChange).toHaveBeenCalledTimes(stateUpdatesBeforeLateConfigure); +}); + +test("reports worker initialization failures without leaking a worker", async () => { + const createWorker = vi.fn(() => { + throw new Error("worker unavailable"); + }); + + render(); + + act(() => { + screen.getByText("Start").click(); + }); + + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("error"); + expect(screen.getByTestId("error")).toHaveTextContent("worker unavailable"); + }); + expect(screen.getByTestId("api")).toHaveTextContent("none"); +}); + +test("ignores a previous worker session that completes after restart", async () => { + let resolveFirstConfigure!: () => void; + const firstConfigure = new Promise((resolve) => { + resolveFirstConfigure = resolve; + }); + const firstApi = createAudioWorkerApi({ + configure: () => firstConfigure, + }); + const secondApi = createAudioWorkerApi(); + const first = createEndpoint(firstApi); + const second = createEndpoint(secondApi); + const workers = [first.worker, second.worker]; + + render( + { + const worker = workers.shift(); + if (!worker) { + throw new Error("unexpected worker"); + } + return worker; + }} + />, + ); + + act(() => { + screen.getByText("Start").click(); + screen.getByText("Start").click(); + }); + + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("ready"); + }); + + await act(async () => { + resolveFirstConfigure(); + await firstConfigure; + }); + + expect(screen.getByTestId("state")).toHaveTextContent("ready"); + expect(first.terminate).toHaveBeenCalledOnce(); +}); diff --git a/packages/react-user-media/src/audio/worker/audio-worker.ts b/packages/react-user-media/src/audio/worker/audio-worker.ts new file mode 100644 index 0000000..03709a9 --- /dev/null +++ b/packages/react-user-media/src/audio/worker/audio-worker.ts @@ -0,0 +1,4 @@ +import * as Comlink from "comlink"; +import { createAudioWorkerApi } from "./audio-worker-api"; + +Comlink.expose(createAudioWorkerApi()); diff --git a/packages/react-user-media/src/audio/worklet-rpc/controller.spec.ts b/packages/react-user-media/src/audio/worklet-rpc/controller.spec.ts new file mode 100644 index 0000000..046919e --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/controller.spec.ts @@ -0,0 +1,47 @@ +import { describe, expect, test, vi } from "vitest"; +import { WorkletRpcController } from "./controller"; + +describe("WorkletRpcController", () => { + test("applies configure synchronously to subsequent process calls", async () => { + const controller = new WorkletRpcController(); + + await controller.configure({ passthrough: false }); + const outputs = [[new Float32Array([99, 99])]]; + + expect(controller.process([[new Float32Array([3, 4])]], outputs, 1)).toBe( + true, + ); + expect(outputs[0]?.[0]).toEqual(new Float32Array([0, 0])); + }); + + test("sends level results to subscribed callbacks", async () => { + const controller = new WorkletRpcController(); + const onResult = vi.fn(); + + await controller.subscribe(onResult); + controller.process( + [[new Float32Array([3, 4])]], + [[new Float32Array(2)]], + 12.5, + ); + + expect(onResult).toHaveBeenCalledWith({ + rms: Math.sqrt(12.5), + peak: 4, + timestamp: 12.5, + }); + }); + + test("fails closed after disposal", async () => { + const controller = new WorkletRpcController(); + const onResult = vi.fn(); + + await controller.subscribe(onResult); + await controller.dispose(); + + await expect(controller.configure({})).rejects.toThrow("disposed"); + await expect(controller.subscribe(onResult)).rejects.toThrow("disposed"); + controller.process([[new Float32Array([1])]], [[new Float32Array(1)]], 1); + expect(onResult).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/react-user-media/src/audio/worklet-rpc/controller.ts b/packages/react-user-media/src/audio/worklet-rpc/controller.ts new file mode 100644 index 0000000..cf3b837 --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/controller.ts @@ -0,0 +1,86 @@ +import type { AudioProcessResult } from "./types"; + +export type AudioWorkletRpcOptions = { + passthrough?: boolean; +}; + +export type { AudioProcessResult } from "./types"; + +export type AudioWorkletRpcApi = { + configure(options: AudioWorkletRpcOptions): Promise; + subscribe(onResult: (result: AudioProcessResult) => void): Promise; + dispose(): Promise; +}; + +/** + * Synchronous DSP state owned by the AudioWorkletProcessor. + * + * The public methods are async only because Comlink serializes calls as + * promises. `process` deliberately remains synchronous. + */ +export class WorkletRpcController implements AudioWorkletRpcApi { + #disposed = false; + #passthrough = true; + #subscriptions = new Set<(result: AudioProcessResult) => void>(); + + async configure(options: AudioWorkletRpcOptions): Promise { + this.#assertActive(); + this.#passthrough = options.passthrough ?? true; + } + + async subscribe( + onResult: (result: AudioProcessResult) => void, + ): Promise { + this.#assertActive(); + this.#subscriptions.add(onResult); + } + + async dispose(): Promise { + this.#disposed = true; + this.#subscriptions.clear(); + } + + process( + inputs: Float32Array[][], + outputs: Float32Array[][], + timestamp: number, + ): boolean { + if (this.#disposed) { + return false; + } + + const samples = inputs[0]?.[0]; + const output = outputs[0]?.[0]; + if (!samples) { + return true; + } + + let sumSquares = 0; + let peak = 0; + for (let index = 0; index < samples.length; index += 1) { + const sample = samples[index] ?? 0; + sumSquares += sample * sample; + peak = Math.max(peak, Math.abs(sample)); + if (output) { + output[index] = this.#passthrough ? sample : 0; + } + } + + const result = { + rms: Math.sqrt(sumSquares / samples.length), + peak, + timestamp, + }; + for (const subscription of this.#subscriptions) { + subscription(result); + } + + return true; + } + + #assertActive(): void { + if (this.#disposed) { + throw new Error("AudioWorklet RPC session is disposed"); + } + } +} diff --git a/packages/react-user-media/src/audio/worklet-rpc/example.tsx b/packages/react-user-media/src/audio/worklet-rpc/example.tsx new file mode 100644 index 0000000..0e4eff4 --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/example.tsx @@ -0,0 +1,30 @@ +import { useMemo, useState } from "react"; +import { type AudioProcessResult, useMediaAudioWorkletRpc } from "./index"; + +/** + * Copy into an application after supplying a microphone MediaStream. + * The module URL override is useful when the package's ESM asset URL cannot be + * resolved by the consuming bundler. + */ +export function AudioWorkletRpcDemo({ + stream, + workletModuleUrl, +}: { + stream: MediaStream | null; + workletModuleUrl?: URL | string; +}) { + const [result, setResult] = useState(); + const options = useMemo( + () => ({ onResult: setResult, workletModuleUrl }), + [workletModuleUrl], + ); + const state = useMediaAudioWorkletRpc(stream, options); + + return ( + + {state.status === "ready" && result + ? `RMS ${result.rms.toFixed(3)}, peak ${result.peak.toFixed(3)}` + : state.status} + + ); +} diff --git a/packages/react-user-media/src/audio/worklet-rpc/hook.spec.tsx b/packages/react-user-media/src/audio/worklet-rpc/hook.spec.tsx new file mode 100644 index 0000000..5969dfb --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/hook.spec.tsx @@ -0,0 +1,34 @@ +import { renderHook, waitFor } from "@testing-library/react"; +import { vi } from "vitest"; +import type { AudioWorkletRpcSession } from "./session"; + +const { createSession } = vi.hoisted(() => ({ + createSession: vi.fn(), +})); + +vi.mock("./session", () => ({ + createAudioWorkletRpcSession: createSession, +})); + +import { useMediaAudioWorkletRpc } from "./hook"; + +test("does not recreate its session for equivalent inline options", async () => { + const stream = {} as MediaStream; + createSession.mockResolvedValue({ + api: {}, + dispose: vi.fn().mockResolvedValue(undefined), + } satisfies AudioWorkletRpcSession); + + const { rerender } = renderHook( + ({ workletModuleUrl }: { workletModuleUrl: string }) => + useMediaAudioWorkletRpc(stream, { workletModuleUrl }), + { initialProps: { workletModuleUrl: "worklet.js" } }, + ); + + await waitFor(() => { + expect(createSession).toHaveBeenCalledTimes(1); + }); + rerender({ workletModuleUrl: "worklet.js" }); + + expect(createSession).toHaveBeenCalledTimes(1); +}); diff --git a/packages/react-user-media/src/audio/worklet-rpc/hook.ts b/packages/react-user-media/src/audio/worklet-rpc/hook.ts new file mode 100644 index 0000000..53c8299 --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/hook.ts @@ -0,0 +1,87 @@ +import { useEffect, useMemo, useState } from "react"; +import type { AudioWorkletRpcApi } from "./controller"; +import { + type AudioWorkletRpcSessionOptions, + createAudioWorkletRpcSession, +} from "./session"; + +const defaultOptions: AudioWorkletRpcSessionOptions = {}; + +export type AudioWorkletRpcState = + | { status: "idle" | "loading"; api?: undefined; error?: undefined } + | { status: "ready"; api: AudioWorkletRpcApi; error?: undefined } + | { status: "error"; api?: undefined; error: Error }; + +/** + * React lifecycle wrapper for the AudioWorklet + Comlink control strategy. + * It never stops tracks owned by the supplied MediaStream. + */ +export function useMediaAudioWorkletRpc( + stream: MediaStream | null | undefined, + options: AudioWorkletRpcSessionOptions = defaultOptions, +): AudioWorkletRpcState { + const [state, setState] = useState({ + status: "idle", + }); + const { + audioContext, + createAudioContext, + createWorkletNode, + onResult, + processorName, + workletModuleUrl, + } = options; + const sessionOptions = useMemo( + () => ({ + audioContext, + createAudioContext, + createWorkletNode, + onResult, + processorName, + workletModuleUrl, + }), + [ + audioContext, + createAudioContext, + createWorkletNode, + onResult, + processorName, + workletModuleUrl, + ], + ); + + useEffect(() => { + if (!stream) { + setState({ status: "idle" }); + return; + } + + let active = true; + let session: Awaited>; + setState({ status: "loading" }); + + void createAudioWorkletRpcSession(stream, sessionOptions) + .then((createdSession) => { + if (!active) { + return createdSession.dispose(); + } + session = createdSession; + setState({ status: "ready", api: createdSession.api }); + }) + .catch((error: unknown) => { + if (active) { + setState({ + status: "error", + error: error instanceof Error ? error : new Error(String(error)), + }); + } + }); + + return () => { + active = false; + void session?.dispose(); + }; + }, [stream, sessionOptions]); + + return state; +} diff --git a/packages/react-user-media/src/audio/worklet-rpc/index.spec.ts b/packages/react-user-media/src/audio/worklet-rpc/index.spec.ts new file mode 100644 index 0000000..088a16a --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/index.spec.ts @@ -0,0 +1,54 @@ +import { expose } from "comlink"; +import { describe, expect, test, vi } from "vitest"; +import { WorkletRpcController } from "./controller"; +import { createAudioWorkletRpcSession } from "./index"; + +describe("createAudioWorkletRpcSession", () => { + test("wires a Comlink control port and tears down without touching stream tracks", async () => { + const channel = new MessageChannel(); + const controller = new WorkletRpcController(); + expose(controller, channel.port1); + const source = { connect: vi.fn(), disconnect: vi.fn() }; + const node = { + connect: vi.fn(), + disconnect: vi.fn(), + port: channel.port2, + }; + const silentGain = { + connect: vi.fn(), + disconnect: vi.fn(), + gain: { value: 1 }, + }; + const context = { + audioWorklet: { addModule: vi.fn().mockResolvedValue(undefined) }, + close: vi.fn().mockResolvedValue(undefined), + createGain: vi.fn(() => silentGain), + createMediaStreamSource: vi.fn(() => source), + destination: {}, + resume: vi.fn().mockResolvedValue(undefined), + }; + const stream = { getTracks: vi.fn() } as unknown as MediaStream; + + const session = await createAudioWorkletRpcSession(stream, { + audioContext: context as unknown as AudioContext, + createWorkletNode: () => node as unknown as AudioWorkletNode, + workletModuleUrl: "worklet-rpc-test.js", + }); + await session.api.configure({ passthrough: false }); + const outputs = [[new Float32Array([99])]]; + controller.process([[new Float32Array([1])]], outputs, 1); + + expect(outputs[0]?.[0]).toEqual(new Float32Array([0])); + await session.dispose(); + + expect(context.audioWorklet.addModule).toHaveBeenCalledWith( + "worklet-rpc-test.js", + ); + expect(context.resume).toHaveBeenCalledOnce(); + expect(source.disconnect).toHaveBeenCalledOnce(); + expect(node.disconnect).toHaveBeenCalledOnce(); + expect(silentGain.disconnect).toHaveBeenCalledOnce(); + expect(context.close).not.toHaveBeenCalled(); + expect(stream.getTracks).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/react-user-media/src/audio/worklet-rpc/index.ts b/packages/react-user-media/src/audio/worklet-rpc/index.ts new file mode 100644 index 0000000..a5547bf --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/index.ts @@ -0,0 +1,12 @@ +export type { + AudioProcessResult, + AudioWorkletRpcApi, + AudioWorkletRpcOptions, +} from "./controller"; +export type { AudioWorkletRpcState } from "./hook"; +export { useMediaAudioWorkletRpc } from "./hook"; +export { + type AudioWorkletRpcSession, + type AudioWorkletRpcSessionOptions, + createAudioWorkletRpcSession, +} from "./session"; diff --git a/packages/react-user-media/src/audio/worklet-rpc/port.spec.ts b/packages/react-user-media/src/audio/worklet-rpc/port.spec.ts new file mode 100644 index 0000000..c807da9 --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/port.spec.ts @@ -0,0 +1,30 @@ +import { proxy, releaseProxy, wrap } from "comlink"; +import { describe, expect, test, vi } from "vitest"; +import type { AudioWorkletRpcApi } from "./controller"; +import { WorkletRpcController } from "./controller"; +import { exposeWorkletRpc } from "./port"; + +describe("exposeWorkletRpc", () => { + test("configures and subscribes through a Comlink MessagePort proxy", async () => { + const channel = new MessageChannel(); + const controller = new WorkletRpcController(); + exposeWorkletRpc(controller, channel.port1); + const api = wrap(channel.port2); + const onResult = vi.fn(); + + await api.configure({ passthrough: false }); + await api.subscribe(proxy(onResult)); + const outputs = [[new Float32Array([99, 99])]]; + controller.process([[new Float32Array([3, 4])]], outputs, 12.5); + + expect(outputs[0]?.[0]).toEqual(new Float32Array([0, 0])); + await vi.waitFor(() => + expect(onResult).toHaveBeenCalledWith({ + rms: Math.sqrt(12.5), + peak: 4, + timestamp: 12.5, + }), + ); + api[releaseProxy](); + }); +}); diff --git a/packages/react-user-media/src/audio/worklet-rpc/port.ts b/packages/react-user-media/src/audio/worklet-rpc/port.ts new file mode 100644 index 0000000..e65f566 --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/port.ts @@ -0,0 +1,15 @@ +import { expose } from "comlink"; +import type { AudioWorkletRpcApi } from "./controller"; + +/** + * Attaches the control plane to an AudioWorkletNode MessagePort. + * + * `process()` does not participate in Comlink RPC: it remains a synchronous + * call on the audio rendering thread. + */ +export function exposeWorkletRpc( + api: AudioWorkletRpcApi, + port: MessagePort, +): void { + expose(api, port); +} diff --git a/packages/react-user-media/src/audio/worklet-rpc/processor.ts b/packages/react-user-media/src/audio/worklet-rpc/processor.ts new file mode 100644 index 0000000..1cb233c --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/processor.ts @@ -0,0 +1,33 @@ +import { WorkletRpcController } from "./controller"; +import { exposeWorkletRpc } from "./port"; + +declare abstract class AudioWorkletProcessor { + readonly port: MessagePort; + abstract process( + inputs: Float32Array[][], + outputs: Float32Array[][], + parameters: Record, + ): boolean; +} + +declare function registerProcessor( + name: string, + processor: typeof AudioWorkletProcessor, +): void; + +declare const currentTime: number; + +class AudioWorkletRpcProcessor extends AudioWorkletProcessor { + #controller = new WorkletRpcController(); + + constructor() { + super(); + exposeWorkletRpc(this.#controller, this.port); + } + + process(inputs: Float32Array[][], outputs: Float32Array[][]): boolean { + return this.#controller.process(inputs, outputs, currentTime); + } +} + +registerProcessor("react-user-media-worklet-rpc", AudioWorkletRpcProcessor); diff --git a/packages/react-user-media/src/audio/worklet-rpc/session.ts b/packages/react-user-media/src/audio/worklet-rpc/session.ts new file mode 100644 index 0000000..45e4c6a --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/session.ts @@ -0,0 +1,77 @@ +import { proxy, releaseProxy, wrap } from "comlink"; +import type { AudioProcessResult, AudioWorkletRpcApi } from "./controller"; + +export type AudioWorkletRpcSessionOptions = { + audioContext?: AudioContext; + createAudioContext?: () => AudioContext; + createWorkletNode?: ( + context: AudioContext, + processorName: string, + ) => AudioWorkletNode; + onResult?: (result: AudioProcessResult) => void; + processorName?: string; + workletModuleUrl?: URL | string; +}; + +export type AudioWorkletRpcSession = { + api: AudioWorkletRpcApi; + dispose(): Promise; +}; + +const defaultProcessorName = "react-user-media-worklet-rpc"; + +export async function createAudioWorkletRpcSession( + stream: MediaStream, + options: AudioWorkletRpcSessionOptions = {}, +): Promise { + const ownsContext = !options.audioContext; + const context = + options.audioContext ?? + options.createAudioContext?.() ?? + new AudioContext(); + const processorName = options.processorName ?? defaultProcessorName; + const moduleUrl = + options.workletModuleUrl ?? new URL("./processor.js", import.meta.url); + + await context.audioWorklet.addModule(moduleUrl); + const source = context.createMediaStreamSource(stream); + const node = + options.createWorkletNode?.(context, processorName) ?? + new AudioWorkletNode(context, processorName); + const silentGain = context.createGain(); + silentGain.gain.value = 0; + source.connect(node); + node.connect(silentGain); + silentGain.connect(context.destination); + await context.resume(); + + const api = wrap(node.port); + const resultCallback = options.onResult ? proxy(options.onResult) : undefined; + if (resultCallback) { + await api.subscribe(resultCallback); + } + + let disposed = false; + return { + api, + async dispose(): Promise { + if (disposed) { + return; + } + disposed = true; + try { + await api.dispose(); + } finally { + resultCallback?.[releaseProxy](); + api[releaseProxy](); + source.disconnect(); + node.disconnect(); + silentGain.disconnect(); + node.port.close(); + if (ownsContext) { + await context.close(); + } + } + }, + }; +} diff --git a/packages/react-user-media/src/audio/worklet-rpc/types.ts b/packages/react-user-media/src/audio/worklet-rpc/types.ts new file mode 100644 index 0000000..581e62d --- /dev/null +++ b/packages/react-user-media/src/audio/worklet-rpc/types.ts @@ -0,0 +1,8 @@ +export type { AudioProcessResult } from "../../audio"; + +/** + * Worklet-rpc configure options (passthrough / metric tuning). + */ +export type AudioWorkletRpcConfigureOptions = { + passthrough?: boolean; +}; diff --git a/packages/react-user-media/src/hooks/index.ts b/packages/react-user-media/src/hooks/index.ts index 08a580d..c68f2eb 100644 --- a/packages/react-user-media/src/hooks/index.ts +++ b/packages/react-user-media/src/hooks/index.ts @@ -1,4 +1,5 @@ export * from "./use-media"; +export * from "./use-media-audio-processor"; export * from "./use-media-devices"; export * from "./use-media-devices-ext"; export * from "./use-media-ext"; diff --git a/packages/react-user-media/src/hooks/use-media-audio-processor.spec.tsx b/packages/react-user-media/src/hooks/use-media-audio-processor.spec.tsx new file mode 100644 index 0000000..7f12799 --- /dev/null +++ b/packages/react-user-media/src/hooks/use-media-audio-processor.spec.tsx @@ -0,0 +1,236 @@ +import "@testing-library/jest-dom"; +import { act, cleanup, render, screen, waitFor } from "@testing-library/react"; +import { userEvent } from "@vitest/browser/context"; +import { afterEach, expect, test, vi } from "vitest"; +import levelMeterWorkletUrl from "../audio-worklet/level-meter.ts?url"; +import { useMediaAudioProcessor } from "./use-media-audio-processor"; + +class MockPort extends EventTarget { + readonly messages: unknown[] = []; + closed = false; + onmessage: ((event: MessageEvent) => void) | null = null; + + postMessage(message: unknown) { + this.messages.push(message); + } + + close() { + this.closed = true; + } + + emit(message: unknown) { + this.onmessage?.(new MessageEvent("message", { data: message })); + } +} + +class MockNode { + readonly port = new MockPort(); + disconnected = false; + + connect = vi.fn(); + + disconnect = vi.fn(() => { + this.disconnected = true; + }); +} + +class MockAudioContext { + readonly audioWorklet = { + addModule: vi.fn<() => Promise>().mockResolvedValue(), + }; + readonly destination = new MockNode(); + readonly source = new MockNode(); + readonly gain = new MockNode() as MockNode & { gain: { value: number } }; + close = vi.fn<() => Promise>().mockResolvedValue(); + resume = vi.fn<() => Promise>().mockResolvedValue(); + + constructor() { + this.gain.gain = { value: 1 }; + } + + createMediaStreamSource = vi.fn(() => this.source); + createGain = vi.fn(() => this.gain); +} + +function AudioProcessorHarness({ + media, + context, + workletModuleUrl = "https://example.test/level-meter.js", +}: { + media: MediaStream; + context: MockAudioContext | AudioContext; + workletModuleUrl?: string; +}) { + const processor = useMediaAudioProcessor({ + strategy: "worklet", + createAudioContext: () => context as unknown as AudioContext, + workletModuleUrl, + }); + + return ( + <> + + + + {processor.isError + ? "error" + : processor.isLoading + ? "loading" + : processor.isReady + ? "ready" + : "idle"} + + {processor.result?.rms ?? "none"} + + ); +} + +afterEach(() => { + cleanup(); + vi.unstubAllGlobals(); +}); + +test("reports a module-loading failure as an error", async () => { + const context = new MockAudioContext(); + context.audioWorklet.addModule.mockRejectedValueOnce( + new Error("module failed"), + ); + vi.stubGlobal("AudioWorkletNode", MockNode); + + render(); + + await act(async () => { + screen.getByRole("button", { name: "start" }).click(); + }); + + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("error"); + }); + expect(context.close).toHaveBeenCalledOnce(); +}); + +test("tears down the graph without stopping supplied tracks", async () => { + const context = new MockAudioContext(); + const stop = vi.fn(); + const media = { + getTracks: () => [{ stop }], + } as unknown as MediaStream; + vi.stubGlobal("AudioWorkletNode", MockNode); + + render(); + + await act(async () => { + screen.getByRole("button", { name: "start" }).click(); + }); + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("ready"); + }); + + await act(async () => { + screen.getByRole("button", { name: "stop" }).click(); + }); + + expect(context.source.disconnect).toHaveBeenCalledOnce(); + expect(context.gain.disconnect).toHaveBeenCalledOnce(); + expect(context.close).toHaveBeenCalledOnce(); + expect(stop).not.toHaveBeenCalled(); +}); + +test("exposes worklet level messages while its session is current", async () => { + const context = new MockAudioContext(); + vi.stubGlobal("AudioWorkletNode", MockNode); + + render(); + + await act(async () => { + screen.getByRole("button", { name: "start" }).click(); + }); + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("ready"); + }); + + await act(async () => { + context.source.connect.mock.calls[0]?.[0].port.emit({ + rms: 0.25, + peak: 0.5, + timestamp: 1, + }); + }); + + expect(screen.getByTestId("rms")).toHaveTextContent("0.25"); +}); + +test("tears down an active session when unmounted without stopping tracks", async () => { + const context = new MockAudioContext(); + const stop = vi.fn(); + const media = { + getTracks: () => [{ stop }], + } as unknown as MediaStream; + vi.stubGlobal("AudioWorkletNode", MockNode); + + const view = render( + , + ); + + await act(async () => { + screen.getByRole("button", { name: "start" }).click(); + }); + await waitFor(() => { + expect(screen.getByTestId("state")).toHaveTextContent("ready"); + }); + + const processor = context.source.connect.mock.calls[0]?.[0] as MockNode; + view.unmount(); + + expect(context.source.disconnect).toHaveBeenCalledOnce(); + expect(processor.disconnect).toHaveBeenCalledOnce(); + expect(context.gain.disconnect).toHaveBeenCalledOnce(); + expect(processor.port.closed).toBe(true); + expect(processor.port.onmessage).toBeNull(); + expect(context.close).toHaveBeenCalledOnce(); + expect(stop).not.toHaveBeenCalled(); +}); + +test("reports non-silent levels from a live synthetic audio graph", async () => { + const inputContext = new AudioContext(); + const processorContext = new AudioContext(); + const oscillator = inputContext.createOscillator(); + const destination = inputContext.createMediaStreamDestination(); + const source = processorContext.createMediaStreamSource(destination.stream); + oscillator.frequency.value = 440; + oscillator.connect(destination); + oscillator.start(); + + await processorContext.audioWorklet.addModule(levelMeterWorkletUrl); + const processor = new AudioWorkletNode( + processorContext, + "react-user-media-level-meter", + ); + const silentGain = processorContext.createGain(); + silentGain.gain.value = 0; + source.connect(processor); + processor.connect(silentGain); + silentGain.connect(processorContext.destination); + + const message = new Promise<{ rms: number; peak: number }>((resolve) => { + processor.port.onmessage = (event) => { + const levels = event.data as { rms: number; peak: number }; + if (levels.rms > 0 && levels.peak > 0) { + resolve(levels); + } + }; + }); + + render(); + await userEvent.click(screen.getByRole("button", { name: "Activate audio" })); + await Promise.all([inputContext.resume(), processorContext.resume()]); + + const levels = await message; + + expect(levels.rms).toBeGreaterThan(0); + expect(levels.peak).toBeGreaterThan(0); + + oscillator.stop(); + await inputContext.close(); + await processorContext.close(); +}); diff --git a/packages/react-user-media/src/hooks/use-media-audio-processor.ts b/packages/react-user-media/src/hooks/use-media-audio-processor.ts new file mode 100644 index 0000000..5fb0662 --- /dev/null +++ b/packages/react-user-media/src/hooks/use-media-audio-processor.ts @@ -0,0 +1,148 @@ +import { useCallback, useEffect, useMemo, useRef, useState } from "react"; +import type { + AudioProcessorState, + AudioProcessResult, + MediaAudioProcessorOptions, +} from "../audio"; +import { getLevelMeterWorkletModuleUrl } from "../audio-worklet"; +import type { ShallowShapeOf } from "../types"; + +interface WorkletGraph { + readonly context: AudioContext; + readonly source: MediaStreamAudioSourceNode; + readonly processor: AudioWorkletNode; + readonly silentGain: GainNode; +} + +function toError(error: unknown): Error { + return error instanceof Error ? error : new Error(String(error)); +} + +function disposeGraph(graph: WorkletGraph | undefined) { + if (!graph) { + return; + } + + graph.processor.port.onmessage = null; + graph.processor.port.close(); + graph.source.disconnect(); + graph.processor.disconnect(); + graph.silentGain.disconnect(); + void graph.context.close(); +} + +/** + * Connects an existing MediaStream to the built-in, realtime AudioWorklet + * level meter. It never obtains media or stops caller-owned stream tracks. + */ +export function useMediaAudioProcessor( + options: MediaAudioProcessorOptions, +): AudioProcessorState { + const [isLoading, setIsLoading] = useState(false); + const [isReady, setIsReady] = useState(false); + const [error, setError] = useState(null); + const [result, setResult] = useState(null); + const sessionId = useRef(0); + const graphRef = useRef(undefined); + + const stop = useCallback(function stopAudioProcessor() { + sessionId.current += 1; + disposeGraph(graphRef.current); + graphRef.current = undefined; + setIsLoading(false); + setIsReady(false); + setError(null); + setResult(null); + }, []); + + const start = useCallback( + function startAudioProcessor(media: MediaStream) { + const currentSessionId = sessionId.current + 1; + sessionId.current = currentSessionId; + disposeGraph(graphRef.current); + graphRef.current = undefined; + setIsLoading(true); + setIsReady(false); + setError(null); + setResult(null); + + void (async () => { + let graph: WorkletGraph | undefined; + let context: AudioContext | undefined; + + try { + context = options.createAudioContext?.() ?? new AudioContext(); + const moduleUrl = + options.workletModuleUrl ?? getLevelMeterWorkletModuleUrl(); + await context.audioWorklet.addModule(moduleUrl); + await context.resume(); + + const source = context.createMediaStreamSource(media); + const processor = new AudioWorkletNode( + context, + "react-user-media-level-meter", + ); + const silentGain = context.createGain(); + silentGain.gain.value = 0; + source.connect(processor); + processor.connect(silentGain); + silentGain.connect(context.destination); + + graph = { context, source, processor, silentGain }; + processor.port.onmessage = ( + event: MessageEvent, + ) => { + if (sessionId.current === currentSessionId) { + setResult(event.data); + } + }; + + if (sessionId.current !== currentSessionId) { + disposeGraph(graph); + return; + } + + graphRef.current = graph; + setIsReady(true); + } catch (cause) { + disposeGraph(graph); + if (!graph) { + void context?.close(); + } + if (sessionId.current === currentSessionId) { + setError(toError(cause)); + } + } finally { + if (sessionId.current === currentSessionId) { + setIsLoading(false); + } + } + })(); + }, + [options], + ); + + useEffect(function disposeAudioProcessorOnUnmount() { + return function teardownAudioProcessor() { + sessionId.current += 1; + disposeGraph(graphRef.current); + graphRef.current = undefined; + }; + }, []); + + const state = useMemo( + () => + ({ + isLoading, + isReady, + isError: error !== null, + error, + result, + start, + stop, + }) satisfies ShallowShapeOf, + [error, isLoading, isReady, result, start, stop], + ); + + return state as AudioProcessorState; +} diff --git a/packages/react-user-media/src/index.ts b/packages/react-user-media/src/index.ts index 1d241d5..a5651a9 100644 --- a/packages/react-user-media/src/index.ts +++ b/packages/react-user-media/src/index.ts @@ -1,3 +1,25 @@ +export type { + AudioFrame, + AudioProcessingStrategy, + AudioProcessOptions, + AudioProcessorErrorState, + AudioProcessorIdleState, + AudioProcessorLoadingState, + AudioProcessorReadyState, + AudioProcessorState, + AudioProcessResult, + MediaAudioProcessorOptions, +} from "./audio"; +export { createAudioWorker } from "./audio/audio-worker"; +export type { + AudioProcessSubscriber, + AudioWorkerApi, +} from "./audio/types"; +export { + type AudioWorkerState, + type UseAudioWorkerOptions, + useAudioWorker, +} from "./audio/use-audio-worker"; export * from "./close-media"; export * from "./components"; export * from "./hooks"; diff --git a/packages/react-user-media/tsconfig.types.json b/packages/react-user-media/tsconfig.types.json index c8a02d4..33842c0 100644 --- a/packages/react-user-media/tsconfig.types.json +++ b/packages/react-user-media/tsconfig.types.json @@ -4,7 +4,11 @@ "outDir": "./dist/types", "declaration": true, "declarationMap": true, - "jsx": "react-jsx" + "target": "ES2020", + "module": "ESNext", + "moduleResolution": "bundler", + "jsx": "react-jsx", + "skipLibCheck": true }, "include": ["src/**/*.ts", "src/**/*.tsx"], "exclude": ["src/**/*.spec.ts", "src/**/*.spec.tsx"] diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index fc5f6ce..2f34d75 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -84,6 +84,10 @@ importers: version: 7.3.6(@types/node@22.20.1)(yaml@2.9.0) packages/react-user-media: + dependencies: + comlink: + specifier: ^4.4.2 + version: 4.4.2 devDependencies: '@testing-library/dom': specifier: ^10.4.0 @@ -883,6 +887,9 @@ packages: color-name@1.1.4: resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==} + comlink@4.4.2: + resolution: {integrity: sha512-OxGdvBmJuNKSCMO4NTl1L47VRp6xn2wG4F/2hYzB6tiCb709otOxtEYCSvK80PtjODfXXZu8ds+Nw5kVCjqd2g==} + commander@4.1.1: resolution: {integrity: sha512-NOKm8xhkzAjzFx8B2v5OAHT+u5pRQc2UCa2Vq9jYL/31o2wi9mxBA7LIFs3sV5VSC49z6pEhfbMULvShKj26WA==} engines: {node: '>= 6'} @@ -2171,6 +2178,8 @@ snapshots: color-name@1.1.4: {} + comlink@4.4.2: {} + commander@4.1.1: {} confbox@0.1.8: {}