docs(openspec): audio processing strategies

.gitignore

@@ -31,3 +31,5 @@ coverage
 docs_out
 
 *.tsbuildinfo
+.worktrees
+.superpowers

openspec/changes/add-audio-processing-strategies/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-23

openspec/changes/add-audio-processing-strategies/design.md

@@ -0,0 +1,158 @@
+## Context
+
+`@bengreenier/react-user-media` acquires streams (`useMedia`), observes audio tracks, plays (`AudioPlayer`), and records (`useMediaRecorder`). There is no signal-processing path on main. Draft PR #6 explored hand-rolled workers + WebCodecs; this change does not revive it.
+
+Consumers need different primitives for different jobs. This design specs **three strategies** under one shared family so callers pick by workload, with common types and lifecycle conventions.
+
+Library constraints: React 19 peer, tsup CJS+ESM, Vitest browser tests, secure context, ESM-first module URLs for worklets/workers.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Shared contracts: `AudioFrame`, `AudioProcessOptions`, `AudioProcessResult`, discriminated idle/loading/ready/error, session invalidation, no default track-stop
+- Strategy A — **AudioWorklet realtime**: graph-native DSP on the audio thread
+- Strategy B — **Dedicated Worker + Comlink**: async buffer jobs via `expose`/`wrap`/`transfer`/`proxy`
+- Strategy C — **Worklet + Comlink RPC**: `process()` stays sync; Comlink models port control/results
+- Clear selection guidance; phased ship order; tree-shake-friendly subpath exports where practical
+- **Forward compatibility:** architecture MUST NOT preclude a future change adding video processing and/or WebCodecs on this library (especially via the worker + Comlink job path)
+
+**Non-Goals (deferred, not forbidden):**
+
+- One strategy pretending to cover all workloads
+- Shipping video strategies or WebCodecs APIs in *this* change
+- Replacing MediaRecorder
+- CJS-default worker/worklet factories without overrides
+- Heavy DSP suite beyond a v1 level-meter proving each path
+
+## Selection guide
+
+```
+  Need                              Use
+  ─────────────────────────────────────────────────────
+  Live mic meters / FX / VAD-lite   audio-worklet
+  Offline / heavy / buffer jobs     audio-worker (Comlink)
+  Live DSP + typed configure/sub    audio-worklet-rpc
+```
+
+## Decisions
+
+### One family, three capabilities
+
+**Choice:** Three OpenSpec capabilities (`audio-worklet`, `audio-worker`, `audio-worklet-rpc`) sharing types and lifecycle rules. React surface:
+
+- `useMediaAudioProcessor(media, { strategy: "worklet" | "worklet-rpc", ... })` for stream-oriented paths
+- `useAudioWorker(options?)` for buffer/job-oriented path (`strategy: "worker"` implicit)
+- Optional unified overload later; not required for v1
+
+**Rationale:** Lets consumers choose tradeoffs without learning three unrelated products; keeps specs separable for phased delivery.
+
+**Alternatives considered:** Single capability with only worklet (too narrow); worker-only + Comlink (rejects realtime); three unrelated hook families (DX tax).
+
+### Shared types and ownership
+
+**Choice:** Public shared types for frames/results/options. Media-capture owns `MediaStream` lifecycle. Processors MUST NOT call `getUserMedia` and MUST NOT stop tracks on unmount by default. Hooks follow boolean discriminants + generation/session ids (ignore late results after restart).
+
+**Rationale:** Matches existing library conventions (`useMedia`, `useMediaRecorder`).
+
+### Forward compatibility: video and WebCodecs
+
+**Choice:** This change ships **audio-only** APIs, but treats video/WebCodecs as a planned extension surface—not a closed door:
+
+- Keep lifecycle/session/ownership rules **media-kind agnostic** (idle/loading/ready/error, no default track-stop, ESM module overrides) so a future `video-*` capability can mirror them
+- Prefer namespaced audio types (`AudioFrame`, `useAudioWorker`, `audio-worklet` subpath) over sealing a single global `MediaProcessor` that only fits PCM—so video can add `VideoFrame` / WebCodecs types beside them without a breaking rename
+- Treat the **Comlink Dedicated Worker** path as the primary future home for WebCodecs (`AudioEncoder`/`VideoEncoder`, transferable `AudioData`/`VideoFrame`): same `configure` / process / `subscribe` / `dispose` + `transfer` shape
+- Do **not** hard-wire worker internals to “Float32 PCM forever” in a way that blocks transferring other frame types later (keep frame handling behind typed process methods / narrow modules)
+- AudioWorklet remains audio-specific (correct); video realtime would be a separate future primitive (e.g. `MediaStreamTrackProcessor` / transform streams), not forced through AudioWorklet
+
+**Rationale:** Priority is audio now; consumers and the library should still be able to grow into video/WebCodecs without throwing away the processing family.
+
+**Alternatives considered:** Listing video/WebCodecs as absolute non-goals—over-constrains the product; implementing WebCodecs in this change—out of priority/scope.
+
+### Strategy A — AudioWorklet realtime
+
+**Choice:**
+
+```
+  MediaStream → MediaStreamAudioSourceNode → AudioWorkletNode → silent Gain / destination
+```
+
+Processor implements sync `process(inputs, outputs, parameters)`. v1 built-in: RMS/peak (and optional passthrough to outputs). Metrics to main via `port.postMessage` at a throttled rate (e.g. ≤60 Hz), not every quantum. Hook handles `audioWorklet.addModule`, context create/resume (user-gesture aware), graph connect/teardown.
+
+**Rationale:** Correct clock for live stream DSP; avoids main-thread frame pull.
+
+**Alternatives considered:** AnalyserNode-only on main — simpler but not an extensible processing strategy; Dedicated Worker as primary — wrong clock.
+
+### Strategy B — Dedicated Worker + Comlink
+
+**Choice:** Depend on [`comlink`](https://www.npmjs.com/package/comlink) (GoogleChromeLabs). Worker `Comlink.expose`s:
+
+```ts
+interface AudioWorkerApi {
+  configure(options: AudioProcessOptions): Promise<void>;
+  processFrame(frame: AudioFrame): Promise<AudioProcessResult>;
+  subscribe(onResult: (r: AudioProcessResult) => void): Promise<void>;
+  dispose(): Promise<void>;
+}
+```
+
+Hot path uses `Comlink.transfer` for `ArrayBuffer`s. Main `Comlink.wrap`; teardown `releaseProxy` + `terminate`. Optional thin bridge can pull PCM from a stream for convenience, but docs steer live realtime use to worklet.
+
+**Rationale:** Best DX for async jobs; transfer avoids structured-clone copies.
+
+### Strategy C — Worklet + Comlink on port
+
+**Choice:** Same realtime `process()` as strategy A. Main thread `Comlink.wrap(audioWorkletNode.port)`; worklet side `Comlink.expose(controlApi, port)` for `configure` / `subscribe` / `dispose`. **No awaits inside `process()`** — configure mutates processor fields; results are queued/posted from process or a timer coalesced on the worklet side without blocking the quantum.
+
+**Rationale:** Keeps audio-thread correctness while giving Comlink-shaped control plane for consumers who want it.
+
+**Alternatives considered:** Hand-typed port protocol only (strategy A) — enough for many apps; forcing Comlink for all worklet users — unnecessary dependency weight.
+
+### Packaging
+
+**Choice:**
+
+- ESM subpaths for worklet module(s) and worker script(s), e.g. `@bengreenier/react-user-media/audio-worklet`, `.../audio-worker`
+- `addModule(url)` / `new Worker(url, { type: "module" })` via `import.meta.url` factories
+- Overrides: `workletModuleUrl`, `createWorker`, etc. for tests/bundlers
+- `comlink` as a dependency of worker and worklet-rpc entry paths; worklet-only consumers should not need to load Comlink if split correctly
+- Document CJS limitation
+
+**Rationale:** Dual publish makes default URL workers/worklets unreliable under `require`.
+
+### Implementation phases
+
+1. Shared types + **audio-worklet** (level meter + hook)
+2. **audio-worker** (Comlink job API + hook)
+3. **audio-worklet-rpc** (Comlink on port; reuse worklet DSP)
+
+**Rationale:** Delivers the common live-mic case first; each phase is independently demoable/testable.
+
+### Contrast with PR #6 / prior change
+
+**Choice:** Do not reuse PR #6’s hand-rolled message enums. Supersede planning change `add-audio-worker-comlink` (worker-only). WebCodecs ideas from that exploration remain valid for a **future** change on the Comlink worker path—not in this delivery.
+
+## Risks / Trade-offs
+
+- **[Risk] Three strategies increase API/doc surface** → Mitigation: selection guide, shared types, phased ship, subpath exports
+- **[Risk] Worklet `process()` glitches if too heavy** → Mitigation: keep v1 metering cheap; document budget; push heavy work to worker strategy
+- **[Risk] Comlink on worklet port misused inside process** → Mitigation: spec forbids async in process; code review + tests
+- **[Risk] Module URL / COOP packaging pain** → Mitigation: overrides + ESM docs; avoid SharedArrayBuffer requirement for v1
+- **[Risk] AudioContext suspended until gesture** → Mitigation: loading/error states; resume helper documented
+- **[Risk] Audio-first naming paints a corner against video** → Mitigation: namespaced audio modules/types; shared lifecycle conventions; worker transfer path kept frame-type-pluggable
+- **[Trade-off] Spec all three before all are implemented** → Accepted; tasks are phased with checkboxes
+- **[Trade-off] Defer WebCodecs/video** → Accepted for priority; extensibility decision above is mandatory
+
+## Migration Plan
+
+- Additive APIs only; no breaking changes to existing hooks
+- Minor version bump when first strategy ships; subsequent strategies additive
+- Remove superseded OpenSpec change `add-audio-worker-comlink` from `openspec/changes/`
+- Rollback: remove new exports/deps; no data migration
+
+## Open Questions
+
+- Exact public names: `useMediaAudioProcessor` vs `useAudioWorklet` / `useAudioWorkletRpc` as separate exports (lean: strategy param on stream hook + dedicated `useAudioWorker`)
+- Whether worklet-rpc is a separate npm subpath or a `strategy: "worklet-rpc"` flag (lean: strategy flag reusing worklet module + thin Comlink adapter)
+- Default metric post rate and whether results are per-channel or mixed (lean: mixed summary + optional per-channel in result type)
+- When to open a follow-up change for WebCodecs/video (after worker path ships vs. parallel spike)

openspec/changes/add-audio-processing-strategies/proposal.md

@@ -0,0 +1,42 @@
+## Why
+
+The library captures media but offers no off-main-thread processing path. Different workloads need different browser primitives: **AudioWorklet** for realtime graph DSP, a **Dedicated Worker + Comlink** for async buffer jobs, and **Comlink on the worklet port** when consumers want typed configure/subscribe without hand-rolled messages. Speccing all three as one family lets consumers pick the right tradeoff. This change prioritizes **audio**, but the patterns (lifecycle, Comlink jobs, transferables, ESM module packaging) MUST leave the door open for later **video / WebCodecs** work on the same library.
+
+## What Changes
+
+- Add a shared media-processing type/lifecycle family for audio first (frame/result/options, idle/loading/ready/error, stream ownership stays in media-capture), structured so video/WebCodecs can reuse the same conventions later
+- Add **AudioWorklet realtime** strategy: `MediaStream` → Web Audio graph → `AudioWorkletProcessor.process()`
+- Add **Dedicated Worker + Comlink** strategy: `expose`/`wrap`/`transfer` job API for PCM buffers (natural on-ramp for future WebCodecs encode/decode jobs)
+- Add **Worklet + Comlink RPC** strategy: realtime `process()` plus Comlink on `AudioWorkletNode.port` for control/results
+- Ship ESM-first module URLs for worklet and worker scripts; allow `create*` / URL overrides
+- Phased implementation: worklet → worker → worklet-rpc; shared types first
+- Examples + browser tests per strategy; selection guide in docs
+
+## Non-goals (this change only)
+
+- Implementing video processors, WebCodecs encode/decode APIs, or video strategies in this change (deferred; must remain possible)
+- Replacing `useMediaRecorder` or owning `getUserMedia` / track teardown by default
+- SharedWorker, ServiceWorker
+- Reviving draft PR #6’s hand-rolled message protocol wholesale (WebCodecs direction may return in a future change on the Comlink worker path)
+- Guaranteeing default factories under CJS `require` without overrides
+- Shipping a full DSP plugin marketplace (v1: level meter + extensibility hooks)
+
+## Capabilities
+
+### New Capabilities
+
+- `audio-worklet`: Realtime AudioWorklet processing of live `MediaStream` audio in the Web Audio graph, with React lifecycle hooks and port-based metrics.
+- `audio-worker`: Comlink-backed Dedicated Worker API and hooks for async PCM frame jobs (`processFrame` + transferables).
+- `audio-worklet-rpc`: Comlink RPC layered on `AudioWorkletNode.port` for typed configure/subscribe while DSP remains in `process()`.
+
+### Modified Capabilities
+
+- (none — `openspec/specs/` is empty; baseline changes are documentation-only)
+
+## Impact
+
+- Package: `packages/react-user-media` — shared audio types, worklet + worker modules, hooks, `comlink` dependency, multi-entry packaging/exports
+- Examples: demos for at least worklet metering; optional worker/rpc demos
+- Tests: Vitest browser coverage per strategy; shared lifecycle contracts
+- Docs: strategy selection guide; ESM module URL / override notes
+- Supersedes planning-only change `add-audio-worker-comlink` (worker-only intent)

openspec/changes/add-audio-processing-strategies/specs/audio-worker/spec.md

@@ -0,0 +1,80 @@
+## ADDED Requirements
+
+### Requirement: Comlink-exposed worker API
+The library SHALL ship a Dedicated Worker module that exposes an audio job API via `Comlink.expose`. The main thread SHALL obtain a typed proxy with `Comlink.wrap`. The shared TypeScript contract SHALL include at least `configure`, `processFrame`, and `dispose`. Proxy method results SHALL be awaitable.
+
+#### Scenario: Wrap and configure
+- **WHEN** the consumer creates the library audio worker and wraps it with Comlink
+- **THEN** awaiting `configure` with valid options resolves and subsequent `processFrame` calls are accepted
+
+#### Scenario: Dispose fails closed
+- **WHEN** the consumer awaits `dispose` on the proxy
+- **THEN** further `processFrame` calls reject or otherwise fail closed and worker-owned state is cleared
+
+### Requirement: Transferable PCM frames
+`processFrame` SHALL accept PCM frame data movable with `Comlink.transfer` so `ArrayBuffer` ownership transfers to the worker on the hot path. After a successful transfer of a buffer used as an argument, that buffer MUST NOT remain usable on the calling thread (detached / zero-length semantics).
+
+#### Scenario: Process transferred frame
+- **WHEN** the consumer calls `processFrame` with channel data wrapped in `Comlink.transfer`
+- **THEN** the worker returns an `AudioProcessResult`
+- **AND** the transferred `ArrayBuffer` is detached on the caller side
+
+### Requirement: Built-in level analysis
+After configuration, `processFrame` SHALL support at least RMS and peak computation over the provided frame, returned as `AudioProcessResult` using the shared type family.
+
+#### Scenario: RMS and peak for a known frame
+- **WHEN** the consumer processes a frame of known constant amplitude
+- **THEN** the result’s peak reflects that amplitude and RMS is within an accepted numeric tolerance
+
+### Requirement: Optional subscription via Comlink.proxy
+The worker API MAY expose `subscribe(onResult)` where `onResult` is passed with `Comlink.proxy`. Unsubscribe or `dispose` MUST stop further callbacks for that session.
+
+#### Scenario: Proxied callback receives results
+- **WHEN** the consumer subscribes with a proxied callback and processes frames
+- **THEN** the callback is invoked with `AudioProcessResult` values
+- **WHEN** the consumer disposes the API
+- **THEN** no further subscription callbacks are delivered for that session
+
+### Requirement: `useAudioWorker` lifecycle hook
+The library SHALL export `useAudioWorker` that manages worker creation and Comlink wrap/release with idle/loading/ready/error state. Unmount and stop MUST `releaseProxy` and `worker.terminate()`. A session/generation id MUST ignore late results after restart.
+
+#### Scenario: Ready after start
+- **WHEN** the consumer starts the hook and the worker initializes successfully
+- **THEN** state is ready and a usable Comlink proxy API is available
+
+#### Scenario: Cleanup on unmount
+- **WHEN** the component unmounts while a worker is active
+- **THEN** the proxy is released and the worker is terminated
+
+#### Scenario: Ignores late results after restart
+- **WHEN** the consumer restarts the worker and a previous instance later delivers a result
+- **THEN** the new session’s ready-state data is not overwritten by the stale result
+
+#### Scenario: Init failure
+- **WHEN** worker construction or Comlink setup fails
+- **THEN** state is error, ready is false, and no leaked worker remains owned by the hook
+
+### Requirement: Stream bridge does not own capture
+If a convenience bridge feeds `MediaStream` PCM into the worker, it MUST NOT call `getUserMedia` and MUST NOT stop tracks on the provided stream by default when the bridge stops. Docs SHOULD steer realtime live-stream DSP to the worklet strategy.
+
+#### Scenario: Bridge stop leaves tracks running
+- **WHEN** a stream-fed worker bridge stops
+- **THEN** tracks on the provided `MediaStream` remain in their prior `readyState`
+
+### Requirement: ESM worker packaging and overrides
+The package SHALL provide an ESM-consumable worker factory or URL. Consumers MUST be able to override worker creation for tests and bundlers. Default CJS `require` without an override MAY be unsupported and MUST be documented.
+
+#### Scenario: Custom createWorker in tests
+- **WHEN** the consumer supplies a `createWorker` override returning a Comlink-compatible endpoint
+- **THEN** `useAudioWorker` can reach ready state without the production worker script
+
+### Requirement: Extensible job surface for future media kinds
+The worker + Comlink lifecycle (`configure`, frame processing, optional `subscribe`, `dispose`, transferable payloads, hook session cleanup) SHALL be structured so a future library change can add video and/or WebCodecs job APIs without replacing that lifecycle. This change MUST NOT ship video or WebCodecs APIs; audio PCM remains the only implemented frame kind.
+
+#### Scenario: Audio-only in this change
+- **WHEN** a consumer uses the shipped worker API from this change
+- **THEN** only audio PCM processing APIs are available (no WebCodecs encode/decode or video frame processors yet)
+
+#### Scenario: Lifecycle shape stays reusable
+- **WHEN** maintainers review the public worker control surface
+- **THEN** it still follows configure / process / subscribe / dispose with transfer-friendly payloads suitable for later non-PCM frame types