From a4c85d03a456336144de5427c419a0e96899f473 Mon Sep 17 00:00:00 2001 From: Tisha Chawla Date: Mon, 7 Sep 2026 21:59:47 +0530 Subject: [PATCH 1/2] docs: design spec for multimodal stream governance Draft design for governing continuous media streams (audio, video, realtime voice) on cost, provider-agnostic. Builds on primitives that already exist: the tick() time-based moment, stream_complete, and the declared-but-unimplemented CANCEL actuator (#67). Core idea is time-windowed accrual driven by tick, not per-frame pricing, so overhead is bounded regardless of frame rate. Pricing section grounded in how real 2026 streaming APIs bill (OpenAI Realtime per-token audio with cached-audio rates; Gemini Live fixed tokens-per-second audio and per-second video): the universal unit is seconds of stream, mapped per provider to a billable quantity, so one price(modality, direction, quantity, cached) signature covers both token-metered and time-metered providers. Status: draft. No implementation yet; phasing in the doc. Co-Authored-By: Claude Opus 4.8 --- .../multimodal-stream-governance-spec.md | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 docs/notes/multimodal-stream-governance-spec.md diff --git a/docs/notes/multimodal-stream-governance-spec.md b/docs/notes/multimodal-stream-governance-spec.md new file mode 100644 index 0000000..531f7da --- /dev/null +++ b/docs/notes/multimodal-stream-governance-spec.md @@ -0,0 +1,183 @@ +# Spec: multimodal stream governance + +Status: draft / design +Scope: cost governance of continuous media streams (audio, video, realtime +voice). Provider-agnostic. Safety and content moderation are explicitly out of +scope; this governs spend, not what the stream contains. + +## Problem + +TokenOps today governs **discrete, post-hoc** work. A model call happens, +returns, is priced (`input_tokens x price + output_tokens x price`), the ledger +updates, and the next call is checked before it dispatches. Every part of that +assumes a call is atomic and short. + +Continuous streams break all three assumptions: + +1. **No discrete boundary.** A voice or video stream has no single return to + price. Cost accrues while frames flow. +2. **Cost is a rate, not a sum.** The runaway failure mode is not "many cheap + calls added up," it is "a stream that should last ten seconds lasted ten + minutes," or "frames arriving faster or larger than budgeted." We govern + duration and throughput. +3. **Pre-call checking is too coarse.** There is no "before the call" when the + call is the whole stream. Governance must happen mid-stream, and at near-zero + overhead: a millisecond-scale check between millisecond-scale frames must not + break real time. + +This spec extends the discrete governor to a continuous one without a new +enforcement engine, by reusing primitives that already exist. + +## What we build on (already present) + +- **`Governor.tick(now)`** is a time-based moment, not call-based. It already + exists and is the natural hook for stream governance. +- **`stream_complete`** exists in `providers/factory.py`. +- **`ActionKind.CANCEL`** is already declared (tracked in issue #67, currently + unimplemented). Its purpose is exactly this: stop a stream mid-flight, keep the + partial output, bill only what was consumed. This spec is what gives CANCEL a + reason to exist. +- **The ledger** already accumulates spend keyed by + `(budget_id, segment_key, period)` and supports `HALT` per run. Stream spend is + more spend against the same accumulators; no new ledger shape is required. + +## Design + +### Unit of accounting: time-windowed accrual + +We do **not** price per frame. Each open stream accrues cost locally as frames +arrive, and the accrued delta is flushed to the ledger on a fixed **accrual +window** (default 500 ms). Budget checks run at flush time, not per frame. + +This bounds governance overhead independent of frame rate, and it matches how the +failure manifests (duration and rate, not individual frames). A 90-frames-per- +second video stream and a 16-kHz audio stream both cost the same governance: +two ledger touches per second. + +### Stream lifecycle and the governance seam + +A governed stream is wrapped the way a call is, but the wrapper sits in the +iteration loop rather than around a return: + +``` +open -> admit (inflight++), start accrual clock +frame -> accrue local cost (no ledger, no check) +window-> flush accrued delta to ledger; run stream detectors; maybe act +close -> flush final delta, complete (inflight--) +``` + +Proposed entry point, mirroring `wrap_complete`: + +```python +governed_stream = wrap_stream( + governor, controls, attr, + provider=..., model=..., modality="audio", + dispatch=stream_complete, + accrual_window_s=0.5, +) +for chunk in governed_stream(...): + ... # CANCEL is raised from inside the loop, tearing the stream down +``` + +`wrap_stream` drives `Governor.tick(now)` on each accrual window. Detectors that +care about streams read the accrued state; detectors that do not are unaffected. + +### Pricing: per-modality rate functions + +Extend the existing `PriceFn` with rate-based pricing. Cost stays micro-USD +(`Micros`, int) end to end, as it is today. + +How real providers bill a stream, as of 2026, tells us the abstraction we need. +Two shapes cover the field: + +- **Token-metered.** OpenAI's Realtime API bills audio per token (roughly $32 + per 1M audio input tokens, $64 per 1M output, with cached audio about 80x + cheaper). Gemini's Live API tokenizes audio at a fixed rate (about 32 tokens + per second of input audio, 25 per second of output) and then prices those + tokens. Either the provider reports token counts on its stream events, or a + fixed tokens-per-second constant derives them from elapsed time. +- **Time-metered.** Gemini prices video at a flat rate per second (about $0.15/s); + some speech models bill per minute. Here the billable quantity is simply + elapsed stream time. + +The unifying observation: **the universal underlying unit is seconds of stream**, +which each provider maps to its billable quantity, directly (time-metered) or via +a fixed tokens-per-second conversion (token-metered). Direction (input vs output) +and a cache-hit flag both change the rate, mirroring how existing text pricing +already handles input/output and cached tokens. + +So the price function signature generalizes to: + +``` +price(modality, direction, quantity, cached) -> Micros +``` + +A provider **stream adapter** maps that provider's stream events to a series of +`(modality, direction, quantity, cached)` records per accrual window; the price +function turns them into `Micros`. Rates are provider-supplied config, never +hardcoded. The governor never sees provider specifics, which is what keeps this +provider-agnostic. + +### Detectors (new, stream-scoped) + +Three, matching the real failure modes, smallest first: + +1. **`stream_duration_cap`** trips when a single stream exceeds a wall-clock + ceiling. Cheapest, no pricing dependency, the natural sibling of `step_cap`. + Moment: `tick`. Action: `CANCEL`. +2. **`stream_spend_cap`** trips when accrued stream cost (this stream, or the + run total including streams) crosses budget. Moment: `tick`. Action: `CANCEL`, + then `HALT` if the run as a whole is over. +3. **`stream_rate_guard`** trips when throughput (bytes/sec, frames/sec, or + resolution) exceeds an expected envelope, catching a stream that is quietly + more expensive than budgeted. Moment: `tick`. Action: `MUTATE` (e.g. request a + lower bitrate or resolution), then `CANCEL`. + +### Actuator: CANCEL (closes issue #67) + +CANCEL is the streaming actuator. On CANCEL: + +1. The wrapper stops iterating and tears down the provider stream cleanly (no + leaked socket or generator). +2. The final accrued delta is flushed, so the ledger reflects exactly what was + consumed, not the full intended stream and not zero. +3. The partial result gathered so far is returned to the caller. + +Billing correctness at teardown (charge for consumed frames only) is the +substance of the implementation, and the reason CANCEL is distinct from HALT. +HALT stops the run; CANCEL stops one stream and lets the run continue. + +## Security notes (in scope, ties to the threat model) + +Streaming widens the attack surface in ways worth stating: + +- **Frame ingress is untrusted-input-facing.** A provider adapter parses + provider frames; a malformed or hostile frame stream must not crash or stall + the governor. This path should be fuzzed. +- **Denial of wallet is the primary risk.** An unmetered or under-metered stream + is exactly the failure this feature exists to prevent; the accrual clock must + be monotonic and must keep flushing even if frames stall, so a stalled-open + stream still trips `stream_duration_cap`. +- **The accrual window is a security parameter, not just performance.** Too large + a window means a runaway can spend a full window before the first check. The + default must be small enough that worst-case unmetered spend is bounded and + documented. + +## Out of scope (v1) + +- Content safety / moderation of stream contents. +- Cross-process stream aggregation (a stream split across hosts). Single-process + accrual first; the shared ledger already handles cross-process spend totals if + needed later. +- Video pricing beyond a simple per-frame or per-megapixel rate. + +## Phasing + +1. `wrap_stream` + accrual clock + `tick`-driven flush, single modality (audio). +2. CANCEL implementation and billing-at-teardown (issue #67). +3. `stream_duration_cap`, then `stream_spend_cap`, then `stream_rate_guard`. +4. Video modality and resolution-scaled pricing. + +Each phase is independently shippable and testable offline with a scripted +frame source, no provider account required, matching how the existing policy +tests run. From 1d0737bf8c985f14a9f2414024cf1de391583e5d Mon Sep 17 00:00:00 2001 From: Tisha Chawla Date: Tue, 8 Sep 2026 21:17:26 +0530 Subject: [PATCH 2/2] docs: make the streaming spec concrete with the three voice pipelines Add a section grounding the abstraction in the three real voice pipelines (speech-to-speech, speech-to-text, text-to-speech): how each flows, the unit it bills in, its distinct denial-of-wallet shape, and an approximate 2026 anchor price. Shows that three billing units (tokens, minutes, characters) and two directions collapse into one price() call and one accrual loop, and motivates why the detectors cover duration, spend, and rate rather than a single one. Co-Authored-By: Claude Opus 4.8 --- .../multimodal-stream-governance-spec.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/notes/multimodal-stream-governance-spec.md b/docs/notes/multimodal-stream-governance-spec.md index 531f7da..80dffca 100644 --- a/docs/notes/multimodal-stream-governance-spec.md +++ b/docs/notes/multimodal-stream-governance-spec.md @@ -118,6 +118,30 @@ function turns them into `Micros`. Rates are provider-supplied config, never hardcoded. The governor never sees provider specifics, which is what keeps this provider-agnostic. +### The three voice pipelines, concretely + +Voice is the first real target, and it is three distinct pipelines. They bill in +three different units, yet the accrual model above covers all three unchanged. +Approximate 2026 prices are given as anchors, not as configuration. + +| Pipeline | Flows | Billed by | Runaway risk | Anchor price | +|---|---|---|---|---| +| Speech-to-speech | audio in and out | audio tokens, both directions (output ~2x input) | a conversation that never ends | Realtime API, ~$64 / 1M audio output tokens | +| Speech-to-text | audio in, text out | minutes of audio in (or audio-in tokens) | an hour-long audio file | Whisper, ~$0.017 / minute | +| Text-to-speech | text in, audio out | characters, or text-in + audio-out tokens | unbounded text to speak | OpenAI TTS, ~$15 / 1M characters | + +The point of the abstraction: three billing units (tokens, minutes, characters) +and two directions collapse into one `price(modality, direction, quantity, +cached)` call and one accrual loop. Speech-to-speech is the hardest case because +it is bidirectional and continuous, so it is the natural first implementation +target; the accrual clock meters input and output as separate directions on the +same window. + +Each pipeline has a distinct denial-of-wallet shape, which is why the detectors +below cover duration, spend, and rate rather than any single one: a +speech-to-speech session runs away on *duration*, a transcription job on input +*size*, and a synthesis job on output *volume*. + ### Detectors (new, stream-scoped) Three, matching the real failure modes, smallest first: