From 8586089fb1b021f172f3a074dcaf18fb069c16d6 Mon Sep 17 00:00:00 2001 From: Jon-Carlos Rivera Date: Thu, 1 Oct 2026 15:57:33 -0700 Subject: [PATCH 01/58] feat(wit)!: ffrwd:av 0.19.0 is the node world, and ten recipes name its SQL One module interface: typed input ports with a clock and a pairing each (lockstep, hold, interval, arrival), typed outputs with a format, a time base and a latency, describe without params and shape(params, bound), one tick per process call with frames fetched on demand, progress from the end of each tick's interval. types, values, encoder and decoder carry on; every other 0.18.0 interface is frozen in worlds/0.18.0 for the host to adapt. Cookbook recipes 145 to 154 are the SQL surface, red until the compiler and sidecar land: rows-only returns, structural row matching, one call evaluated once, mixed kinds, optional inputs, many-ports, several outputs, ffrwd.merge_spans, explain's delays, and a node with no inputs as a source. Co-Authored-By: Claude Fable 5.1 --- docs/examples.md | 269 +++++++++ sidecar/wit/av.wit | 1032 ++++++++++++++-------------------- sidecar/worlds/0.18.0/av.wit | 995 ++++++++++++++++++++++++++++++++ 3 files changed, 1698 insertions(+), 598 deletions(-) create mode 100644 sidecar/worlds/0.18.0/av.wit diff --git a/docs/examples.md b/docs/examples.md index a03ecb0..0c6cd3d 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1117,3 +1117,272 @@ ffmpeg -i tests/fixtures/av.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f nut Both `notes` calls and `pad_rows` run in one sidecar, and the rows never leave it: `n`'s notes ride the frames on `pad_rows`' first pad, and `o`'s, on the second, are dropped before the module is called. `pad_rows` is a stand-in that writes one row per call saying what reached it, so `seen.ndjson` holds sixty rows, `"first-0"` on frame 0, nothing on frame 1, `"first-2"` on frame 2, and never a note from `o`. The first stream has to be the one those rows ride, untouched: `pad_rows(o.v, n.v, notes => n.notes)` is refused, and so is an ffmpeg filter between `n.v` and the call, since ffmpeg carries the frames on and drops the rows. Selecting both halves is a WITH body's business alone; a SELECT that writes its columns reads `.notes` on its own, as a track. + +## 145. A detector returns its rows, and the picture stays where it was + +A module reading one stream need not hand it back. `spot` finds a mark in each frame and returns rows alone - `RETURNS STRUCT(...)[]` with no stream beside it is a data stream the query reads while it runs - and `ring` draws them, reading the picture from the source and the rows from the detector. Both arguments descend from `f.video[1]`, so the rows pair with the frames by pts, frame for frame, and nothing is copied that the detector did not make: + +```pgsql +CREATE FUNCTION spot(v video_stream, every number DEFAULT 30) +RETURNS STRUCT(start_t number, id number, x number, y number, w number, h number)[] + AS '../sidecar/modules/target/wasm32-wasip2/release/spot.wasm', 'spot' + LANGUAGE wasm; + +CREATE FUNCTION ring(v video_stream, + spots STRUCT(start_t number, id number, + x number, y number, w number, h number)[]) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/ring.wasm', 'ring' + LANGUAGE wasm; + +COPY ( + SELECT ring(f.video[1], spot(f.video[1])) + FROM input('tests/fixtures/testsrc.mp4') f +) TO 'ringed.mp4' WITH (video_codec 'libx264', crf 20) +``` + +``` + +``` + +`spot` writes one row per frame for as long as the mark is in view, and every row of one mark carries the pts it was first seen at as `start_t`: the row for frame t says what is true at t, so a reader needs no look-ahead and a run split across workers agrees on the ids. The old spelling, a module returning `STRUCT(v video_stream, spots ...)` with the picture untouched, is what a package keeps when its own module has not moved: inside the sidecar the rows ride the frames exactly as before. A migrated package that wants the old reading back writes it in SQL - `CREATE FUNCTION spotted(v video_stream) RETURNS STRUCT(v video_stream, spots STRUCT(...)[]) AS $$ SELECT v, spot(v) AS spots $$ LANGUAGE sql` - and `ring(spotted(v))` reads the record as the stream and the rows, as a call over a two-part result always has. + +## 146. A reader names only the fields it reads + +`dim` wants a box; it does not care who found it or when. Its parameter declares the four fields it reads, and `spot`'s six-field rows are accepted because every field `dim` names is there with the type it names - the others pass through untouched. Row matching is structural, so one reader serves every detector whose rows carry a box: + +```pgsql +CREATE FUNCTION spot(v video_stream, every number DEFAULT 30) +RETURNS STRUCT(start_t number, id number, x number, y number, w number, h number)[] + AS '../sidecar/modules/target/wasm32-wasip2/release/spot.wasm', 'spot' + LANGUAGE wasm; + +CREATE FUNCTION dim(v video_stream, boxes STRUCT(x number, y number, w number, h number)[], + amount number DEFAULT 0.5) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/dim.wasm', 'dim' + LANGUAGE wasm; + +COPY ( + SELECT dim(f.video[1], + ARRAY(SELECT s FROM unnest(spot(f.video[1])) s WHERE s.w * s.h > 400)) + FROM input('tests/fixtures/testsrc.mp4') f +) TO 'dimmed.mp4' WITH (video_codec 'libx264', crf 20) +``` + +``` + +``` + +A field the reader names that the producer's rows lack, or carries with another type, is still refused at compile time, naming both. The WHERE runs inside the sidecar as before, and it may test fields the reader never sees. + +## 147. One call, however many places read it + +`hear` listens to the sound and writes a cue per window. The query reads it twice, as a caption track of the file and as the words `burn` paints onto the picture, and that is one node: a call with the same arguments written anywhere in one query is evaluated once, and its data output is split to every reader. Today those were two runs of the model, since a caption track feeds the muxer and a module's rows feed a stage of its own: + +```pgsql +CREATE FUNCTION hear(a audio_stream) RETURNS cue[] + AS '../sidecar/modules/target/wasm32-wasip2/release/hear.wasm', 'hear' + LANGUAGE wasm; + +CREATE FUNCTION burn(v video_stream, a audio_stream DEFAULT NULL, + words cue[] DEFAULT NULL) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/burn.wasm', 'burn' + LANGUAGE wasm; + +COPY ( + SELECT burn(f.video[1], words => hear(f.audio[1])), f.audio[1], hear(f.audio[1]) + FROM input('tests/fixtures/av.mp4') f +) TO 'heard.mkv' WITH (video_codec 'libx264', crf 20, audio_codec 'copy') +``` + +``` + +``` + +Two calls whose arguments differ are still two nodes. The split is of a data edge, so what it costs is a second copy of each row, not a second run. + +## 148. A node reads the picture, the sound and the words at once + +A module's inputs are any mix of kinds in any order: `burn` takes the picture as its clock, the sound beside it frame for frame, and the words by their time. Each is a port of its own with its own pairing, declared by the module and read by the compiler, so the query writes the call and nothing about how the three are lined up: + +```pgsql +CREATE FUNCTION hear(a audio_stream) RETURNS cue[] + AS '../sidecar/modules/target/wasm32-wasip2/release/hear.wasm', 'hear' + LANGUAGE wasm; + +CREATE FUNCTION burn(v video_stream, a audio_stream DEFAULT NULL, + words cue[] DEFAULT NULL) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/burn.wasm', 'burn' + LANGUAGE wasm; + +COPY ( + SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), f.audio[1] + FROM input('tests/fixtures/av.mp4') f +) TO 'burned.mp4' WITH (video_codec 'libx264', crf 20, audio_codec 'aac') +``` + +``` + +``` + +`hear` works two seconds of sound at a time and says so, and a window's cues leave with the window. `burn` reads them by interval, so the host holds each picture until the window holding its time is done, and the picture leaves `burn` two seconds behind the sound that enters it. [Recipe 153](#153-see-what-each-node-waits-for) shows where that number is printed. + +## 149. Leave an input out + +Any stream parameter may carry `DEFAULT NULL`, and a call leaves it off or writes `NULL`: the port is unbound and the module is told so. `burn` with the picture alone paints nothing; with the sound and no words it paints a level meter; `inset` shows a feed that connects later, by port, over the picture, and runs on the picture alone until something connects: + +```pgsql +CREATE FUNCTION burn(v video_stream, a audio_stream DEFAULT NULL, + words cue[] DEFAULT NULL) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/burn.wasm', 'burn' + LANGUAGE wasm; + +CREATE FUNCTION inset(v video_stream, feed video_stream DEFAULT NULL, + port number DEFAULT 9000, lead number DEFAULT 0.5) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/inset.wasm', 'inset' + LANGUAGE wasm; + +COPY ( + SELECT inset(burn(f.video[1], f.audio[1]), port => 9100), f.audio[1] + FROM input('tests/fixtures/av.mp4') f +) TO 'inset.mp4' WITH (video_codec 'libx264', crf 20, audio_codec 'aac') +``` + +``` + +``` + +`feed` is a hold input: whatever connects to port 9100 is shown at the picture's pace from `lead` seconds after its first frame arrives, the last frame held while it runs late, and the picture alone again when it ends. The compile listing names the port and the process that owns it, as it does for a switch's feeders. An input with no `DEFAULT NULL` is required, and a call that leaves it off is refused. + +## 150. Tile any number of pictures + +`video_stream[]` declares a port that takes as many streams as the call gives it, each arriving with its own size, rate and tags. `tile` lays them out in a grid. It ticks at the first picture's rate, or at `fps` when the call says so, and at every tick shows each stream's newest frame, so the pictures need not share a rate or a start: + +```pgsql +CREATE FUNCTION tile(v video_stream[], columns number DEFAULT 2, fps number DEFAULT NULL) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/tile.wasm', 'tile' + LANGUAGE wasm; + +COPY ( + SELECT tile(ARRAY[a.video[1], b.video[1], c.video[1]], 3) + FROM input('tests/fixtures/av.mp4') a, + input('tests/fixtures/av2.mp4') b, + input('tests/fixtures/testsrc.mp4') c +) TO 'tiled.mp4' WITH (video_codec 'libx264', crf 20) +``` + +``` + +``` + +A bare array column broadcasts over a filter, one call per element; over a module port declared as an array it is the port's whole list, and a module that wants one call per element is called under `unnest`. `audio_stream[]` and a rows parameter with `[]` on the record work the same way. A port that takes several streams cannot be the module's clock, which is why `tile` keeps time itself; the rate it keeps is read off the first picture by the compiler, which knows every stream's rate before anything runs. + +## 151. A node makes a matte and the rows that go with it + +A module that produces two things returns a record naming both: `matte` makes a mask of the mark it finds and a row per mark, and both leave the one node. Read either field off the call, or every field at once with `.*` in a WITH body; however the fields are read, the call is one instance: + +```pgsql +CREATE FUNCTION matte(v video_stream, every number DEFAULT 30) +RETURNS STRUCT(mask video_stream, + spots STRUCT(start_t number, id number, + x number, y number, w number, h number)[]) + AS '../sidecar/modules/target/wasm32-wasip2/release/matte.wasm', 'matte' + LANGUAGE wasm; + +CREATE FUNCTION dim(v video_stream, boxes STRUCT(x number, y number, w number, h number)[], + amount number DEFAULT 0.5) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/dim.wasm', 'dim' + LANGUAGE wasm; + +COPY ( + WITH m AS (SELECT (matte(f.video[1])).* FROM input('tests/fixtures/testsrc.mp4') f) + SELECT dim(m.mask, m.spots), m.spots + FROM m +) TO 'matte.mkv' WITH (video_codec 'ffv1') +``` + +``` + +``` + +Each field is an output port with a format and a time base of its own, declared by the module for the call's parameters. [Recipe 94](#94-blur-the-people-and-only-the-people)'s `segment` is this shape, and its rows no longer ride the map's frames: they are a data stream beside it, which is why `mask_select` can read them from a call `segment` is not part of. + +## 152. Spans from the rows that said so, frame by frame + +A row per frame is the honest shape for a thing known as it happens, and a span is what a subtitle track or a record wants. `ffrwd.merge_spans` is the reducer between them, a node the host provides: it groups rows by `start_t`, keeps the last row's fields, and ends each span at the last frame that carried it plus that frame's duration. A frame that misses a row is a gap inside the span, not the end of it: + +```pgsql +CREATE FUNCTION spot(v video_stream, every number DEFAULT 30) +RETURNS STRUCT(start_t number, id number, x number, y number, w number, h number)[] + AS '../sidecar/modules/target/wasm32-wasip2/release/spot.wasm', 'spot' + LANGUAGE wasm; + +COPY ( + SELECT ffrwd.merge_spans(spot(f.video[1]), max_span => 10) + FROM input('tests/fixtures/testsrc.mp4') f +) TO 'spots.ndjson' +``` + +``` + +``` + +The rows out carry `start_t` and `end_t` beside the fields in, one row per span, so `spots.ndjson` holds one line per mark rather than one per frame. Rows whose fields are `start_t` and `text` reduce to cues, and selecting them beside a picture writes a subtitle track. A span row leaves when its span ends, so a span is as late as it is long; `max_span` bounds that, and it is what a reader pairing by time waits for. A span still open after ten seconds is written as it stands and goes on as a new one. The reducer closes a span on its producer's progress, not on the next row, so the last span of a run ends where the rows did. + +## 153. See what each node waits for + +Every node declares the window it works in and how late its rows may leave, and the compiler adds the waits up along each path. `explain` prints them: per node, its window in streaming SQL's words (per-frame, tumbling, hopping, sliding), and per output, how far behind the source it runs. The query is [recipe 148](#148-a-node-reads-the-picture-the-sound-and-the-words-at-once)'s: + +```pgsql +CREATE FUNCTION hear(a audio_stream) RETURNS cue[] + AS '../sidecar/modules/target/wasm32-wasip2/release/hear.wasm', 'hear' + LANGUAGE wasm; + +CREATE FUNCTION burn(v video_stream, a audio_stream DEFAULT NULL, + words cue[] DEFAULT NULL) +RETURNS video_stream + AS '../sidecar/modules/target/wasm32-wasip2/release/burn.wasm', 'burn' + LANGUAGE wasm; + +COPY ( + SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), f.audio[1] + FROM input('tests/fixtures/av.mp4') f +) TO 'burned.mp4' WITH (video_codec 'libx264', crf 20, audio_codec 'aac') +``` + +``` +$ ffrwd explain -f query.sql + +``` + +`hear` is a tumbling window of 2 s, so its cues trail the sound by up to 2 s and nothing more; `burn`'s picture is 2 s behind the source, and the sound written beside it waits in its pipe for the same 2 s, which `compile` sizes. On a live input the same sums decide whether a query can run at all: a node that must act ahead of time (an ad decision that needs `announce_before_s`, a playout that needs `lead_s`) fed by a path later than that lead is refused at compile time as `LIVE_LEAD`, naming the node, the lead it needs and the delay of the path feeding it. A file run has no such rule, since nothing there is late. + +## 154. A page with no inputs is a source + +A module with no stream parameters is a source: it declares its outputs for its parameters, ticks at the rate it declares, and reads nothing. `ticker` draws a line of text crossing a canvas. `RETURNS source` puts it in FROM, where the alias carries the stream columns the module declared, and the query reads them as it reads a file's: + +```pgsql +CREATE FUNCTION ticker(text text, width number DEFAULT 1280, height number DEFAULT 720, + fps number DEFAULT 30) RETURNS source + AS '../sidecar/modules/target/wasm32-wasip2/release/ticker.wasm', 'ticker' + LANGUAGE wasm; + +COPY ( + SELECT s.video[1] + FROM ticker('Nothing to see here') s + WHERE s.t < 10 +) TO 'ticker.mp4' WITH (video_codec 'libx264', crf 20) +``` + +``` + +``` + +In a file run the source runs as fast as its reader drains it; in a live run it is paced to the wall clock. `WHERE s.t < 10` ends it after ten seconds, as it would any source. A network source is the same shape with a clock of its own: it emits when it has something, and `shape` may reach the network at compile time to learn its outputs, as a manifest is probed. `ffrwd.blitz.compose` with no streams, a page that animates on its own, is this recipe's shape too. diff --git a/sidecar/wit/av.wit b/sidecar/wit/av.wit index 9d02fd0..ec8b66b 100644 --- a/sidecar/wit/av.wit +++ b/sidecar/wit/av.wit @@ -1,4 +1,15 @@ -package ffrwd:av@0.18.0; +// ffrwd:av 0.19.0: the node world. +// +// A node has any number of typed input ports, any number of typed output +// ports, and a clock. One tick of the clock is one `process` call: the host +// assembles what every input holds for that tick, the node emits on its +// outputs. A frame's bytes cross into the module only on `fetch`. +// +// `types`, `values`, `encoder` and `decoder` carry on from 0.18.0. Every +// other 0.18.0 interface is frozen in worlds/0.18.0 and adapted onto `node` +// by the host. + +package ffrwd:av@0.19.0; interface types { /// Static description of a module, available without running anything. @@ -161,7 +172,7 @@ interface types { /// One raw frame crossing a codec: a video frame's pixels in the /// instance's pix-fmt, or a run of interleaved audio samples in its - /// sample-fmt. Shared between `encoder` and `decoder`. + /// sample-fmt. What a codec and a node hand over. record raw-frame { /// When it is presented, in the stream's time base. pts: s64, @@ -247,522 +258,453 @@ interface types { } } -interface filter { - use types.{meta, stream-info}; - - record frame-info { - width: u32, - height: u32, - /// Seconds since the start of the stream. - time: f64, +interface node-types { + use types.{rational, color-info, video-format, audio-format, coded-stream, + packet, stream-info, rendition-meta, wants}; + + enum port-kind { video, audio, data, packets } + + /// What a node does with the rows that arrive on an input. + /// `per-frame`: reads a tick's rows while handling that tick and keeps + /// nothing. `state`: folds rows into state later ticks depend on, so a + /// frame-parallel instance must also see the rows of the ticks it did not + /// process (`earlier-rows`). + enum rows-use { ignore, per-frame, state } + + /// How a hold input's offset between source time and clock time is fixed. + /// `shared-clock`: the source's pts are on the clock's epoch; offset 0, and + /// the host holds the first frame until the clock reaches it (a timed + /// feeder). `first-frame`: the offset is fixed once the host holds `lead` + /// seconds of the source, or its end, and the start is scheduled `lead` + /// seconds ahead of the clock (an untimed feeder). `tagged`: shared-clock + /// for a feed whose tags carry this name set to 1, first-frame + /// otherwise (the switch's `smart_timed`). + variant anchor { shared-clock, first-frame, tagged(string) } + + /// A frame input paired by time: the newest frame at or before the tick. + /// The host keeps the source buffered ahead, repeats the last frame while + /// the source falls behind, skips forward when it catches up, and reports + /// both. What ffrwd/switch did for itself, for any node. + record hold { + anchor: anchor, + /// Seconds ahead of the clock a first-frame start is scheduled. + lead: f64, + /// Seconds the last frame stays after the source ends; none drops it + /// at once. + linger: option, + /// Seconds of programme time without a frame before a live source is + /// given up on and its feed ended; none waits for it. + timeout: option, + /// Hold inputs of one group arrive on one connection from one source + /// (a feeder's picture and sound): one offset, fixed by the group's + /// first picture, and feed starts and ends on the same tick. + group: option, + /// The param that carries this input's loopback port. Bound to a + /// stream, the host writes the port it picked there; left unbound, the + /// host listens on the port the call wrote, and whatever connects is + /// the source. + port-param: option, + } + + /// A message input paired by time: every message whose pts falls in the + /// tick's interval, extended by `ahead`, each delivered once at the first + /// tick that holds it, held while it is ahead of the clock. The host + /// settles the interval once the producer's progress has passed its end + /// plus `ahead`; `latency` bounds that wait, and a message later than the + /// bound is delivered at the next tick and reported. + record interval { + /// The most this input waits, in seconds: the interval settles once the + /// clock input has arrived this long past its end, progress or not. + /// None waits for the producer's progress alone; a producer that sends + /// none (a lateral, a data feeder) needs a bound set. + latency: option, + /// Seconds past the interval's end whose messages are handed with it, + /// for a node that acts before a stamp (a fade in before a cue). + ahead: f64, + } + + variant pairing { + /// The clock's pts exactly: same source, one frame per tick. The input + /// `clock` names is declared lockstep. + lockstep, + /// Frame kinds only. + hold(hold), + /// Message kinds only. + interval(interval), + /// Delivered as it arrives, unpaired: packets to a sink, data a node + /// orders itself. + arrival, + } + + /// What an input accepts. Empty lists accept anything of the kind. + record accepts { + pixel-formats: list, + sample-formats: list, + sample-rates: list, + channel-counts: list, + codecs: list, + wants: wants, + /// Conformed by the host to that input's stream: a video to its size, + /// an audio to its rate and layout. The named input is single. + like: option, } - variant output { - /// A new frame, same byte length as the input. - frame(list), - /// The input frame is the output; the host copies nothing. - passthrough, + record input-port { + name: string, + kind: port-kind, + required: bool, + /// Any number of streams bound to this one port, each with its own + /// metadata and time base (a publisher's renditions, a weave's rows + /// inputs). + many: bool, + pairing: pairing, + rows: rows-use, + /// The clock input: frames (video) or samples (audio) a call sees, and + /// how many it consumes. Every other input hands the tick's interval + /// and declares 1/1. + window: u32, + stride: u32, + accepts: accepts, + /// Data inputs: the JSON schema of the rows the node reads here, as + /// `rows-module` declared its input rows. None accepts any row. + schema: option, } - /// What one frame produced. - record outcome { - output: output, - /// JSON objects matching `meta.rows-schema`, zero or more per frame. - rows: list, + /// An input's format with one field overridden: a matte the size of its + /// picture, in gray. The named input is single and a frame kind. + record like-input { + port: string, + pixel-format: option, + sample-format: option, } - describe: func() -> meta; - - /// Called once per instance. `pix-fmt` is one of `meta.pixel-formats`, - /// chosen by whatever drives the host. The frame size is fixed for the - /// life of the instance. This interface carries video alone: a module - /// publishing sample formats exports `window-filter` instead. - init: func(width: u32, height: u32, pix-fmt: string, stream-info: stream-info, - params: string) -> result<_, string>; - - /// Replaces the parameters between frames. The module keeps the state - /// it built in `init`; rejecting the new parameters must leave the - /// previous ones in force. - set-params: func(params: string) -> result<_, string>; - - /// Whether `process` depends only on the current frame. False makes - /// frame-parallel hosting unsound and a host must refuse it. Callable - /// after `init`; the answer may depend on the parameters. - frame-independent: func() -> bool; - - process: func(info: frame-info, frame: list) -> outcome; -} - -/// Optional beside `filter`, the way `values` is optional: a module that also -/// exports this one reads the rows an upstream module emitted for the same -/// frame. Everything else - `describe`, `init`, `set-params`, -/// `frame-independent` - still comes from `filter`, so a module exporting -/// this exports both. A host that finds this export calls `process-meta` -/// instead of `process`; one that does not calls `process` as before. -interface meta-filter { - use filter.{frame-info, outcome}; - - /// One frame plus the rows that arrived with it, each an NDJSON line an - /// upstream module produced. Empty when the frame carries none. - process-meta: func(info: frame-info, frame: list, rows-in: list) -> outcome; -} - -/// The window `window-filter`'s `process` borrows, held by the host for -/// exactly that call. Earlier worlds copied every payload of the window into -/// the call; here the metadata is cheap and bytes move only on `fetch`, so a -/// module that reads one payload of a wide window never pays for the rest. -interface window-source { - /// One call's input payloads. Indices run oldest first, 0 to `len() - 1` - - /// or pad order, for a module reading several streams - and an index at or - /// past `len()` is a fault that stops the run. - /// - /// The handle is the call's alone: it cannot be kept past the return, and - /// bytes a module means to keep are its own copy the moment `fetch` hands - /// them over. - resource in-window { - /// How many payloads this call carries. - len: func() -> u32; - - /// Payload `i`'s timestamp. For an audio instance it is the first - /// sample's tick. - pts: func(i: u32) -> s64; - - /// The rows that arrived with payload `i` - each an NDJSON line an - /// upstream module produced, empty when nothing came with it. Rows ride - /// pad 0: a module reading several streams is handed whatever arrived on - /// pad 0, and the rows that arrived on any other pad are dropped before - /// the call - so this is empty on every pad past the first. - rows: func(i: u32) -> list; - - /// Payload `i`'s bytes, copied on demand: the only per-payload copy. - /// - /// For a video instance the bytes are one frame's pixels. For an audio - /// instance they are interleaved samples in the instance's sample - /// format, the whole window in one contiguous piece: the host re-cuts - /// whatever packets arrived, so a module never sees a packet boundary. - fetch: func(i: u32) -> list; - } -} - -/// A window at a time, superseding `filter`'s per-frame `process` and -/// `meta-filter`'s `process-meta`. A module exports this one *or* those; a -/// host adapts a module that exports the older pair as window 1, stride 1, -/// one-to-one. `describe`, `init` and `set-params` mean here what they mean -/// there, so the two differ only in how the payload arrives and leaves. -/// -/// This is the only interface audio reaches. `init` is handed a `format`, and -/// which arm it carries is the instance's kind for the rest of its life. -interface window-filter { - use types.{format, meta, stream-info}; - use window-source.{in-window}; - - /// Where an output payload's bytes come from. - variant frame-payload { - /// New bytes: one frame's pixels, or interleaved samples. - new(list), - /// An input's bytes, which the module never copied out: the input this - /// call received at the same pts, or - when the call received exactly one - /// input - that one, whatever pts the output carries. For a module - /// reading several streams this is pad 0's bytes, since every pad shares - /// the call's one pts. An audio module may say this only when its stride - /// is its window, since overlapping windows passed through would emit - /// every sample more than once. - same, - } - - /// One output. Its pts is the module's to choose: a module may move its - /// payload in time, drop it, or emit several from one. - record out-frame { - pts: s64, - frame: frame-payload, - rows: list, + variant output-format { + video(video-format), + audio(audio-format), + /// The codec, "json" today. + data(string), + packets(coded-stream), + like(like-input), } - /// `meta`, plus how the host must drive this module. - record window-meta { - meta: meta, - /// What one `process` call receives: frames for a video module, SAMPLES - /// for an audio one. 1 is plain streaming for video; an audio module - /// names a chunk it is happy to work in, and the host cuts that chunk out - /// of whatever packets arrive. - window: u32, - /// How much is consumed per call, never more than `window`, in the same - /// unit. `window` makes the windows disjoint. - stride: u32, - /// Whether a call depends only on what it was handed. False makes - /// parallel hosting unsound and a host must refuse it. + record output-port { + name: string, + kind: port-kind, + /// None: the clock input's format, as most filters leave it. + format: option, + /// None: the clock input's. + time-base: option, + /// How far behind the end of its tick's interval a stamp may fall, in + /// seconds; 0 when everything leaves with its tick. A window's + /// length is not counted here: a recogniser that emits a window's + /// results with the window declares 0. Consumers pairing by interval + /// and the compiler's delay sums read it. + latency: f64, + /// Data outputs: the JSON schema of the rows this port writes. + schema: option, + /// A source read in FROM: the row of `relation` this output belongs + /// to. Outputs of one row are one rendition. + row: option, + } + + variant clock { + input(string), + /// A generator: ticks at this rate, paced to real time in a live run + /// and as fast as the outputs drain otherwise. For a node with inputs + /// that only need turns (a publisher's network session): at least this + /// often, and the last call comes once every input has ended, unless + /// the node finished first. + rate(rational), + /// A rate clock at the named input's rate, which the compiler reads + /// off that port's first stream; the input itself is held like any + /// other. A tile of pictures at the first picture's rate. + rate-of(string), + /// The node emits when it has something; its outputs' pts are the + /// timeline (a network source). + self-clocked, + } + + record node-shape { + inputs: list, + outputs: list, + clock: clock, + /// Every call depends only on what it was handed, counting a state + /// input's earlier rows as handed. A host spreads a pure node over + /// workers; an impure one runs as one instance, in order. pure: bool, - /// For video: whether every call returns exactly one output per frame it - /// consumed, each at that frame's own pts. For audio: whether the samples - /// leaving over the instance's life are the samples that arrived, neither - /// dropped nor invented - per-call counts may differ, but the output runs - /// on without overlap, and a gap only where the input had one: a sample - /// either follows the last one out or keeps the timestamp the input gave - /// it. Neither window nor stride settles this, so it is declared. + /// Frame kinds: one frame out per frame in on the clock's outputs, each at + /// its tick's pts. Audio: samples conserved and contiguous. one-to-one: bool, - /// Whether this module ACTS on the rows arriving with its payloads. Every - /// call receives them whatever this says, so availability is no answer: a - /// module that only carries rows through declares false, and one that - /// reads what an upstream module detected declares true. - reads-rows: bool, - /// Whether upstream rows may appear on this module's own output. False - /// means the rows leaving are the module's own, or none. It speaks of pad - /// 0: rows arrive there and nowhere else. - forwards-rows: bool, - /// How many streams this module reads. Frames arrive one per pad, in pad - /// order, all at the same pts. 1 is the ordinary module, reading one - /// stream. More than 1 is window 1, stride 1 - a call takes one frame off - /// each pad and hands one back - and a host refuses any other shape. - inputs: u32, - /// Arguments of the call that are FEEDERS rather than pads: a stream - /// that may be empty, which the module reads itself over a loopback - /// connection instead of being handed its frames. Empty for every - /// module that has none. - feeders: list, - } - - /// One feeder argument. The host picks a loopback port, writes it into - /// the param `port-param` names, and delivers the stream passed in that - /// position there as NUT. A call that passes a number in that position - /// instead is naming a port itself, and the host wires nothing. - record feeder { - /// Which argument of the call it is, counting stream arguments from 0. - input: u32, - /// The param the host puts the port it picked in. - port-param: string, - /// "video" or "audio". - kind: string, - /// Feeders sharing one connection: every feeder in a query fed by the - /// same source and naming the same group gets ONE port, written into - /// each call's `port-param`, and ONE NUT carrying every stream those - /// feeders name, video then audio. Empty: a connection of its own. - group: string, + /// A rate or self-clocked node: whether it ends by itself. A live one + /// does not, and the compiler plans it as live. Ignored on an input + /// clock. + bounded: bool, + /// A source read in FROM: one JSON object per relation row (rendition, + /// bandwidth, codecs, geometry), which its outputs name by index. + relation: list, + } + + /// One stream bound to an input port at `init`. + record bound-stream { + port: string, + /// Names the stream for the instance's life: distinct across the + /// bound streams, and what every tick call and `same` take. + id: u32, + /// Its time base is `info.time-base`. + info: stream-info, + /// The format the stream arrives in, resolved: never `like`. Packets + /// inputs carry `coded`. + format: option, + rendition: rendition-meta, + /// The relation row the stream came from, where the call reads rows (a + /// sink handed a ladder); the publisher names its tracks by it. + row: option, + /// Packets inputs: how deep the stream reorders, as `input-stream` + /// carries it at 0.18. 0 elsewhere. + decode-delay: u32, + /// Data inputs: the producer's declared latency, when it has one. + latency: option, } - /// What one call produced. - record processed { - frames: list, - /// Rows with nothing to ride. Only the last call may fill this. - trailing: list, + record message { + pts: s64, + data: list, } - describe: func() -> window-meta; + record timed-rows { + pts: s64, + rows: list, + } - /// Called once per instance. The arm of `format` is the one the module's - /// own declaration asked for: a module publishing pixel formats is opened - /// with `video`, one publishing sample formats with `audio`. - init: func(format: format, stream-info: stream-info, - params: string) -> result<_, string>; + record feed-start { + /// The source's tags, `smart_timed` among them. + tags: list>, + /// Its first frame's pts, in its own time base. + first-pts: s64, + /// The clock time that first frame stands at, in the clock's time base: + /// what maps the source's pts onto the tick. + at: s64, + } + + /// A hold input's feed: one source, from the tick its first frame shows + /// on to the last its last frame shows on, `linger` included. The source + /// going backwards, or forwards by more than a second, ends it and + /// starts the next. + record feed { + start: feed-start, + /// The clock time of the last tick its last frame shows on, once the + /// host can foretell it: the source ended, or timed out. + ends: option, + } + + /// One frame as a tick sees it, without its bytes: a picture, or the + /// tick's audio samples as one run, which is a frame as codecs count + /// them. + record frame { + /// In the stream's time base; for audio, the first sample's. + pts: s64, + /// Its place in the stream's `frames` this tick, oldest first from 0: + /// what `fetch` and `same` name it by. + index: u32, + /// How long it is presented, in the stream's time base; none where + /// nothing settles it. + duration: option, + /// The rows that arrived with it, each a JSON object from an upstream + /// module. Empty on an input whose `rows` is `ignore`. + rows: list, + } +} - /// Replaces the parameters between windows, as `filter`'s does. - set-params: func(params: string) -> result<_, string>; +/// What one tick holds. +interface node-tick { + use types.{rational, packet, stream-info}; + use node-types.{frame, message, feed, timed-rows}; - /// One window. `last` marks the final call of the instance's life: it - /// carries whatever the last stride left over, which may be nothing, and - /// happens exactly once. There is no separate flush. + /// One call's inputs, held by the host for exactly that call. A stream + /// is named by the `id` `init` gave it; an id `init` did not give, or an + /// index at or past the length of that stream's `frames`, is a + /// fault that stops the run. /// - /// `window` holds the call's payloads, borrowed for the call: the window - /// for a module reading one stream. For a module reading several it is one - /// frame per pad, in pad order, every one of them at the same pts - which - /// is why such a module is window 1, stride 1. + /// The tick's interval runs from its time to the next tick's: one stride + /// of a frame clock, one packet of a packets clock, every message + /// sharing one pts of a data clock, one period of a rate clock. A last + /// call's runs to the end of its inputs. Every input that is not the + /// clock hands what falls in it. /// - /// `trailing` is the rows an upstream module had nothing to put them on; - /// only the last call carries any. Like `in-window`'s rows, it follows pad - /// 0: what reached the other pads with nothing to ride never arrives here. - /// - /// Output timestamps never decrease, within a call or between calls. - process: func(window: borrow, trailing: list, last: bool) -> processed; + /// The handle is the call's alone: it cannot be kept past the return, + /// and bytes a module means to keep are its own copy the moment `fetch` + /// hands them over. + resource tick { + /// The tick's time in `time-base`. An input clock: its first frame or + /// message this call, or the running maximum of dts over a packets + /// clock (pts where dts is unset), or, on a last call that hands none, + /// the time just past its last. A rate clock: the tick's number, from + /// 0. A self-clocked node: microseconds since its first call. + pts: func() -> s64; + + /// The clock input's time base, the inverse of a rate clock's rate, or + /// 1/1000000 for a self-clocked node. + time-base: func() -> rational; + + /// Whether this is the instance's final call. It carries whatever the + /// inputs have left, which may be nothing, and happens exactly once. + /// There is no separate flush. + last: func() -> bool; + + /// The streams bound to input `port`, in `init`'s order: the same every + /// tick, empty for an optional port with nothing bound. A name the + /// shape does not declare is a fault. + streams: func(port: string) -> list; + + /// The stream as the host knows it this tick, its time base included. + /// A hold input's tags and time base are its current source's, from + /// each feed's start on. + info: func(id: u32) -> stream-info; + + /// A hold input's feed as it stands this tick: the same record on + /// every tick the source shows on, so a worker that missed the start + /// still sees it; none while nothing shows, and on every other input. + /// A feed that ends and another that starts within one tick are two + /// ticks. + feed: func(id: u32) -> option; + + /// Frame kinds: the frames this tick hands, oldest first. The clock and + /// a lockstep input hand their window. A hold input hands the frame it + /// shows, or none before its feed's first frame and once its feed has + /// ended. Audio hands one frame, the tick's samples re-cut from + /// whatever packets arrived; a hold audio input hands none while its + /// source is behind, and the node fills the silence. Empty on message + /// kinds. + frames: func(id: u32) -> list; + + /// Item `index`'s bytes, copied on demand: the only copy of a frame + /// kind's payload. Video: one frame's pixels in the stream's format, + /// tightly packed. Audio: interleaved samples in its sample format. + fetch: func(id: u32, index: u32) -> list; + + /// Data inputs: the messages this tick hands under the input's pairing, + /// in pts order, each delivered once. Empty on other kinds. + messages: func(id: u32) -> list; + + /// Packets inputs: the packets this tick hands, in decode order, each + /// delivered once. Empty on other kinds. + packets: func(id: u32) -> list; + + /// For an input whose `rows` is `state`: the rows of every tick this + /// instance did not process since its previous call, oldest first, + /// each beside the pts it arrived at, all earlier than this call's. An + /// instance's first call is handed every one before it. A data input's + /// rows are its messages. Empty on every other input. + earlier-rows: func(id: u32) -> list; + } } -/// A sink for encoded packets. Where `window-filter` is handed decoded -/// payloads, this interface is handed the encoder's own output - compressed -/// bytes with their timestamps and keyframe flags - and hands back rows -/// alone. A packet sink consumes the streams: no frames leave, so a module -/// exports this one instead of the frame interfaces, and what it emits goes -/// to a rows output or nowhere. -/// -/// A sink is a filter with no output pads, and reads streams the way one -/// does: several at once, of either kind. Earlier worlds carried exactly one -/// coded stream and one packet list; here `init` is handed every stream the -/// query named, in that order, and `process` is handed one packet list per -/// pad in the same order. -interface packet-sink { - use types.{arity, input-stream, meta, pad-packets, wants}; - - /// `meta`, plus what this sink accepts. A packet sink names codecs the way - /// a frame module names pixel formats; `meta`'s own format lists stay - /// empty, since no decoded payload ever reaches it. - record packet-sink-meta { - meta: meta, - /// ffmpeg's names for the VIDEO codecs accepted, most preferred first. - /// Empty is every codec. - video-codecs: list, - /// ffmpeg's names for the AUDIO codecs accepted, most preferred first. - /// Empty is every codec. - audio-codecs: list, - /// How many video streams this sink reads. - video: arity, - /// How many audio streams this sink reads. - audio: arity, - /// How many data streams this sink reads. A world before 0.17.0 read - /// none, which is what one is adapted to. - data: arity, - /// How much of the stream this sink has to see. `all` is the answer for - /// a sink that reads every packet, and the answer a world before 0.16.0 - /// is adapted to, since none of them could say otherwise. - wants: wants, +/// A node: typed inputs, typed outputs and a clock, its shape from its +/// params. +interface node { + use types.{meta, packet, raw-frame}; + use node-types.{bound-stream, message, node-shape}; + use node-tick.{tick}; + + /// An input frame's bytes leaving on an output, never copied into the + /// module: frame `index` of stream `id`'s `frames` this tick. Its format + /// must be the port's (size and pixel format, or sample format, rate and + /// channels); the host ends the run otherwise. Audio may leave this way + /// only from an input whose stride is its window, since overlapping + /// windows would emit samples twice. + record same-frame { + pts: s64, + duration: option, + id: u32, + index: u32, } - /// What one call produced. Rows only: a packet sink's rows are its whole - /// product. - record processed { - /// JSON objects matching `meta.rows-schema`, zero or more per call. - rows: list, - /// Rows with no packet left to prompt them. Only the last call may fill - /// this. - trailing: list, + /// One thing leaving. Its pts and duration are in its port's time base, + /// and the module's to choose. + variant payload { + /// New bytes in the port's format: one frame's pixels, tightly packed, + /// or a run of interleaved samples. + frame(raw-frame), + same(same-frame), + /// For codec "json", one UTF-8 JSON object. Empty: a progress mark, + /// never delivered; the port's progress stands at its pts. + message(message), + packet(packet), } - describe: func() -> packet-sink-meta; - - /// Called once per instance, with every stream the sink reads: video pads - /// first, then audio, then data, each group in the order the query named - /// them. A sink declaring `one` of a kind is opened with exactly one of - /// it, and one declaring `zero` with none. - init: func(streams: list, params: string) -> result<_, string>; - - /// Replaces the parameters between calls, as the other interfaces do. - set-params: func(params: string) -> result<_, string>; - - /// Packets in decode order, one list per pad in `init`'s order. Pads run - /// independently: packets are not frames, so nothing pairs a pad's packet - /// with another pad's, and a call may carry packets on some pads and none - /// on others. `last` marks the final call of the instance's life: it - /// carries whatever is left, which may be no packets at all, and happens - /// exactly once. - /// - /// A call that is not the last may carry no packets on any pad. A module - /// runs only inside a call, so a host calls a sink nothing has reached for - /// a short while anyway, and one that drives something of its own there - - /// a network session - keeps it running between packets. Such a call is - /// answered like any other, and a module with nothing to do returns at - /// once. - process: func(pads: list, last: bool) -> processed; -} + record emission { + /// An output's name in the node's shape. + port: string, + payload: payload, + } -/// A transform over encoded packets: packets in, packets out, with rows -/// arriving beside them. Where `packet-sink` is a terminus - the encoder's -/// output goes in and rows alone come back - a filter hands the packets on, -/// so it sits BETWEEN the encoder that wrote them and whatever muxes or -/// publishes them. Nothing here decodes: the bytes a filter does not touch -/// are the bytes that leave. -/// -/// It reads pads the way a sink does, and declares what it accepts the same -/// way, so `describe`, `init` and `set-params` mean here what they mean -/// there. The two differences are that `init` answers the streams leaving - -/// ordinarily the ones that arrived - and that `process` answers packets as -/// well as rows. -/// -/// ROWS are the second input. A module weaving data into a stream needs the -/// data, and it arrives as JSON objects with times of their own rather than -/// on the packets: the host hands over whatever rows it has at each call, -/// and the module decides what to do with one whose time is ahead of or -/// behind the packets it has seen. A row ahead is held; a row behind is the -/// module's to place or drop. Nothing here pairs a row with a packet, so a -/// file of rows produced by an earlier stage and a live feed arriving while -/// packets flow are the same interface. -/// -/// What a filter must not break is the container on the other side: -/// -/// - Packet ORDER per pad is preserved. What arrives in decode order leaves -/// in decode order, and a module that holds packets back releases them in -/// that same order. -/// - ONE packet in, one packet out per access unit. A filter may hold a -/// packet back across calls and release it later, but over the instance's -/// life the packets leaving a pad are the packets that arrived on it, -/// neither dropped nor invented. A muxer counts frames; a filter that -/// changed the count would write a file whose timestamps no longer -/// describe its pictures. -/// - `pts`, `dts` and `duration` pass through untouched unless the module -/// is rewriting timestamps on purpose. They are the arriving packet's -/// own, and a module that rewrites them keeps dts non-decreasing. -/// - `keyframe` says of the OUTGOING packet what it said of the incoming -/// one: whether decoding can start there. Adding a parameter set or an -/// SEI to a keyframe leaves it a keyframe; nothing a filter does to the -/// bytes makes a non-keyframe one. -/// -/// A filter that rewrites a stream's out-of-band header - a new SPS and -/// PPS, say - says so by answering `init` with the changed `extradata`, and -/// the host writes that header rather than the one it read. Extradata is -/// the only part a filter may change. The codec, the time base and the -/// geometry belong to the stream that arrived; the profile and the level -/// are not carried beside the packets at all, but read back out of the -/// header itself, so a filter that raises a profile does it by writing the -/// parameter set that says so. -interface packet-filter { - use types.{arity, coded-stream, input-stream, meta, pad-packets}; - - /// `meta`, plus what this filter accepts. A packet filter names codecs - /// the way a frame module names pixel formats; `meta`'s own format lists - /// stay empty, since no decoded payload ever reaches it. - record packet-filter-meta { - meta: meta, - /// ffmpeg's names for the VIDEO codecs accepted, most preferred first. - /// Empty is every codec. - video-codecs: list, - /// ffmpeg's names for the AUDIO codecs accepted, most preferred first. - /// Empty is every codec. - audio-codecs: list, - /// How many video streams this filter reads. - video: arity, - /// How many audio streams this filter reads. - audio: arity, - /// How many data streams this filter reads. A world before 0.17.0 read - /// none, which is what one is adapted to. - data: arity, - /// Whether this filter ACTS on the rows handed to `process`. Every call - /// receives whatever rows have arrived whatever this says, so - /// availability is no answer: a filter that only rewrites packets - /// declares false, and one weaving rows into the stream declares true. - /// A host with no rows to give refuses a filter declaring true rather - /// than run it blind. - reads-rows: bool, - } - - /// What one call produced. - record filtered { - /// The packets leaving, one list per pad in `init`'s order, each in - /// decode order. A pad's list is empty when the call released nothing - /// on it - which a filter holding packets back does, and which says - /// nothing about whether more are coming. - pads: list, - /// JSON objects matching `meta.rows-schema`, zero or more per call: - /// what the filter itself has to say, exactly as a sink's `processed` - /// carries them. + /// What one tick produced. + record emitted { + /// Per port, frames and messages never go back in pts, within a call + /// or across calls, and packets keep decode order with dts never + /// decreasing. Ports interleave freely. + items: list, + /// Rows for the run's rows output, as a sink's are. rows: list, - /// Rows with no packet left to prompt them. Only the last call may fill - /// this. - trailing: list, + /// Nothing more will leave: the host makes the last call and then ends + /// every output. A node on an input clock ends with that input and need + /// not set it. + finished: bool, } - describe: func() -> packet-filter-meta; + /// Takes no params, so the compiler validates params before it asks for + /// a shape. A node leaves the format lists empty, since its ports say + /// what they accept in `shape`; `rows-schema` describes `emitted.rows`, + /// each data port carries its own `schema`, and `rows-language` tags + /// every "json" output. + describe: func() -> meta; - /// Called once per instance, with every stream the filter reads: video - /// pads first, then audio, then data, each group in the order the query - /// named them. A filter declaring `one` of a kind is opened with exactly - /// one of it, and one declaring `zero` with none. + /// The ports and clock for these params, and for `bound`: the inputs + /// the call binds, by the declaration's names, so a shape may turn on + /// what is given (a page with no picture ticks at a rate; an output + /// `like` an unbound input is left out). Called at compile time with the + /// call's static params, once per distinct (module, params, bound), and + /// cached; a source may read the network here to learn its outputs, as + /// `probe` did. Callable at any point in an instance's life: it reads + /// nothing `init` or `set-params` set. /// - /// The list answered is the streams LEAVING, one per input pad and in the - /// same order: ordinarily the ones that arrived, handed straight back. A - /// filter that rewrites a stream's out-of-band header answers the changed - /// `extradata` instead, and the host writes that. Everything else in the - /// record is the arriving stream's: the codec, the time base and the - /// geometry because the packets leaving are packets of the stream that - /// arrived, and the profile and the level because nothing carries them - /// beside the packets - they are read back out of the header. A host - /// refuses any of those changed, and a list of any other length. - init: func(streams: list, params: string) -> result, string>; - - /// Replaces the parameters between calls, as the other interfaces do. + /// The host refuses a shape in which `clock` names no input, or one that + /// is optional, `many` or not lockstep; an input is lockstep with no + /// input clock, or paired other than `arrival` on a self-clocked node; + /// `hold` is on a message kind or `interval` on a frame kind; a data + /// input's `rows` is `ignore`; `like` names an input that is not single + /// and bound; or a stride is 0 or above its window. + shape: func(params: string, bound: list) -> result; + + /// Opens an instance for `shape(params, bound)` on the streams bound to + /// its inputs, one entry each: ports in the shape's order, a port's + /// streams in the order the query named them, none for an optional port + /// with nothing bound. Each `id` is distinct and names its stream, in + /// every tick call and in `same`, for the instance's life. `latched` + /// names the outputs the query latched onto; the node may leave the rest + /// unmade. + init: func(bound: list, latched: list, params: string) -> result<_, string>; + + /// Replaces the params between ticks. Params whose shape differs from + /// the instance's are refused by the host and never reach this call, so + /// a pairing's fields and a port's latency never change under an + /// instance; an Err leaves the previous params in force. set-params: func(params: string) -> result<_, string>; - /// Packets in decode order, one list per pad in `init`'s order, and the - /// rows that have arrived since the last call. Pads run independently: - /// packets are not frames, so nothing pairs a pad's packet with another - /// pad's, and a call may carry packets on some pads and none on others. - /// - /// `rows` is whatever the host had ready. They arrive in the order they - /// were written and never arrive twice, and a module needing a row's time - /// reads it out of the row. - /// - /// A filter may be given SEVERAL rows inputs, one per rows argument the - /// query wrote. They are not separate lists: every row arrives in this one, - /// and a row from a NAMED input carries an extra `"_arg": ""` field - /// saying which argument it filled. The host writes that field itself and - /// refuses a producer row that already carries one, so a module may read it - /// as the host's word rather than the producer's. An input the host was - /// given no name for adds nothing, and its rows arrive exactly as written. - /// - /// WHEN they arrive depends on where they come from, and a module may rely - /// on this much: - /// - /// - From a FILE, every row is in hand before the first call, as many as - /// the host's own buffer holds. A file whose rows outrun that buffer - /// fills it, and the rest arrive over the calls that follow as the - /// module takes them. So a filter fed a file of any ordinary size sees - /// the lot before packet one, and one fed a very large file sees them in - /// order as it drains them. SEVERAL files share that one buffer, so what - /// is in hand before the first call is the first bufferful of all of them - /// together, in whatever order their readers interleaved. - /// - From a PIPE, they arrive whenever they are written, which may be none - /// for many calls and many at once. Packets never wait on a pipe: a live - /// run whose rows are still being produced keeps flowing. + /// One tick. An Err ends the run with its message, as a panic does; a + /// tick with nothing to emit returns Ok with nothing. A self-clocked + /// node's call may wait until it has something, and is called again as + /// soon as it returns. /// - /// Either way nothing pairs a row with a packet, and a module that needs - /// every row before it commits to anything holds its packets back until - /// the call that brings them. - /// - /// `last` marks the final call of the instance's life: it carries whatever - /// packets and rows are left, which may be none of either, and happens - /// exactly once. Everything held back leaves on it - a packet still held - /// when `last` returns is a packet the container never gets. - process: func(pads: list, rows: list, last: bool) -> filtered; + /// Unless the node is self-clocked, returning promises that no output + /// still has anything to send stamped earlier than the end of the tick's + /// interval less the port's `latency`. The host sends that time down + /// each data edge as the node's progress, and never less than the last + /// message's pts; a message that breaks the promise reaches its + /// consumers late, and is reported. + process: func(tick: borrow) -> result; } -/// A source of encoded packets, the mirror of `packet-sink`: no input pads, -/// one or more output tracks. Nothing arrives to push a source, so the host -/// pulls: `next` is called once per batch until it answers none. -/// -/// `probe` reads the whole catalog at compile time - every track the source -/// publishes and whether it ever ends - which is what makes those tracks -/// known before anything runs. `open` is told which of them the run reads, -/// and the source subscribes to exactly those: a track nobody asked for is -/// never pulled off the wire. -interface packet-source { - use types.{coded-stream, meta, pad-packets, rendition-meta, stream-info}; - - /// One track the source publishes, fixed for the life of the instance. - record source-track { - coded: coded-stream, - info: stream-info, - /// Which relation row this track belongs to. - row: u32, - /// What the source read of that row. - rendition: rendition-meta, - } - - /// Every track a source publishes, and whether it ever ends. - record catalog { - tracks: list, - bounded: bool, - } - - describe: func() -> meta; - - /// Reads the catalog at compile time: the source connects or opens its - /// file and reports every track it would publish, without producing - /// packets. - probe: func(params: string) -> result; - - /// Opens the source for a run, subscribing to `tracks` alone: indices into - /// the catalog `probe` read, distinct, in the order the run wants them. - /// The catalog returned is that restriction - one track per index, in that - /// order - and it is the order every later call speaks in. An index the - /// source does not publish is refused by name. - open: func(params: string, tracks: list) -> result; - - /// One pull: a packet list per track `open` was given, in that order, or - /// none once the source has nothing left. - /// - /// On a data track a packet holding nothing but whitespace is no message - /// but a HEARTBEAT: the source's word that no message before its pts is - /// still to come on that track. The host carries it to the track's - /// readers, and no module downstream is ever handed one. The host says - /// nothing of its own about how far a data track has got past its start, - /// since tracks run independently, so a source that knows hands these, - /// and one that cannot know hands none. - next: func() -> result>, string>; -} interface values { /// Static description of one value function, available without calling it. @@ -781,114 +723,6 @@ interface values { invoke: func(name: string, args: string) -> result; } -/// Rows in, rows out, with no stream at all: every other module either reads -/// or writes frames or packets and carries rows beside them, but a module -/// exporting this interface has nothing beside - JSON objects are its whole -/// input and its whole output. -interface rows-module { - use types.{meta}; - - /// `meta`, plus the schema of the row this module reads. `meta`'s own - /// `rows-schema` already means "what this module emits" for every other - /// interface, so the schema it consumes belongs beside `meta` rather than - /// inside it - the way `packet-sink-meta` adds sink-only fields beside the - /// shared record instead of widening it for every module. - record rows-module-meta { - meta: meta, - /// JSON Schema for one row object `process` reads. Empty when the module - /// reads no shape in particular. - input-rows-schema: string, - } - - describe: func() -> rows-module-meta; - - init: func(params: string) -> result<_, string>; - - /// JSON rows in, JSON rows out, in one call: no stream carries them, so - /// nothing else drives this module. A call may answer zero rows for an - /// input row, or several - a filter that drops some, or a module that - /// splits one row into several. - process: func(rows: list) -> result, string>; - - /// Whatever the module held back across every `process` call. Called once, - /// after the last `process`. - finish: func() -> result, string>; -} - -/// Messages in, messages out: a data stream is a sequence of messages, each -/// one JSON object timed by its pts, and a module exporting this interface -/// reads data streams and writes them. It is not one-to-one: a call may -/// answer any number of messages, or none, and may hold some back for a -/// later call. -/// -/// It may also read CLOCK pads: a video or audio argument whose frames the -/// module never sees, only their time. That is how a module with no data -/// input still acts at programme times - the host hands it `now` on every -/// call - and how one reading messages knows what time it is between them. -interface data-filter { - use types.{meta, rational}; - - /// One message: for codec "json", one UTF-8 JSON object. - record message { - pts: s64, - data: list, - } - - /// What one argument of the call is. - enum pad-kind { - /// A data stream: its messages arrive in `process`. - data, - /// A video or audio stream read for its clock alone. - clock, - } - - record pad-info { - kind: pad-kind, - /// For a data pad, its codec: "json". Empty for a clock pad. - codec: string, - /// The unit a data pad's pts, or a clock pad's `now`, is counted in. - time-base: rational, - } - - record data-filter-meta { - meta: meta, - /// The codec of each data stream the module writes, in output order: - /// ["json"] for one. - outputs: list, - /// The unit the outputs' pts are counted in. - time-base: rational, - } - - /// The messages that arrived on one data pad since the last call, in pts - /// order. A clock pad has none. - record pad-messages { - pad: u32, - messages: list, - } - - record processed { - /// The messages written, one list per output in output order, each in - /// pts order. Across calls an output's pts never decrease. - outputs: list>, - /// Rows for the run's rows output, as a sink's are. - rows: list, - } - - describe: func() -> data-filter-meta; - - /// Called once, with every argument of the call in order. - init: func(pads: list, params: string) -> result<_, string>; - - /// One batch: every message that arrived since the last call, across the - /// data pads, and `now`, the programme time in ticks of the first clock - /// pad's time base - none when the call has no clock pad. A module with - /// a clock pad is called as its clock advances, whether or not a message - /// arrived; one without is called when messages arrive. `last` marks the - /// final call, which happens once and carries whatever is left. - process: func(input: list, now: option, - last: bool) -> result; -} - /// A codec a package implements, where ffmpeg's own encoder would stand: /// raw frames in, coded packets out. A query names it in an output's options /// (`video_codec => ffrwd.pyrowave.encode(bitrate => 200000000)`) and its @@ -981,15 +815,17 @@ interface decoder { decode: func(packets: list, last: bool) -> result, string>; } -world video-module { export filter; } -world meta-module { export filter; export meta-filter; } -world window-module { import window-source; export window-filter; } -world packet-sink-module { export packet-sink; } -world packet-filter-module { export packet-filter; } -world packet-source-module { export packet-source; } +/// A node module. A module that also imports `wasi:nn`, `wasi:webgpu`, +/// `wasi:http` or `wasi:sockets` includes this world in its own. +/// +/// `values`, `encoder`, `decoder` and their worlds stay as they are at +/// 0.18.0. Any of them may ride beside `node` in one component, whose +/// world includes this one and theirs. +world node-module { + import node-tick; + export node; +} world values-module { export values; } -world rows-module-host { export rows-module; } -world data-filter-module { export data-filter; } world encoder-module { export encoder; } world decoder-module { export decoder; } world codec-module { export encoder; export decoder; } diff --git a/sidecar/worlds/0.18.0/av.wit b/sidecar/worlds/0.18.0/av.wit new file mode 100644 index 0000000..9d02fd0 --- /dev/null +++ b/sidecar/worlds/0.18.0/av.wit @@ -0,0 +1,995 @@ +package ffrwd:av@0.18.0; + +interface types { + /// Static description of a module, available without running anything. + record meta { + name: string, + version: string, + /// JSON Schema for the `params` string. + params-schema: string, + /// JSON Schema for one row object; empty when the module emits none. + rows-schema: string, + /// Pixel formats accepted, most preferred first. Empty means this is not + /// a video module. + pixel-formats: list, + /// Sample formats accepted, most preferred first: "f32" or "s16", always + /// interleaved. Empty means this is not an audio module. A module is one + /// kind or the other, so exactly one of this and `pixel-formats` is + /// filled in. + sample-formats: list, + /// Sample rates accepted. Empty is every rate. + sample-rates: list, + /// Channel counts accepted. Empty is every count. + channel-counts: list, + /// Ordered param names; the module's rows' language is the first of these + /// params that is set at the call; empty list = none declared. + rows-language: list, + } + + /// A ratio of two whole numbers. + record rational { + num: s32, + den: s32, + } + + /// A stream's colorimetry, in ffmpeg's own names: "tv" or "pc" for the + /// range, "bt709" and the like for the rest. A field the wire does not + /// settle is "unknown", ffmpeg's own spelling for it. + record color-info { + range: string, + primaries: string, + trc: string, + space: string, + } + + /// The frames of a video instance, fixed for the life of the instance. + /// Frames cross this boundary square-pixel: the host rescales anamorphic + /// input before it arrives, so no aspect ratio is carried here. + record video-format { + width: u32, + height: u32, + pix-fmt: string, + /// The colorimetry the stream header declared; none where it did not. + color: option, + } + + /// The samples of an audio instance, fixed for the life of the instance. + record audio-format { + sample-rate: u32, + channels: u32, + /// "f32" or "s16", interleaved. + sample-fmt: string, + /// ffmpeg's name for the channel layout ("stereo", "5.1"), which a + /// count alone cannot say; none where the wire does not carry one. + channel-layout: option, + } + + /// What an instance is opened for, which is what tells its kind. A module + /// is handed the arm its own declaration asked for; the host refuses the + /// other one before `init`. + variant format { + video(video-format), + audio(audio-format), + } + + /// The stream an instance is attached to, told once at init. + record stream-info { + index: u32, + kind: string, + codec: string, + /// Seconds; absent when unknown. + duration: option, + /// The stream's own tags. + tags: list>, + /// The unit timestamps are counted in, as a fraction of a second. The host + /// always supplies it; `den` is always positive. A windowed module is + /// handed timestamps rather than seconds, so this is how one whose + /// parameters are in seconds converts them. For audio it is what turns a + /// sample position into a timestamp: at 1/48000 one tick is one sample. + time-base: rational, + } + + /// The frames a coded video stream declares. Shared between `packet-sink` + /// and `packet-source`. + record coded-video { + width: u32, + height: u32, + /// The pixel aspect ratio the stream declares. Packets cannot be + /// host-rescaled, so a packager writes it through (mp4's `pasp`); + /// none where the wire does not say. + sample-aspect-ratio: option, + /// The colorimetry the stream header declared; none where it did not. + color: option, + } + + /// The samples a coded audio stream declares. + record coded-audio { + sample-rate: u32, + channels: u32, + /// ffmpeg's name for the channel layout ("stereo", "5.1"), which a + /// count alone cannot say; none where the wire does not carry one. + channel-layout: option, + } + + /// What a coded stream carries, which is what tells its kind. + variant coded-format { + video(coded-video), + audio(coded-audio), + /// A data stream: messages rather than pictures or sound, each packet + /// one message at its own pts. For codec "json" a message is one UTF-8 + /// JSON object. It has no geometry, and every packet is a keyframe. + data, + } + + /// One encoded stream an instance is opened for, fixed for its life. + record coded-stream { + /// ffmpeg's name for the codec, e.g. "h264". + codec: string, + /// The unit `pts` and `dts` are counted in, as a fraction of a second. + time-base: rational, + /// The geometry the stream declares, and its kind. + format: coded-format, + /// The codec's out-of-band header, exactly as the stream header carried + /// it: for h264, the SPS and PPS a downstream packager needs before the + /// first packet. + extradata: list, + /// The codec's profile, in the codec's own numbering - what ffmpeg's + /// codec parameters carry; none where the wire does not say. + profile: option, + /// The codec's level, likewise. + level: option, + } + + /// One encoded packet, exactly as the encoder emitted it. + record packet { + /// When the picture is presented, in the stream's time base. Packets + /// arrive in decode order, so where the stream reorders frames this is + /// not monotonic. + pts: s64, + /// When the packet is decoded; never decreasing. Absent for the first + /// packets of a reordering stream, where the wire does not settle it. + dts: option, + /// How long the packet is presented, in the stream's time base: the + /// next packet's pts minus this one's, in presentation order. None + /// where the wire does not settle it - unknown is never spelled 0. + duration: option, + /// Whether decoding can start at this packet. + keyframe: bool, + /// The encoded bytes, untouched. + data: list, + } + + /// One raw frame crossing a codec: a video frame's pixels in the + /// instance's pix-fmt, or a run of interleaved audio samples in its + /// sample-fmt. Shared between `encoder` and `decoder`. + record raw-frame { + /// When it is presented, in the stream's time base. + pts: s64, + /// How long it is presented, in the stream's time base; none where + /// nothing settles it - unknown is never spelled 0. + duration: option, + data: list, + } + + /// One pad's packets from one call, in decode order. Empty when nothing + /// arrived on that pad; a packet sink's list of these is as long as its + /// `init`'s streams and in the same order, so the index is the pad. + record pad-packets { + packets: list, + } + + /// What the source read of one relation row: a rendition's name, bitrate + /// and codec string, exactly as the manifest or catalog said them. None + /// where nothing said so. Shared between `packet-sink` and + /// `packet-source`. + record rendition-meta { + name: option, + bandwidth: option, + codecs: option, + language: option, + } + + /// How much of a stream a sink has to be handed to do its work. A sink + /// that reads what a writer put on keyframes, or what a stream's first + /// packet declares, does not need the rest - and a host reading a file to + /// answer a question at compile time can copy far less of it. + /// + /// It is a REQUEST, not a promise: a host may hand over more than was + /// asked for, and every sink has to work when it does. What a host may not + /// do is hand over less. + enum wants { + /// Every packet of every stream. What a sink that counts, tallies or + /// republishes needs, and what a host with no way to skip hands over. + all, + /// The keyframes, and whatever else the host could not cheaply drop. + /// What a sink reading data a writer placed on keyframes needs. + keyframes, + /// The first packet of each stream, and no more. What a sink reading + /// only what a stream declares about itself needs. + first, + } + + /// How many streams of one kind a module reading encoded packets takes. + /// Shared between `packet-sink` and `packet-filter`. + enum arity { + /// None of this kind reaches the module. + zero, + /// Exactly one, which is what every sink before 0.12.0 read. + one, + /// One or more, as many as the query hands over. + many, + /// As many as the query hands over, none included: the module reads + /// them when they are there and works when they are not. + any, + } + + /// One pad's stream, told once at init: what the wire's own header said + /// about the encoding, and what the process driving the host said about + /// the stream. Shared between `packet-sink` and `packet-filter`. + record input-stream { + coded: coded-stream, + info: stream-info, + /// Which relation row this pad belongs to - the moq sink learns + /// renditions from the rows instead of from argument names. + row: u32, + /// What the source read of that row. + rendition: rendition-meta, + /// How deep this stream reorders: how many packets the decoder holds + /// back before the first picture leaves it. 0 where decode order IS + /// presentation order, and higher wherever B-frames do. + /// + /// It is the bound a module that must see packets in PRESENTATION + /// order works with: a packet's presentation successor has arrived + /// once `decode-delay` more packets have. It is also why `packet.dts` + /// is none for a pad's first `decode-delay` packets, which is the wire + /// not having settled them rather than the host withholding them. + decode-delay: u32, + } +} + +interface filter { + use types.{meta, stream-info}; + + record frame-info { + width: u32, + height: u32, + /// Seconds since the start of the stream. + time: f64, + } + + variant output { + /// A new frame, same byte length as the input. + frame(list), + /// The input frame is the output; the host copies nothing. + passthrough, + } + + /// What one frame produced. + record outcome { + output: output, + /// JSON objects matching `meta.rows-schema`, zero or more per frame. + rows: list, + } + + describe: func() -> meta; + + /// Called once per instance. `pix-fmt` is one of `meta.pixel-formats`, + /// chosen by whatever drives the host. The frame size is fixed for the + /// life of the instance. This interface carries video alone: a module + /// publishing sample formats exports `window-filter` instead. + init: func(width: u32, height: u32, pix-fmt: string, stream-info: stream-info, + params: string) -> result<_, string>; + + /// Replaces the parameters between frames. The module keeps the state + /// it built in `init`; rejecting the new parameters must leave the + /// previous ones in force. + set-params: func(params: string) -> result<_, string>; + + /// Whether `process` depends only on the current frame. False makes + /// frame-parallel hosting unsound and a host must refuse it. Callable + /// after `init`; the answer may depend on the parameters. + frame-independent: func() -> bool; + + process: func(info: frame-info, frame: list) -> outcome; +} + +/// Optional beside `filter`, the way `values` is optional: a module that also +/// exports this one reads the rows an upstream module emitted for the same +/// frame. Everything else - `describe`, `init`, `set-params`, +/// `frame-independent` - still comes from `filter`, so a module exporting +/// this exports both. A host that finds this export calls `process-meta` +/// instead of `process`; one that does not calls `process` as before. +interface meta-filter { + use filter.{frame-info, outcome}; + + /// One frame plus the rows that arrived with it, each an NDJSON line an + /// upstream module produced. Empty when the frame carries none. + process-meta: func(info: frame-info, frame: list, rows-in: list) -> outcome; +} + +/// The window `window-filter`'s `process` borrows, held by the host for +/// exactly that call. Earlier worlds copied every payload of the window into +/// the call; here the metadata is cheap and bytes move only on `fetch`, so a +/// module that reads one payload of a wide window never pays for the rest. +interface window-source { + /// One call's input payloads. Indices run oldest first, 0 to `len() - 1` - + /// or pad order, for a module reading several streams - and an index at or + /// past `len()` is a fault that stops the run. + /// + /// The handle is the call's alone: it cannot be kept past the return, and + /// bytes a module means to keep are its own copy the moment `fetch` hands + /// them over. + resource in-window { + /// How many payloads this call carries. + len: func() -> u32; + + /// Payload `i`'s timestamp. For an audio instance it is the first + /// sample's tick. + pts: func(i: u32) -> s64; + + /// The rows that arrived with payload `i` - each an NDJSON line an + /// upstream module produced, empty when nothing came with it. Rows ride + /// pad 0: a module reading several streams is handed whatever arrived on + /// pad 0, and the rows that arrived on any other pad are dropped before + /// the call - so this is empty on every pad past the first. + rows: func(i: u32) -> list; + + /// Payload `i`'s bytes, copied on demand: the only per-payload copy. + /// + /// For a video instance the bytes are one frame's pixels. For an audio + /// instance they are interleaved samples in the instance's sample + /// format, the whole window in one contiguous piece: the host re-cuts + /// whatever packets arrived, so a module never sees a packet boundary. + fetch: func(i: u32) -> list; + } +} + +/// A window at a time, superseding `filter`'s per-frame `process` and +/// `meta-filter`'s `process-meta`. A module exports this one *or* those; a +/// host adapts a module that exports the older pair as window 1, stride 1, +/// one-to-one. `describe`, `init` and `set-params` mean here what they mean +/// there, so the two differ only in how the payload arrives and leaves. +/// +/// This is the only interface audio reaches. `init` is handed a `format`, and +/// which arm it carries is the instance's kind for the rest of its life. +interface window-filter { + use types.{format, meta, stream-info}; + use window-source.{in-window}; + + /// Where an output payload's bytes come from. + variant frame-payload { + /// New bytes: one frame's pixels, or interleaved samples. + new(list), + /// An input's bytes, which the module never copied out: the input this + /// call received at the same pts, or - when the call received exactly one + /// input - that one, whatever pts the output carries. For a module + /// reading several streams this is pad 0's bytes, since every pad shares + /// the call's one pts. An audio module may say this only when its stride + /// is its window, since overlapping windows passed through would emit + /// every sample more than once. + same, + } + + /// One output. Its pts is the module's to choose: a module may move its + /// payload in time, drop it, or emit several from one. + record out-frame { + pts: s64, + frame: frame-payload, + rows: list, + } + + /// `meta`, plus how the host must drive this module. + record window-meta { + meta: meta, + /// What one `process` call receives: frames for a video module, SAMPLES + /// for an audio one. 1 is plain streaming for video; an audio module + /// names a chunk it is happy to work in, and the host cuts that chunk out + /// of whatever packets arrive. + window: u32, + /// How much is consumed per call, never more than `window`, in the same + /// unit. `window` makes the windows disjoint. + stride: u32, + /// Whether a call depends only on what it was handed. False makes + /// parallel hosting unsound and a host must refuse it. + pure: bool, + /// For video: whether every call returns exactly one output per frame it + /// consumed, each at that frame's own pts. For audio: whether the samples + /// leaving over the instance's life are the samples that arrived, neither + /// dropped nor invented - per-call counts may differ, but the output runs + /// on without overlap, and a gap only where the input had one: a sample + /// either follows the last one out or keeps the timestamp the input gave + /// it. Neither window nor stride settles this, so it is declared. + one-to-one: bool, + /// Whether this module ACTS on the rows arriving with its payloads. Every + /// call receives them whatever this says, so availability is no answer: a + /// module that only carries rows through declares false, and one that + /// reads what an upstream module detected declares true. + reads-rows: bool, + /// Whether upstream rows may appear on this module's own output. False + /// means the rows leaving are the module's own, or none. It speaks of pad + /// 0: rows arrive there and nowhere else. + forwards-rows: bool, + /// How many streams this module reads. Frames arrive one per pad, in pad + /// order, all at the same pts. 1 is the ordinary module, reading one + /// stream. More than 1 is window 1, stride 1 - a call takes one frame off + /// each pad and hands one back - and a host refuses any other shape. + inputs: u32, + /// Arguments of the call that are FEEDERS rather than pads: a stream + /// that may be empty, which the module reads itself over a loopback + /// connection instead of being handed its frames. Empty for every + /// module that has none. + feeders: list, + } + + /// One feeder argument. The host picks a loopback port, writes it into + /// the param `port-param` names, and delivers the stream passed in that + /// position there as NUT. A call that passes a number in that position + /// instead is naming a port itself, and the host wires nothing. + record feeder { + /// Which argument of the call it is, counting stream arguments from 0. + input: u32, + /// The param the host puts the port it picked in. + port-param: string, + /// "video" or "audio". + kind: string, + /// Feeders sharing one connection: every feeder in a query fed by the + /// same source and naming the same group gets ONE port, written into + /// each call's `port-param`, and ONE NUT carrying every stream those + /// feeders name, video then audio. Empty: a connection of its own. + group: string, + } + + /// What one call produced. + record processed { + frames: list, + /// Rows with nothing to ride. Only the last call may fill this. + trailing: list, + } + + describe: func() -> window-meta; + + /// Called once per instance. The arm of `format` is the one the module's + /// own declaration asked for: a module publishing pixel formats is opened + /// with `video`, one publishing sample formats with `audio`. + init: func(format: format, stream-info: stream-info, + params: string) -> result<_, string>; + + /// Replaces the parameters between windows, as `filter`'s does. + set-params: func(params: string) -> result<_, string>; + + /// One window. `last` marks the final call of the instance's life: it + /// carries whatever the last stride left over, which may be nothing, and + /// happens exactly once. There is no separate flush. + /// + /// `window` holds the call's payloads, borrowed for the call: the window + /// for a module reading one stream. For a module reading several it is one + /// frame per pad, in pad order, every one of them at the same pts - which + /// is why such a module is window 1, stride 1. + /// + /// `trailing` is the rows an upstream module had nothing to put them on; + /// only the last call carries any. Like `in-window`'s rows, it follows pad + /// 0: what reached the other pads with nothing to ride never arrives here. + /// + /// Output timestamps never decrease, within a call or between calls. + process: func(window: borrow, trailing: list, last: bool) -> processed; +} + +/// A sink for encoded packets. Where `window-filter` is handed decoded +/// payloads, this interface is handed the encoder's own output - compressed +/// bytes with their timestamps and keyframe flags - and hands back rows +/// alone. A packet sink consumes the streams: no frames leave, so a module +/// exports this one instead of the frame interfaces, and what it emits goes +/// to a rows output or nowhere. +/// +/// A sink is a filter with no output pads, and reads streams the way one +/// does: several at once, of either kind. Earlier worlds carried exactly one +/// coded stream and one packet list; here `init` is handed every stream the +/// query named, in that order, and `process` is handed one packet list per +/// pad in the same order. +interface packet-sink { + use types.{arity, input-stream, meta, pad-packets, wants}; + + /// `meta`, plus what this sink accepts. A packet sink names codecs the way + /// a frame module names pixel formats; `meta`'s own format lists stay + /// empty, since no decoded payload ever reaches it. + record packet-sink-meta { + meta: meta, + /// ffmpeg's names for the VIDEO codecs accepted, most preferred first. + /// Empty is every codec. + video-codecs: list, + /// ffmpeg's names for the AUDIO codecs accepted, most preferred first. + /// Empty is every codec. + audio-codecs: list, + /// How many video streams this sink reads. + video: arity, + /// How many audio streams this sink reads. + audio: arity, + /// How many data streams this sink reads. A world before 0.17.0 read + /// none, which is what one is adapted to. + data: arity, + /// How much of the stream this sink has to see. `all` is the answer for + /// a sink that reads every packet, and the answer a world before 0.16.0 + /// is adapted to, since none of them could say otherwise. + wants: wants, + } + + /// What one call produced. Rows only: a packet sink's rows are its whole + /// product. + record processed { + /// JSON objects matching `meta.rows-schema`, zero or more per call. + rows: list, + /// Rows with no packet left to prompt them. Only the last call may fill + /// this. + trailing: list, + } + + describe: func() -> packet-sink-meta; + + /// Called once per instance, with every stream the sink reads: video pads + /// first, then audio, then data, each group in the order the query named + /// them. A sink declaring `one` of a kind is opened with exactly one of + /// it, and one declaring `zero` with none. + init: func(streams: list, params: string) -> result<_, string>; + + /// Replaces the parameters between calls, as the other interfaces do. + set-params: func(params: string) -> result<_, string>; + + /// Packets in decode order, one list per pad in `init`'s order. Pads run + /// independently: packets are not frames, so nothing pairs a pad's packet + /// with another pad's, and a call may carry packets on some pads and none + /// on others. `last` marks the final call of the instance's life: it + /// carries whatever is left, which may be no packets at all, and happens + /// exactly once. + /// + /// A call that is not the last may carry no packets on any pad. A module + /// runs only inside a call, so a host calls a sink nothing has reached for + /// a short while anyway, and one that drives something of its own there - + /// a network session - keeps it running between packets. Such a call is + /// answered like any other, and a module with nothing to do returns at + /// once. + process: func(pads: list, last: bool) -> processed; +} + +/// A transform over encoded packets: packets in, packets out, with rows +/// arriving beside them. Where `packet-sink` is a terminus - the encoder's +/// output goes in and rows alone come back - a filter hands the packets on, +/// so it sits BETWEEN the encoder that wrote them and whatever muxes or +/// publishes them. Nothing here decodes: the bytes a filter does not touch +/// are the bytes that leave. +/// +/// It reads pads the way a sink does, and declares what it accepts the same +/// way, so `describe`, `init` and `set-params` mean here what they mean +/// there. The two differences are that `init` answers the streams leaving - +/// ordinarily the ones that arrived - and that `process` answers packets as +/// well as rows. +/// +/// ROWS are the second input. A module weaving data into a stream needs the +/// data, and it arrives as JSON objects with times of their own rather than +/// on the packets: the host hands over whatever rows it has at each call, +/// and the module decides what to do with one whose time is ahead of or +/// behind the packets it has seen. A row ahead is held; a row behind is the +/// module's to place or drop. Nothing here pairs a row with a packet, so a +/// file of rows produced by an earlier stage and a live feed arriving while +/// packets flow are the same interface. +/// +/// What a filter must not break is the container on the other side: +/// +/// - Packet ORDER per pad is preserved. What arrives in decode order leaves +/// in decode order, and a module that holds packets back releases them in +/// that same order. +/// - ONE packet in, one packet out per access unit. A filter may hold a +/// packet back across calls and release it later, but over the instance's +/// life the packets leaving a pad are the packets that arrived on it, +/// neither dropped nor invented. A muxer counts frames; a filter that +/// changed the count would write a file whose timestamps no longer +/// describe its pictures. +/// - `pts`, `dts` and `duration` pass through untouched unless the module +/// is rewriting timestamps on purpose. They are the arriving packet's +/// own, and a module that rewrites them keeps dts non-decreasing. +/// - `keyframe` says of the OUTGOING packet what it said of the incoming +/// one: whether decoding can start there. Adding a parameter set or an +/// SEI to a keyframe leaves it a keyframe; nothing a filter does to the +/// bytes makes a non-keyframe one. +/// +/// A filter that rewrites a stream's out-of-band header - a new SPS and +/// PPS, say - says so by answering `init` with the changed `extradata`, and +/// the host writes that header rather than the one it read. Extradata is +/// the only part a filter may change. The codec, the time base and the +/// geometry belong to the stream that arrived; the profile and the level +/// are not carried beside the packets at all, but read back out of the +/// header itself, so a filter that raises a profile does it by writing the +/// parameter set that says so. +interface packet-filter { + use types.{arity, coded-stream, input-stream, meta, pad-packets}; + + /// `meta`, plus what this filter accepts. A packet filter names codecs + /// the way a frame module names pixel formats; `meta`'s own format lists + /// stay empty, since no decoded payload ever reaches it. + record packet-filter-meta { + meta: meta, + /// ffmpeg's names for the VIDEO codecs accepted, most preferred first. + /// Empty is every codec. + video-codecs: list, + /// ffmpeg's names for the AUDIO codecs accepted, most preferred first. + /// Empty is every codec. + audio-codecs: list, + /// How many video streams this filter reads. + video: arity, + /// How many audio streams this filter reads. + audio: arity, + /// How many data streams this filter reads. A world before 0.17.0 read + /// none, which is what one is adapted to. + data: arity, + /// Whether this filter ACTS on the rows handed to `process`. Every call + /// receives whatever rows have arrived whatever this says, so + /// availability is no answer: a filter that only rewrites packets + /// declares false, and one weaving rows into the stream declares true. + /// A host with no rows to give refuses a filter declaring true rather + /// than run it blind. + reads-rows: bool, + } + + /// What one call produced. + record filtered { + /// The packets leaving, one list per pad in `init`'s order, each in + /// decode order. A pad's list is empty when the call released nothing + /// on it - which a filter holding packets back does, and which says + /// nothing about whether more are coming. + pads: list, + /// JSON objects matching `meta.rows-schema`, zero or more per call: + /// what the filter itself has to say, exactly as a sink's `processed` + /// carries them. + rows: list, + /// Rows with no packet left to prompt them. Only the last call may fill + /// this. + trailing: list, + } + + describe: func() -> packet-filter-meta; + + /// Called once per instance, with every stream the filter reads: video + /// pads first, then audio, then data, each group in the order the query + /// named them. A filter declaring `one` of a kind is opened with exactly + /// one of it, and one declaring `zero` with none. + /// + /// The list answered is the streams LEAVING, one per input pad and in the + /// same order: ordinarily the ones that arrived, handed straight back. A + /// filter that rewrites a stream's out-of-band header answers the changed + /// `extradata` instead, and the host writes that. Everything else in the + /// record is the arriving stream's: the codec, the time base and the + /// geometry because the packets leaving are packets of the stream that + /// arrived, and the profile and the level because nothing carries them + /// beside the packets - they are read back out of the header. A host + /// refuses any of those changed, and a list of any other length. + init: func(streams: list, params: string) -> result, string>; + + /// Replaces the parameters between calls, as the other interfaces do. + set-params: func(params: string) -> result<_, string>; + + /// Packets in decode order, one list per pad in `init`'s order, and the + /// rows that have arrived since the last call. Pads run independently: + /// packets are not frames, so nothing pairs a pad's packet with another + /// pad's, and a call may carry packets on some pads and none on others. + /// + /// `rows` is whatever the host had ready. They arrive in the order they + /// were written and never arrive twice, and a module needing a row's time + /// reads it out of the row. + /// + /// A filter may be given SEVERAL rows inputs, one per rows argument the + /// query wrote. They are not separate lists: every row arrives in this one, + /// and a row from a NAMED input carries an extra `"_arg": ""` field + /// saying which argument it filled. The host writes that field itself and + /// refuses a producer row that already carries one, so a module may read it + /// as the host's word rather than the producer's. An input the host was + /// given no name for adds nothing, and its rows arrive exactly as written. + /// + /// WHEN they arrive depends on where they come from, and a module may rely + /// on this much: + /// + /// - From a FILE, every row is in hand before the first call, as many as + /// the host's own buffer holds. A file whose rows outrun that buffer + /// fills it, and the rest arrive over the calls that follow as the + /// module takes them. So a filter fed a file of any ordinary size sees + /// the lot before packet one, and one fed a very large file sees them in + /// order as it drains them. SEVERAL files share that one buffer, so what + /// is in hand before the first call is the first bufferful of all of them + /// together, in whatever order their readers interleaved. + /// - From a PIPE, they arrive whenever they are written, which may be none + /// for many calls and many at once. Packets never wait on a pipe: a live + /// run whose rows are still being produced keeps flowing. + /// + /// Either way nothing pairs a row with a packet, and a module that needs + /// every row before it commits to anything holds its packets back until + /// the call that brings them. + /// + /// `last` marks the final call of the instance's life: it carries whatever + /// packets and rows are left, which may be none of either, and happens + /// exactly once. Everything held back leaves on it - a packet still held + /// when `last` returns is a packet the container never gets. + process: func(pads: list, rows: list, last: bool) -> filtered; +} + +/// A source of encoded packets, the mirror of `packet-sink`: no input pads, +/// one or more output tracks. Nothing arrives to push a source, so the host +/// pulls: `next` is called once per batch until it answers none. +/// +/// `probe` reads the whole catalog at compile time - every track the source +/// publishes and whether it ever ends - which is what makes those tracks +/// known before anything runs. `open` is told which of them the run reads, +/// and the source subscribes to exactly those: a track nobody asked for is +/// never pulled off the wire. +interface packet-source { + use types.{coded-stream, meta, pad-packets, rendition-meta, stream-info}; + + /// One track the source publishes, fixed for the life of the instance. + record source-track { + coded: coded-stream, + info: stream-info, + /// Which relation row this track belongs to. + row: u32, + /// What the source read of that row. + rendition: rendition-meta, + } + + /// Every track a source publishes, and whether it ever ends. + record catalog { + tracks: list, + bounded: bool, + } + + describe: func() -> meta; + + /// Reads the catalog at compile time: the source connects or opens its + /// file and reports every track it would publish, without producing + /// packets. + probe: func(params: string) -> result; + + /// Opens the source for a run, subscribing to `tracks` alone: indices into + /// the catalog `probe` read, distinct, in the order the run wants them. + /// The catalog returned is that restriction - one track per index, in that + /// order - and it is the order every later call speaks in. An index the + /// source does not publish is refused by name. + open: func(params: string, tracks: list) -> result; + + /// One pull: a packet list per track `open` was given, in that order, or + /// none once the source has nothing left. + /// + /// On a data track a packet holding nothing but whitespace is no message + /// but a HEARTBEAT: the source's word that no message before its pts is + /// still to come on that track. The host carries it to the track's + /// readers, and no module downstream is ever handed one. The host says + /// nothing of its own about how far a data track has got past its start, + /// since tracks run independently, so a source that knows hands these, + /// and one that cannot know hands none. + next: func() -> result>, string>; +} + +interface values { + /// Static description of one value function, available without calling it. + record function-meta { + name: string, + /// JSON Schema for `invoke`'s `args` object. + params-schema: string, + /// JSON Schema for the value `invoke` returns. + result-schema: string, + } + + list-functions: func() -> list; + + /// `args` is one JSON object keyed by parameter name. The ok string is one + /// JSON value; the err string is the module's own message. + invoke: func(name: string, args: string) -> result; +} + +/// Rows in, rows out, with no stream at all: every other module either reads +/// or writes frames or packets and carries rows beside them, but a module +/// exporting this interface has nothing beside - JSON objects are its whole +/// input and its whole output. +interface rows-module { + use types.{meta}; + + /// `meta`, plus the schema of the row this module reads. `meta`'s own + /// `rows-schema` already means "what this module emits" for every other + /// interface, so the schema it consumes belongs beside `meta` rather than + /// inside it - the way `packet-sink-meta` adds sink-only fields beside the + /// shared record instead of widening it for every module. + record rows-module-meta { + meta: meta, + /// JSON Schema for one row object `process` reads. Empty when the module + /// reads no shape in particular. + input-rows-schema: string, + } + + describe: func() -> rows-module-meta; + + init: func(params: string) -> result<_, string>; + + /// JSON rows in, JSON rows out, in one call: no stream carries them, so + /// nothing else drives this module. A call may answer zero rows for an + /// input row, or several - a filter that drops some, or a module that + /// splits one row into several. + process: func(rows: list) -> result, string>; + + /// Whatever the module held back across every `process` call. Called once, + /// after the last `process`. + finish: func() -> result, string>; +} + +/// Messages in, messages out: a data stream is a sequence of messages, each +/// one JSON object timed by its pts, and a module exporting this interface +/// reads data streams and writes them. It is not one-to-one: a call may +/// answer any number of messages, or none, and may hold some back for a +/// later call. +/// +/// It may also read CLOCK pads: a video or audio argument whose frames the +/// module never sees, only their time. That is how a module with no data +/// input still acts at programme times - the host hands it `now` on every +/// call - and how one reading messages knows what time it is between them. +interface data-filter { + use types.{meta, rational}; + + /// One message: for codec "json", one UTF-8 JSON object. + record message { + pts: s64, + data: list, + } + + /// What one argument of the call is. + enum pad-kind { + /// A data stream: its messages arrive in `process`. + data, + /// A video or audio stream read for its clock alone. + clock, + } + + record pad-info { + kind: pad-kind, + /// For a data pad, its codec: "json". Empty for a clock pad. + codec: string, + /// The unit a data pad's pts, or a clock pad's `now`, is counted in. + time-base: rational, + } + + record data-filter-meta { + meta: meta, + /// The codec of each data stream the module writes, in output order: + /// ["json"] for one. + outputs: list, + /// The unit the outputs' pts are counted in. + time-base: rational, + } + + /// The messages that arrived on one data pad since the last call, in pts + /// order. A clock pad has none. + record pad-messages { + pad: u32, + messages: list, + } + + record processed { + /// The messages written, one list per output in output order, each in + /// pts order. Across calls an output's pts never decrease. + outputs: list>, + /// Rows for the run's rows output, as a sink's are. + rows: list, + } + + describe: func() -> data-filter-meta; + + /// Called once, with every argument of the call in order. + init: func(pads: list, params: string) -> result<_, string>; + + /// One batch: every message that arrived since the last call, across the + /// data pads, and `now`, the programme time in ticks of the first clock + /// pad's time base - none when the call has no clock pad. A module with + /// a clock pad is called as its clock advances, whether or not a message + /// arrived; one without is called when messages arrive. `last` marks the + /// final call, which happens once and carries whatever is left. + process: func(input: list, now: option, + last: bool) -> result; +} + +/// A codec a package implements, where ffmpeg's own encoder would stand: +/// raw frames in, coded packets out. A query names it in an output's options +/// (`video_codec => ffrwd.pyrowave.encode(bitrate => 200000000)`) and its +/// packets leave through whatever that output is. +/// +/// ffmpeg knows no codec a package brings, so the codec's identity on the +/// wire is its four-character tag: what a container writes for it, and what +/// a `decoder` names as the tags it reads. +interface encoder { + use types.{coded-stream, format, meta, packet, raw-frame, rational, stream-info}; + + record encoder-meta { + /// `meta.pixel-formats` or `meta.sample-formats` name the frames it + /// takes, most preferred first, the way a filter's do, and say its kind. + meta: meta, + /// The codec's name as a query, a message or a catalog spells it: + /// "pyrowave". + codec: string, + /// The four-character tag a container writes for it: "PYRW". Four + /// ASCII characters, exactly. + fourcc: string, + /// How many frames it takes in before its first packet leaves: 0 for an + /// intra-only codec that answers every frame in the call that brings it. + delay: u32, + /// How deep its packets reorder: how many packets a decoder holds back + /// before the first frame leaves it. 0 where decode order is + /// presentation order. + decode-delay: u32, + /// For audio, the samples every frame but the last must hold, which the + /// host cuts the stream into before it arrives; 0 takes a run of any + /// length. Always 0 for video. + frame-samples: u32, + } + + describe: func() -> encoder-meta; + + /// Called once per instance. `format` is the frames it takes: the arm is + /// its kind, and the pix-fmt or sample-fmt one `meta` named. `info` is the + /// stream being encoded; its time base is the one frames arrive in. + /// `frame-rate` is the stream's nominal frame rate for video, what a + /// bitrate-driven encoder divides its budget by; none for audio, and where + /// nothing says it. + /// + /// Answers the coded stream it writes: `codec` its own name, the time base + /// its packets are counted in (ordinarily `info.time-base`), the geometry, + /// and `extradata`, the codec's out-of-band header, which a container + /// writes before the first packet. + init: func(format: format, info: stream-info, frame-rate: option, params: string) -> result; + + /// Frames in presentation order, as many as have arrived. Answers the + /// packets that left, in decode order, which may be none while the codec + /// holds frames back. `last` marks the final call, which happens exactly + /// once and may carry no frame: every frame still held leaves on it. + /// + /// A video frame's `data` is tightly packed, exactly as ffmpeg's rawvideo + /// carries it in NUT: the planes one after another, each row directly + /// after the last with no padding between. The host hands the payload + /// through unchanged. + encode: func(frames: list, last: bool) -> result, string>; +} + +/// The mirror of `encoder`: coded packets in, raw frames out. A package +/// declares the tags it reads, and an input whose stream carries one of them +/// is decoded by it where ffmpeg would have decoded it. +interface decoder { + use types.{coded-stream, format, meta, packet, raw-frame, stream-info}; + + record decoder-meta { + /// `meta.pixel-formats` or `meta.sample-formats` name the frames it can + /// write, most preferred first, and say its kind. + meta: meta, + /// The four-character tags it reads, as its encoder writes them. + fourccs: list, + /// How many packets it takes in before its first frame leaves. + delay: u32, + } + + describe: func() -> decoder-meta; + + /// Called once per instance with the coded stream as the container + /// declared it: its tag in `codec`, its extradata, its geometry and its + /// time base. Answers the frames it writes, whose arm is the stream's kind + /// and whose pix-fmt or sample-fmt is one `meta` named. + init: func(coded: coded-stream, info: stream-info, params: string) -> result; + + /// Packets in decode order, as many as have arrived. Answers the frames + /// that left, in presentation order, in the stream's time base. `last` + /// marks the final call, which happens exactly once and may carry no + /// packet: every frame still held leaves on it. + decode: func(packets: list, last: bool) -> result, string>; +} + +world video-module { export filter; } +world meta-module { export filter; export meta-filter; } +world window-module { import window-source; export window-filter; } +world packet-sink-module { export packet-sink; } +world packet-filter-module { export packet-filter; } +world packet-source-module { export packet-source; } +world values-module { export values; } +world rows-module-host { export rows-module; } +world data-filter-module { export data-filter; } +world encoder-module { export encoder; } +world decoder-module { export decoder; } +world codec-module { export encoder; export decoder; } From ad260f8aebb065306ece6bbe640f1c571224aa06 Mon Sep 17 00:00:00 2001 From: Jon-Carlos Rivera Date: Thu, 1 Oct 2026 16:08:45 -0700 Subject: [PATCH 02/58] chore(wit): the world bump's other two moves, and the 0.18.0 bindings read their frozen copy Co-Authored-By: Claude Fable 5.1 --- cli/ffrwd/wasm.py | 1 + sidecar/runtime/src/runtime.rs | 24 ++++++++++++------------ sidecar/wasm/ffrwd.json | 2 +- 3 files changed, 14 insertions(+), 13 deletions(-) diff --git a/cli/ffrwd/wasm.py b/cli/ffrwd/wasm.py index fd07e52..5b791aa 100644 --- a/cli/ffrwd/wasm.py +++ b/cli/ffrwd/wasm.py @@ -170,6 +170,7 @@ "ffrwd:av@0.16.0", "ffrwd:av@0.17.0", "ffrwd:av@0.18.0", + "ffrwd:av@0.19.0", ) # The world a module scaffolded today is built against: the newest of those, diff --git a/sidecar/runtime/src/runtime.rs b/sidecar/runtime/src/runtime.rs index e78de44..5132f6c 100644 --- a/sidecar/runtime/src/runtime.rs +++ b/sidecar/runtime/src/runtime.rs @@ -310,11 +310,11 @@ mod world_0180 { } pub mod video { - wasmtime::component::bindgen!({ path: "../wit", world: "video-module" }); + wasmtime::component::bindgen!({ path: "../worlds/0.18.0", world: "video-module" }); } pub mod meta { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "meta-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types, @@ -324,7 +324,7 @@ mod world_0180 { } pub mod window { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "window-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types, @@ -335,52 +335,52 @@ mod world_0180 { } pub mod packet { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "packet-sink-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); } pub mod packet_source { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "packet-source-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); } pub mod values { - wasmtime::component::bindgen!({ path: "../wit", world: "values-module" }); + wasmtime::component::bindgen!({ path: "../worlds/0.18.0", world: "values-module" }); } pub mod rows { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "rows-module-host", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); } pub mod packet_filter { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "packet-filter-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); } pub mod data_filter { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "data-filter-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); } pub mod encoder { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "encoder-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); } pub mod decoder { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "decoder-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types }, }); @@ -389,7 +389,7 @@ mod world_0180 { // so one conversion serves a module whichever codec world it declared. pub mod codec { wasmtime::component::bindgen!({ - path: "../wit", + path: "../worlds/0.18.0", world: "codec-module", with: { "ffrwd:av/types": crate::runtime::world_0180::video::ffrwd::av::types, diff --git a/sidecar/wasm/ffrwd.json b/sidecar/wasm/ffrwd.json index 6c7bfed..a5de5f7 100644 --- a/sidecar/wasm/ffrwd.json +++ b/sidecar/wasm/ffrwd.json @@ -1,6 +1,6 @@ { "name": "ffrwd/wasm", - "version": "0.18.0", + "version": "0.19.0", "description": "The ffrwd:av wit world a wasm module is built against, carried as a package: this version's number is the world's. Assets only - no exports, no recipes, no module.", "license": "MIT", "files": [ From 2add1a917870654c7c8030653373f95d50204530 Mon Sep 17 00:00:00 2001 From: Jon-Carlos Rivera Date: Thu, 1 Oct 2026 17:16:38 -0700 Subject: [PATCH 03/58] feat(compiler): node declarations, shape(params, bound), ports and structural rows A LANGUAGE wasm function over a node module reads describe for its params schema and asks the sidecar for its shape per distinct (module, params, bound) call, cached. Arguments bind ports by position and name: DEFAULT NULL on any stream or rows parameter, arrays on many-ports, mixed kinds in any order, typed rows parameters as data ports with a schema, a hold port given as a port number. A rows-only return is a run-time data stream; a STRUCT return names the outputs the node makes; RETURNS source on a node with no streams is a relation in FROM. A SQL function may return a stream and its rows as one record. Row matching is structural: a reader's fields must be in the producer's port schema with the same type, and extra fields pass. One call with the same module, arguments and params is one node wherever it is written. A signature only a node can carry keeps the older world's refusal on the declaration and raises it at lowering once describe says the module is no node, so the seven tests that pinned that refusal now read it there. LIVE_LEAD is declared, with its errors.md entry; timing.py holds the delay sums and is not yet wired into compile or explain. Co-Authored-By: Claude Fable 5.1 --- cli/ffrwd/cli.py | 8 +- cli/ffrwd/compiler.py | 71 ++- cli/ffrwd/errors.py | 11 + cli/ffrwd/functions.py | 490 ++++++++++++++++- cli/ffrwd/ir.py | 42 ++ cli/ffrwd/lower.py | 974 +++++++++++++++++++++++++++++++++- cli/ffrwd/macros.py | 17 + cli/ffrwd/parser.py | 72 ++- cli/ffrwd/prompt.py | 7 + cli/ffrwd/pts.py | 2 + cli/ffrwd/shapes.py | 715 +++++++++++++++++++++++++ cli/ffrwd/split.py | 2 + cli/ffrwd/timing.py | 392 ++++++++++++++ cli/ffrwd/warnings.py | 1 + cli/ffrwd/wasm.py | 25 +- cli/tests/conftest.py | 18 + cli/tests/test_data_filter.py | 16 +- cli/tests/test_feeders.py | 11 +- cli/tests/test_lower.py | 15 +- cli/tests/test_node_world.py | 704 ++++++++++++++++++++++++ cli/tests/test_wasm.py | 3 +- docs/error-schema.json | 1 + docs/errors.md | 8 + 23 files changed, 3542 insertions(+), 63 deletions(-) create mode 100644 cli/ffrwd/shapes.py create mode 100644 cli/ffrwd/timing.py create mode 100644 cli/tests/test_node_world.py diff --git a/cli/ffrwd/cli.py b/cli/ffrwd/cli.py index af0763e..525544a 100644 --- a/cli/ffrwd/cli.py +++ b/cli/ffrwd/cli.py @@ -1489,8 +1489,12 @@ def _cmd_explain(args: argparse.Namespace, on_warning: OnWarning) -> int: payload: object = graphs[0].to_dict() if len(graphs) == 1 else [ graph.to_dict() for graph in graphs ] - if compiled.plan is not None: - payload = {"graph": payload, "plan": compiled.plan.to_dict()} + if compiled.plan is not None or compiled.timing is not None: + payload = {"graph": payload} + if compiled.plan is not None: + payload["plan"] = compiled.plan.to_dict() + if compiled.timing is not None: + payload["timing"] = compiled.timing.to_dict() print(json.dumps(payload, indent=2)) return 0 diff --git a/cli/ffrwd/compiler.py b/cli/ffrwd/compiler.py index 7ad11da..1602fc7 100644 --- a/cli/ffrwd/compiler.py +++ b/cli/ffrwd/compiler.py @@ -44,6 +44,7 @@ from dataclasses import dataclass, replace from . import registry as registry_module +from . import shapes as shapes_module from . import wasm from .emit import Emitted, emit from .errors import ErrorCode, FfrwdError @@ -64,14 +65,17 @@ check_spellable, external_filters, from_commands, + is_live, + is_live_probe, partition, ) from .project import ModelPin, PackageSet from .pts import insert_pts_resets from .split import insert_splits from .table import TableSink +from .timing import HOLD_LIMIT, Timing, check_live_leads, timing from .vars import substitute -from .warnings import OnWarning +from .warnings import FfrwdWarning, OnWarning, WarningCode from .wasm import Described __all__ = [ @@ -354,6 +358,8 @@ def _negotiable( if declared.module not in describes or declared.reads_rows_from_select: continue # a row-reading sink has no single kind; its pads are the rows' described = describes[declared.module] + if described.node: + continue # its formats are its shape's, per call and per edge if _declared_kind(declared, described) != kind: continue if described.packet_sink or described.packet_filter: @@ -477,7 +483,7 @@ def _module_shapes( return { declared.module: describes[declared.module].shape for declared in declared_stream.values() - if declared.module in describes + if declared.module in describes and not describes[declared.module].node } @@ -526,6 +532,9 @@ class Compiled: plan: ProcessPlan | None = None default_timeout: float | None = None duration: float | None = None + # How late each node module's outputs run behind the source, for a query + # that calls one (:mod:`ffrwd.timing`). + timing: Timing | None = None def _input_duration(probes: Mapping[str, ProbeResult | None]) -> float | None: @@ -655,6 +664,7 @@ def compile_all( unset: Mapping[tuple[int, int], str] | None = None, describe: wasm.Describe = wasm.describe, invoke: wasm.Invoke = wasm.invoke, + shape: shapes_module.Shape = shapes_module.shape, ) -> Compiled: """Compile SQL `text` into its commands, and the plan that runs them. @@ -668,7 +678,8 @@ def compile_all( lowering's `describes` is: a caller with no sidecar can still compile. `invoke` is the same for a VALUE-returning module: lowering runs it once per call site to fold the result, and a test hands over its own so - folding spawns nothing. + folding spawns nothing. `shape` is the same for a node module's shape, + asked once per distinct call. Raises ``FfrwdError`` — and nothing else — on every rejection. """ @@ -685,6 +696,7 @@ def compile_all( describes=describes, invoke=invoke, probe_failures=probe_failures, + shapes=shapes_module.ShapeCache(shape), ) ready = [insert_splits(insert_pts_resets(graph)) for graph in graphs] ready[0] = replace( @@ -694,13 +706,20 @@ def compile_all( for lateral in ready[0].laterals ], ) + timed = timing(ready[0], probes) + if timed is not None: + if _runs_live(res, probes, ready[0]): + check_live_leads(ready[0], probes, _module_anchors(res)) + _warn_held(timed, on_warning) budget = _default_timeout(_input_duration(probes)) span = _run_duration(ready, _probed_paths(res, probes)) stream_wasm = _stream_wasm(res) hosted = _hosted_wasm(res) leaky = any(node.filter == LEAKY for node in ready[0].nodes.values()) if not hosted and not ready[0].module_sources and not leaky: - return Compiled(graphs=ready, default_timeout=budget, duration=span) + return Compiled( + graphs=ready, default_timeout=budget, duration=span, timing=timed + ) try: plan = partition( ready[0], @@ -717,7 +736,7 @@ def compile_all( except FfrwdError as err: raise _anchored(err, stream_wasm) from err return Compiled( - graphs=ready, plan=plan, default_timeout=budget, duration=span + graphs=ready, plan=plan, default_timeout=budget, duration=span, timing=timed ) except FfrwdError: raise @@ -739,6 +758,48 @@ def compile_all( ) from err +def _runs_live( + res: Resolved, probes: Mapping[str, ProbeResult | None], graph: Graph +) -> bool: + """Whether the query reads anything that does not end: a live input, or a + node read in FROM that never finishes by itself.""" + for alias, index in res.sources.items(): + options = graph.input_options.get(alias) + if is_live(res.input_paths[index], options) or is_live_probe(probes.get(alias)): + return True + return any( + graph.node_shapes.get(name, {}).get("bounded") is False + for name in graph.node_sources.values() + ) + + +def _module_anchors(res: Resolved) -> dict[str, tuple[int, int, str]]: + """Each module path -> where the query declared it, and what it is called.""" + found: dict[str, tuple[int, int, str]] = {} + for declared in res.wasm.values(): + found.setdefault(declared.module, (declared.line, declared.col, declared.called)) + return found + + +def _warn_held(timed: Timing, on_warning: OnWarning | None) -> None: + """Say so where a stream written beside a later one holds a lot of it.""" + if on_warning is None: + return + for output in timed.outputs: + if output.held is None or output.held <= HOLD_LIMIT: + continue + on_warning( + FfrwdWarning( + WarningCode.HELD_STREAM, + "", + f"'{output.ref}' waits {round(output.holds, 3):g} s for the stream " + f"written beside it, about {output.held // (1024 * 1024)} MiB of it", + hint="the stream waits in its pipe until the later one catches up; " + "a shorter window or latency on the later path holds less", + ) + ) + + # What a run-time lateral's body is resolved with at compile time, a value of # each declared type standing in for what a message will bind. _STAND_INS = {"number": "1", "text": "x", "boolean": "true"} diff --git a/cli/ffrwd/errors.py b/cli/ffrwd/errors.py index 26a5cfe..d9eac9d 100644 --- a/cli/ffrwd/errors.py +++ b/cli/ffrwd/errors.py @@ -36,6 +36,7 @@ class ErrorCode(str, Enum): PLAYER_NOT_FOUND = "PLAYER_NOT_FOUND" # --show asked for, ffplay not on PATH RUNTIME_NOT_FOUND = "RUNTIME_NOT_FOUND" # setup nn: no ONNX Runtime pinned for this platform UNBOUNDED_LIVE_INPUT = "UNBOUNDED_LIVE_INPUT" # one-open input, uncountable paths + LIVE_LEAD = "LIVE_LEAD" # a live node fed later than the lead it needs BUFFER_OVERFLOW = "BUFFER_OVERFLOW" # a run-time edge outgrew its computed bound INPUT_NEVER_OPENED = "INPUT_NEVER_OPENED" # a run-time consumer never opened its end STARTUP_DEADLOCK = "STARTUP_DEADLOCK" # no pipe order lets every process start @@ -72,6 +73,10 @@ def __init__( self.hint = hint super().__init__(str(self)) + def __reduce__(self) -> tuple[object, ...]: + """Copied and pickled whole: a refusal can wait inside a query's tree.""" + return (_rebuilt, (self.code, self.message, self.line, self.col, self.hint)) + def to_dict(self) -> dict[str, object]: return { "line": self.line, @@ -91,3 +96,9 @@ def __str__(self) -> str: if self.hint is not None: parts.append(f" (hint: {self.hint})") return "".join(parts) + + +def _rebuilt( + code: ErrorCode, message: str, line: int | None, col: int | None, hint: str | None +) -> FfrwdError: + return FfrwdError(code, message, line=line, col=col, hint=hint) diff --git a/cli/ffrwd/functions.py b/cli/ffrwd/functions.py index adc13fc..18698d5 100644 --- a/cli/ffrwd/functions.py +++ b/cli/ffrwd/functions.py @@ -112,6 +112,8 @@ FILTER_NAMESPACE, MACRO_NAMESPACE, MERGE_CUES, + NODE_REFUSAL, + OLDER_WORLD, SINK_ALIAS, SINK_STREAMS, ModuleExport, @@ -149,6 +151,7 @@ "SINK_ALIAS", "SINK_STREAMS", "WASM_DATA", + "WASM_NODE", "WASM_STREAM_NAMES", "WASM_STREAM_TYPES", "Annotation", @@ -160,6 +163,9 @@ "WasmFunction", "expanded", "is_number_argument", + "is_port", + "spread", + "struct_entries", "package_modules", "package_signatures", "package_sources", @@ -337,6 +343,13 @@ class RuntimeLateral: **WASM_STREAM_TYPES, WASM_DATA: "data", } +# What a declaration RETURNS when its signature is one only a node module +# has: kinds mixed in any order, a stream left out, an array of streams, rows +# beside several outputs. Which it is waits on the module's describe, so the +# declaration keeps the refusal an older world's module earns (`refusal`). +WASM_NODE = "node" +# The types a node reads as an input port rather than as a value. +_PORT_TYPES = frozenset({_WASM_STREAM, _WASM_AUDIO_STREAM, WASM_DATA}) _WASM_DATA_HINT = ( "a data filter reads its streams first -- data_stream for messages, " "video_stream or audio_stream for the time alone -- then the values it is " @@ -372,6 +385,14 @@ class RuntimeLateral: "write RETURNS TABLE( , ...), one name per column the body selects" ) _VALUE_BODY_HINT = "a value-returning function's body is one SELECT of one column" +_STRUCT_BODY_HINT = ( + "a struct-returning function's body is one SELECT, one column per field, in order" +) +_STRUCT_RETURN_HINT = ( + "a function returning a struct names a stream and rows beside it: RETURNS " + "STRUCT( video_stream, STRUCT( , ...)[]), or " + "audio_stream and cue[] the same way" +) _TABLE_BODY_HINT = ( "a table-returning function's body is one SELECT, one column per RETURNS TABLE column" ) @@ -583,6 +604,20 @@ def _leading_streams(params: tuple[Parameter, ...]) -> tuple[Parameter, ...]: return params[:end] +def is_port(param: Parameter) -> bool: + """Whether a node reads `param` as an input port: a stream, or rows.""" + return param.annotation is not None or element_type(param.type) in _PORT_TYPES + + +def _written_outputs(outputs: tuple[Parameter, ...]) -> str: + """A node's RETURNS as a signature spells it.""" + if not outputs: + return WASM_SOURCE + if len(outputs) == 1 and not outputs[0].name: + return outputs[0].type + return "STRUCT(" + ", ".join(f"{o.name} {o.type}" for o in outputs) + ")" + + @dataclass(frozen=True) class WasmFunction: """One ``LANGUAGE wasm`` declaration: a module, an export, and a signature. @@ -636,12 +671,30 @@ class WasmFunction: # data_stream, whose one output is the call itself, and for every other # kind. data_fields: tuple[str, ...] = () + # What a node module makes, read off the RETURNS: one unnamed entry for a + # stream or rows, one per field of a STRUCT, none for a source, whose + # outputs its shape names. None where no node returns that. + outputs: tuple[Parameter, ...] | None = None + # What a module of an older world refuses this declaration with, for one + # only a node can carry (`returns` is WASM_NODE). + refusal: FfrwdError | None = field(default=None, compare=False) + + @property + def is_node_only(self) -> bool: + """True for a declaration only a node module can carry.""" + return self.returns == WASM_NODE + + @property + def ports(self) -> tuple[Parameter, ...]: + """Every parameter a node reads as an input port, in declared order.""" + return tuple(param for param in self.params if is_port(param)) @property def is_value(self) -> bool: """True for a function returning a compile-time value, not a stream.""" return ( self.returns not in WASM_STREAM_TYPES + and not self.is_node_only and not self.is_data_filter and not self.is_sink and not self.is_packets @@ -774,6 +827,8 @@ def stream_kind(self) -> StreamType: ) if self.is_data_filter: return "data" + if self.is_node_only: + raise ValueError(f"'{self.name}' is a node; each port has a kind of its own") written = ( self.params[0].type if (self.is_sink or self.is_packets or self.is_packet_rows) and self.params @@ -834,6 +889,8 @@ def stream_params(self) -> tuple[Parameter, ...]: """ if self.is_value or self.is_rows: return () + if self.is_node_only: + return tuple(p for p in self.ports if p.annotation is None) return _leading_streams(self.params) @property @@ -858,7 +915,7 @@ def reads_params(self) -> tuple[Parameter, ...]: together, and a parameter one skipped would be a value parameter the other kept. """ - if self.is_value or self.is_rows or self.is_packet_rows: + if self.is_value or self.is_rows or self.is_packet_rows or self.is_node_only: return () found: list[Parameter] = [] for param in self.params[self.stream_arity :]: @@ -876,7 +933,7 @@ def reads(self) -> Annotation | None: riding one: :attr:`rows_param` is that one. None for a PACKET ROWS function too, whose rows are what it hands back. """ - if self.is_value or self.is_rows or self.is_packet_rows: + if self.is_value or self.is_rows or self.is_packet_rows or self.is_node_only: return None after = self.stream_arity return self.params[after].annotation if len(self.params) > after else None @@ -910,6 +967,8 @@ def value_params(self) -> tuple[Parameter, ...]: return () if self.is_value: return self.params + if self.is_node_only: + return tuple(param for param in self.params if not is_port(param)) skip = self.stream_arity + len(self.reads_params) return self.params[skip:] @@ -925,7 +984,7 @@ def written_params(self) -> tuple[Parameter, ...]: it and any producer, so the rows reach it as arguments of their own and every one of them is written at the call. """ - if self.is_value or self.is_rows: + if self.is_value or self.is_rows or self.is_node_only: return self.params if self.is_packets: return (*self.stream_params, *self.reads_params, *self.value_params) @@ -956,6 +1015,8 @@ def written_returns(self) -> str: if self.data_fields: written = ", ".join(f"{field} {WASM_DATA}" for field in self.data_fields) return f"STRUCT({written})" + if self.is_node_only and self.outputs is not None: + return _written_outputs(self.outputs) if self.emits is None: return self.returns stream, annotation = self.stream_field, self.emits @@ -1008,6 +1069,9 @@ class _Function: used: bool = False package: str = "" package_version: str = "" + # A ``RETURNS STRUCT( , )``'s fields, one per + # column the body selects. None for every other return. + fields: tuple[Parameter, ...] | None = None @property def returns_rows(self) -> bool: @@ -1099,6 +1163,7 @@ def expanded( yield script except FfrwdError as err: raise expander.translate(err) from err + expander.settle_older_world(expanded_tree) expander.settle(expanded_tree) @@ -1225,6 +1290,21 @@ def _annotation( return Annotation(name=column, fields=tuple(declared)) +def _many_annotation( + node: exp.Expr | None, column: str, name: str, anchor: exp.Expr +) -> Annotation | None: + """The record an array of record arrays declares, ``STRUCT(...)[][]``, or None. + + A node port that takes any number of rows streams, each of that record. + """ + if not isinstance(node, exp.DataType) or node.this is not exp.DataType.Type.ARRAY: + return None + inner = node.expressions[0] if len(node.expressions) == 1 else None + if not isinstance(inner, exp.DataType) or inner.this is not exp.DataType.Type.ARRAY: + return None + return _annotation(inner, column, name, anchor) + + def _record_array( node: exp.Expr | None, column: str, name: str, anchor: exp.Expr ) -> Annotation | None: @@ -1350,14 +1430,25 @@ def _reanchor(err: FfrwdError, name: str, anchor: exp.Expr) -> FfrwdError: def _body_select( - text: str, name: str, anchor: exp.Expr, columns: tuple[Parameter, ...] | None + text: str, + name: str, + anchor: exp.Expr, + columns: tuple[Parameter, ...] | None, + fields: tuple[Parameter, ...] | None = None, ) -> exp.Select: """Parse and shape-check one body: a single SELECT of the declared width. - A value's body is one column; a table's is one per ``RETURNS TABLE`` name. + A value's body is one column; a table's is one per ``RETURNS TABLE`` name, + and a struct's one per field. """ - wanted = 1 if columns is None else len(columns) - shape = _VALUE_BODY_HINT if columns is None else _TABLE_BODY_HINT + wanted = len(fields) if fields is not None else 1 if columns is None else len(columns) + shape = ( + _STRUCT_BODY_HINT + if fields is not None + else _VALUE_BODY_HINT + if columns is None + else _TABLE_BODY_HINT + ) try: parsed = parse(text) except FfrwdError as err: @@ -1397,7 +1488,9 @@ def _body_select( if written != wanted: plural = "" if written == 1 else "s" said = ( - "and a value is one column" + f"but its RETURNS STRUCT declares {wanted} fields" + if fields is not None + else "and a value is one column" if columns is None else f"but its RETURNS TABLE declares {wanted}" ) @@ -1570,6 +1663,23 @@ def _column_defs( if kind.allow_annotation else None ) + many = ( + _many_annotation(node.args.get("kind"), written, name, anchor) + if kind.allow_annotation and annotation is None + else None + ) + if many is not None: + if default is not None and not isinstance(default, exp.Null): + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"function '{name}' gives the {kind.noun} '{written}' a DEFAULT", + anchor, + fallback=create, + hint="rows are produced by the call that fills them; DEFAULT " + "NULL, the only default they can carry, makes them optional", + ) + declared.append(Parameter(written, f"{many.written}[]", default, many)) + continue if annotation is not None: # DEFAULT NULL makes the column optional: a call over a plain # stream wires no rows in. Any other default would be a value, @@ -1983,6 +2093,131 @@ def _define_wasm( params: tuple[Parameter, ...], returns_prop: exp.ReturnsProperty, body: exp.Expr | None, +) -> WasmFunction: + """One validated ``LANGUAGE wasm`` declaration, as every world reads it. + + The kinds an older world's module has are decided here, by the signature + alone (:func:`_define_wasm_kind`). A node module's reading rides beside + every one of them (`outputs`), since only its describe says a module is a + node. A signature no older kind has, and a node can, is a node's alone: + it keeps the refusal an older module earns, for lowering to raise once + the module turns out not to be a node. + """ + node = returns_prop.this if isinstance(returns_prop.this, exp.Expr) else None + try: + declared = _define_wasm_kind(create, name, identifier, params, returns_prop, body) + except FfrwdError as refusal: + found = _node_declaration( + create, name, identifier, params, returns_prop, body, refusal + ) + if found is None: + raise + return found + if declared.is_value or declared.is_codec or declared.is_sink or declared.is_packets: + return declared + outputs = _node_outputs(node, name, identifier) + return declared if outputs is None else replace(declared, outputs=outputs) + + +def _node_outputs( + node: exp.Expr | None, name: str, identifier: exp.Identifier +) -> tuple[Parameter, ...] | None: + """What a node module makes, as a RETURNS says it, or None where no node + returns that. + + A stream or a data stream is one output, and so are rows (``STRUCT(...)[]``, + ``cue[]``); a ``STRUCT`` of those names several, one per field; a + ``source`` names none, its shape naming them. + """ + written = _type_name(node) + if written in _PORT_TYPES: + return (Parameter("", written),) + if written == WASM_SOURCE: + return () + try: + rows = _annotation(node, _ROWS_RETURN, name, identifier) + except FfrwdError: + return None + if rows is not None: + return (Parameter("", written or rows.written, annotation=rows),) + fields = _struct_fields(node) + if not fields: + return None + outputs: list[Parameter] = [] + for field_node in fields: + field_name = ( + _ident_name(field_node.this) if isinstance(field_node.this, exp.Identifier) else "" + ) + if not field_name or any(o.name == field_name for o in outputs): + return None + kind = field_node.args.get("kind") + field_type = _type_name(kind) + if field_type in _PORT_TYPES: + outputs.append(Parameter(field_name, field_type)) + continue + try: + record = _annotation(kind, field_name, name, identifier) + except FfrwdError: + return None + if record is None: + return None + outputs.append(Parameter(field_name, field_type or record.written, annotation=record)) + return tuple(outputs) + + +def _node_declaration( + create: exp.Create, + name: str, + identifier: exp.Identifier, + params: tuple[Parameter, ...], + returns_prop: exp.ReturnsProperty, + body: exp.Expr | None, + refusal: FfrwdError, +) -> WasmFunction | None: + """The declaration as a node module reads it, or None where no node can be it. + + Its parameters are ports (streams, data streams and rows, each maybe an + array, each maybe ``DEFAULT NULL``) and values (text, number, boolean, + vector), in any order; its RETURNS is one a node makes, and names at + least one output, since a node that reads ports and makes nothing is a + sink. + """ + if returns_prop.args.get("is_table"): + return None + try: + module, export, _ = _module_export(body, name, identifier, create) + except FfrwdError: + return None + node = returns_prop.this if isinstance(returns_prop.this, exp.Expr) else None + outputs = _node_outputs(node, name, identifier) + if not outputs: + return None + if not any(is_port(param) for param in params): + return None + for param in params: + if not is_port(param) and param.type not in _ANNOTATION_FIELD_TYPES: + return None + line, col = _pos(identifier, create) + return WasmFunction( + name=name, + module=module, + export=export, + params=params, + returns=WASM_NODE, + line=line, + col=col, + outputs=outputs, + refusal=refusal, + ) + + +def _define_wasm_kind( + create: exp.Create, + name: str, + identifier: exp.Identifier, + params: tuple[Parameter, ...], + returns_prop: exp.ReturnsProperty, + body: exp.Expr | None, ) -> WasmFunction: """One validated ``LANGUAGE wasm`` declaration: a stream filter or a value. @@ -2509,14 +2744,18 @@ def _define(create: exp.Create, *, packaged: bool = False) -> _Function | WasmFu hint="a module is hosted, not inlined; say LANGUAGE wasm", ) columns: tuple[Parameter, ...] | None = None + fields: tuple[Parameter, ...] | None = None if returns_prop.args.get("is_table"): columns = _table_columns(returns_prop, name, identifier, create) returns = "TABLE(" + ", ".join(f"{c.name} {c.type}" for c in columns) + ")" else: node = returns_prop.this if isinstance(returns_prop.this, exp.Expr) else None - returns = _checked_type(node, name, identifier) + fields = _sql_struct_return(node, name, identifier) + returns = _written_outputs(fields) if fields else _checked_type(node, name, identifier) - body = _body_select(_body_text(create, name, identifier), name, identifier, columns) + body = _body_select( + _body_text(create, name, identifier), name, identifier, columns, fields + ) aliases = _body_aliases(body, name, params, identifier) _check_body_scope(body, name, params, aliases, identifier) return _Function( @@ -2528,9 +2767,34 @@ def _define(create: exp.Create, *, packaged: bool = False) -> _Function | WasmFu aliases=aliases, position=0, columns=columns, + fields=fields, ) +def _sql_struct_return( + node: exp.Expr | None, name: str, identifier: exp.Identifier +) -> tuple[Parameter, ...] | None: + """A sql function's ``RETURNS STRUCT( , )``. + + The fields a call over it is read as: the stream and the rows beside it, + which a node reading both takes as two arguments. None for a RETURNS that + is not a struct at all. + """ + fields = _struct_fields(node) + if fields is None: + return None + outputs = _node_outputs(node, name, identifier) + streams = [o for o in outputs or () if o.annotation is None] + if outputs is None or len(outputs) != 2 or len(streams) != 1 or outputs[0] != streams[0]: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"function '{name}' returns a struct that is not a stream and rows", + identifier, + hint=_STRUCT_RETURN_HINT, + ) + return outputs + + def _in_lib( err: FfrwdError, name: str, path: Path, anchor: exp.Expr | None ) -> FfrwdError: @@ -3568,7 +3832,13 @@ def _wasm_argument_kind(call: exp.Anonymous, wasm: Mapping[str, WasmFunction] | if _call_name(call) in _VECTOR_BUILTIN_ARITY: return "number" declared = wasm.get(_call_name(call)) if wasm is not None else None - return _declared_kind(declared.returns) if declared is not None else "stream" + if declared is None: + return "stream" + if declared.is_node_only: + outputs = declared.outputs or () + one = len(outputs) == 1 and not outputs[0].name + return _declared_kind(outputs[0].type) if one else "stream" + return _declared_kind(declared.returns) def _positional_and_named( @@ -3726,6 +3996,8 @@ def _struct_fields_of(declared: WasmFunction) -> tuple[str, ...]: return declared.data_fields if declared.emits is not None: return (declared.stream_field, declared.emits.name) + if declared.is_node_only and declared.outputs: + return tuple(output.name for output in declared.outputs if output.name) return () @@ -3808,6 +4080,73 @@ def _declared_kind(declared: str) -> str: return "stream" if TYPES[element].kind != "scalar" else declared +def _struct_of(fields: tuple[Parameter, ...], projections: Sequence[exp.Expr]) -> exp.Struct: + """A struct-returning call's value: each field, the column the body selects for it.""" + entries: list[exp.Expr] = [] + for field_param, projection in zip(fields, projections): + value = projection.this if isinstance(projection, exp.Alias) else projection + assert isinstance(value, exp.Expr) + entries.append( + exp.PropertyEQ(this=exp.to_identifier(field_param.name), expression=value) + ) + return exp.Struct(expressions=entries) + + +def struct_entries(node: exp.Expr) -> list[tuple[str, exp.Expr]] | None: + """A written ``STRUCT( AS , ...)`` as its fields in order, or None.""" + node = _unparen(node) + if not isinstance(node, exp.Struct): + return None + entries: list[tuple[str, exp.Expr]] = [] + for entry in node.expressions: + if not isinstance(entry, exp.PropertyEQ) or not isinstance(entry.expression, exp.Expr): + return None + entries.append((_ident_name(entry.this), entry.expression)) + return entries + + +def _struct_star(projection: exp.Expr) -> list[tuple[str, exp.Expr]] | None: + """``().*``: its fields, each read once, or None.""" + dot = _unparen(projection) + if not isinstance(dot, exp.Dot) or not isinstance(dot.expression, exp.Star): + return None + return struct_entries(dot.this) if isinstance(dot.this, exp.Expr) else None + + +def spread(arguments: Sequence[exp.Expr]) -> list[exp.Expr]: + """Positional arguments with each struct a call became read as its fields. + + A call over a stream and the rows beside it hands a node both, the way a + call over a two-part result always has: ``ring(spotted(v))`` is + ``ring(, )``. + """ + spread_out: list[exp.Expr] = [] + for argument in arguments: + entries = None if isinstance(argument, exp.Kwarg) else struct_entries(argument) + if entries is None: + spread_out.append(argument) + continue + spread_out.extend(value for _, value in entries) + return spread_out + + +def _struct_field_read(value: exp.Expr) -> tuple[exp.Expr, exp.Expr] | None: + """``.`` around a struct a call became: the read, and the field.""" + inner: exp.Expr = value + parent = value.parent + while isinstance(parent, exp.Paren): + inner, parent = parent, parent.parent + if not isinstance(parent, exp.Dot) or parent.this is not inner: + return None + field_node = parent.args.get("expression") + entries = struct_entries(value) + if entries is None or not isinstance(field_node, exp.Identifier): + return None + wanted = _ident_name(field_node) + found = next((entry for name, entry in entries if name == wanted), None) + return None if found is None else (parent, found) + + def _record_rows(argument: exp.Expr) -> list[exp.Expr] | None: """The rows a ``STRUCT(...)[]`` argument writes, or None for anything else.""" node = _unparen(argument) @@ -4378,6 +4717,11 @@ def _expand_within( site.node.replace(replacement) if site.node is root: root = replacement + read = _struct_field_read(replacement) + if read is not None: + if read[0] is root: + root = read[1] + read[0].replace(read[1]) def _next_call(self, root: exp.Expr, position: int) -> _CallSite | _WasmSite | None: """The first call to a defined function in `root`'s own query, if any. @@ -4499,6 +4843,11 @@ def _expand_data_stars(self, statement: exp.Expr) -> None: projections: list[exp.Expr] = [] expanded = False for projection in select.expressions: + entries = _struct_star(projection) + if entries is not None: + expanded = True + projections.extend(exp.alias_(value, name) for name, value in entries) + continue call = self._data_star_call(projection) if call is None: projections.append(projection) @@ -4666,8 +5015,9 @@ def _check_wasm_calls(self, statement: exp.Expr, position: int) -> None: if declared.is_packet_rows: # As above: a Table-position call is skipped before this # loop sees it, so anything reaching here is written where - # a stream or a value belongs. - raise _error( + # a stream or a value belongs. A node answers it with a data + # stream, so the refusal waits for the module's describe. + node.meta[OLDER_WORLD] = _error( ErrorCode.UNSUPPORTED_SQL, f"function '{declared.name}' returns rows read off a " "stream's packets, and this call is not in FROM", @@ -4678,9 +5028,111 @@ def _check_wasm_calls(self, statement: exp.Expr, position: int) -> None: arguments = [ argument for argument in node.expressions if isinstance(argument, exp.Expr) ] - self._check_wasm_arguments(declared, node, arguments) + if declared.is_node_only or OLDER_WORLD in node.meta: + try: + self._check_node_arguments(declared, node, arguments) + except FfrwdError as refusal: + node.meta[NODE_REFUSAL] = refusal + else: + self._check_any_world_arguments(declared, node, arguments) self.wasm_used.add(declared.name) + def _check_any_world_arguments( + self, declared: WasmFunction, call: exp.Anonymous, arguments: list[exp.Expr] + ) -> None: + """A call's arguments as an older world's module reads them, or as a node does. + + A call only a node reads keeps the older refusal for lowering, which + raises it once the module's describe says it is not a node. + """ + try: + self._check_wasm_arguments(declared, call, arguments) + except FfrwdError as refusal: + if declared.outputs is None: + raise + try: + self._check_node_arguments(declared, call, arguments) + except FfrwdError: + raise refusal from None + call.meta[OLDER_WORLD] = refusal + + def _check_node_arguments( + self, declared: WasmFunction, call: exp.Anonymous, arguments: list[exp.Expr] + ) -> None: + """A call to a node against its signature: ports and values alike. + + The positionals fill the parameters in declared order, whatever each + is, and a name fills the one it names; each parameter once, and one + with no DEFAULT always. A port takes a stream, rows, an array of + either or NULL, and a number where a held input may be given as a port; + which of those fits is the module's shape to say, in lowering. + """ + positional, named = _positional_and_named(spread(arguments)) + params = declared.params + plural = "" if len(positional) == 1 else "s" + if len(positional) > len(params): + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() got {len(positional)} argument{plural}, but it " + f"declares {len(params)}", + call, + hint=declared.signature, + ) + filled: dict[str, exp.Expr] = { + param.name: argument for param, argument in zip(params, positional) + } + for name, value in named: + param = next((p for p in params if p.name == name), None) + if param is None: + listed = ", ".join(f"'{p.name}'" for p in params) + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() has no parameter '{name}'", + value, + fallback=call, + hint=f"its parameters are {listed}", + ) + if name in filled: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() gets '{name}' twice: positionally and by name", + value, + fallback=call, + hint=f"write '{name}' once: {declared.signature}", + ) + filled[name] = value + unfilled = next( + (p for p in params if p.default is None and p.name not in filled), None + ) + if unfilled is not None: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() does not write '{unfilled.name}', which has no " + "DEFAULT", + call, + hint=declared.signature, + ) + for param in params: + argument = filled.get(param.name) + if argument is None: + continue + if not is_port(param): + self._check_wasm_argument(declared, call, param, argument) + continue + if _call_name(argument) == _INPUT: + self._check_wasm_argument(declared, call, param, argument) + written = _argument_kind(argument, self.wasm) + if written in (None, "stream", "number", _RECORD_KIND) or _is_null(argument): + continue + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() takes {param.type} as its '{param.name}' " + f"argument, got {_KIND_NAMES.get(written, written)}", + argument, + fallback=call, + hint=declared.signature, + ) + def _reject_wasm_row_source(self, item: exp.Table) -> None: """A wasm function in FROM: refused unless it is a source, which is exactly a table. @@ -5096,6 +5548,8 @@ def _expand_call( with self._scoped(function.identity): self._expand_within(body, host, position, (*stack, function.qualified)) _splice(host, body) + if function.fields is not None: + return _struct_of(function.fields, body.expressions) projection: exp.Expr = body.expressions[0] inner = projection.this if isinstance(projection, exp.Alias) else None return inner if isinstance(inner, exp.Expr) else projection @@ -5666,6 +6120,14 @@ def translate(self, err: FfrwdError) -> FfrwdError: hint=err.hint, ) + def settle_older_world(self, script: exp.Expr) -> None: + """Say every refusal kept for lowering at the call site, as one raised now is.""" + for node in script.walk(): + for key in (OLDER_WORLD, NODE_REFUSAL): + refusal = node.meta.get(key) + if isinstance(refusal, FfrwdError): + node.meta[key] = self.translate(refusal) + def settle(self, script: exp.Expr) -> None: """Flatten every stamped position onto its call site. diff --git a/cli/ffrwd/ir.py b/cli/ffrwd/ir.py index c45eb63..9907725 100644 --- a/cli/ffrwd/ir.py +++ b/cli/ffrwd/ir.py @@ -79,6 +79,11 @@ # the same way, and its one argument is the gap rows still merge across. ROWMERGE = "rowmerge" MAX_DISTANCE = "max_distance" +# The same node over rows written once per tick, each carrying the pts its +# span began at as `start_t`: runs of one `start_t` become one span, cut at +# `max_span` seconds, which is also how late a span may leave. +MAX_SPAN = "max_span" +MERGE_SPANS = "merge_spans" # The node that drops the pictures of a live stream that fall too far behind # the wall clock. Hosted the same way; its arguments are how late, in seconds @@ -173,6 +178,10 @@ class Node: # carries rows and no frames. Only a ROWS MODULE has any: it reads rows # and writes rows, so it has no `inputs` and no `outputs` at all. rows_inputs: list[str] = field(default_factory=list) + # A node module's port each input binds, one per input, and the port + # each output is, one per output. Empty for every other node. + ports: list[str] = field(default_factory=list) + out_ports: list[str] = field(default_factory=list) @property def rows_only(self) -> bool: @@ -191,6 +200,10 @@ def to_dict(self) -> dict[str, object]: written["reads_annotations"] = True if self.rows_inputs: written["rows_inputs"] = list(self.rows_inputs) + if self.ports: + written["ports"] = list(self.ports) + if self.out_ports: + written["out_ports"] = list(self.out_ports) return written @classmethod @@ -204,9 +217,13 @@ def from_dict(cls, d: dict[str, object]) -> Node: assert isinstance(node_filter, str) assert isinstance(node_args, dict) raw_rows_inputs = d.get("rows_inputs") or [] + raw_ports = d.get("ports") or [] + raw_out_ports = d.get("out_ports") or [] assert isinstance(node_inputs, list) assert isinstance(node_outputs, list) assert isinstance(raw_rows_inputs, list) + assert isinstance(raw_ports, list) + assert isinstance(raw_out_ports, list) return cls( id=node_id, filter=node_filter, @@ -215,6 +232,8 @@ def from_dict(cls, d: dict[str, object]) -> Node: outputs=[_parse_stream_type(x) for x in node_outputs], reads_annotations=bool(d.get("reads_annotations", False)), rows_inputs=[str(x) for x in raw_rows_inputs], + ports=[str(x) for x in raw_ports], + out_ports=[str(x) for x in raw_out_ports], ) @@ -916,6 +935,12 @@ class Graph: # Each run-time lateral: its data stream is a sink of its own, written to # the loopback port `tap` names, and its instances are started per message. laterals: list[Lateral] = field(default_factory=list) + # Each node module's node id -> the shape its call was given, as the + # sidecar wrote it: its ports, their pairings and its clock. + node_shapes: dict[str, dict[str, object]] = field(default_factory=dict) + # A node read in FROM: its alias -> the node making the alias's streams, + # which every ``src:`` ref has been rewritten to a pad of. + node_sources: dict[str, str] = field(default_factory=dict) @property def outputs(self) -> list[Output]: @@ -995,6 +1020,10 @@ def to_dict(self) -> dict[str, object]: } if self.laterals: d["laterals"] = [lateral.to_dict() for lateral in self.laterals] + if self.node_shapes: + d["node_shapes"] = {name: dict(shape) for name, shape in self.node_shapes.items()} + if self.node_sources: + d["node_sources"] = dict(self.node_sources) return d @classmethod @@ -1130,6 +1159,17 @@ def from_dict(cls, d: dict[str, object]) -> Graph: assert isinstance(raw_laterals, list) laterals = [Lateral.from_dict(one) for one in raw_laterals if isinstance(one, dict)] + raw_node_shapes = d.get("node_shapes") or {} + assert isinstance(raw_node_shapes, dict) + node_shapes = { + str(name): dict(shape) + for name, shape in raw_node_shapes.items() + if isinstance(shape, dict) + } + raw_node_sources = d.get("node_sources") or {} + assert isinstance(raw_node_sources, dict) + node_sources = {str(alias): str(name) for alias, name in raw_node_sources.items()} + return cls( input_paths=[str(p) for p in raw_inputs], sources={str(k): int(v) for k, v in raw_sources.items()}, @@ -1151,6 +1191,8 @@ def from_dict(cls, d: dict[str, object]) -> Graph: dropped_aliases=dropped_aliases, feeders=feeders, laterals=laterals, + node_shapes=node_shapes, + node_sources=node_sources, ) diff --git a/cli/ffrwd/lower.py b/cli/ffrwd/lower.py index 308fc29..624d465 100644 --- a/cli/ffrwd/lower.py +++ b/cli/ffrwd/lower.py @@ -303,7 +303,9 @@ RuntimeLateral, WasmFunction, is_number_argument, + is_port, named_annotation_parameter, + spread, wasm_named_parameter, ) from ffrwd.inputs import render_options, rendered_options @@ -312,6 +314,7 @@ FEEDER_HOST, LEAKY, MAX_DISTANCE, + MERGE_SPANS, NO_CHAPTERS, NO_METADATA, PIPE, @@ -356,6 +359,8 @@ MACRO_NAMESPACE, MAP_COLUMNS, MERGE_CUES, + NODE_REFUSAL, + OLDER_WORLD, ROW_MERGE, ROW_PREDICATE, ROW_STREAM, @@ -420,6 +425,7 @@ from ffrwd.probe import probe as probe_one_path from ffrwd.processes import CLOCK_SIZE, COPY_CODEC, NUT, RAWVIDEO, ref_type from ffrwd.registry import DynamicFilter, FilterOption, Registry, SourceFilter +from ffrwd.shapes import InputPort, NodeShape, Shape, ShapeCache, row_mismatch from ffrwd.sink import ( CODEC_PARAMS_FLAGS, COLOR_OPTIONS, @@ -483,6 +489,7 @@ FFMPEG_SAMPLE_FMTS, PACKET_FILTER_WORLD, PACKET_SOURCE_WORLD, + SAMPLE_FMT_CODECS, WIRE_AUDIO_CODECS, WIRE_PIX_FMTS, WIRE_VIDEO_CODECS, @@ -495,6 +502,7 @@ PacketRead, ProbeSource, ReadPackets, + _grant_args, audio_encoder_codec, catalog_as_probe, encoder_codec, @@ -3859,6 +3867,122 @@ class _Env: # ExpandCtx +@dataclass(frozen=True) +class _NodeInstance: + """One node module's instance in the graph, and the shape it was given.""" + + ref: str + shape: NodeShape + + +# The kind of port each node parameter is, and the type a declaration spells +# one of each kind with. +_PORT_KINDS: Mapping[str, StreamType] = { + "video_stream": "video", + "audio_stream": "audio", + WASM_DATA: "data", +} +_PORT_TYPE_NAMES: Mapping[str, str] = { + "video": "video_stream", + "audio": "audio_stream", + "data": "data_stream or STRUCT(...)[]", + "packets": "packets", +} + +# What a JSON schema calls each type an annotation field may be. +_SCHEMA_TYPES: Mapping[str, dict[str, object]] = { + "number": {"type": "number"}, + "text": {"type": "string"}, + "boolean": {"type": "boolean"}, + "vector": {"type": "array", "items": {"type": "number"}}, +} + + +def _node_source_probe(shape: NodeShape) -> ProbeResult: + """A node source's outputs as a :class:`~ffrwd.probe.ProbeResult`. + + One stream per output, counted per kind the way ffprobe counts a file's. + A rate clock is the pictures' rate. Outputs naming a relation row group + into one rendition per row, which carries that row's name, bandwidth, + codecs and language; a source naming none is one row, as a file is. + """ + fps = f"{shape.clock.rate[0]}/{shape.clock.rate[1]}" if shape.clock.rate else None + counted: dict[str, int] = {} + streams: list[StreamMeta] = [] + by_row: dict[int, list[StreamMeta]] = {} + for output in shape.outputs: + index = counted.get(output.kind, 0) + counted[output.kind] = index + 1 + found = output.format + video = found is not None and found.kind == "video" + audio = found is not None and found.kind == "audio" + stream = StreamMeta( + type=cast(StreamType, output.kind), + index=index, + metadata={}, + width=found.width if found is not None and video else None, + height=found.height if found is not None and video else None, + fps=fps if output.kind == "video" else None, + sample_rate=found.sample_rate if found is not None and audio else None, + codec=_NODE_SOURCE_CODECS.get( + (found.sample_format if found is not None and audio else None) or output.kind + ), + channels=found.channels if found is not None and audio else None, + ) + streams.append(stream) + if output.row is not None: + by_row.setdefault(output.row, []).append(stream) + renditions: list[RenditionMeta] = [] + for row, members in sorted(by_row.items()): + said = shape.relation[row] if row < len(shape.relation) else {} + picture = next((s for s in members if s.type == "video"), None) + bandwidth = said.get("bandwidth") + renditions.append( + RenditionMeta( + streams=members, + bandwidth=bandwidth if isinstance(bandwidth, int) else None, + width=picture.width if picture is not None else None, + height=picture.height if picture is not None else None, + codecs=_text_or_none(said.get("codecs")), + name=_text_or_none(said.get("name")), + language=_text_or_none(said.get("language")), + program_id=None, + ) + ) + return ProbeResult( + streams=streams, format_name="node", renditions=renditions, live=not shape.bounded + ) + + +# What a node source's stream is on the wire, by its kind or, for sound, its +# sample format: what the probe of a file would have said its codec is. +_NODE_SOURCE_CODECS: Mapping[str, str] = { + "video": RAWVIDEO, + "audio": SAMPLE_FMT_CODECS["f32"], + "data": JSON_CODEC, + **SAMPLE_FMT_CODECS, +} + + +def _text_or_none(value: object) -> str | None: + return value if isinstance(value, str) else None + + +def _port_kind(param: Parameter) -> StreamType: + """The kind of stream a node reads or writes for `param`: rows are data.""" + if param.annotation is not None: + return "data" + return _PORT_KINDS.get(element_type(param.type), "data") + + +def _record_schema(record: Annotation) -> dict[str, object]: + """A declared record as the JSON schema of one row of it.""" + return { + "type": "object", + "properties": {f.name: dict(_SCHEMA_TYPES[f.type]) for f in record.fields}, + } + + class _NodeFactory: """Mints ``n1, n2, ...`` node ids into a graph, in creation order. @@ -3916,8 +4040,27 @@ def __init__( probe_source: ProbeSource = wasm_probe_source, probe_path: ProbePath = probe_one_path, read_packets: ReadPackets = wasm_read_packet_rows, + shapes: Shape | None = None, ) -> None: self.res = res + # Asks a node module for its shape, once per distinct call. + self.shapes: Shape = shapes if shapes is not None else ShapeCache() + # (call text, id(env)) -> the instances a node call lowered to, and + # whether it broadcast: every read of one call in one branch is one. + self._node_calls: dict[ + tuple[str, int], tuple[tuple[_NodeInstance, ...], bool] + ] = {} + # (module, export, bound inputs, params) -> the instance: one call is + # one node wherever the query writes it. + self._node_refs: dict[ + tuple[str, str, tuple[tuple[str, FrameRef], ...], str], _NodeInstance + ] = {} + # A node's data pad -> the function writing it and the schema of its + # rows, which a reader's port is matched against. + self._data_schemas: dict[FrameRef, tuple[str, Mapping[str, object]]] = {} + # A node's rows pad -> its declaration, its record and the call, for + # the track or rows file a COPY selecting it writes. + self._node_rows: dict[FrameRef, tuple[WasmFunction, Annotation, exp.Anonymous]] = {} self.probes = probes # Why an alias in `probes` maps to None, when there is a specific # answer -- unset (or no answer for this alias) reads the same as an @@ -4281,6 +4424,7 @@ def run(self) -> Graph: is None). When there are sinks they are just a mirror of ``sinks[0]`` and walking them again would lower the first group twice. """ + self._check_declared_worlds() self._lower_ctes() if self.res.sinks: self.graph.sinks = self._lower_sinks() @@ -4305,6 +4449,7 @@ def run(self) -> Graph: attachments=list(self.attachments), ) ] + self._place_node_sources() self._check_feeder_groups() self._place_feeders() self._place_laterals() @@ -5093,7 +5238,9 @@ def _rows_file(self, raw: RawSink) -> str: ] sole = _unwrap(written[0]) if len(written) == 1 else None if sole is not None and ( - self._rows_projection(sole) is not None or self._rows_call(sole) is not None + self._rows_projection(sole) is not None + or self._rows_call(sole) is not None + or self._node_rows_column(sole) ): return path raise _error( @@ -7357,6 +7504,8 @@ def _lower_branch(self, select: exp.Select, *, tags: _TagScope) -> list[_Column] written, self.rows_file = self.rows_file, document or self.rows_file try: value = self._branch_value(projection, env, select) + if tags == "sink": + value = self._node_rows_at_sink(value, projection, env, select) finally: self.rows_file = written column = _Column( @@ -8616,6 +8765,10 @@ def _add_module_source( """ call = _call_parts(inner) assert call is not None # inner is exp.Anonymous; _call_parts always answers + node_module = self._node_module(declared) + if node_module is not None: + self._add_node_source(alias, inner, declared, node_module, call, join, env, select) + return described = self._described_source(declared, inner, select) if not described.source: self._add_url_source( @@ -8661,6 +8814,87 @@ def _add_module_source( bounded=catalog.bounded, ) + def _add_node_source( + self, + alias: str, + inner: exp.Anonymous, + declared: WasmFunction, + described: Described, + call: _Call, + join: RawRowJoin | None, + env: _Env, + select: exp.Select, + ) -> None: + """``FROM () alias``: a node reading nothing, bound as a + probed input is. + + Its shape for the call's params names its outputs, and each is one of + the alias's streams, ``s.video[1]`` the first picture; its relation + rows are the alias's renditions. The node making them is in the graph + like any other, and every ``src:`` ref is a pad of it once + lowering ends (:meth:`_place_node_sources`). A source that never ends + reads as a live input does. + """ + params = self._wasm_params(declared, described, call, inner, select, env, {}, first=0) + shape = self._node_shape(declared, described, params, [], inner, select) + required = next((port for port in shape.inputs if port.required), None) + if required is not None: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() is read in FROM, and the module " + f"'{declared.module}' requires an input '{required.name}'", + inner, + fallback=select, + hint="a source reads nothing; a node reading a stream is called " + "over that stream in the SELECT list", + ) + packets = next((o for o in shape.outputs if o.kind == "packets"), None) + if packets is not None: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"the module '{declared.module}' writes packets on its " + f"'{packets.name}' output, which a node source cannot hand on yet", + inner, + fallback=select, + hint="a node source writing coded packets is not wired yet", + ) + result = _node_source_probe(shape) + self.probes[alias] = result + env.bindings[alias] = _InputBinding(alias=alias) + self._bind_renditions(alias, join, env, select) + kinds = [cast(StreamType, output.kind) for output in shape.outputs] + ref = self.ctx.node(declared.module, params, [], kinds) + self.graph.nodes[ref].out_ports = [output.name for output in shape.outputs] + self.graph.node_shapes[ref] = dict(shape.raw) + self.graph.node_sources[alias] = ref + + def _place_node_sources(self) -> None: + """Every ``src:`` ref of a node read in FROM, as that node's pad. + + The alias binds as a probed input does, so what reads it reads a + source ref; the node is what makes the stream, and is what the ref + now names. + """ + if not self.graph.node_sources: + return + pads: dict[FrameRef, FrameRef] = {} + for alias, name in self.graph.node_sources.items(): + made = self.graph.nodes[name] + counted: dict[StreamType, int] = {} + for pad, kind in enumerate(made.outputs): + index = counted.get(kind, 0) + counted[kind] = index + 1 + ref = f"src:{alias}:{_TYPE_MARKERS[kind]}:{index}" + pads[ref] = name if len(made.outputs) == 1 else f"{name}:{pad}" + for node in self.graph.nodes.values(): + node.inputs = [pads.get(ref, ref) for ref in node.inputs] + for unit in self.graph.sinks: + for output in unit.outputs: + output.ref = pads.get(output.ref, output.ref) + self.graph.rows_sinks = { + pads.get(ref, ref): sink for ref, sink in self.graph.rows_sinks.items() + } + # -- FROM () alias over a VALUES module: a URL table ---- def _add_url_source( @@ -11289,7 +11523,9 @@ def _collect_trims( hint=_TIME_HINT, ) column, low, high, strict = parsed - if strict: + table_node = column.args.get("table") + ticks = table_node is not None and _fold(table_node) in self.graph.node_sources + if strict and not ticks: raise _error( ErrorCode.UNSUPPORTED_SQL, "strict inequalities are not supported", @@ -11297,7 +11533,6 @@ def _collect_trims( fallback=where, hint=_TIME_HINT, ) - table_node = column.args.get("table") if table_node is None: raise _error( ErrorCode.UNSUPPORTED_SQL, @@ -11307,6 +11542,15 @@ def _collect_trims( hint=_TIME_HINT, ) alias = _fold(table_node) + if ticks and low is not None: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"'{alias}' is a node read in FROM, so it starts at 0 and " + f"'WHERE {alias}.t' can only end it", + conjunct, + fallback=where, + hint=f"write an end alone, e.g. WHERE {alias}.t < 10", + ) if _fold(column.this) != TIME_COLUMN: raise _error( ErrorCode.UNSUPPORTED_SQL, @@ -11660,6 +11904,12 @@ def _check_declared_stream( def _lower_stream_expr(self, node: exp.Expr, env: _Env, select: exp.Select) -> _Value: node = _unwrap(node) + self._check_older_world(node) + # A node's outputs are its own, whatever the declaration would mean to + # a module of an older world. + lowered = self._lower_node_expr(node, env, select) + if lowered is not None: + return lowered # An array of cue records IS a subtitle track, so it lowers here, in a # stream position, and not as an output column the way `chapters` does. cues = self._lower_cue_array(node, env, select) @@ -14546,6 +14796,8 @@ def _wasm_params( first: int, params_schema: Mapping[str, object] | None = None, ports: _Ports | None = None, + written: Mapping[str, exp.Expr] | None = None, + value_params: Sequence[Parameter] | None = None, ) -> dict[str, object]: """The value arguments as the module's own parameters, schema-checked. @@ -14561,18 +14813,23 @@ def _wasm_params( whose parameters belong to one FUNCTION of the module rather than to the module's single export. `ports` is what the call's feeders settled: a port the host picked is written here, whatever the - declaration's DEFAULT says. + declaration's DEFAULT says. `written` and `value_params` are a node + call's, whose values stand anywhere among its ports. """ schema_source = ( described.params_schema if params_schema is None else params_schema ) properties = schema_source.get("properties") known = properties if isinstance(properties, dict) else {} - self._check_value_param_schemas(declared, known, node, select) - written = self._wasm_written(declared, call, node, select, first=first, ports=ports) + values = declared.value_params if value_params is None else tuple(value_params) + self._check_value_param_schemas(declared, known, node, select, values) + if written is None: + written = self._wasm_written( + declared, call, node, select, first=first, ports=ports + ) owned = ports.owned if ports is not None else {} params: dict[str, object] = {} - for param in declared.value_params: + for param in values: argument = written.get(param.name) anchor = argument if argument is not None else node value: RowValue @@ -14705,6 +14962,7 @@ def _check_value_param_schemas( known: Mapping[str, object], node: exp.Expr, select: exp.Select, + values: Sequence[Parameter] | None = None, ) -> None: """Every value parameter the declaration names, against what SQL can write. @@ -14715,7 +14973,7 @@ def _check_value_param_schemas( here beats the module failing on a key it never got. A schema naming nothing judgeable is left alone, as it is at the argument. """ - for param in declared.value_params: + for param in declared.value_params if values is None else values: kinds = _schema_types(known.get(param.name)) if not kinds or any(kind in _JSON_TYPES for kind in kinds): continue @@ -15309,6 +15567,10 @@ def _lower_wasm_call( :meth:`_written_annotation` lowers the producer once and hands back the pad the module reads. """ + if self._node_module(declared) is not None: + lowered = self._lower_node_expr(node, env, select) + assert lowered is not None # a call to a node module + return lowered described = self._described(declared, node, select) if declared.is_packets: return self._lower_packets_call( @@ -15476,6 +15738,685 @@ def build(values: list[object], element: int) -> FrameRef: ) return lowered + # -- nodes -- + + def _node_module(self, declared: WasmFunction) -> Described | None: + """The describe of `declared`'s module, where that module is a node.""" + described = self.describes.get(declared.module) + return described if described is not None and described.node else None + + def _check_declared_worlds(self) -> None: + """A declaration only a node can carry, naming a module that is not one.""" + for declared in self.res.wasm.values(): + if declared.refusal is not None and self._node_module(declared) is None: + raise declared.refusal + + def _node_call( + self, node: exp.Expr + ) -> tuple[exp.Anonymous, WasmFunction, str | None] | None: + """```` or ``.`` over a call to a node module: the + call, its declaration and the field read. None for anything else.""" + node = _unwrap(node) + field_name: str | None = None + base: exp.Expr = node + if isinstance(node, exp.Dot): + identifier = node.args.get("expression") + inner = _unwrap(node.this) if isinstance(node.this, exp.Expr) else None + if not isinstance(identifier, exp.Identifier) or not isinstance( + inner, exp.Anonymous + ): + return None + field_name, base = _fold(identifier), inner + if not isinstance(base, exp.Anonymous): + return None + declared = self.res.wasm.get(str(base.name).lower()) + if declared is None or self._node_module(declared) is None: + return None + return base, declared, field_name + + def _check_older_world(self, node: exp.Expr) -> None: + """What resolve kept for a module of an older world, raised for one. + + A call or field read only a node makes sense of carries the refusal + an older module earns; a node module reads it as written. + """ + refusal = _unwrap(node).meta.get(OLDER_WORLD) + if isinstance(refusal, FfrwdError) and self._node_call(node) is None: + raise refusal + + def _node_output( + self, + declared: WasmFunction, + field_name: str | None, + node: exp.Expr, + select: exp.Select, + ) -> Parameter: + """The declared output a call read whole, or the field it read, names.""" + outputs = declared.outputs or () + if field_name is not None: + found = next((o for o in outputs if o.name == field_name), None) + if found is not None: + return found + listed = ", ".join(f"'{o.name}'" for o in outputs if o.name) + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() returns no field '{field_name}'", + node, + fallback=select, + hint=f"it returns {listed}" if listed else declared.signature, + ) + if len(outputs) == 1 and not outputs[0].name: + return outputs[0] + fields = ", ".join(f".{o.name}" for o in outputs if o.name) + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() returns {declared.written_returns}, and a struct " + "is not a stream", + node, + fallback=select, + hint=f"read one output off the call, {declared.name}(...){fields.split(',')[0]}: " + f"it writes {fields}" + if fields + else declared.signature, + ) + + def _node_output_kind(self, declared: WasmFunction, field_name: str | None) -> str: + """The kind of what a node call or field read is, for the classifier.""" + outputs = declared.outputs or () + if field_name is not None: + found = next((o for o in outputs if o.name == field_name), None) + else: + found = outputs[0] if len(outputs) == 1 and not outputs[0].name else None + return _UNSUPPORTED_KIND if found is None else _port_kind(found) + + def _lower_node_expr( + self, node: exp.Expr, env: _Env, select: exp.Select + ) -> _Value | None: + """A call to a node module, or a field read off one; None for anything else. + + Every read of one call is one instance (:meth:`_node_instances`), so a + caption track and a module reading the same rows read one node's + output, split. + """ + found = self._node_call(node) + if found is None: + return None + base, declared, field_name = found + refusal = base.meta.get(NODE_REFUSAL) + if isinstance(refusal, FfrwdError): + raise refusal + call = _call_parts(base) + assert call is not None # an Anonymous always splits + output = self._node_output(declared, field_name, node, select) + instances, broadcast = self._node_instances(base, declared, call, env, select) + streams = tuple( + _Stream(ref=ref, type=kind) + for ref, kind in ( + self._node_pad(instance, declared, output, base, select) + for instance in instances + ) + ) + value = _Value( + type=streams[0].type, + streams=streams, + is_array=broadcast, + ) + return self._row_filtered(value, _unwrap(node)) + + def _node_written( + self, + declared: WasmFunction, + call: _Call, + node: exp.Expr, + select: exp.Select, + ) -> dict[str, exp.Expr]: + """Each parameter the call writes, ports and values alike, keyed by name. + + The positionals fill the signature in declared order and each name the + one it names, as resolve already checked. + """ + written: dict[str, exp.Expr] = {} + positional = self._spread_node_outputs(spread(call.args)) + for param, argument in zip(declared.params, positional): + written[param.name] = argument + for named in call.named: + if not any(p.name == named.name for p in declared.params): + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() has no parameter '{named.name}'", + named.value, + fallback=node, + hint=declared.signature, + ) + written[named.name] = named.value + if len(positional) > len(declared.params): + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() got {len(positional)} arguments, but it declares " + f"{len(declared.params)}", + node, + fallback=select, + hint=declared.signature, + ) + return written + + def _spread_node_outputs(self, arguments: Sequence[exp.Expr]) -> list[exp.Expr]: + """A node call read whole where it makes a stream and the rows beside it, + as those two fields read off it, in its own order.""" + spread_out: list[exp.Expr] = [] + for argument in arguments: + found = self._node_call(argument) + outputs = found[1].outputs or () if found is not None else () + two_part = ( + found is not None + and found[2] is None + and len(outputs) == 2 + and outputs[0].annotation is None + and outputs[1].annotation is not None + ) + if found is None or not two_part: + spread_out.append(argument) + continue + spread_out.extend( + exp.Dot(this=found[0].copy(), expression=exp.to_identifier(output.name)) + for output in outputs + ) + return spread_out + + def _lower_port( + self, + declared: WasmFunction, + param: Parameter, + argument: exp.Expr, + env: _Env, + select: exp.Select, + ) -> _Value: + """One port's argument: a stream, rows, or an array of either.""" + inner = _unwrap(argument) + if isinstance(inner, exp.Array) and inner.expressions: + elements = [ + self._lower_expr(element, env, select) + for element in inner.expressions + if isinstance(element, exp.Expr) + ] + streams = tuple(stream for value in elements for stream in value.streams) + value = _Value(type=elements[0].type, streams=streams, is_array=True) + else: + value = self._lower_expr(argument, env, select) + wanted = _port_kind(param) + got = next((s.type for s in value.streams if s.type != wanted), None) + if got is not None: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() takes {param.type} as its '{param.name}' " + f"argument, and its argument is a {got} stream", + argument, + fallback=select, + hint=declared.signature, + ) + return value + + def _node_instances( + self, + base: exp.Anonymous, + declared: WasmFunction, + call: _Call, + env: _Env, + select: exp.Select, + ) -> tuple[tuple[_NodeInstance, ...], bool]: + """The node instances a call is: one, or one per element it broadcasts over, + and whether it broadcasts. + + A port declared without ``[]`` given an array is one instance per + element, as a filter is; a value read off a row is one per row. A + call written twice in one branch is lowered once, and two calls whose + module, inputs and params agree are one instance wherever they are + written (:meth:`_node_instance`). + """ + key = (base.sql(), id(env)) + found = self._node_calls.get(key) + if found is not None: + return found + described = self._node_module(declared) + assert described is not None # what `_node_call` checked + written = self._node_written(declared, call, base, select) + params_by_name = {param.name: param for param in declared.params} + ports: dict[str, exp.Expr] = {} + values: dict[str, exp.Expr] = {} + for name, argument in written.items(): + if not is_port(params_by_name[name]): + values[name] = argument + elif not isinstance(_unwrap(argument), exp.Null): + ports[name] = argument + node_values = tuple(param for param in declared.params if not is_port(param)) + numbers = {name: arg for name, arg in ports.items() if is_number_argument(arg)} + streams = { + name: self._lower_port(declared, params_by_name[name], argument, env, select) + for name, argument in ports.items() + if name not in numbers + } + tuples = env.relation.tuples if env.relation is not None else [] + per_row = any(_reads_row_column(argument, env) for argument in values.values()) + broadcast = { + name: value + for name, value in streams.items() + if value.is_array and not is_array(params_by_name[name].type) + } + length = self._node_length(declared, base, broadcast, per_row, len(tuples), select) + instances: list[_NodeInstance] = [] + for element in range(1 if length is None else length): + row = tuples[element] if per_row and element < len(tuples) else ( + tuples[0] if len(tuples) == 1 else {} + ) + bound_values = { + name: _scalar(value.at(element)) if name in broadcast else value + for name, value in streams.items() + } + params = self._wasm_params( + declared, described, call, base, select, env, row, first=0, + written=values, value_params=node_values, + ) + shape_params = params + if per_row: + fixed = { + name: argument + for name, argument in values.items() + if not _reads_row_column(argument, env) + } + shape_params = self._wasm_params( + declared, described, call, base, select, env, {}, first=0, + written=fixed, value_params=node_values, + ) + instances.append( + self._node_instance( + declared, + described, + params, + shape_params, + bound_values, + numbers, + values, + base, + select, + ) + ) + self._node_calls[key] = (tuple(instances), length is not None) + return self._node_calls[key] + + def _node_length( + self, + declared: WasmFunction, + base: exp.Anonymous, + broadcast: Mapping[str, _Value], + per_row: bool, + rows: int, + select: exp.Select, + ) -> int | None: + """How many instances a call broadcasts to, or None for one.""" + lengths = {name: len(value.streams) for name, value in broadcast.items()} + if per_row: + lengths["the rows its values are read off"] = rows + if not lengths: + return None + distinct = sorted(set(lengths.values())) + if len(distinct) > 1: + named = ", ".join(f"{name} has {count}" for name, count in lengths.items()) + raise _error( + ErrorCode.BROADCAST_MISMATCH, + f"{declared.name}() cannot broadcast over arrays of different " + f"lengths: {named}", + base, + fallback=select, + hint=_ZIP_HINT, + ) + return distinct[0] + + def _node_shape( + self, + declared: WasmFunction, + described: Described, + params: Mapping[str, object], + bound: Sequence[str], + base: exp.Anonymous, + select: exp.Select, + ) -> NodeShape: + """The module's shape for this call, said at the call when it refuses.""" + try: + return self.shapes( + declared.module, + json.dumps(params, sort_keys=True), + bound, + _grant_args(described, declared.module), + ) + except FfrwdError as err: + raise _error( + err.code, f"{declared.name}(): {err.message}", base, fallback=select, + hint=err.hint, + ) from None + + def _node_instance( + self, + declared: WasmFunction, + described: Described, + params: dict[str, object], + shape_params: dict[str, object], + streams: Mapping[str, _Value], + numbers: Mapping[str, exp.Expr], + written: Mapping[str, exp.Expr], + base: exp.Anonymous, + select: exp.Select, + ) -> _NodeInstance: + """One instance: the shape for its params and bound ports, checked + against the declaration, and the node it is. `written` is the values + the call wrote, which a port number may not write again.""" + bound = [param.name for param in declared.ports if param.name in streams] + held = { + name: self._held_port(declared, name, argument, base, select) + for name, argument in numbers.items() + } + shape = self._node_shape( + declared, described, {**shape_params, **held}, bound, base, select + ) + self._check_node_ports(declared, shape, streams, numbers, base, select) + for name, argument in numbers.items(): + port = shape.input(name) + hold = port.pairing.hold if port is not None else None + if hold is None or hold.port_param is None: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() takes '{name}' as a stream, and the call " + "writes a number there", + argument, + fallback=base, + hint="a number stands for a port only where the module holds " + "an input on a port it names; pass a stream here", + ) + if hold.port_param in written: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() writes '{hold.port_param}' twice: as a " + f"number in '{name}' and as itself", + argument, + fallback=base, + hint=f"write the port once: {declared.signature}", + ) + params[hold.port_param] = held[name] + inputs: list[FrameRef] = [] + names: list[str] = [] + for port in shape.inputs: + value = streams.get(port.name) + if value is None: + continue + for stream in value.streams: + inputs.append(stream.ref) + names.append(port.name) + if port.kind == "data" and port.schema is not None: + self._match_rows(declared, port, value, base, select) + kinds: list[StreamType] = [] + for output in shape.outputs: + if output.kind == "packets": + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"the module '{declared.module}' writes packets on its " + f"'{output.name}' output, which a node call cannot read yet", + base, + fallback=select, + hint="read its frames or its rows; a node writing coded " + "packets is not wired yet", + ) + kinds.append(cast(StreamType, output.kind)) + key = ( + declared.module, + declared.export, + tuple(zip(names, inputs)), + json.dumps(params, sort_keys=True), + ) + found = self._node_refs.get(key) + if found is not None: + return found + ref = self.ctx.node(declared.module, params, inputs, kinds) + made = self.graph.nodes[ref] + made.ports = names + made.out_ports = [output.name for output in shape.outputs] + self.graph.node_shapes[ref] = dict(shape.raw) + instance = _NodeInstance(ref=ref, shape=shape) + self._node_refs[key] = instance + return instance + + def _held_port( + self, + declared: WasmFunction, + name: str, + argument: exp.Expr, + base: exp.Anonymous, + select: exp.Select, + ) -> int: + """A port number written where a held input goes: a whole number in range.""" + value = _number(argument) + if isinstance(value, float) and value.is_integer(): + value = int(value) + if not isinstance(value, int) or not 0 < value < 65536: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}()'s '{name}' is a port, and {_sql_text(argument)} " + "is not one", + argument, + fallback=base, + hint="a port is a whole number from 1 to 65535", + ) + return value + + def _check_node_ports( + self, + declared: WasmFunction, + shape: NodeShape, + streams: Mapping[str, _Value], + numbers: Mapping[str, exp.Expr], + base: exp.Anonymous, + select: exp.Select, + ) -> None: + """The declaration's ports against the shape's inputs. + + A parameter the shape has no port for is fine left unbound and + refused bound, naming it; an input the module requires is bound; each + bound port is the kind the module reads there, and only a port taking + many is handed several. + """ + declared_ports = {param.name: param for param in declared.ports} + for name in [*streams, *numbers]: + if shape.input(name) is not None: + continue + known = ", ".join(f"'{port.name}'" for port in shape.inputs) or "none" + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() binds '{name}', and for these params the module " + f"'{declared.module}' reads no input '{name}'", + base, + fallback=select, + hint=f"leave '{name}' out of this call; the inputs it reads here " + f"are {known}", + ) + for port in shape.inputs: + param = declared_ports.get(port.name) + if param is None: + if port.required: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"the module '{declared.module}' requires an input " + f"'{port.name}', and {declared.name}() declares none", + base, + fallback=select, + hint=f"declare '{port.name}' as a " + f"{_PORT_TYPE_NAMES[port.kind]} parameter", + ) + continue + if port.required and port.name not in streams and port.name not in numbers: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() leaves '{port.name}' out, and the module " + f"'{declared.module}' requires it", + base, + fallback=select, + hint=f"pass a stream for '{port.name}': {declared.signature}", + ) + if _port_kind(param) != port.kind: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() declares '{port.name}' as {param.type}, and " + f"the module '{declared.module}' reads {port.kind} there", + base, + fallback=select, + hint=f"declare '{port.name}' as a " + f"{_PORT_TYPE_NAMES[port.kind]} parameter", + ) + value = streams.get(port.name) + if value is not None and len(value.streams) > 1 and not port.many: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() hands {len(value.streams)} streams to " + f"'{port.name}', and the module '{declared.module}' reads one " + "there", + base, + fallback=select, + hint=f"pass one stream for '{port.name}'", + ) + + def _match_rows( + self, + declared: WasmFunction, + port: InputPort, + value: _Value, + base: exp.Anonymous, + select: exp.Select, + ) -> None: + """Rows a port reads against the rows its producer writes, by their schemas. + + Every field the port's schema names has to be in the producer's, with + a type it takes; fields the producer writes beyond them pass. + """ + assert port.schema is not None # checked by the caller + for stream in value.streams: + producer = self._under_rows_nodes(stream.ref) + written = self._data_schemas.get(producer) + if written is None: + continue + name, schema = written + mismatch = row_mismatch(port.schema, schema) + if mismatch is None: + continue + field_name, wants, writes = mismatch + said = "does not write it" if writes == "nothing" else f"writes {writes}" + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{declared.name}() reads '{field_name}' as {wants} on its " + f"'{port.name}' input, and {name}() {said}", + base, + fallback=select, + hint=f"pass rows carrying '{field_name}' as {wants}; a reader names " + "only the fields it reads, and a producer may write more", + ) + + def _node_pad( + self, + instance: _NodeInstance, + declared: WasmFunction, + output: Parameter, + base: exp.Anonymous, + select: exp.Select, + ) -> tuple[FrameRef, StreamType]: + """The pad of `instance` a declared output reads, and its kind. + + A named output is the module's output of that name; one unnamed is the + module's one output, or its one of the kind the RETURNS says. + """ + shape = instance.shape + wanted = _port_kind(output) + if output.name: + found = shape.output(output.name) + if found is None: + made = ", ".join(f"'{o.name}'" for o in shape.outputs) or "none" + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() returns the field '{output.name}', and the " + f"module '{declared.module}' makes no output '{output.name}'", + base, + fallback=select, + hint=f"name the fields after the outputs it makes: {made}", + ) + else: + kinds = [o for o in shape.outputs if o.kind == wanted] + if len(shape.outputs) == 1: + found = shape.outputs[0] + elif len(kinds) == 1: + found = kinds[0] + else: + made = ", ".join(f"'{o.name}' ({o.kind})" for o in shape.outputs) + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() returns {output.type}, and the module " + f"'{declared.module}' makes " + + (f"{len(kinds)} outputs of that kind" if kinds else "none of that kind"), + base, + fallback=select, + hint=f"name the one to read with RETURNS STRUCT( " + f", ...); it makes {made or 'nothing'}", + ) + if found.kind != wanted: + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() returns {output.type}" + + (f" as '{output.name}'" if output.name else "") + + f", and the module '{declared.module}' writes {found.kind} there", + base, + fallback=select, + hint=f"declare it as {_PORT_TYPE_NAMES[found.kind]}", + ) + index = shape.outputs.index(found) + ref = instance.ref if len(shape.outputs) == 1 else f"{instance.ref}:{index}" + if found.kind == "data": + schema = found.schema + if schema is None and output.annotation is not None: + schema = _record_schema(output.annotation) + if schema is not None: + self._data_schemas[ref] = (declared.name, schema) + if output.annotation is not None: + self._node_rows[ref] = (declared, output.annotation, base) + return ref, found.kind + + def _node_rows_column(self, node: exp.Expr) -> bool: + """Whether `node` is a node's rows, through any spans reduced off them.""" + node = _unwrap(node) + call = _call_parts(node) + if call is not None and call.is_macro and call.name.lower() == MERGE_SPANS: + return bool(call.args) and self._node_rows_column(call.args[0]) + found = self._node_call(node) + if found is None: + return False + outputs = found[1].outputs or () + if found[2] is None: + return len(outputs) == 1 and outputs[0].annotation is not None + return any(o.name == found[2] and o.annotation is not None for o in outputs) + + def _node_rows_at_sink( + self, value: _Value, node: exp.Expr, env: _Env, select: exp.Select + ) -> _Value: + """A node's rows selected as a column: a track of their own, or the rows. + + Rows read by another node stay a data stream; the ones a COPY writes + become what a module's rows have always become there, a rows file at a + rows destination and a subtitle track anywhere else. + """ + if value.is_array or len(value.streams) != 1: + return value + stream = value.streams[0] + found = self._node_rows.get(self._under_rows_nodes(stream.ref)) + if stream.type != "data" or found is None: + return value + declared, annotation, call = found + return self._rows_output( + stream.ref, "data", declared, annotation, call, node, env, select + ) + # -- feeders -- def _check_feeders( @@ -17054,7 +17995,8 @@ def _lower_macro_call( stream_param = macro.params[stream_pos] self._reject_null_stream(call.display, call.args[stream_pos], select) kind = self._classify(call.args[stream_pos], env, select) - self._reject_passthrough_args(call.display, [kind], call, call.args[stream_pos]) + if stream_param.stream_type != "data": + self._reject_passthrough_args(call.display, [kind], call, call.args[stream_pos]) if kind != stream_param.stream_type: hint = macro.kind_hints.get( kind, @@ -18549,6 +19491,9 @@ def _classify(self, node: exp.Expr, env: _Env, select: exp.Select) -> str: and not node.this.is_string ): return "num" + found = self._node_call(node) + if found is not None: + return self._node_output_kind(found[1], found[2]) # An output of a data filter is a data stream. if _data_projection(node, self.res.wasm) is not None: return "data" @@ -18643,6 +19588,7 @@ def _classify(self, node: exp.Expr, env: _Env, select: exp.Select) -> str: def run_table(self) -> list[TableSink]: """One :class:`~ffrwd.table.TableSink` per COPY, or one bare-select.""" + self._check_declared_worlds() self._lower_ctes() self.table_mode = True sinks: list[TableSink] = [] @@ -19811,6 +20757,7 @@ def lower( probe_source: ProbeSource = wasm_probe_source, read_packets: ReadPackets = wasm_read_packet_rows, probe_path: ProbePath = probe_one_path, + shapes: Shape | None = None, ) -> Graph: """Lower a resolved query into an IR graph -- its FIRST command's. @@ -19835,14 +20782,15 @@ def lower( call site's arguments, to fold its result -- a parameter for the same reason. `probe_source` runs one ``RETURNS source`` module's ``probe``, once per FROM alias that calls one, to bind its catalog -- a parameter - for the same reason. + for the same reason. `shapes` asks a node module for its shape per call, + and is one too. Raises ``FfrwdError`` — and nothing else — on every rejection. """ return lower_commands( res, probes, registry=registry, on_warning=on_warning, describes=describes, invoke=invoke, probe_failures=probe_failures, probe_source=probe_source, - read_packets=read_packets, probe_path=probe_path, + read_packets=read_packets, probe_path=probe_path, shapes=shapes, )[0] @@ -19858,6 +20806,7 @@ def lower_commands( probe_source: ProbeSource = wasm_probe_source, read_packets: ReadPackets = wasm_read_packet_rows, probe_path: ProbePath = probe_one_path, + shapes: Shape | None = None, ) -> list[Graph]: """Lower a resolved query into one IR graph per ffmpeg COMMAND. @@ -19874,11 +20823,13 @@ def lower_commands( Same probing/registry contract as :func:`lower`; raises ``FfrwdError`` -- and nothing else -- on every rejection. """ + asked = shapes if shapes is not None else ShapeCache() try: shared = _Lowerer( res, probes, registry, fanout_sinks=True, on_warning=on_warning, describes=describes, invoke=invoke, probe_failures=probe_failures, probe_source=probe_source, read_packets=read_packets, probe_path=probe_path, + shapes=asked, ) graph = shared.run() count = shared.fanout_count @@ -19894,6 +20845,7 @@ def lower_commands( res, probes, registry, fanout_index=index, on_warning=on_warning, describes=describes, invoke=invoke, probe_failures=probe_failures, probe_source=probe_source, read_packets=read_packets, probe_path=probe_path, + shapes=asked, ).run() for index in range(count) ] diff --git a/cli/ffrwd/macros.py b/cli/ffrwd/macros.py index 2972aaf..b348435 100644 --- a/cli/ffrwd/macros.py +++ b/cli/ffrwd/macros.py @@ -26,7 +26,10 @@ DEFAULT_MAX_SPREAD, LEAKY, MAX_LATENESS, + MAX_SPAN, MAX_SPREAD, + MERGE_SPANS, + ROWMERGE, StreamType, ) @@ -148,6 +151,12 @@ def _leaky(values: list[object], node: NodeBuilder, options: dict[str, object]) return node(LEAKY, limits, [str(f)], ["video"]) +def _merge_spans(values: list[object], node: NodeBuilder, options: dict[str, object]) -> str: + """The host's own rows node, grouping per-tick rows into spans by `start_t`.""" + (rows,) = values + return node(ROWMERGE, {MERGE_SPANS: True, **options}, [str(rows)], ["data"]) + + _LEAKY_SOUND_HINT = ( "ffrwd.leaky() drops late pictures and never sound: pass the picture " "through it and the sound beside it, e.g. ffrwd.leaky(s.video[1]), s.audio[1]" @@ -196,6 +205,14 @@ def _leaky(values: list[object], node: NodeBuilder, options: dict[str, object]) positive=(MAX_LATENESS,), nonnegative=(MAX_SPREAD,), ), + MERGE_SPANS: Macro( + name=MERGE_SPANS, + params=(MacroParam("rows", "stream", "data"),), + output="data", + expand=_merge_spans, + options=(MAX_SPAN,), + positive=(MAX_SPAN,), + ), "loudnorm2": Macro( name="loudnorm2", params=(MacroParam("stream", "stream", "audio"),), diff --git a/cli/ffrwd/parser.py b/cli/ffrwd/parser.py index 782faeb..b11c02d 100644 --- a/cli/ffrwd/parser.py +++ b/cli/ffrwd/parser.py @@ -2717,6 +2717,15 @@ def _projection_field_name(node: exp.Expr) -> str | None: # with, for lowering to mint the node that applies it. ROW_PREDICATE = "row_predicate" +# Where a call or a field read only a node module can make sense of keeps the +# refusal an older world's module earns, for lowering to raise once the +# module's describe says it is not a node. +OLDER_WORLD = "older_world" + +# The same for the refusal a node module earns: a call written for an older +# world's module, which only a node's describe can show to be wrong. +NODE_REFUSAL = "node_refusal" + # Where `merge_cues(...)` over a module's rows leaves the distance it was # written with, for lowering to mint the node that applies it. ROW_MERGE = "row_merge" @@ -2951,6 +2960,25 @@ def annotation_projection( return (base, declared) if _ident_name(field) == declared.emits.name else None +def node_rows_call(node: object, wasm: Mapping[str, WasmFunction]) -> Annotation | None: + """The record a call's rows carry, for a call a node may answer with rows alone. + + A declaration whose RETURNS is one array of records over a stream: a + packet sink's in FROM, a node's run-time data stream anywhere else. None + for every other expression. + """ + if not isinstance(node, exp.Expr): + return None + call = _unwrap_paren(node) + if not isinstance(call, exp.Anonymous): + return None + declared = wasm.get(str(call.name).lower()) + outputs = declared.outputs if declared is not None else None + if not outputs or len(outputs) != 1 or outputs[0].name: + return None + return outputs[0].annotation + + def is_annotation_argument(node: object, wasm: Mapping[str, WasmFunction]) -> bool: """Whether this argument WRITES a consumer's annotation column. @@ -5149,6 +5177,31 @@ def _check_wasm_field( A data filter's struct is its outputs, each a data stream of its own, and every field of it is one. """ + if declared.is_node_only: + names = tuple(o.name for o in declared.outputs or () if o.name) + if path in names: + return + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"{declared.name}() returns no field '{path}'", + sub, + fallback=select, + hint="it returns " + ", ".join(f"'{name}'" for name in names) + if names + else f"it returns one {declared.written_returns}: the call itself is " + "the stream", + ) + try: + self._check_older_wasm_field(declared, path, sub, select) + except FfrwdError as refusal: + if not any(o.name == path for o in declared.outputs or ()): + raise + sub.meta[OLDER_WORLD] = refusal + + def _check_older_wasm_field( + self, declared: WasmFunction, path: str, sub: exp.Dot, select: exp.Select + ) -> None: + """:meth:`_check_wasm_field` as a module of an older world reads it.""" if declared.is_data_filter: if path in declared.data_fields: return @@ -5860,11 +5913,17 @@ def _row_gather(self, array_node: exp.Array) -> bool: arguments = item.expressions source = arguments[0] if len(arguments) == 1 else None found = annotation_projection(source, self.wasm) - if found is None or not isinstance(source, exp.Expr): + rows = node_rows_call(source, self.wasm) + if (found is None and rows is None) or not isinstance(source, exp.Expr): self._no_gather_over_rows_function(source, array_node) return False - emits = found[1].emits - assert emits is not None # what annotation_projection matched on + if found is None: + try: + self._no_gather_over_rows_function(source, array_node) + except FfrwdError as refusal: + _unwrap_paren(source).meta.setdefault(OLDER_WORLD, refusal) + emits = found[1].emits if found is not None else rows + assert emits is not None # what annotation_projection or node_rows_call matched on _check_query_args( subquery, frozenset({"expressions", "from_", "where"}), "row gather" ) @@ -7930,7 +7989,11 @@ def _check_where( hint=_WHERE_HINT, ) column, low, high, strict = parsed - if strict: + table_node = column.args.get("table") + # A node read in FROM ends on a tick, so `t < 10` says where; + # which wasm sources are nodes is the module's to say, in lowering. + ticks = table_node is not None and _ident_name(table_node) in self.wasm_sources + if strict and not ticks: raise _error( ErrorCode.UNSUPPORTED_SQL, "strict inequalities are not supported", @@ -7938,7 +8001,6 @@ def _check_where( fallback=where, hint=_STRICT_HINT, ) - table_node = column.args.get("table") if table_node is None: raise _error( ErrorCode.UNSUPPORTED_SQL, diff --git a/cli/ffrwd/prompt.py b/cli/ffrwd/prompt.py index f0204a6..e6b37aa 100644 --- a/cli/ffrwd/prompt.py +++ b/cli/ffrwd/prompt.py @@ -1698,6 +1698,13 @@ def _examples() -> str: "two paths meet -- apply it after they meet, or drop it -- or record " "the input to a file first and write the query over the file." ), + ErrorCode.LIVE_LEAD: ( + "A live query feeds a node an input later than the lead the node " + "needs on it; `message` names the node, the input, the lead and how " + "far behind the path feeding it runs. Feed that input from a path " + "with less delay -- fewer windowed nodes on the way, shorter windows " + "-- or give the node a longer lead where it takes one as a parameter." + ), ErrorCode.STARTUP_DEADLOCK: ( "The query splits one input's streams apart and brings them back " "together in a way whose processes would each be waiting on the next " diff --git a/cli/ffrwd/pts.py b/cli/ffrwd/pts.py index 3b01fa9..eebfe2b 100644 --- a/cli/ffrwd/pts.py +++ b/cli/ffrwd/pts.py @@ -100,6 +100,8 @@ def rewire(ref: FrameRef, *, protected: bool) -> FrameRef: # Rows carry no timestamps a reset could touch; the edge rides # through naming the same producer. rows_inputs=list(node.rows_inputs), + ports=list(node.ports), + out_ports=list(node.out_ports), ) # A trim/atrim mapped straight to an output file, with no filter in diff --git a/cli/ffrwd/shapes.py b/cli/ffrwd/shapes.py new file mode 100644 index 0000000..4e6e820 --- /dev/null +++ b/cli/ffrwd/shapes.py @@ -0,0 +1,715 @@ +"""A node module's shape for one call: its ports, how each pairs, its clock. + +A module exporting ``ffrwd:av@0.19.0``'s ``node`` says what it reads and +writes per call, not once: its ports and formats turn on its params and on +which inputs the call binds. The compiler asks the sidecar for each distinct +call, ``ffrwd-wasm --shape --params --bound ``, which +prints the WIT's ``node-shape`` record as JSON (:class:`NodeShape`). +:func:`shape` runs that, and like :func:`ffrwd.wasm.describe` it is a seam: a +lowering test hands over its own and nothing is spawned. + +A shape is a pure function of the module's bytes, the params and the bound +port names, so :class:`ShapeCache` asks once per distinct triple. + +Also here, since every reader of a shape needs them: the streaming words a +window is said in (:func:`window_words`), and structural row matching +(:func:`row_mismatch`), which compares the JSON schemas two data ports carry. +""" + +from __future__ import annotations + +import hashlib +import json +import subprocess +import tempfile +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass, field +from fractions import Fraction +from pathlib import Path +from typing import Literal, cast + +from . import binaries +from .errors import ErrorCode, FfrwdError + +__all__ = [ + "Accepts", + "Anchor", + "Clock", + "Hold", + "InputPort", + "Interval", + "NodeShape", + "OutputFormat", + "OutputPort", + "Pairing", + "Shape", + "ShapeCache", + "node_shape", + "row_mismatch", + "shape", + "window_words", +] + +PortKind = Literal["video", "audio", "data", "packets"] +RowsUse = Literal["ignore", "per-frame", "state"] +PairingKind = Literal["lockstep", "hold", "interval", "arrival"] +AnchorKind = Literal["shared-clock", "first-frame", "tagged"] +ClockKind = Literal["input", "rate", "rate-of", "self-clocked"] +FormatKind = Literal["video", "audio", "data", "packets", "like"] +Wants = Literal["all", "keyframes", "first"] + +_PORT_KINDS: tuple[PortKind, ...] = ("video", "audio", "data", "packets") +_ROWS_USES: tuple[RowsUse, ...] = ("ignore", "per-frame", "state") +_WANTS: tuple[Wants, ...] = ("all", "keyframes", "first") +_ANCHORS: tuple[AnchorKind, ...] = ("shared-clock", "first-frame", "tagged") + +_SHAPE_FLAG = "--shape" +_PARAMS_FLAG = "--params" +_PARAMS_FROM_FLAG = "--params-from" +_BOUND_FLAG = "--bound" + +# Params longer than this go in a file rather than on the command line, which +# Windows caps at 32,767 characters for everything on it. +PARAMS_INLINE_LIMIT = 4096 + +_UNKNOWN_HINT = "the module may be built against a sidecar this ffrwd does not know" + + +@dataclass(frozen=True) +class Anchor: + """How a hold input's offset is fixed; `tag` names the tag `tagged` reads.""" + + kind: AnchorKind + tag: str = "" + + +@dataclass(frozen=True) +class Hold: + anchor: Anchor + lead: float + linger: float | None = None + timeout: float | None = None + group: str | None = None + port_param: str | None = None + + +@dataclass(frozen=True) +class Interval: + latency: float | None = None + ahead: float = 0.0 + + +@dataclass(frozen=True) +class Pairing: + kind: PairingKind + hold: Hold | None = None + interval: Interval | None = None + + +@dataclass(frozen=True) +class Accepts: + pixel_formats: tuple[str, ...] = () + sample_formats: tuple[str, ...] = () + sample_rates: tuple[int, ...] = () + channel_counts: tuple[int, ...] = () + codecs: tuple[str, ...] = () + wants: Wants = "all" + like: str | None = None + + +@dataclass(frozen=True) +class InputPort: + name: str + kind: PortKind + required: bool + many: bool + pairing: Pairing + rows: RowsUse + window: int + stride: int + accepts: Accepts + schema: Mapping[str, object] | None = None + + +@dataclass(frozen=True) +class OutputFormat: + """One arm of ``output-format``. + + `video`: width, height, `pixel_format`. `audio`: `sample_rate`, + `channels`, `sample_format`. `data`: `codec`. `packets`: `codec` and + `time_base`. `like`: `port`, with `pixel_format` or `sample_format` the + field it overrides. + """ + + kind: FormatKind + width: int | None = None + height: int | None = None + pixel_format: str | None = None + sample_rate: int | None = None + channels: int | None = None + sample_format: str | None = None + codec: str | None = None + time_base: tuple[int, int] | None = None + port: str | None = None + + +@dataclass(frozen=True) +class OutputPort: + name: str + kind: PortKind + format: OutputFormat | None = None + time_base: tuple[int, int] | None = None + latency: float = 0.0 + schema: Mapping[str, object] | None = None + row: int | None = None + + +@dataclass(frozen=True) +class Clock: + """`port` for `input` and `rate-of`, `rate` for `rate`.""" + + kind: ClockKind + port: str = "" + rate: tuple[int, int] | None = None + + +@dataclass(frozen=True) +class NodeShape: + inputs: tuple[InputPort, ...] + outputs: tuple[OutputPort, ...] + clock: Clock + pure: bool = True + one_to_one: bool = False + bounded: bool = True + relation: tuple[Mapping[str, object], ...] = () + # The JSON the sidecar printed, kept for `explain` and the IR. + raw: Mapping[str, object] = field(default_factory=dict, compare=False, repr=False) + + def input(self, name: str) -> InputPort | None: + return next((port for port in self.inputs if port.name == name), None) + + def output(self, name: str) -> OutputPort | None: + return next((port for port in self.outputs if port.name == name), None) + + @property + def clock_input(self) -> InputPort | None: + """The input the clock names, for an input clock; None otherwise.""" + return self.input(self.clock.port) if self.clock.kind == "input" else None + + def to_dict(self) -> dict[str, object]: + return dict(self.raw) + + +def _reject(message: str, hint: str = _UNKNOWN_HINT) -> FfrwdError: + return FfrwdError(ErrorCode.UNSUPPORTED_SQL, message, hint=hint) + + +# -- reading the JSON ------------------------------------------------------- + + +def _object(value: object, what: str, module: str) -> Mapping[str, object]: + if not isinstance(value, dict): + raise _reject(f"the sidecar's shape of {module} has {what} that is not an object") + return cast(Mapping[str, object], value) + + +def _kind(value: Mapping[str, object], what: str, module: str) -> str: + kind = value.get("kind") + if not isinstance(kind, str): + raise _reject(f"the sidecar's shape of {module} has {what} with no kind") + return kind + + +def _arm(value: Mapping[str, object], kind: str) -> object: + """A variant's payload: nested under its kind's own name, or the object itself.""" + nested = value.get(kind.replace("-", "_")) + if nested is None: + nested = value.get("value") + return value if nested is None else nested + + +def _text(value: object) -> str | None: + return value if isinstance(value, str) else None + + +def _number(value: object) -> float | None: + if isinstance(value, bool) or not isinstance(value, int | float): + return None + return float(value) + + +def _whole(value: object) -> int | None: + if isinstance(value, bool) or not isinstance(value, int): + return None + return value + + +def _texts(value: object) -> tuple[str, ...]: + if not isinstance(value, list): + return () + return tuple(item for item in value if isinstance(item, str)) + + +def _wholes(value: object) -> tuple[int, ...]: + if not isinstance(value, list): + return () + return tuple(found for item in value if (found := _whole(item)) is not None) + + +def _rational(value: object) -> tuple[int, int] | None: + if isinstance(value, dict): + num, den = _whole(value.get("num")), _whole(value.get("den")) + elif isinstance(value, list) and len(value) == 2: + num, den = _whole(value[0]), _whole(value[1]) + else: + return None + if num is None or den is None or den == 0: + return None + return (num, den) + + +def _schema(value: object) -> Mapping[str, object] | None: + """A port's JSON schema: written as a string of JSON, or as the object.""" + if isinstance(value, str): + if not value.strip(): + return None + try: + value = json.loads(value) + except ValueError: + return None + return cast(Mapping[str, object], value) if isinstance(value, dict) else None + + +def _choice(value: object, choices: tuple[str, ...], fallback: str) -> str: + if isinstance(value, str): + spelled = value.replace("_", "-") + if spelled in choices: + return spelled + return fallback + + +def _anchor(value: object, module: str) -> Anchor: + if isinstance(value, str): + kind = _choice(value, ("shared-clock", "first-frame"), "first-frame") + return Anchor(cast(AnchorKind, kind)) + raw = _object(value, "a hold anchor", module) + kind = _choice(_kind(raw, "a hold anchor", module), _ANCHORS, "") + if not kind: + raise _reject(f"the sidecar's shape of {module} names an anchor this ffrwd does not know") + tag = "" + if kind == "tagged": + arm = _arm(raw, kind) + tag = _text(arm) or (_text(raw.get("name")) or _text(raw.get("tag")) or "") + return Anchor(cast(AnchorKind, kind), tag) + + +def _pairing(value: object, module: str, port: str) -> Pairing: + what = f"input '{port}''s pairing" + if isinstance(value, str): + kind = _choice(value, ("lockstep", "arrival"), "") + if not kind: + raise _reject(f"the sidecar's shape of {module} gives {what} no fields") + return Pairing(cast(PairingKind, kind)) + raw = _object(value, what, module) + kind = _choice(_kind(raw, what, module), ("lockstep", "hold", "interval", "arrival"), "") + if kind == "hold": + arm = _object(_arm(raw, kind), what, module) + return Pairing( + "hold", + hold=Hold( + anchor=_anchor(arm.get("anchor"), module), + lead=_number(arm.get("lead")) or 0.0, + linger=_number(arm.get("linger")), + timeout=_number(arm.get("timeout")), + group=_text(arm.get("group")), + port_param=_text(arm.get("port_param")), + ), + ) + if kind == "interval": + arm = _object(_arm(raw, kind), what, module) + return Pairing( + "interval", + interval=Interval( + latency=_number(arm.get("latency")), ahead=_number(arm.get("ahead")) or 0.0 + ), + ) + if kind in ("lockstep", "arrival"): + return Pairing(cast(PairingKind, kind)) + raise _reject( + f"the sidecar's shape of {module} pairs input '{port}' in a way this ffrwd does not know" + ) + + +def _accepts(value: object) -> Accepts: + if not isinstance(value, dict): + return Accepts() + return Accepts( + pixel_formats=_texts(value.get("pixel_formats")), + sample_formats=_texts(value.get("sample_formats")), + sample_rates=_wholes(value.get("sample_rates")), + channel_counts=_wholes(value.get("channel_counts")), + codecs=_texts(value.get("codecs")), + wants=cast(Wants, _choice(value.get("wants"), _WANTS, "all")), + like=_text(value.get("like")), + ) + + +def _port_kind(value: object, module: str, port: str) -> PortKind: + kind = _choice(value, _PORT_KINDS, "") + if not kind: + raise _reject( + f"the sidecar's shape of {module} gives port '{port}' a kind this ffrwd does not know" + ) + return cast(PortKind, kind) + + +def _name(raw: Mapping[str, object], what: str, module: str) -> str: + name = raw.get("name") + if not isinstance(name, str) or not name: + raise _reject(f"the sidecar's shape of {module} has {what} with no name") + return name + + +def _input_port(value: object, module: str) -> InputPort: + raw = _object(value, "an input port", module) + name = _name(raw, "an input port", module) + window = _whole(raw.get("window")) or 1 + stride = _whole(raw.get("stride")) or 1 + return InputPort( + name=name, + kind=_port_kind(raw.get("kind"), module, name), + required=raw.get("required") is True, + many=raw.get("many") is True, + pairing=_pairing(raw.get("pairing"), module, name), + rows=cast(RowsUse, _choice(raw.get("rows"), _ROWS_USES, "ignore")), + window=max(window, 1), + stride=max(stride, 1), + accepts=_accepts(raw.get("accepts")), + schema=_schema(raw.get("schema")), + ) + + +def _output_format(value: object, module: str, port: str) -> OutputFormat | None: + if value is None: + return None + what = f"output '{port}''s format" + raw = _object(value, what, module) + kind = _choice(_kind(raw, what, module), ("video", "audio", "data", "packets", "like"), "") + arm = _arm(raw, kind) + if kind == "data": + codec = _text(arm) if not isinstance(arm, dict) else _text(arm.get("codec")) + return OutputFormat("data", codec=codec or "json") + body = _object(arm, what, module) + if kind == "video": + return OutputFormat( + "video", + width=_whole(body.get("width")), + height=_whole(body.get("height")), + pixel_format=_text(body.get("pix_fmt")) or _text(body.get("pixel_format")), + ) + if kind == "audio": + return OutputFormat( + "audio", + sample_rate=_whole(body.get("sample_rate")), + channels=_whole(body.get("channels")), + sample_format=_text(body.get("sample_fmt")) or _text(body.get("sample_format")), + ) + if kind == "packets": + return OutputFormat( + "packets", codec=_text(body.get("codec")), time_base=_rational(body.get("time_base")) + ) + if kind == "like": + return OutputFormat( + "like", + port=_text(body.get("port")), + pixel_format=_text(body.get("pixel_format")), + sample_format=_text(body.get("sample_format")), + ) + raise _reject( + f"the sidecar's shape of {module} gives output '{port}' a format this ffrwd does not know" + ) + + +def _output_port(value: object, module: str) -> OutputPort: + raw = _object(value, "an output port", module) + name = _name(raw, "an output port", module) + return OutputPort( + name=name, + kind=_port_kind(raw.get("kind"), module, name), + format=_output_format(raw.get("format"), module, name), + time_base=_rational(raw.get("time_base")), + latency=max(_number(raw.get("latency")) or 0.0, 0.0), + schema=_schema(raw.get("schema")), + row=_whole(raw.get("row")), + ) + + +def _clock(value: object, module: str) -> Clock: + if isinstance(value, str): + if value.replace("_", "-") == "self-clocked": + return Clock("self-clocked") + raise _reject(f"the sidecar's shape of {module} names a clock with no fields") + raw = _object(value, "a clock", module) + kind = _choice(_kind(raw, "a clock", module), ("input", "rate", "rate-of", "self-clocked"), "") + arm = _arm(raw, kind) + if kind in ("input", "rate-of"): + port = _text(arm) if not isinstance(arm, dict) else _text(arm.get("port")) + if not port: + raise _reject(f"the sidecar's shape of {module} names a clock with no input") + return Clock(cast(ClockKind, kind), port=port) + if kind == "rate": + nested = isinstance(arm, dict) and "num" not in arm + rate = _rational(arm.get("rate") if nested and isinstance(arm, dict) else arm) + if rate is None or rate[0] <= 0 or rate[1] <= 0: + raise _reject(f"the sidecar's shape of {module} names a rate clock with no rate") + return Clock("rate", rate=rate) + if kind == "self-clocked": + return Clock("self-clocked") + raise _reject(f"the sidecar's shape of {module} names a clock this ffrwd does not know") + + +def _relation(value: object) -> tuple[Mapping[str, object], ...]: + if not isinstance(value, list): + return () + rows: list[Mapping[str, object]] = [] + for item in value: + found = _schema(item) + rows.append(found if found is not None else {}) + return tuple(rows) + + +def node_shape(module: str, payload: object) -> NodeShape: + """One ``--shape`` document as a :class:`NodeShape`, or a rejection naming `module`.""" + raw = _object(payload, "a document", module) + inputs = raw.get("inputs") + outputs = raw.get("outputs") + if not isinstance(inputs, list) or not isinstance(outputs, list): + raise _reject(f"the sidecar's shape of {module} names no inputs or no outputs") + return NodeShape( + inputs=tuple(_input_port(port, module) for port in inputs), + outputs=tuple(_output_port(port, module) for port in outputs), + clock=_clock(raw.get("clock"), module), + pure=raw.get("pure") is not False, + one_to_one=raw.get("one_to_one") is True, + bounded=raw.get("bounded") is not False, + relation=_relation(raw.get("relation")), + raw=dict(raw), + ) + + +# -- asking the sidecar ----------------------------------------------------- + + +# Runs one module's `shape` for one call: :func:`shape` is the real one, and a +# lowering test passes its own. `grants` are the sidecar flags a module's own +# imports need to answer at all (a source reading the network for its +# outputs), ahead of the flag that dispatches the call. +Shape = Callable[[str, str, Sequence[str], Sequence[str]], NodeShape] + + +def shape(module: str, params: str, bound: Sequence[str], grants: Sequence[str] = ()) -> NodeShape: + """Ask the sidecar for the shape of `module` under `params` with `bound` bound. + + Raises ``FfrwdError`` and nothing else, unanchored: the caller anchors it + on the call that named the module. + """ + from .wasm import ( # deferred: wasm imports processes, which reads shapes + INSTALL_HINT, + _first_line, + timeout_seconds, + ) + + sidecar = binaries.ffrwd_wasm_path() + if sidecar is None: + raise _reject( + f"the ffrwd-wasm sidecar is not installed, and the shape of '{module}' " + "needs it to read the module", + hint=INSTALL_HINT, + ) + argv = [sidecar, *grants, _SHAPE_FLAG, module] + if bound: + argv += [_BOUND_FLAG, ",".join(bound)] + budget = timeout_seconds() + with tempfile.TemporaryDirectory(prefix="ffrwd-shape-") as scratch: + if len(params) > PARAMS_INLINE_LIMIT: + written = Path(scratch) / "params.json" + written.write_text(params, encoding="utf-8") + argv += [_PARAMS_FROM_FLAG, str(written)] + elif params and params != "{}": + argv += [_PARAMS_FLAG, params] + done = _run_shape(sidecar, module, argv, budget) + if done.returncode != 0: + raise _reject( + f"the module '{module}' refused the shape for these params: " + f"{_first_line(done.stderr)}", + hint="check the arguments match what the module declares", + ) + try: + payload = json.loads(done.stdout) + except ValueError as err: + raise _reject( + f"the ffrwd-wasm sidecar's shape of {module} is not JSON", + hint="the sidecar on PATH may be a different version than this ffrwd", + ) from err + return node_shape(module, payload) + + +def _run_shape( + sidecar: str, module: str, argv: list[str], budget: float +) -> subprocess.CompletedProcess[str]: + """One ``--shape`` run, refused by name when it cannot be run at all.""" + from .wasm import INSTALL_HINT, _budget_hint # deferred: wasm imports processes + + try: + return subprocess.run( + argv, + capture_output=True, + encoding="utf-8", + errors="replace", + timeout=budget, + check=False, + ) + except (OSError, ValueError) as err: + raise _reject( + f"could not run the ffrwd-wasm sidecar at {sidecar}: " + f"{getattr(err, 'strerror', None) or err}", + hint=INSTALL_HINT, + ) from err + except subprocess.TimeoutExpired as err: + raise _reject( + f"the ffrwd-wasm sidecar did not shape {module} within {budget:g}s", + hint=_budget_hint(budget), + ) from err + + +def _module_hash(module: str) -> str: + """The module file's own digest, or its path where the file cannot be read. + + A path that names nothing readable still shapes the same within one + compile, which is all a test's fake module needs. + """ + try: + return hashlib.sha256(Path(module).read_bytes()).hexdigest() + except OSError: + return f"path:{module}" + + +class ShapeCache: + """One :data:`Shape` asked once per (module bytes, params, bound ports).""" + + def __init__(self, ask: Shape = shape) -> None: + self._ask = ask + self._hashes: dict[str, str] = {} + self._shapes: dict[tuple[str, str, frozenset[str]], NodeShape] = {} + + def __call__( + self, module: str, params: str, bound: Sequence[str], grants: Sequence[str] = () + ) -> NodeShape: + digest = self._hashes.get(module) + if digest is None: + digest = self._hashes[module] = _module_hash(module) + key = (digest, params, frozenset(bound)) + found = self._shapes.get(key) + if found is None: + found = self._shapes[key] = self._ask(module, params, bound, grants) + return found + + +# -- windows, in streaming words ------------------------------------------- + + +def window_words(port: InputPort, rate: Fraction | None = None) -> str: + """The clock input's window as streaming SQL says it. + + `rate` is items per second (frames, or samples for audio); with it the + window is said in seconds, without it in items. + """ + if port.window <= 1: + return "per-frame" + + def span(items: int) -> str: + if rate is None or rate <= 0: + unit = "samples" if port.kind == "audio" else "frames" + return f"{items} {unit}" + return f"{_seconds(Fraction(items) / rate)} s" + + if port.stride >= port.window: + return f"tumbling {span(port.window)}" + if port.stride == 1 and port.kind != "audio": + return f"sliding {span(port.window)}" + return f"hopping {span(port.window)} every {span(port.stride)}" + + +def _seconds(value: Fraction | float) -> str: + """Seconds as a reader writes them: 2, 0.5, 0.033.""" + rounded = round(float(value), 3) + return f"{rounded:g}" + + +# -- structural row matching ------------------------------------------------ + + +def _types(schema: Mapping[str, object]) -> frozenset[str] | None: + written = schema.get("type") + if isinstance(written, str): + return frozenset({written}) + if isinstance(written, list): + return frozenset(item for item in written if isinstance(item, str)) + return None + + +def _covers(reader: frozenset[str], producer: frozenset[str]) -> bool: + """Whether every type the producer may write is one the reader takes. + + JSON Schema's integer is a number, so a reader of numbers takes a + producer of integers; the reverse is not so. + """ + return all(t in reader or (t == "integer" and "number" in reader) for t in producer) + + +def _type_text(types: frozenset[str] | None) -> str: + if not types: + return "no type" + return " or ".join(sorted(types)) + + +def row_mismatch( + reader: Mapping[str, object], producer: Mapping[str, object] +) -> tuple[str, str, str] | None: + """The first field `reader` names that `producer` lacks or types otherwise. + + ``(field, what the reader wants, what the producer writes)``, or None + where every field the reader names is there with a type it takes. Fields + the producer writes beyond those pass. Nested objects and arrays are + compared the same way, a nested field named by its dotted path. + """ + wanted = reader.get("properties") + if not isinstance(wanted, dict): + return None + written = producer.get("properties") + given: Mapping[str, object] = written if isinstance(written, dict) else {} + for name, want in wanted.items(): + if not isinstance(want, dict): + continue + have = given.get(name) + want_types = _types(want) + if not isinstance(have, dict): + return (name, _type_text(want_types), "nothing") + have_types = _types(have) + if want_types is not None and (have_types is None or not _covers(want_types, have_types)): + return (name, _type_text(want_types), _type_text(have_types)) + nested = row_mismatch(want, have) + if nested is not None: + return (f"{name}.{nested[0]}", nested[1], nested[2]) + items_want, items_have = want.get("items"), have.get("items") + if isinstance(items_want, dict) and isinstance(items_have, dict): + item_types_want, item_types_have = _types(items_want), _types(items_have) + if item_types_want is not None and ( + item_types_have is None or not _covers(item_types_want, item_types_have) + ): + return ( + f"{name}[]", + _type_text(item_types_want), + _type_text(item_types_have), + ) + return None diff --git a/cli/ffrwd/split.py b/cli/ffrwd/split.py index 99291f5..12f925a 100644 --- a/cli/ffrwd/split.py +++ b/cli/ffrwd/split.py @@ -174,6 +174,8 @@ def rewire(ref: FrameRef) -> FrameRef: # A rows edge is not a pad and never fans out: nothing to split, # so it rides through naming the same producer it always did. rows_inputs=list(node.rows_inputs), + ports=list(node.ports), + out_ports=list(node.out_ports), ) # Units in order, each unit's outputs in list order: pad assignment is diff --git a/cli/ffrwd/timing.py b/cli/ffrwd/timing.py new file mode 100644 index 0000000..6aeb509 --- /dev/null +++ b/cli/ffrwd/timing.py @@ -0,0 +1,392 @@ +"""How late each node module's outputs run behind the source, and why. + +Every node declares the window its clock input reads and how late each output +may leave; every input says how it pairs with the clock. Summed along the +paths of a graph, those say how far behind its source each stream a query +writes runs (:func:`timing`): + +- a source's stream runs at its source's time, 0; +- a node is ready for a tick once every input it waits for has arrived: + the clock, each lockstep input, and each interval input with its `ahead`, + an interval input waiting no longer than its own bound past the clock. + A held input and one delivered on arrival hold nothing up; +- a node's window is how long its tick takes to fill, a tumbling 2 s + window 2 s; a node's output adds the output's declared latency; +- the host's span reducer adds its `max_span`. + +Where streams that run at different delays meet, in one node or in one file +written, the earlier one waits: :attr:`OutputTiming.holds` says for how long. +A live query cannot give a node an interval input later than the bound the +node set on it, which is :data:`~ffrwd.errors.ErrorCode.LIVE_LEAD` +(:func:`check_live_leads`). +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from fractions import Fraction + +from .errors import ErrorCode, FfrwdError +from .ir import ( + MAX_SPAN, + MERGE_SPANS, + ROWMERGE, + FrameRef, + Graph, + StreamType, + is_src, + src_parts, +) +from .probe import ProbeResult +from .shapes import InputPort, NodeShape, node_shape, window_words + +__all__ = [ + "HOLD_LIMIT", + "NodeTiming", + "OutputTiming", + "Timing", + "check_live_leads", + "timing", +] + +# How many bytes a stream may wait for the one written beside it before the +# compile says so: a picture held behind 30 s of late words is about 2 GB of +# 1080p, and that is worth a line. +HOLD_LIMIT = 512 * 1024 * 1024 + +# Bytes per picture element of a raw picture on an edge, yuv420p's; and of +# one sample of one channel, f32's. +_PICTURE_BYTES = Fraction(3, 2) +_SAMPLE_BYTES = 4 + + +@dataclass(frozen=True) +class Wait: + """One input of a node: how it pairs, and how late what it reads runs.""" + + port: str + pairing: str + delay: float | None + bound: float | None = None + + +@dataclass(frozen=True) +class NodeTiming: + """One node module: its window in streaming words, and when it is ready. + + `delay` is how far behind the source a tick's outputs leave, before each + output's own latency; None where something on the way has no size. + """ + + node: str + module: str + window: str + waits: tuple[Wait, ...] + delay: float | None + + def to_dict(self) -> dict[str, object]: + return { + "node": self.node, + "module": self.module, + "window": self.window, + "inputs": [ + { + "port": wait.port, + "pairing": wait.pairing, + "delay": wait.delay, + **({"bound": wait.bound} if wait.bound is not None else {}), + } + for wait in self.waits + ], + "delay": self.delay, + } + + +@dataclass(frozen=True) +class OutputTiming: + """One stream a query writes: how late it runs, how long it waits for + the latest one written beside it, and about how many bytes that wait + holds, None where the stream's size is not known.""" + + ref: FrameRef + delay: float | None + holds: float + held: int | None = None + + def to_dict(self) -> dict[str, object]: + written: dict[str, object] = { + "ref": self.ref, + "delay": self.delay, + "holds": self.holds, + } + if self.held: + written["held_bytes"] = self.held + return written + + +@dataclass(frozen=True) +class Timing: + nodes: tuple[NodeTiming, ...] + outputs: tuple[OutputTiming, ...] + + def to_dict(self) -> dict[str, object]: + return { + "nodes": [node.to_dict() for node in self.nodes], + "outputs": [output.to_dict() for output in self.outputs], + } + + +class _Paths: + """Delays over one graph, each ref's counted once.""" + + def __init__(self, graph: Graph, probes: Mapping[str, ProbeResult | None]) -> None: + self.graph = graph + self.probes = probes + self.shapes = { + name: node_shape(graph.nodes[name].filter, raw) + for name, raw in graph.node_shapes.items() + if name in graph.nodes + } + self._delays: dict[FrameRef, float | None] = {} + self._ready: dict[str, tuple[float | None, tuple[Wait, ...]]] = {} + + def delay(self, ref: FrameRef) -> float | None: + if ref not in self._delays: + self._delays[ref] = self._count(ref) + return self._delays[ref] + + def _count(self, ref: FrameRef) -> float | None: + if is_src(ref): + return 0.0 + name, _, pad = ref.partition(":") + node = self.graph.nodes.get(name) + if node is None: + return 0.0 + shape = self.shapes.get(name) + if shape is not None: + ready = self.ready(name)[0] + output = shape.outputs[int(pad) if pad else 0] if shape.outputs else None + latency = output.latency if output is not None else 0.0 + return None if ready is None else ready + latency + found: list[float | None] = [self.delay(one) for one in node.inputs] + if any(one is None for one in found): + return None + above = max((one for one in found if one is not None), default=0.0) + if node.filter == ROWMERGE and node.args.get(MERGE_SPANS): + span = node.args.get(MAX_SPAN) + return above + float(span) if isinstance(span, int | float) else None + return above + + def ready(self, name: str) -> tuple[float | None, tuple[Wait, ...]]: + """When node `name`'s tick has all it waits for, its window filled.""" + if name in self._ready: + return self._ready[name] + node = self.graph.nodes[name] + shape = self.shapes[name] + by_port: dict[str, list[FrameRef]] = {} + for bound, ref in zip(node.ports, node.inputs): + by_port.setdefault(bound, []).append(ref) + clock = shape.clock_input + clock_delay: float | None = 0.0 + if clock is not None: + clock_delay = self._latest(by_port.get(clock.name, [])) + waits: list[Wait] = [] + ready = clock_delay + for port in shape.inputs: + refs = by_port.get(port.name, []) + if not refs: + continue + arrived = self._latest(refs) + pairing = port.pairing + interval = pairing.interval + if pairing.kind == "lockstep": + waited = arrived + said = "clock" if clock is not None and port.name == clock.name else "lockstep" + elif interval is not None: + said = "interval" + waited = None if arrived is None else arrived + interval.ahead + if interval.latency is not None and clock_delay is not None: + limit = clock_delay + interval.latency + interval.ahead + waited = limit if waited is None else min(waited, limit) + else: + waits.append(Wait(port.name, pairing.kind, arrived)) + continue + waits.append( + Wait(port.name, said, arrived, interval.latency if interval else None) + ) + ready = None if ready is None or waited is None else max(ready, waited) + window = self.window_seconds(name, shape, by_port) + total = None if ready is None or window is None else ready + window + self._ready[name] = (total, tuple(waits)) + return self._ready[name] + + def _latest(self, refs: list[FrameRef]) -> float | None: + found = [self.delay(ref) for ref in refs] + if any(one is None for one in found): + return None + return max((one for one in found if one is not None), default=0.0) + + def window_seconds( + self, name: str, shape: NodeShape, by_port: Mapping[str, list[FrameRef]] + ) -> float | None: + """How long the clock input's window takes to fill; 0 for one frame.""" + clock = shape.clock_input + if clock is None or clock.window <= 1: + return 0.0 + rate = self.clock_rate(clock, by_port.get(clock.name, [])) + return None if rate is None else float(Fraction(clock.window) / rate) + + def clock_rate(self, port: InputPort, refs: list[FrameRef]) -> Fraction | None: + """Items per second of what the clock input reads: frames, or samples.""" + if port.kind == "audio" and port.accepts.sample_rates: + return Fraction(port.accepts.sample_rates[0]) + return self.rate(refs[0], port.kind) if refs else None + + def bytes_per_second(self, ref: FrameRef) -> Fraction | None: + """About how many bytes a second of the raw stream `ref` is on an edge.""" + origin = self.origin(ref) + if origin is None: + return None + alias, kind, index = origin + probe = self.probes.get(alias) + streams = probe.by_type(kind) if probe is not None else [] + if index >= len(streams): + return None + stream = streams[index] + if kind == "video": + rate = _fraction(stream.fps) + if rate is None or not stream.width or not stream.height: + return None + return stream.width * stream.height * _PICTURE_BYTES * rate + if kind == "audio" and stream.sample_rate and stream.channels: + return Fraction(stream.sample_rate * stream.channels * _SAMPLE_BYTES) + return None + + def origin(self, ref: FrameRef) -> tuple[str, StreamType, int] | None: + """The source stream `ref` was made from, along first inputs.""" + while not is_src(ref): + node = self.graph.nodes.get(ref.partition(":")[0]) + if node is None or not node.inputs: + return None + ref = node.inputs[0] + alias, kind, index = src_parts(ref) + return alias, kind, index + + def rate(self, ref: FrameRef, kind: str) -> Fraction | None: + """The rate of the stream `ref` is, read back to where it came from.""" + if is_src(ref): + alias, stream_kind, index = src_parts(ref) + probe = self.probes.get(alias) + streams = probe.by_type(stream_kind) if probe is not None else [] + if index >= len(streams): + return None + stream = streams[index] + if kind == "audio": + return Fraction(stream.sample_rate) if stream.sample_rate else None + return _fraction(stream.fps) + name, _, _ = ref.partition(":") + node = self.graph.nodes.get(name) + if node is None: + return None + shape = self.shapes.get(name) + if shape is not None and shape.clock.rate is not None and kind == "video": + return Fraction(*shape.clock.rate) + return self.rate(node.inputs[0], kind) if node.inputs else None + + +def _fraction(fps: str | None) -> Fraction | None: + numerator, _, denominator = (fps or "").partition("/") + try: + found = Fraction(int(numerator), int(denominator) if denominator else 1) + except (ValueError, ZeroDivisionError): + return None + return found if found > 0 else None + + +def timing(graph: Graph, probes: Mapping[str, ProbeResult | None]) -> Timing | None: + """Each node module's window and readiness, and each written stream's + delay; None for a graph with no node module in it.""" + if not graph.node_shapes: + return None + paths = _Paths(graph, probes) + nodes: list[NodeTiming] = [] + for name, shape in paths.shapes.items(): + node = graph.nodes[name] + by_port: dict[str, list[FrameRef]] = {} + for bound, ref in zip(node.ports, node.inputs): + by_port.setdefault(bound, []).append(ref) + clock = shape.clock_input + if clock is not None: + rate = paths.clock_rate(clock, by_port.get(clock.name, [])) + window = window_words(clock, rate) + elif shape.clock.kind == "rate" and shape.clock.rate is not None: + window = f"rate {_rate_words(shape.clock.rate)}" + else: + window = shape.clock.kind + ready, waits = paths.ready(name) + nodes.append(NodeTiming(name, node.filter, window, waits, ready)) + outputs: list[OutputTiming] = [] + for unit in graph.sinks: + delays = [paths.delay(output.ref) for output in unit.outputs] + latest = max((d for d in delays if d is not None), default=0.0) + for output, delay in zip(unit.outputs, delays): + holds = 0.0 if delay is None else latest - delay + per_second = paths.bytes_per_second(output.ref) if holds else None + held = None if per_second is None else int(per_second * Fraction(holds)) + outputs.append(OutputTiming(output.ref, delay, holds, held)) + for ref in graph.rows_sinks: + if not any(output.ref == ref for output in outputs): + outputs.append(OutputTiming(ref, paths.delay(ref), 0.0)) + return Timing(tuple(nodes), tuple(outputs)) + + +def _rate_words(rate: tuple[int, int]) -> str: + num, den = rate + return f"{num}/s" if den == 1 else f"{num}/{den}/s" + + +def check_live_leads( + graph: Graph, + probes: Mapping[str, ProbeResult | None], + anchors: Mapping[str, tuple[int, int, str]], +) -> None: + """Refuse a node whose interval input runs later than the bound it set. + + A node bounding how long it waits for an input (`interval.latency`) is + one that acts ahead of the rows it reads, and in a live run what arrives + past the bound is late for good. `anchors` maps a module path to where + the query named it and the function's name. + """ + paths = _Paths(graph, probes) + for name, shape in paths.shapes.items(): + node = graph.nodes[name] + clock = shape.clock_input + by_port: dict[str, list[FrameRef]] = {} + for bound, ref in zip(node.ports, node.inputs): + by_port.setdefault(bound, []).append(ref) + clock_delay = paths._latest(by_port.get(clock.name, [])) if clock is not None else 0.0 + for port in shape.inputs: + interval = port.pairing.interval + if interval is None or interval.latency is None: + continue + arrived = paths._latest(by_port.get(port.name, [])) + if arrived is None or clock_delay is None: + continue + late = arrived - clock_delay + if late <= interval.latency: + continue + line, col, called = anchors.get(node.filter, (1, 1, node.filter)) + raise FfrwdError( + ErrorCode.LIVE_LEAD, + f"{called}() needs '{port.name}' {_seconds(interval.latency)} ahead of " + f"its clock, and the path feeding it runs {_seconds(late)} behind", + line=line, + col=col, + hint=f"feed '{port.name}' from a path no later than " + f"{_seconds(interval.latency)}, or give {called}() a longer lead", + ) + + +def _seconds(value: float) -> str: + return f"{round(value, 3):g} s" diff --git a/cli/ffrwd/warnings.py b/cli/ffrwd/warnings.py index c35330d..58121a8 100644 --- a/cli/ffrwd/warnings.py +++ b/cli/ffrwd/warnings.py @@ -38,6 +38,7 @@ class WarningCode(str, Enum): MISSING_LICENSE = "MISSING_LICENSE" # a package published with no "license" RECIPE_DOES_NOT_COMPILE = "RECIPE_DOES_NOT_COMPILE" # a recipe the probe compile rejected UNSHAPED_LISTENER = "UNSHAPED_LISTENER" # a listener input probed, for want of a shape + HELD_STREAM = "HELD_STREAM" # a stream waits long beside a later one it is written with @dataclass(frozen=True) diff --git a/cli/ffrwd/wasm.py b/cli/ffrwd/wasm.py index 5b791aa..3adef73 100644 --- a/cli/ffrwd/wasm.py +++ b/cli/ffrwd/wasm.py @@ -83,6 +83,7 @@ "DEFAULT_TIMEOUT_SECONDS", "LANGUAGE_TAGS", "MODEL_SUFFIX", + "NODE_WORLD", "PACKET_FILTER_WORLD", "PACKET_SOURCE_WORLD", "FFMPEG_SAMPLE_FMTS", @@ -243,6 +244,11 @@ # The first world whose sidecar hosts a codec package's encoder and decoder. CODEC_WORLD = "ffrwd:av@0.18.0" +# What a module exporting the 0.19.0 world's `node` describes as its world: +# its ports are not in its describe at all but in its shape, per call +# (:mod:`ffrwd.shapes`). +NODE_WORLD = "node-module" + # The sample formats one can carry, the pcm each of them travels as, and # the name ffmpeg's own options spell it by. WIRE_SAMPLE_FMTS: tuple[str, ...] = ("f32", "s16") @@ -631,6 +637,8 @@ class Described: # reads. None for every other module; a codec module fills both. encoder: EncoderInfo | None = None decoder: DecoderInfo | None = None + # A node: what it reads and writes is its shape's to say, per call. + node: bool = False @property def packet_sink(self) -> bool: @@ -776,12 +784,13 @@ def _described(path: str, payload: object) -> Described: f"the sidecar described {path} with something that is not an object", hint="the module may be built against a sidecar this ffrwd does not know", ) - world = payload.get("world") - if not isinstance(world, str): + written = payload.get("world") + if not isinstance(written, str): raise _reject( f"the sidecar's description of {path} names no world", hint="the module may be built against a sidecar this ffrwd does not know", ) + world = _hosted_world(written) name = payload.get("name") functions = _functions(payload.get("functions")) if not isinstance(name, str) and not functions: @@ -847,9 +856,21 @@ def _described(path: str, payload: object) -> Described: feeders=_feeders(payload.get("feeders")), encoder=_encoder_info(payload.get("encoder")), decoder=_decoder_info(payload.get("decoder")), + node=written == NODE_WORLD or payload.get("node") is True, ) +def _hosted_world(world: str) -> str: + """The world the sidecar hosts a module in, as the compiler's checks read it. + + The sidecar describes a module by the world it hosts it in, not the one + it was built against: a node is `node-module`, which is 0.19.0's, and + every older module is adapted into the newest world it knows. Each + `hosts_*` check asks what that world can host. + """ + return WORLDS[-1] if world == NODE_WORLD else world + + def _encoder_info(value: object) -> EncoderInfo | None: """The describe's ``encoder`` object, or None where there is none.""" if not isinstance(value, dict): diff --git a/cli/tests/conftest.py b/cli/tests/conftest.py index 9084fdf..8143cfa 100644 --- a/cli/tests/conftest.py +++ b/cli/tests/conftest.py @@ -31,6 +31,8 @@ import pytest from ffrwd import registry as registry_module +from ffrwd.errors import FfrwdError +from ffrwd.parser import parse, resolve from ffrwd.registry import Registry, load_reference SNAPSHOT_PATH = Path(__file__).resolve().parent / "data" / "reference_registry.json" @@ -73,6 +75,22 @@ def pinned_ffmpeg() -> None: pytest.skip(message) +def older_world_refusal(text: str) -> FfrwdError: + """What a module of an older world refuses `text`'s declaration with. + + Resolve refuses it outright, or, for a signature a node module may + carry, keeps the refusal on the declaration for lowering to raise once + the module's describe says it is no node. + """ + try: + resolved = resolve(parse(text)) + except FfrwdError as err: + return err + kept = [d.refusal for d in resolved.wasm.values() if d.refusal is not None] + assert kept, "the declaration was neither refused nor kept a refusal" + return kept[0] + + def clear_leaks(home: Path, paths: Iterable[Path]) -> None: """Remove each of `paths`, and refuse any that is not under `home`. diff --git a/cli/tests/test_data_filter.py b/cli/tests/test_data_filter.py index bbb1c64..eb0fd0a 100644 --- a/cli/tests/test_data_filter.py +++ b/cli/tests/test_data_filter.py @@ -40,6 +40,7 @@ from ffrwd.registry import Registry, load_reference from ffrwd.split import insert_splits from ffrwd.wasm import WORLDS, Described +from tests.conftest import older_world_refusal SNAPSHOT_PATH = Path(__file__).resolve().parent / "data" / "reference_registry.json" @@ -400,15 +401,12 @@ def test_the_shapes_a_data_filter_is_declared_in() -> None: def test_a_data_filter_declaration_is_refused_by_what_it_gets_wrong( signature: str, needle: str ) -> None: - with pytest.raises(FfrwdError) as caught: - resolve( - parse( - f"CREATE FUNCTION f{signature} AS 'm.wasm', 'm' LANGUAGE wasm;\n" - "COPY (SELECT f(f.data[1])" + _FROM - ) - ) - assert caught.value.code is ErrorCode.UNSUPPORTED_SQL - assert needle in caught.value.message + refusal = older_world_refusal( + f"CREATE FUNCTION f{signature} AS 'm.wasm', 'm' LANGUAGE wasm;\n" + "COPY (SELECT f(f.data[1])" + _FROM + ) + assert refusal.code is ErrorCode.UNSUPPORTED_SQL + assert needle in refusal.message def test_a_field_a_data_filter_does_not_return_is_refused_at_resolve() -> None: diff --git a/cli/tests/test_feeders.py b/cli/tests/test_feeders.py index 4e2a2df..73b3ddb 100644 --- a/cli/tests/test_feeders.py +++ b/cli/tests/test_feeders.py @@ -47,6 +47,7 @@ from ffrwd.relay import Relay from ffrwd.split import insert_splits from ffrwd.wasm import WORLDS, Described, Feeder +from tests.conftest import older_world_refusal execute = sys.modules["ffrwd.execute"] lower = sys.modules["ffrwd.lower"] @@ -413,11 +414,11 @@ def test_no_feeder_wires_nothing_and_the_port_keeps_its_default(call: str) -> No def test_only_a_feeder_may_default_to_null_at_the_declaration( declaration: str, needle: str ) -> None: - with pytest.raises(FfrwdError) as caught: - resolve(parse(declaration + "\nCOPY (SELECT f(p.video[1]) FROM input('p.mp4') p) " - "TO 'out.mp4'")) - assert caught.value.code is ErrorCode.UNSUPPORTED_SQL - assert needle in caught.value.message + refusal = older_world_refusal( + declaration + "\nCOPY (SELECT f(p.video[1]) FROM input('p.mp4') p) TO 'out.mp4'" + ) + assert refusal.code is ErrorCode.UNSUPPORTED_SQL + assert needle in refusal.message def test_a_null_default_the_module_reads_as_a_pad_is_refused() -> None: diff --git a/cli/tests/test_lower.py b/cli/tests/test_lower.py index f9f2c1c..1c85f5b 100644 --- a/cli/tests/test_lower.py +++ b/cli/tests/test_lower.py @@ -71,6 +71,7 @@ from ffrwd.warnings import FfrwdWarning, OnWarning, WarningCode from ffrwd.wasm import WORLDS, Described, SourceCatalog, SourceRendition from ffrwd.wasm import SourceTrack as WasmSourceTrack +from tests.conftest import older_world_refusal PROJECT_ROOT = Path(__file__).resolve().parent.parent REPO_ROOT = PROJECT_ROOT.parent @@ -14865,15 +14866,11 @@ def test_only_a_packet_filter_declares_several_annotation_columns() -> None: " RETURNS video_stream\n" f" AS '{ROWS_MODULE}', 'shots' LANGUAGE wasm;\n" ) - with pytest.raises(FfrwdError) as caught: - resolve( - parse( - declare - + "COPY (SELECT blur(f.video[1]) FROM input('f.mp4') f) TO 'o.mp4'" - ) - ) - assert caught.value.code is ErrorCode.UNSUPPORTED_SQL - assert "takes the annotation column 'b' in position 3" in caught.value.message + refusal = older_world_refusal( + declare + "COPY (SELECT blur(f.video[1]) FROM input('f.mp4') f) TO 'o.mp4'" + ) + assert refusal.code is ErrorCode.UNSUPPORTED_SQL + assert "takes the annotation column 'b' in position 3" in refusal.message def test_a_packets_call_inside_a_cte_body_is_refused() -> None: diff --git a/cli/tests/test_node_world.py b/cli/tests/test_node_world.py new file mode 100644 index 0000000..d5fa13a --- /dev/null +++ b/cli/tests/test_node_world.py @@ -0,0 +1,704 @@ +"""Tests for calls to node modules: declarations, shapes, ports and outputs. + +Bare-machine, as tests/test_data_filter.py is: every module is a synthetic +:class:`~ffrwd.wasm.Described` describing a node, every shape is a JSON +document read by :func:`ffrwd.shapes.node_shape` exactly as the sidecar's +``--shape`` answer is, and nothing is spawned. +""" + +from __future__ import annotations + +import functools +import json +from collections.abc import Callable, Mapping, Sequence +from pathlib import Path + +import pytest + +from ffrwd import shapes +from ffrwd.errors import ErrorCode, FfrwdError +from ffrwd.ir import Graph +from ffrwd.lower import lower +from ffrwd.parser import parse, resolve +from ffrwd.probe import ProbeResult, StreamMeta +from ffrwd.registry import Registry, load_reference +from ffrwd.timing import check_live_leads, timing +from ffrwd.wasm import WORLDS, Described + +SNAPSHOT_PATH = Path(__file__).resolve().parent / "data" / "reference_registry.json" + +_ROWS = { + "type": "object", + "properties": { + name: {"type": "number"} for name in ("start_t", "id", "x", "y", "w", "h") + }, +} +_BOX = { + "type": "object", + "properties": {name: {"type": "number"} for name in ("x", "y", "w", "h")}, +} +_CUE = { + "type": "object", + "properties": { + "text": {"type": "string"}, + "start_t": {"type": "number"}, + "end_t": {"type": "number"}, + }, +} + + +def _clock(name: str, kind: str = "video", window: int = 1) -> dict[str, object]: + return { + "name": name, + "kind": kind, + "required": True, + "many": False, + "pairing": {"kind": "lockstep"}, + "rows": "ignore", + "window": window, + "stride": window, + "accepts": {}, + } + + +def _input( + name: str, + kind: str, + pairing: dict[str, object], + *, + required: bool = False, + many: bool = False, + schema: Mapping[str, object] | None = None, +) -> dict[str, object]: + port: dict[str, object] = { + "name": name, + "kind": kind, + "required": required, + "many": many, + "pairing": pairing, + "rows": "per-frame" if kind == "data" else "ignore", + "window": 1, + "stride": 1, + "accepts": {}, + } + if schema is not None: + port["schema"] = json.dumps(schema) + return port + + +def _output( + name: str, kind: str, *, schema: Mapping[str, object] | None = None, latency: float = 0 +) -> dict[str, object]: + port: dict[str, object] = {"name": name, "kind": kind, "latency": latency} + if kind == "data": + port["format"] = {"kind": "data", "codec": "json"} + if schema is not None: + port["schema"] = json.dumps(schema) + return port + + +def _shape( + inputs: list[dict[str, object]], + outputs: list[dict[str, object]], + clock: dict[str, object], + **rest: object, +) -> dict[str, object]: + return { + "inputs": inputs, + "outputs": outputs, + "clock": clock, + "pure": True, + "one_to_one": False, + "bounded": True, + "relation": [], + **rest, + } + + +def _spot(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("v")], [_output("spots", "data", schema=_ROWS)], {"kind": "input", "port": "v"} + ) + + +def _reader(port: str, schema: Mapping[str, object]) -> Callable[..., dict[str, object]]: + def shaped(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("v"), _input(port, "data", {"kind": "lockstep"}, required=True, + schema=schema)], + [_output("v", "video")], + {"kind": "input", "port": "v"}, + ) + + return shaped + + +def _hear(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("a", "audio", window=96000)], + [_output("cues", "data", schema=_CUE)], + {"kind": "input", "port": "a"}, + ) + + +def _burn(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [ + _clock("v"), + _input("a", "audio", {"kind": "lockstep"}), + _input("words", "data", {"kind": "interval", "ahead": 0}, schema=_CUE), + ], + [_output("v", "video")], + {"kind": "input", "port": "v"}, + ) + + +def _inset(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + hold = { + "kind": "hold", + "anchor": {"kind": "first-frame"}, + "lead": params.get("lead", 0.5), + "port_param": "port", + } + return _shape( + [_clock("v"), _input("feed", "video", hold)], + [_output("v", "video")], + {"kind": "input", "port": "v"}, + ) + + +def _tile(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + hold = {"kind": "hold", "anchor": {"kind": "shared-clock"}, "lead": 0} + clock = ( + {"kind": "rate", "rate": {"num": int(str(params["fps"])), "den": 1}} + if "fps" in params + else {"kind": "rate-of", "port": "v"} + ) + return _shape( + [_input("v", "video", hold, required=True, many=True)], + [_output("v", "video")], + clock, + ) + + +def _matte(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("v")], + [_output("mask", "video"), _output("spots", "data", schema=_ROWS)], + {"kind": "input", "port": "v"}, + ) + + +def _ticker(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + canvas = { + "kind": "video", + "width": params.get("width", 1280), + "height": params.get("height", 720), + "pix_fmt": "rgba", + } + return _shape( + [], + [{"name": "v", "kind": "video", "format": canvas, "latency": 0}], + {"kind": "rate", "rate": {"num": int(str(params.get("fps", 30))), "den": 1}}, + bounded=False, + ) + + +SHAPES: dict[str, Callable[..., dict[str, object]]] = { + "spot.wasm": _spot, + "ring.wasm": _reader("spots", _ROWS), + "dim.wasm": _reader("boxes", _BOX), + "hear.wasm": _hear, + "burn.wasm": _burn, + "inset.wasm": _inset, + "tile.wasm": _tile, + "matte.wasm": _matte, + "ticker.wasm": _ticker, +} + +_PARAMS = { + "spot.wasm": {"every": {"type": "number"}}, + "dim.wasm": {"amount": {"type": "number"}}, + "inset.wasm": {"port": {"type": "integer"}, "lead": {"type": "number"}}, + "tile.wasm": {"columns": {"type": "number"}, "fps": {"type": "number"}}, + "matte.wasm": {"every": {"type": "number"}}, + "ticker.wasm": { + name: {"type": "number" if name != "text" else "string"} + for name in ("text", "width", "height", "fps") + }, +} + + +def _node(path: str) -> Described: + return Described( + world="node-module", + name=path.removesuffix(".wasm"), + version="0.1.0", + params_schema={"type": "object", "properties": _PARAMS.get(path, {})}, + node=True, + ) + + +class _Asked: + """A fake `shape` seam that counts what it is asked.""" + + def __init__(self) -> None: + self.asked: list[tuple[str, dict[str, object], tuple[str, ...]]] = [] + + def __call__( + self, module: str, params: str, bound: Sequence[str], grants: Sequence[str] = () + ) -> shapes.NodeShape: + decoded = json.loads(params) + self.asked.append((module, decoded, tuple(bound))) + return shapes.node_shape(module, SHAPES[module](decoded, bound)) + + +_DECLARATIONS = { + "spot": "CREATE FUNCTION spot(v video_stream, every number DEFAULT 30) " + "RETURNS STRUCT(start_t number, id number, x number, y number, w number, h number)[] " + "AS 'spot.wasm', 'spot' LANGUAGE wasm;", + "ring": "CREATE FUNCTION ring(v video_stream, spots STRUCT(start_t number, id number, " + "x number, y number, w number, h number)[]) RETURNS video_stream " + "AS 'ring.wasm', 'ring' LANGUAGE wasm;", + "dim": "CREATE FUNCTION dim(v video_stream, boxes STRUCT(x number, y number, " + "w number, h number)[], amount number DEFAULT 0.5) RETURNS video_stream " + "AS 'dim.wasm', 'dim' LANGUAGE wasm;", + "hear": "CREATE FUNCTION hear(a audio_stream) RETURNS cue[] " + "AS 'hear.wasm', 'hear' LANGUAGE wasm;", + "burn": "CREATE FUNCTION burn(v video_stream, a audio_stream DEFAULT NULL, " + "words cue[] DEFAULT NULL) RETURNS video_stream AS 'burn.wasm', 'burn' LANGUAGE wasm;", + "inset": "CREATE FUNCTION inset(v video_stream, feed video_stream DEFAULT NULL, " + "port number DEFAULT 9000, lead number DEFAULT 0.5) RETURNS video_stream " + "AS 'inset.wasm', 'inset' LANGUAGE wasm;", + "tile": "CREATE FUNCTION tile(v video_stream[], columns number DEFAULT 2, " + "fps number DEFAULT NULL) RETURNS video_stream AS 'tile.wasm', 'tile' LANGUAGE wasm;", + "matte": "CREATE FUNCTION matte(v video_stream, every number DEFAULT 30) " + "RETURNS STRUCT(mask video_stream, spots STRUCT(start_t number, id number, " + "x number, y number, w number, h number)[]) AS 'matte.wasm', 'matte' LANGUAGE wasm;", + "ticker": "CREATE FUNCTION ticker(text text, width number DEFAULT 1280, " + "height number DEFAULT 720, fps number DEFAULT 30) RETURNS source " + "AS 'ticker.wasm', 'ticker' LANGUAGE wasm;", +} + + +def _declared(query: str) -> str: + called = [text for name, text in _DECLARATIONS.items() if f"{name}(" in query] + return "\n".join([*called, query]) + + +@functools.cache +def _registry() -> Registry: + return load_reference(SNAPSHOT_PATH) + + +def _probes() -> dict[str, ProbeResult | None]: + def media() -> ProbeResult: + return ProbeResult( + streams=[ + StreamMeta( + type="video", index=0, metadata={}, width=320, height=240, + fps="25/1", sample_rate=None, codec="h264", + ), + StreamMeta( + type="audio", index=0, metadata={}, width=None, height=None, + fps=None, sample_rate=48000, codec="aac", channels=2, + ), + ] + ) + + return {"f": media(), "a": media(), "b": media(), "c": media()} + + +def _lowered( + query: str, + modules: Mapping[str, Described] | None = None, + asked: _Asked | None = None, +) -> Graph: + return lower( + resolve(parse(_declared(query))), + _probes(), + registry=_registry(), + describes=dict(modules) if modules is not None else {p: _node(p) for p in SHAPES}, + shapes=asked if asked is not None else _Asked(), + ) + + +def _refused(query: str, modules: Mapping[str, Described] | None = None) -> FfrwdError: + with pytest.raises(FfrwdError) as caught: + _lowered(query, modules) + return caught.value + + +_FROM = " FROM input('f.mp4') f) TO 'out.mkv'" + + +# -- the shape document ------------------------------------------------------ + + +def test_a_shape_document_reads_every_field_the_wit_names() -> None: + read = shapes.node_shape( + "m.wasm", + { + "inputs": [ + _clock("v"), + _input( + "feed", + "video", + { + "kind": "hold", + "hold": { + "anchor": {"kind": "tagged", "tagged": "smart_timed"}, + "lead": 0.3, + "linger": 1.5, + "group": "switch", + "port_param": "port", + }, + }, + ), + _input("words", "data", {"kind": "interval", "latency": 2, "ahead": 0.1}, + schema=_CUE), + ], + "outputs": [ + { + "name": "mask", + "kind": "video", + "format": {"kind": "like", "port": "v", "pixel_format": "gray"}, + "latency": 0, + }, + { + "name": "clock", + "kind": "data", + "format": {"kind": "data", "data": "json"}, + "time_base": {"num": 1, "den": 1000000}, + "latency": 0.25, + "row": 0, + }, + ], + "clock": {"kind": "rate", "rate": {"num": 30000, "den": 1001}}, + "pure": False, + "one_to_one": True, + "bounded": False, + "relation": ['{"name": "720p"}'], + }, + ) + feed, words = read.inputs[1], read.inputs[2] + assert feed.pairing.hold == shapes.Hold( + anchor=shapes.Anchor("tagged", "smart_timed"), + lead=0.3, + linger=1.5, + group="switch", + port_param="port", + ) + assert words.pairing.interval == shapes.Interval(latency=2.0, ahead=0.1) + assert words.schema == _CUE + assert read.outputs[0].format == shapes.OutputFormat( + "like", port="v", pixel_format="gray" + ) + assert read.outputs[1].format == shapes.OutputFormat("data", codec="json") + assert (read.outputs[1].time_base, read.outputs[1].latency, read.outputs[1].row) == ( + (1, 1_000_000), + 0.25, + 0, + ) + assert read.clock == shapes.Clock("rate", rate=(30000, 1001)) + assert (read.pure, read.one_to_one, read.bounded) == (False, True, False) + assert read.relation == ({"name": "720p"},) + + +def test_a_shape_document_missing_its_ports_is_refused_naming_the_module() -> None: + with pytest.raises(FfrwdError) as caught: + shapes.node_shape("m.wasm", {"clock": {"kind": "self-clocked"}}) + assert "m.wasm" in caught.value.message + + +@pytest.mark.parametrize( + ("port", "words"), + [ + ((1, 1), "per-frame"), + ((96000, 96000), "tumbling 2 s"), + ((96000, 48000), "hopping 2 s every 1 s"), + ((15, 1), "sliding 0.6 s"), + ], +) +def test_a_window_is_said_in_streaming_words(port: tuple[int, int], words: str) -> None: + window, stride = port + kind = "audio" if window > 100 else "video" + rate = 48000 if kind == "audio" else 25 + read = shapes.node_shape( + "m.wasm", + _shape([{**_clock("x", kind, window), "stride": stride}], [], + {"kind": "input", "port": "x"}), + ) + from fractions import Fraction + + assert shapes.window_words(read.inputs[0], Fraction(rate)) == words + + +def test_rows_match_by_their_fields_and_extra_fields_pass() -> None: + assert shapes.row_mismatch(_BOX, _ROWS) is None + integer = {"type": "object", "properties": {"x": {"type": "integer"}}} + assert shapes.row_mismatch(_BOX, integer | {"properties": { + name: {"type": "integer"} for name in ("x", "y", "w", "h")}}) is None + assert shapes.row_mismatch(integer, _BOX) == ("x", "integer", "number") + assert shapes.row_mismatch(_CUE, _ROWS) == ("text", "string", "nothing") + + +def test_one_shape_is_asked_once_per_module_params_and_bound_ports() -> None: + asked = _Asked() + cache = shapes.ShapeCache(asked) + cache("spot.wasm", "{}", ["v"]) + cache("spot.wasm", "{}", ["v"]) + cache("spot.wasm", '{"every": 5}', ["v"]) + assert [one[1] for one in asked.asked] == [{}, {"every": 5}] + + +# -- declarations ------------------------------------------------------------ + + +def test_a_signature_only_a_node_reads_resolves_and_waits_for_the_module() -> None: + res = resolve(parse(_declared("COPY (SELECT burn(f.video[1], f.audio[1])" + _FROM))) + burn = res.wasm["burn"] + assert burn.is_node_only + assert [port.name for port in burn.ports] == ["v", "a", "words"] + assert burn.refusal is not None + old = Described(world=WORLDS[-1], name="burn", pixel_formats=("rgba",)) + error = _refused("COPY (SELECT burn(f.video[1], f.audio[1])" + _FROM, {"burn.wasm": old}) + assert error.message == burn.refusal.message + + +def test_a_module_of_an_older_world_refuses_a_rows_call_outside_from() -> None: + old = Described(world=WORLDS[-1], name="spot", video_codecs=("h264",)) + error = _refused("COPY (SELECT spot(f.video[1])" + _FROM, {"spot.wasm": old}) + assert "this call is not in FROM" in error.message + + +# -- ports and outputs ------------------------------------------------------- + + +def test_a_detector_returns_rows_and_the_reader_takes_the_picture_from_the_source() -> None: + graph = _lowered("COPY (SELECT ring(f.video[1], spot(f.video[1]))" + _FROM) + nodes = {node.filter: node for node in graph.nodes.values()} + spot, ring = nodes["spot.wasm"], nodes["ring.wasm"] + assert (spot.inputs, spot.ports, spot.outputs, spot.out_ports) == ( + ["src:f:v:0"], ["v"], ["data"], ["spots"] + ) + assert (ring.inputs, ring.ports) == (["src:f:v:0", spot.id], ["v", "spots"]) + assert graph.node_shapes[spot.id]["clock"] == {"kind": "input", "port": "v"} + + +def test_a_reader_naming_a_field_the_producer_lacks_is_refused_naming_both() -> None: + SHAPES["dim.wasm"] = _reader("boxes", { + "type": "object", "properties": {"x": {"type": "number"}, "z": {"type": "number"}}, + }) + try: + error = _refused("COPY (SELECT dim(f.video[1], spot(f.video[1]))" + _FROM) + finally: + SHAPES["dim.wasm"] = _reader("boxes", _BOX) + assert error.message == ( + "dim() reads 'z' as number on its 'boxes' input, and spot() does not write it" + ) + + +def test_one_call_read_twice_is_one_node() -> None: + graph = _lowered( + "COPY (SELECT burn(f.video[1], words => hear(f.audio[1])), f.audio[1], " + "hear(f.audio[1])" + _FROM + ) + hears = [node for node in graph.nodes.values() if node.filter == "hear.wasm"] + assert len(hears) == 1 + (burn,) = [node for node in graph.nodes.values() if node.filter == "burn.wasm"] + assert burn.inputs == ["src:f:v:0", hears[0].id] + assert burn.ports == ["v", "words"] + assert graph.rows_sinks[hears[0].id].container == "webvtt" + + +def test_kinds_mix_in_one_call_and_a_left_out_port_is_unbound() -> None: + asked = _Asked() + graph = _lowered( + "COPY (SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), " + "burn(f.video[1]) AS plain" + _FROM, + asked=asked, + ) + burns = [node for node in graph.nodes.values() if node.filter == "burn.wasm"] + assert [node.ports for node in burns] == [["v", "a", "words"], ["v"]] + assert [one[2] for one in asked.asked if one[0] == "burn.wasm"] == [ + ("v", "a", "words"), + ("v",), + ] + + +def test_a_held_input_left_unbound_keeps_its_port_param() -> None: + graph = _lowered("COPY (SELECT inset(f.video[1], port => 9100)" + _FROM) + (inset,) = [node for node in graph.nodes.values() if node.filter == "inset.wasm"] + assert (inset.ports, inset.args) == (["v"], {"port": 9100, "lead": 0.5}) + + +def test_a_port_number_where_a_held_input_goes_writes_its_port_param() -> None: + graph = _lowered("COPY (SELECT inset(f.video[1], 9200)" + _FROM) + (inset,) = [node for node in graph.nodes.values() if node.filter == "inset.wasm"] + assert (inset.ports, inset.args["port"]) == (["v"], 9200) + + +def test_an_array_fills_a_port_taking_many() -> None: + graph = _lowered( + "COPY (SELECT tile(ARRAY[a.video[1], b.video[1], c.video[1]], 3) " + "FROM input('a.mp4') a, input('b.mp4') b, input('c.mp4') c) TO 'out.mkv'" + ) + (tile,) = [node for node in graph.nodes.values() if node.filter == "tile.wasm"] + assert tile.inputs == ["src:a:v:0", "src:b:v:0", "src:c:v:0"] + assert tile.ports == ["v", "v", "v"] + + +def test_every_field_of_a_struct_return_is_an_output_of_one_node() -> None: + graph = _lowered( + "COPY (WITH m AS (SELECT (matte(f.video[1])).* FROM input('f.mp4') f) " + "SELECT dim(m.mask, m.spots), m.spots FROM m) TO 'out.mkv'" + ) + (matte,) = [node for node in graph.nodes.values() if node.filter == "matte.wasm"] + (dim,) = [node for node in graph.nodes.values() if node.filter == "dim.wasm"] + assert matte.out_ports == ["mask", "spots"] + assert dim.inputs == [f"{matte.id}:0", f"{matte.id}:1"] + assert graph.rows_sinks[f"{matte.id}:1"].container == "webvtt" + + +def test_a_declared_port_the_shape_has_none_for_is_refused_bound() -> None: + SHAPES["burn.wasm"] = lambda params, bound: _shape( + [_clock("v")], [_output("v", "video")], {"kind": "input", "port": "v"} + ) + try: + unbound = _lowered("COPY (SELECT burn(f.video[1])" + _FROM) + error = _refused("COPY (SELECT burn(f.video[1], f.audio[1])" + _FROM) + finally: + SHAPES["burn.wasm"] = _burn + assert any(node.filter == "burn.wasm" for node in unbound.nodes.values()) + assert error.message == ( + "burn() binds 'a', and for these params the module 'burn.wasm' reads no input 'a'" + ) + + +def test_a_port_the_module_requires_is_refused_unbound() -> None: + SHAPES["burn.wasm"] = lambda params, bound: _shape( + [_clock("v"), _input("a", "audio", {"kind": "lockstep"}, required=True)], + [_output("v", "video")], + {"kind": "input", "port": "v"}, + ) + try: + error = _refused("COPY (SELECT burn(f.video[1])" + _FROM) + finally: + SHAPES["burn.wasm"] = _burn + assert error.message == "burn() leaves 'a' out, and the module 'burn.wasm' requires it" + + +def test_a_gather_over_a_nodes_rows_narrows_them_on_the_data_edge() -> None: + graph = _lowered( + "COPY (SELECT dim(f.video[1], ARRAY(SELECT s FROM unnest(spot(f.video[1])) s " + "WHERE s.w > 20))" + _FROM + ) + (spot,) = [node for node in graph.nodes.values() if node.filter == "spot.wasm"] + (narrow,) = [node for node in graph.nodes.values() if node.filter == "rowfilter"] + (dim,) = [node for node in graph.nodes.values() if node.filter == "dim.wasm"] + assert (narrow.inputs, narrow.outputs) == ([spot.id], ["data"]) + assert dim.inputs == ["src:f:v:0", narrow.id] + + +def test_spans_reduce_a_nodes_rows_into_a_rows_file() -> None: + graph = _lowered( + "COPY (SELECT ffrwd.merge_spans(spot(f.video[1]), max_span => 10) " + "FROM input('f.mp4') f) TO 'spots.ndjson'" + ) + (spot,) = [node for node in graph.nodes.values() if node.filter == "spot.wasm"] + (spans,) = [node for node in graph.nodes.values() if node.filter == "rowmerge"] + assert (spans.inputs, spans.args) == ([spot.id], {"merge_spans": True, "max_span": 10}) + assert graph.rows_sinks[spans.id].path == "spots.ndjson" + + +def test_a_node_reading_nothing_is_a_source_in_from() -> None: + graph = _lowered( + "COPY (SELECT s.video[1] FROM ticker('Nothing to see here') s WHERE s.t < 10) " + "TO 'ticker.mp4'" + ) + (ticker,) = [node for node in graph.nodes.values() if node.filter == "ticker.wasm"] + assert (ticker.inputs, ticker.out_ports) == ([], ["v"]) + assert ticker.args == {"text": "Nothing to see here", "width": 1280, "height": 720, + "fps": 30} + assert graph.node_sources == {"s": ticker.id} + assert [output.ref for output in graph.sinks[0].outputs] == [ticker.id] + + +def test_a_sql_function_returning_a_stream_and_rows_hands_a_node_both() -> None: + graph = _lowered( + "CREATE FUNCTION spotted(v video_stream) RETURNS STRUCT(v video_stream, " + "spots STRUCT(start_t number, id number, x number, y number, w number, h number)[]) " + "AS $$ SELECT v, spot(v) AS spots $$ LANGUAGE sql;\n" + "COPY (SELECT ring(spotted(f.video[1]))" + _FROM + ) + (spot,) = [node for node in graph.nodes.values() if node.filter == "spot.wasm"] + (ring,) = [node for node in graph.nodes.values() if node.filter == "ring.wasm"] + assert (ring.inputs, ring.ports) == (["src:f:v:0", spot.id], ["v", "spots"]) + + +def test_a_node_making_a_stream_and_its_rows_hands_a_reader_both() -> None: + graph = _lowered("COPY (SELECT ring(matte(f.video[1]))" + _FROM) + (matte,) = [node for node in graph.nodes.values() if node.filter == "matte.wasm"] + (ring,) = [node for node in graph.nodes.values() if node.filter == "ring.wasm"] + assert ring.inputs == [f"{matte.id}:0", f"{matte.id}:1"] + + +def test_a_struct_a_sql_function_returns_is_a_stream_and_rows_only() -> None: + with pytest.raises(FfrwdError) as caught: + resolve(parse( + "CREATE FUNCTION two(v video_stream) RETURNS STRUCT(a video_stream, " + "b video_stream) AS $$ SELECT v, v $$ LANGUAGE sql;\n" + "COPY (SELECT two(f.video[1]).a" + _FROM + )) + assert caught.value.message == "function 'two' returns a struct that is not a stream and rows" + + +# -- what each node waits for ------------------------------------------------ + + +def test_each_nodes_window_and_each_outputs_delay_add_up_along_the_path() -> None: + graph = _lowered( + "COPY (SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), f.audio[1]" + _FROM + ) + timed = timing(graph, _probes()) + assert timed is not None + by_module = {node.module: node for node in timed.nodes} + assert (by_module["hear.wasm"].window, by_module["hear.wasm"].delay) == ( + "tumbling 2 s", + 2.0, + ) + burn = by_module["burn.wasm"] + assert (burn.window, burn.delay) == ("per-frame", 2.0) + assert [(wait.port, wait.pairing, wait.delay) for wait in burn.waits] == [ + ("v", "clock", 0.0), + ("a", "lockstep", 0.0), + ("words", "interval", 2.0), + ] + picture, sound = timed.outputs + assert (picture.delay, picture.holds) == (2.0, 0.0) + assert (sound.ref, sound.delay, sound.holds, sound.held) == ("src:f:a:0", 0.0, 2.0, 768000) + + +def test_a_live_node_fed_later_than_its_bound_is_refused() -> None: + bounded = {"kind": "interval", "latency": 1.0, "ahead": 0} + + def burn(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("v"), _input("words", "data", bounded, schema=_CUE)], + [_output("v", "video")], + {"kind": "input", "port": "v"}, + ) + + SHAPES["burn.wasm"] = burn + try: + graph = _lowered("COPY (SELECT burn(f.video[1], words => hear(f.audio[1]))" + _FROM) + finally: + SHAPES["burn.wasm"] = _burn + with pytest.raises(FfrwdError) as caught: + check_live_leads(graph, _probes(), {"burn.wasm": (3, 4, "burn")}) + assert caught.value.code is ErrorCode.LIVE_LEAD + assert caught.value.message == ( + "burn() needs 'words' 1 s ahead of its clock, and the path feeding it runs 2 s behind" + ) + assert (caught.value.line, caught.value.col) == (3, 4) diff --git a/cli/tests/test_wasm.py b/cli/tests/test_wasm.py index 399893e..2646883 100644 --- a/cli/tests/test_wasm.py +++ b/cli/tests/test_wasm.py @@ -37,7 +37,7 @@ ) from ffrwd.errors import ErrorCode, FfrwdError from ffrwd.execute import CHAIN, PIPELINE, PipeEdge, plan_argv, render_plan -from ffrwd.functions import WasmFunction, package_modules +from ffrwd.functions import Parameter, WasmFunction, package_modules from ffrwd.ir import ROWFILTER, Graph, RowsSink from ffrwd.lower import lower, lower_table from ffrwd.parser import ModuleExport, Resolved, parse, resolve @@ -193,6 +193,7 @@ def test_a_declaration_rides_out_on_the_resolved_query() -> None: returns="video_stream", line=declared.line, col=declared.col, + outputs=(Parameter("", "video_stream"),), ) assert [(p.name, p.type) for p in declared.params] == [("v", "video_stream")] diff --git a/docs/error-schema.json b/docs/error-schema.json index c5ace01..36f6ecd 100644 --- a/docs/error-schema.json +++ b/docs/error-schema.json @@ -42,6 +42,7 @@ "PLAYER_NOT_FOUND", "RUNTIME_NOT_FOUND", "UNBOUNDED_LIVE_INPUT", + "LIVE_LEAD", "BUFFER_OVERFLOW", "INPUT_NEVER_OPENED", "STARTUP_DEADLOCK", diff --git a/docs/errors.md b/docs/errors.md index 4729375..fd1afc3 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -729,6 +729,14 @@ The third message under this code refuses a live input to a packet sink read at {"line": 5, "col": 38, "code": "UNBOUNDED_LIVE_INPUT", "message": "'live.m3u8' never ends, and a compile-time read reads a stream to the end", "hint": "a live stream is read at run time, by the same module written as a destination: COPY (SELECT ...) TO records()"} ``` +## LIVE_LEAD + +**Meaning:** A node bounds how long it waits for one of its inputs: it acts some time ahead of what that input says (an ad decision announced before the break, a playout that starts its next item early), and its shape says so as the input's `interval.latency`. In a live run what arrives past that bound is late for good. The compiler adds up how far behind the source the path feeding that input runs - every window and declared latency on the way - and refuses the query when that is more than the bound. [Recipe 153](examples.md#153-see-what-each-node-waits-for) shows where the sums are printed. + +**Fires when:** the query has a live input, or a node read in FROM that never ends, and a node's input paired by interval with a bound is fed by a path that runs later than its clock by more than that bound. Never for a file run: nothing there is late, and the node waits for its input as long as it takes. + +The anchor is the declaration of the node that needs the lead. + ## BUFFER_OVERFLOW **Meaning:** Not a compile rejection - the one code a RUN produces. The buffers a plan sized from its bounds were not deep enough, and the pipeline wedged: nothing crossed any pipe of the stage, and no process of it used any CPU, while every one was still alive and one of them was still waiting to hand its bytes over. The CPU is half the test: a stage's pumped pipes are not all the pipes it has, so a process computing over what it read stands still on all of them without being wedged. `ffrwd run` reports it instead of letting the stage sit until the timeout, so the message names the edge, the depth it was given, and how long nothing moved - never a bare "timed out", and never a silently dropped frame. From 6a6a61656dc3e91d00f7037156ad4630d9a61270 Mon Sep 17 00:00:00 2001 From: Jon-Carlos Rivera Date: Thu, 1 Oct 2026 17:19:16 -0700 Subject: [PATCH 04/58] test(modules): the nine node stand-ins the cookbook's node recipes name spot, ring, dim, hear, burn, inset, tile, matte and ticker, written with ffrwd-node 0.1.0 (pinned by tag) and a shared stand-ins crate for the mark finder, drawing and a bitmap font. Each exports ffrwd:av/node at 0.19.0 and is the fixture its recipe compiles against. Co-Authored-By: Claude Fable 5.1 --- sidecar/modules/Cargo.lock | 135 ++++++ sidecar/modules/Cargo.toml | 12 + sidecar/modules/burn/Cargo.toml | 12 + sidecar/modules/burn/src/lib.rs | 208 ++++++++ sidecar/modules/dim/Cargo.toml | 12 + sidecar/modules/dim/src/lib.rs | 143 ++++++ sidecar/modules/hear/Cargo.toml | 12 + sidecar/modules/hear/src/lib.rs | 127 +++++ sidecar/modules/inset/Cargo.toml | 12 + sidecar/modules/inset/src/lib.rs | 149 ++++++ sidecar/modules/matte/Cargo.toml | 12 + sidecar/modules/matte/src/lib.rs | 136 ++++++ sidecar/modules/ring/Cargo.toml | 12 + sidecar/modules/ring/src/lib.rs | 130 +++++ sidecar/modules/spot/Cargo.toml | 12 + sidecar/modules/spot/src/lib.rs | 113 +++++ sidecar/modules/stand-ins/Cargo.toml | 9 + sidecar/modules/stand-ins/src/font.rs | 593 +++++++++++++++++++++++ sidecar/modules/stand-ins/src/lib.rs | 21 + sidecar/modules/stand-ins/src/mark.rs | 160 ++++++ sidecar/modules/stand-ins/src/picture.rs | 232 +++++++++ sidecar/modules/ticker/Cargo.toml | 12 + sidecar/modules/ticker/src/lib.rs | 137 ++++++ sidecar/modules/tile/Cargo.toml | 12 + sidecar/modules/tile/src/lib.rs | 234 +++++++++ 25 files changed, 2647 insertions(+) create mode 100644 sidecar/modules/burn/Cargo.toml create mode 100644 sidecar/modules/burn/src/lib.rs create mode 100644 sidecar/modules/dim/Cargo.toml create mode 100644 sidecar/modules/dim/src/lib.rs create mode 100644 sidecar/modules/hear/Cargo.toml create mode 100644 sidecar/modules/hear/src/lib.rs create mode 100644 sidecar/modules/inset/Cargo.toml create mode 100644 sidecar/modules/inset/src/lib.rs create mode 100644 sidecar/modules/matte/Cargo.toml create mode 100644 sidecar/modules/matte/src/lib.rs create mode 100644 sidecar/modules/ring/Cargo.toml create mode 100644 sidecar/modules/ring/src/lib.rs create mode 100644 sidecar/modules/spot/Cargo.toml create mode 100644 sidecar/modules/spot/src/lib.rs create mode 100644 sidecar/modules/stand-ins/Cargo.toml create mode 100644 sidecar/modules/stand-ins/src/font.rs create mode 100644 sidecar/modules/stand-ins/src/lib.rs create mode 100644 sidecar/modules/stand-ins/src/mark.rs create mode 100644 sidecar/modules/stand-ins/src/picture.rs create mode 100644 sidecar/modules/ticker/Cargo.toml create mode 100644 sidecar/modules/ticker/src/lib.rs create mode 100644 sidecar/modules/tile/Cargo.toml create mode 100644 sidecar/modules/tile/src/lib.rs diff --git a/sidecar/modules/Cargo.lock b/sidecar/modules/Cargo.lock index ca66d37..3d6d346 100644 --- a/sidecar/modules/Cargo.lock +++ b/sidecar/modules/Cargo.lock @@ -231,6 +231,15 @@ version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" +[[package]] +name = "burn" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "bytemuck" version = "1.25.2" @@ -511,6 +520,15 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "dim" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "displaydoc" version = "0.2.7" @@ -522,6 +540,15 @@ dependencies = [ "syn 3.0.4", ] +[[package]] +name = "document-features" +version = "0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4b8a88685455ed29a21542a33abd9cb6510b6b129abadabdcef0f4c55bc8f61" +dependencies = [ + "litrs", +] + [[package]] name = "double" version = "0.1.0" @@ -626,6 +653,18 @@ dependencies = [ "regex-syntax", ] +[[package]] +name = "fast_image_resize" +version = "6.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9c50201dc184ba6553da1695aac20a042efffbe2d84542cee31917c86c3ab1e" +dependencies = [ + "cfg-if", + "document-features", + "num-traits", + "thiserror 2.0.20", +] + [[package]] name = "fauxlate" version = "0.1.0" @@ -657,11 +696,29 @@ dependencies = [ "wit-bindgen 0.57.1", ] +[[package]] +name = "ffrwd-frame" +version = "0.1.0" +source = "git+https://github.com/imbcmdth/ffrwd-frame?tag=v0.1.0#de7db7834977ad93545f1af24345b297df2d670f" +dependencies = [ + "fast_image_resize", +] + [[package]] name = "ffrwd-nal" version = "0.1.0" source = "git+https://github.com/imbcmdth/ffrwd-nal?tag=v0.1.0#bd2ffcf559065b8364369bbc0b24df7220b96eb8" +[[package]] +name = "ffrwd-node" +version = "0.1.0" +source = "git+https://github.com/imbcmdth/ffrwd-node?tag=v0.1.0#84216905037bae949623b6df22febc8fbbc50e17" +dependencies = [ + "serde", + "serde_json", + "wit-bindgen 0.57.1", +] + [[package]] name = "ffrwd-nut" version = "0.1.3" @@ -977,6 +1034,15 @@ dependencies = [ "foldhash", ] +[[package]] +name = "hear" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "heck" version = "0.5.0" @@ -1013,6 +1079,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "inset" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "invert" version = "0.1.0" @@ -1073,6 +1148,12 @@ version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" +[[package]] +name = "litrs" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11d3d7f243d5c5a8b9bb5d6dd2b1602c0cb0b9db1621bafc7ed66e35ff9fe092" + [[package]] name = "log" version = "0.4.34" @@ -1115,6 +1196,15 @@ dependencies = [ "wit-bindgen 0.57.1", ] +[[package]] +name = "matte" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "memchr" version = "2.8.3" @@ -1672,6 +1762,15 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" +[[package]] +name = "ring" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "rms" version = "0.1.0" @@ -1875,12 +1974,30 @@ dependencies = [ "unicode-segmentation", ] +[[package]] +name = "spot" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "stable_deref_trait" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" +[[package]] +name = "stand-ins" +version = "0.1.0" +dependencies = [ + "ffrwd-frame", + "ffrwd-node", + "serde", +] + [[package]] name = "static_assertions" version = "1.1.0" @@ -2011,6 +2128,24 @@ dependencies = [ "syn 3.0.4", ] +[[package]] +name = "ticker" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + +[[package]] +name = "tile" +version = "0.1.0" +dependencies = [ + "ffrwd-node", + "serde", + "stand-ins", +] + [[package]] name = "tokenizers" version = "0.23.1" diff --git a/sidecar/modules/Cargo.toml b/sidecar/modules/Cargo.toml index 9d3b561..7656a56 100644 --- a/sidecar/modules/Cargo.toml +++ b/sidecar/modules/Cargo.toml @@ -66,6 +66,18 @@ members = [ "adapted-0150", "adapted-0160", "adapted-0170", + # The node world's stand-ins, written with ffrwd-node: the modules + # cookbook recipes 145 to 154 name. `stand-ins` is what they share. + "stand-ins", + "spot", + "ring", + "dim", + "hear", + "burn", + "inset", + "tile", + "matte", + "ticker", ] resolver = "2" diff --git a/sidecar/modules/burn/Cargo.toml b/sidecar/modules/burn/Cargo.toml new file mode 100644 index 0000000..f00a7ed --- /dev/null +++ b/sidecar/modules/burn/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "burn" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/burn/src/lib.rs b/sidecar/modules/burn/src/lib.rs new file mode 100644 index 0000000..2317d56 --- /dev/null +++ b/sidecar/modules/burn/src/lib.rs @@ -0,0 +1,208 @@ +//! Burns what it hears and reads onto the picture: a level meter for the +//! sound beside each frame, and the text of every cue showing at the frame's +//! time. The picture is the clock; the sound comes frame for frame and the +//! words by their time, and either may be left out. With neither, the +//! picture leaves untouched. Recipes 147, 148, 149 and 153. + +use ffrwd_node::{ + Bound, Cue, Cues, Init, Input, NoParams, Node, Out, Output, Result, Shape, StateRow, Tick, +}; +use stand_ins::{text_size, Picture, Rect}; + +const METER: [u8; 4] = [64, 220, 64, 255]; +const SHADE: [u8; 4] = [0, 0, 0, 160]; +const TEXT: [u8; 4] = [255, 255, 255, 255]; + +struct Burn { + v: u32, + a: Option<(u32, usize)>, + width: usize, + height: usize, + cues: Cues, +} + +/// The level of `samples` from 0 at -60 dB of full scale or quieter to 1 at +/// full scale. +fn level(samples: &[f32]) -> f64 { + if samples.is_empty() { + return 0.0; + } + let power = samples + .iter() + .map(|s| (*s as f64) * (*s as f64)) + .sum::() + / samples.len() as f64; + let db = 10.0 * power.max(1e-12).log10(); + ((db + 60.0) / 60.0).clamp(0.0, 1.0) +} + +impl Burn { + /// Where the meter's track lies: along the bottom left, a third of the + /// picture wide. + fn track(&self) -> Rect { + let margin = (self.width / 40).max(2); + let tall = (self.height / 40).max(4); + let y1 = self.height.saturating_sub(margin); + Rect { + x0: margin, + y0: y1.saturating_sub(tall), + x1: margin + self.width / 3, + y1, + } + } + + fn meter(&self, picture: &mut Picture, level: f64) { + let track = self.track(); + picture.fill(track, SHADE); + let lit = track.x0 + (track.width() as f64 * level).round() as usize; + picture.fill(Rect { x1: lit, ..track }, METER); + } + + /// `lines` centred above the meter, each on a shaded band. + fn caption(&self, picture: &mut Picture, lines: &[&str]) { + let scale = (self.height / 120).max(1); + let gap = scale * 2; + let (_, line) = text_size("", scale); + let bottom = self.track().y0.saturating_sub(gap * 2); + let mut y = bottom.saturating_sub(lines.len() * (line + gap)); + for text in lines { + let (w, h) = text_size(text, scale); + let x = self.width.saturating_sub(w) / 2; + let band = Rect { + x0: x.saturating_sub(gap), + y0: y.saturating_sub(gap / 2), + x1: x + w + gap, + y1: y + h + gap / 2, + }; + picture.fill(band, SHADE); + picture.text(x as i64, y as i64, scale, text, TEXT); + y += line + gap; + } + } +} + +impl Node for Burn { + const NAME: &'static str = "burn"; + const VERSION: &'static str = "0.1.0"; + type Params = NoParams; + + fn shape(_: &NoParams, _: &Bound) -> Result { + Ok(Shape::new() + .input(Input::video("v").clock().pixel_formats(&["rgba"])) + .input(Input::audio("a").optional().sample_formats(&["f32"])) + .input( + Input::rows("words") + .optional() + .interval() + .state() + .schema::(), + ) + .output(Output::like("v")) + .pure() + .one_to_one()) + } + + fn init(_: NoParams, init: &Init) -> Result { + let v = init.stream("v")?; + let video = v.video_format().ok_or("`v` is a video input")?; + let a = match init.optional("a") { + Some(a) => { + let audio = a.audio_format().ok_or("`a` is an audio input")?; + Some((a.id, audio.channels.max(1) as usize)) + } + None => None, + }; + Ok(Burn { + v: v.id, + a, + width: video.width as usize, + height: video.height as usize, + cues: Cues::new(), + }) + } + + fn fold(&mut self, row: StateRow) -> Result<()> { + self.cues.add(row.row::()?); + Ok(()) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let Some(frame) = tick.frame(self.v) else { + return Ok(()); + }; + let t = tick.time_base().seconds(frame.pts); + self.cues.drop_ended(t); + let lines: Vec<&str> = self.cues.at(t).map(|cue| cue.text.as_str()).collect(); + let sound = self.a.map(|(a, _)| match tick.frame(a) { + Some(heard) => { + let bytes = tick.fetch(a, heard.index); + let (chunks, _) = bytes.as_chunks::<4>(); + let samples: Vec = chunks + .iter() + .map(|chunk| f32::from_le_bytes(*chunk)) + .collect(); + level(&samples) + } + None => 0.0, + }); + if sound.is_none() && lines.is_empty() { + return Ok(out.pass("v", self.v, &frame)?); + } + let mut picture = Picture::new(tick.fetch(self.v, frame.index), self.width, self.height)?; + if let Some(level) = sound { + self.meter(&mut picture, level); + } + self.caption(&mut picture, &lines); + Ok(out.frame("v", frame.pts, frame.duration, picture.data)?) + } +} + +ffrwd_node::export!(Burn); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Payload, Rational}; + + const TB: Rational = Rational::new(1, 10); + + #[test] + fn the_picture_alone_passes() { + let v = BoundStream::video("v", 0, 32, 24, "rgba", TB); + let mut burn = Harness::::new("", vec![v]).unwrap(); + assert_eq!(burn.shape().inputs.len(), 3); + let emitted = burn + .process(&burn.tick(0).frame(0, 0, vec![0; 32 * 24 * 4])) + .unwrap(); + assert!(matches!(emitted.on("v")[..], [Payload::Same { .. }])); + } + + #[test] + fn a_cue_shows_until_it_ends_on_any_worker() { + let bound = vec![ + BoundStream::video("v", 0, 120, 80, "rgba", TB), + BoundStream::rows("words", 1, TB), + ]; + let mut burn = Harness::::new("", bound).unwrap(); + let cue = r#"{"start_t":0.0,"end_t":2.0,"text":"-20 dB"}"#; + let first = burn + .tick(15) + .frame(0, 15, vec![0; 120 * 80 * 4]) + .earlier(1, 0, &[cue]); + let shown = burn.process(&first).unwrap(); + assert!(matches!(shown.on("v")[..], [Payload::Frame { .. }])); + assert_eq!(burn.node().cues.len(), 1); + let after = burn.tick(20).frame(0, 20, vec![0; 120 * 80 * 4]); + let gone = burn.process(&after).unwrap(); + assert!(matches!(gone.on("v")[..], [Payload::Same { .. }])); + assert!(burn.node().cues.is_empty()); + } + + #[test] + fn the_meter_follows_the_sound() { + assert_eq!(level(&[1.0, -1.0]), 1.0); + assert_eq!(level(&[0.0; 4]), 0.0); + assert!((level(&[0.1; 4]) - 2.0 / 3.0).abs() < 1e-6); + } +} diff --git a/sidecar/modules/dim/Cargo.toml b/sidecar/modules/dim/Cargo.toml new file mode 100644 index 0000000..ac8da47 --- /dev/null +++ b/sidecar/modules/dim/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "dim" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/dim/src/lib.rs b/sidecar/modules/dim/src/lib.rs new file mode 100644 index 0000000..4639aa1 --- /dev/null +++ b/sidecar/modules/dim/src/lib.rs @@ -0,0 +1,143 @@ +//! Dims the picture inside every box it is handed by `amount`: 0 leaves it, +//! 1 makes it black. It reads four fields, `{x, y, w, h}`, and any row +//! carrying them will do. Recipes 146 and 151. + +use ffrwd_node::{Bound, Init, Input, Node, Out, Output, Result, Shape, Tick}; +use serde::{Deserialize, Serialize}; +use stand_ins::{Picture, Rect}; + +#[derive(Deserialize)] +struct Params { + amount: f64, +} + +#[derive(Default, Serialize, Deserialize)] +struct Box { + x: f64, + y: f64, + w: f64, + h: f64, +} + +struct Dim { + v: u32, + boxes: u32, + width: usize, + height: usize, + amount: f64, +} + +/// One flag a pixel: inside any of `boxes`. +fn covered(boxes: &[Box], width: usize, height: usize) -> Vec { + let mut inside = vec![false; width * height]; + for found in boxes { + let Some(rect) = Rect::padded(found.x, found.y, found.w, found.h, 0.0, width, height) + else { + continue; + }; + for y in rect.y0..rect.y1 { + inside[y * width + rect.x0..y * width + rect.x1].fill(true); + } + } + inside +} + +impl Node for Dim { + const NAME: &'static str = "dim"; + const VERSION: &'static str = "0.1.0"; + const PARAMS_SCHEMA: &'static str = r#"{"type":"object","properties":{"amount":{"type":"number","minimum":0,"maximum":1,"default":0.5}},"additionalProperties":false}"#; + type Params = Params; + + fn shape(_: &Params, _: &Bound) -> Result { + Ok(Shape::new() + .input(Input::video("v").clock().pixel_formats(&["rgba"])) + .input(Input::rows("boxes").schema::()) + .output(Output::like("v")) + .pure() + .one_to_one()) + } + + fn init(params: Params, init: &Init) -> Result { + let v = init.stream("v")?; + let video = v.video_format().ok_or("`v` is a video input")?; + Ok(Dim { + v: v.id, + boxes: init.stream("boxes")?.id, + width: video.width as usize, + height: video.height as usize, + amount: params.amount, + }) + } + + fn set_params(&mut self, params: Params) -> Result<()> { + self.amount = params.amount; + Ok(()) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let Some(frame) = tick.frame(self.v) else { + return Ok(()); + }; + let boxes: Vec = tick.rows(self.boxes)?; + if boxes.is_empty() || self.amount == 0.0 { + return Ok(out.pass("v", self.v, &frame)?); + } + let mut picture = Picture::new(tick.fetch(self.v, frame.index), self.width, self.height)?; + picture.darken(&covered(&boxes, self.width, self.height), self.amount); + Ok(out.frame("v", frame.pts, frame.duration, picture.data)?) + } +} + +ffrwd_node::export!(Dim); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Payload, Rational}; + + #[test] + fn overlapping_boxes_dim_once() { + let inside = covered( + &[ + Box { + x: 1.0, + y: 1.0, + w: 2.0, + h: 2.0, + }, + Box { + x: 2.0, + y: 2.0, + w: 9.0, + h: 9.0, + }, + ], + 4, + 4, + ); + let count = inside.iter().filter(|inside| **inside).count(); + assert_eq!(count, 3 + 4); + } + + #[test] + fn six_field_rows_are_read_for_their_box() { + let tb = Rational::new(1, 15); + let bound = vec![ + BoundStream::video("v", 0, 4, 4, "rgba", tb), + BoundStream::rows("boxes", 1, tb), + ]; + let mut dim = Harness::::new(r#"{"amount":0.75}"#, bound).unwrap(); + let spot = r#"{"start_t":0,"id":0,"x":0,"y":0,"w":2,"h":1}"#; + let tick = dim + .tick(0) + .frame(0, 0, vec![200; 4 * 4 * 4]) + .message(1, 0, spot.as_bytes()); + let emitted = dim.process(&tick).unwrap(); + let [Payload::Frame { data, .. }] = emitted.on("v")[..] else { + panic!("no frame") + }; + assert_eq!(&data[..4], &[50, 50, 50, 200]); + assert_eq!(&data[8..12], &[200, 200, 200, 200]); + } +} diff --git a/sidecar/modules/hear/Cargo.toml b/sidecar/modules/hear/Cargo.toml new file mode 100644 index 0000000..875fcdc --- /dev/null +++ b/sidecar/modules/hear/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "hear" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/hear/src/lib.rs b/sidecar/modules/hear/src/lib.rs new file mode 100644 index 0000000..76e1ef8 --- /dev/null +++ b/sidecar/modules/hear/src/lib.rs @@ -0,0 +1,127 @@ +//! Listens two seconds of sound at a time and writes one cue a window saying +//! how loud it was: `-23 dB`. A tumbling window whose cues leave with it, so +//! its output's latency is 0. Recipes 147, 148 and 153. + +use ffrwd_node::{Bound, Cue, Init, Input, NoParams, Node, Out, Output, Result, Shape, Tick}; + +const RATE: u32 = 48_000; +const WINDOW: u32 = 2 * RATE; + +struct Hear { + a: u32, + channels: usize, +} + +/// How loud `samples` are, as a cue's text: their RMS in dB of full scale. +fn loudness(samples: &[f32]) -> String { + if samples.is_empty() { + return "silence".to_owned(); + } + let power = samples + .iter() + .map(|s| (*s as f64) * (*s as f64)) + .sum::() + / samples.len() as f64; + let db = 10.0 * power.log10(); + if db < -90.0 { + "silence".to_owned() + } else { + format!("{db:.0} dB") + } +} + +fn samples(bytes: &[u8]) -> Vec { + let (chunks, _) = bytes.as_chunks::<4>(); + chunks + .iter() + .map(|chunk| f32::from_le_bytes(*chunk)) + .collect() +} + +impl Node for Hear { + const NAME: &'static str = "hear"; + const VERSION: &'static str = "0.1.0"; + type Params = NoParams; + + fn shape(_: &NoParams, _: &Bound) -> Result { + Ok(Shape::new() + .input( + Input::audio("a") + .clock() + .window(WINDOW, WINDOW) + .sample_formats(&["f32"]) + .sample_rates(&[RATE]), + ) + .output(Output::rows("cues").schema::()) + .pure()) + } + + fn init(_: NoParams, init: &Init) -> Result { + let a = init.stream("a")?; + let audio = a.audio_format().ok_or("`a` is an audio input")?; + Ok(Hear { + a: a.id, + channels: audio.channels.max(1) as usize, + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let Some(frame) = tick.frame(self.a) else { + return Ok(()); + }; + let samples = samples(&tick.fetch(self.a, frame.index)); + let start = tick.time_base().seconds(frame.pts); + let length = (samples.len() / self.channels) as f64 / RATE as f64; + let cue = Cue::new(start, start + length, loudness(&samples)); + Ok(out.row("cues", frame.pts, &cue)?) + } +} + +ffrwd_node::export!(Hear); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::BoundStream; + + fn bytes(samples: &[f32]) -> Vec { + samples.iter().flat_map(|s| s.to_le_bytes()).collect() + } + + #[test] + fn loudness_in_db_of_full_scale() { + assert_eq!(loudness(&[1.0, -1.0]), "0 dB"); + assert_eq!(loudness(&[0.1; 8]), "-20 dB"); + assert_eq!(loudness(&[0.0; 8]), "silence"); + assert_eq!(loudness(&[]), "silence"); + } + + #[test] + fn a_cue_a_window() { + let a = BoundStream::audio("a", 0, RATE, 2, "f32"); + let mut hear = Harness::::new("", vec![a]).unwrap(); + assert_eq!(hear.shape().inputs[0].window, WINDOW); + let mut cues = Vec::new(); + for (window, length) in [(0, WINDOW), (1, WINDOW), (2, RATE / 2)] { + let pts = (window * WINDOW) as i64; + let tick = hear + .tick(pts) + .frame(0, pts, bytes(&vec![0.1; length as usize * 2])); + let tick = if length < WINDOW { tick.last() } else { tick }; + cues.extend(hear.process(&tick).unwrap().messages("cues")); + } + let read: Vec = cues + .iter() + .map(|(_, json)| ffrwd_node::parse(json).unwrap()) + .collect(); + assert_eq!( + cues.iter().map(|(pts, _)| *pts).collect::>(), + [0, 96_000, 192_000] + ); + assert_eq!(read[1], Cue::new(2.0, 4.0, "-20 dB")); + assert_eq!(read[2], Cue::new(4.0, 4.5, "-20 dB")); + let none = hear.process(&hear.tick(288_000).last()).unwrap(); + assert!(none.items.is_empty()); + } +} diff --git a/sidecar/modules/inset/Cargo.toml b/sidecar/modules/inset/Cargo.toml new file mode 100644 index 0000000..6cda2f1 --- /dev/null +++ b/sidecar/modules/inset/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "inset" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/inset/src/lib.rs b/sidecar/modules/inset/src/lib.rs new file mode 100644 index 0000000..f0297d5 --- /dev/null +++ b/sidecar/modules/inset/src/lib.rs @@ -0,0 +1,149 @@ +//! Shows a feed over the picture, a third of its size in the lower right +//! corner, while the feed has a frame to show; the picture alone otherwise. +//! The feed is a hold input on the loopback port `port` names, shown from +//! `lead` seconds after its first frame arrives, and the host conforms it to +//! the picture's size. Recipe 149. + +use ffrwd_node::{Bound, Init, Input, Node, Out, Output, Result, Shape, Tick}; +use serde::Deserialize; +use stand_ins::{Picture, Rect}; + +const BORDER: [u8; 4] = [255, 255, 255, 255]; + +#[derive(Deserialize)] +struct Params { + lead: f64, +} + +struct Inset { + v: u32, + feed: Option, + width: usize, + height: usize, +} + +/// Where the feed goes on a `width` x `height` picture: a third of it, a +/// fortieth of its width in from the lower right corner. +fn placed(width: usize, height: usize) -> Rect { + let margin = (width / 40).max(1); + let (w, h) = ((width / 3).max(1), (height / 3).max(1)); + let x1 = width.saturating_sub(margin); + let y1 = height.saturating_sub(margin); + Rect { + x0: x1.saturating_sub(w), + y0: y1.saturating_sub(h), + x1, + y1, + } +} + +impl Node for Inset { + const NAME: &'static str = "inset"; + const VERSION: &'static str = "0.1.0"; + const PARAMS_SCHEMA: &'static str = r#"{"type":"object","properties":{"port":{"type":"integer","minimum":0,"maximum":65535,"default":9000},"lead":{"type":"number","minimum":0,"default":0.5}},"additionalProperties":false}"#; + type Params = Params; + + fn shape(params: &Params, _: &Bound) -> Result { + Ok(Shape::new() + .input(Input::video("v").clock().pixel_formats(&["rgba"])) + .input( + Input::video("feed") + .optional() + .hold() + .lead(params.lead) + .port_param("port") + .like("v") + .pixel_formats(&["rgba"]), + ) + .output(Output::like("v")) + .pure() + .one_to_one()) + } + + fn init(_: Params, init: &Init) -> Result { + let v = init.stream("v")?; + let video = v.video_format().ok_or("`v` is a video input")?; + Ok(Inset { + v: v.id, + feed: init.optional("feed").map(|feed| feed.id), + width: video.width as usize, + height: video.height as usize, + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let Some(frame) = tick.frame(self.v) else { + return Ok(()); + }; + let shown = self.feed.and_then(|feed| Some((feed, tick.frame(feed)?))); + let Some((feed, held)) = shown else { + return Ok(out.pass("v", self.v, &frame)?); + }; + let mut picture = Picture::new(tick.fetch(self.v, frame.index), self.width, self.height)?; + let fed = Picture::new(tick.fetch(feed, held.index), self.width, self.height)?; + let at = placed(self.width, self.height); + let border = (self.width / 320).max(1); + picture.fill( + Rect { + x0: at.x0.saturating_sub(border), + y0: at.y0.saturating_sub(border), + x1: at.x1 + border, + y1: at.y1 + border, + }, + BORDER, + ); + picture.put(at.x0, at.y0, &fed.resized(at.width(), at.height())); + Ok(out.frame("v", frame.pts, frame.duration, picture.data)?) + } +} + +ffrwd_node::export!(Inset); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Pairing, Payload, Rational}; + + const TB: Rational = Rational::new(1, 15); + + #[test] + fn the_feed_is_held_on_its_port() { + let v = BoundStream::video("v", 0, 120, 90, "rgba", TB); + let inset = Harness::::new(r#"{"port":9100,"lead":0.25}"#, vec![v]).unwrap(); + let feed = inset.shape().find_input("feed").unwrap(); + let Pairing::Hold(hold) = &feed.pairing else { + panic!("not held") + }; + assert_eq!( + (hold.lead, hold.port_param.as_deref()), + (0.25, Some("port")) + ); + assert_eq!(feed.accepts.like.as_deref(), Some("v")); + } + + #[test] + fn the_feed_shows_in_the_corner_while_it_has_a_frame() { + let bound = vec![ + BoundStream::video("v", 0, 120, 90, "rgba", TB), + BoundStream::video("feed", 1, 120, 90, "rgba", TB), + ]; + let mut inset = Harness::::new("", bound).unwrap(); + let picture = Picture::filled(120, 90, [0, 0, 0, 255]).data; + let fed = Picture::filled(120, 90, [0, 0, 255, 255]).data; + let alone = inset.tick(0).frame(0, 0, picture.clone()); + assert!(matches!( + inset.process(&alone).unwrap().on("v")[..], + [Payload::Same { .. }] + )); + let both = inset.tick(1).frame(0, 1, picture).frame(1, 7, fed); + let emitted = inset.process(&both).unwrap(); + let [Payload::Frame { data, .. }] = emitted.on("v")[..] else { + panic!("no frame") + }; + let at = |x: usize, y: usize| &data[(y * 120 + x) * 4..(y * 120 + x) * 4 + 4]; + let corner = placed(120, 90); + assert_eq!(at(corner.x0 + 5, corner.y0 + 5), [0, 0, 255, 255]); + assert_eq!(at(5, 5), [0, 0, 0, 255]); + } +} diff --git a/sidecar/modules/matte/Cargo.toml b/sidecar/modules/matte/Cargo.toml new file mode 100644 index 0000000..b948a9a --- /dev/null +++ b/sidecar/modules/matte/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "matte" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/matte/src/lib.rs b/sidecar/modules/matte/src/lib.rs new file mode 100644 index 0000000..d5a2ccd --- /dev/null +++ b/sidecar/modules/matte/src/lib.rs @@ -0,0 +1,136 @@ +//! Makes a matte of the mark it finds and a row per mark, both from one +//! node: `mask`, the picture's size in gray, white inside the mark and black +//! elsewhere, and `spots`, the rows `spot` writes. An output the query does +//! not read is not made. Recipe 151. + +use ffrwd_node::{Bound, Init, Input, Node, Out, Output, Result, Shape, Tick}; +use serde::Deserialize; +use stand_ins::{Rgba, Spot, Spotter}; + +#[derive(Deserialize)] +struct Params { + every: u64, +} + +struct Matte { + v: u32, + width: usize, + height: usize, + spotter: Spotter, + mask: bool, + spots: bool, +} + +/// A gray matte of `width` x `height`, white inside `spot`. +fn matte(spot: Option<&Spot>, width: usize, height: usize) -> Vec { + let mut mask = vec![0u8; width * height]; + if let Some(spot) = spot { + let (x0, y0) = (spot.x as usize, spot.y as usize); + let x1 = (x0 + spot.w as usize).min(width); + for y in y0..(y0 + spot.h as usize).min(height) { + mask[y * width + x0.min(x1)..y * width + x1].fill(255); + } + } + mask +} + +impl Node for Matte { + const NAME: &'static str = "matte"; + const VERSION: &'static str = "0.1.0"; + const PARAMS_SCHEMA: &'static str = r#"{"type":"object","properties":{"every":{"type":"integer","minimum":1,"default":30}},"additionalProperties":false}"#; + type Params = Params; + + fn shape(_: &Params, _: &Bound) -> Result { + Ok(Shape::new() + .input(Input::video("v").clock().pixel_formats(&["rgba"])) + .output(Output::video("mask").pixel_format("gray")) + .output(Output::rows("spots").schema::()) + .one_to_one()) + } + + fn init(params: Params, init: &Init) -> Result { + let v = init.stream("v")?; + let video = v.video_format().ok_or("`v` is a video input")?; + Ok(Matte { + v: v.id, + width: video.width as usize, + height: video.height as usize, + spotter: Spotter::new(params.every), + mask: init.latched("mask"), + spots: init.latched("spots"), + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let Some(frame) = tick.frame(self.v) else { + return Ok(()); + }; + let bytes = tick.fetch(self.v, frame.index); + let picture = Rgba::new(&bytes, self.width, self.height)?; + let spot = self + .spotter + .see(tick.time_base().seconds(frame.pts), &picture); + if self.mask { + let mask = matte(spot.as_ref(), self.width, self.height); + out.frame("mask", frame.pts, frame.duration, mask)?; + } + if let Some(spot) = spot.filter(|_| self.spots) { + out.row("spots", frame.pts, &spot)?; + } + Ok(()) + } +} + +ffrwd_node::export!(Matte); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Format, Payload, Rational}; + use stand_ins::{Picture, Rect}; + + #[test] + fn the_mask_follows_the_picture_in_gray() { + let v = BoundStream::video("v", 0, 64, 48, "rgba", Rational::new(1, 15)); + let matte = Harness::::new("", vec![v]).unwrap(); + let mask = matte.shape().find_output("mask").unwrap(); + let like = mask.like.as_ref().unwrap(); + assert_eq!( + (like.port.as_deref(), like.pixel_format.as_deref()), + (Some("v"), Some("gray")) + ); + let spots = matte.shape().find_output("spots").unwrap(); + assert_eq!(spots.format, Some(Format::Data("json".to_owned()))); + } + + #[test] + fn a_mask_and_a_row_from_one_frame() { + let v = BoundStream::video("v", 0, 64, 48, "rgba", Rational::new(1, 15)); + let mut matte = Harness::::new("", vec![v]).unwrap(); + let mut picture = Picture::filled(64, 48, [200, 30, 30, 255]); + picture.fill( + Rect { + x0: 8, + y0: 4, + x1: 24, + y1: 20, + }, + [128, 128, 128, 255], + ); + let emitted = matte + .process(&matte.tick(2).frame(0, 2, picture.data)) + .unwrap(); + let [Payload::Frame { pts: 2, data, .. }] = emitted.on("mask")[..] else { + panic!("no mask") + }; + assert_eq!(data.len(), 64 * 48); + assert_eq!( + (data[4 * 64 + 8], data[4 * 64 + 7], data[20 * 64 + 8]), + (255, 0, 0) + ); + let rows = emitted.messages("spots"); + let spot: Spot = ffrwd_node::parse(&rows[0].1).unwrap(); + assert_eq!((spot.x, spot.y, spot.w, spot.h), (8, 4, 16, 16)); + } +} diff --git a/sidecar/modules/ring/Cargo.toml b/sidecar/modules/ring/Cargo.toml new file mode 100644 index 0000000..d8d419e --- /dev/null +++ b/sidecar/modules/ring/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "ring" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/ring/src/lib.rs b/sidecar/modules/ring/src/lib.rs new file mode 100644 index 0000000..a0a8779 --- /dev/null +++ b/sidecar/modules/ring/src/lib.rs @@ -0,0 +1,130 @@ +//! Draws every six-field row it is handed on the picture the rows came from: +//! a box outline at `x`, `y`, `w`, `h`, its colour picked by `id`. A frame +//! with no rows leaves untouched. Recipe 145. + +use ffrwd_node::{Bound, Init, Input, NoParams, Node, Out, Output, Result, Shape, Tick}; +use serde::{Deserialize, Serialize}; +use stand_ins::{Picture, Rect, PALETTE}; + +/// The row `ring` reads: what `spot` writes, every field a number. +#[derive(Default, Serialize, Deserialize)] +struct Ringed { + start_t: f64, + id: f64, + x: f64, + y: f64, + w: f64, + h: f64, +} + +struct Ring { + v: u32, + spots: u32, + width: usize, + height: usize, +} + +/// The outline's width: two pixels on a small picture, more on a large one. +fn thickness(width: usize) -> usize { + (width / 160).max(2) +} + +impl Node for Ring { + const NAME: &'static str = "ring"; + const VERSION: &'static str = "0.1.0"; + type Params = NoParams; + + fn shape(_: &NoParams, _: &Bound) -> Result { + Ok(Shape::new() + .input(Input::video("v").clock().pixel_formats(&["rgba"])) + .input(Input::rows("spots").schema::()) + .output(Output::like("v")) + .pure() + .one_to_one()) + } + + fn init(_: NoParams, init: &Init) -> Result { + let v = init.stream("v")?; + let video = v.video_format().ok_or("`v` is a video input")?; + Ok(Ring { + v: v.id, + spots: init.stream("spots")?.id, + width: video.width as usize, + height: video.height as usize, + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let Some(frame) = tick.frame(self.v) else { + return Ok(()); + }; + let rows: Vec = tick.rows(self.spots)?; + if rows.is_empty() { + return Ok(out.pass("v", self.v, &frame)?); + } + let mut picture = Picture::new(tick.fetch(self.v, frame.index), self.width, self.height)?; + for row in &rows { + let Some(rect) = Rect::padded(row.x, row.y, row.w, row.h, 0.0, self.width, self.height) + else { + continue; + }; + let colour = PALETTE[row.id.max(0.0) as usize % PALETTE.len()]; + picture.outline(rect, thickness(self.width), colour); + } + Ok(out.frame("v", frame.pts, frame.duration, picture.data)?) + } +} + +ffrwd_node::export!(Ring); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Payload, Rational}; + + fn harness() -> Harness { + let tb = Rational::new(1, 15); + let bound = vec![ + BoundStream::video("v", 0, 32, 24, "rgba", tb), + BoundStream::rows("spots", 1, tb), + ]; + Harness::new("", bound).unwrap() + } + + #[test] + fn a_frame_without_rows_passes_untouched() { + let mut ring = harness(); + let tick = ring + .tick(4) + .frame_with(0, 4, Some(1), &[], vec![0; 32 * 24 * 4]); + let emitted = ring.process(&tick).unwrap(); + assert_eq!( + emitted.on("v"), + [&Payload::Same { + pts: 4, + duration: Some(1), + id: 0, + index: 0 + }] + ); + } + + #[test] + fn a_row_is_outlined_in_its_colour() { + let mut ring = harness(); + let row = r#"{"start_t":0.2,"id":1,"x":4,"y":4,"w":10,"h":8}"#; + let tick = ring + .tick(3) + .frame(0, 3, vec![0; 32 * 24 * 4]) + .message(1, 3, row.as_bytes()); + let emitted = ring.process(&tick).unwrap(); + let [Payload::Frame { pts: 3, data, .. }] = emitted.on("v")[..] else { + panic!("no frame: {emitted:?}") + }; + let at = |x: usize, y: usize| &data[(y * 32 + x) * 4..(y * 32 + x) * 4 + 4]; + assert_eq!(at(4, 4)[..3], PALETTE[1][..3]); + assert_eq!(at(13, 11)[..3], PALETTE[1][..3]); + assert_eq!(at(8, 8), [0, 0, 0, 0]); + } +} diff --git a/sidecar/modules/spot/Cargo.toml b/sidecar/modules/spot/Cargo.toml new file mode 100644 index 0000000..e16727b --- /dev/null +++ b/sidecar/modules/spot/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "spot" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/spot/src/lib.rs b/sidecar/modules/spot/src/lib.rs new file mode 100644 index 0000000..331c51b --- /dev/null +++ b/sidecar/modules/spot/src/lib.rs @@ -0,0 +1,113 @@ +//! Finds a mark in each frame and returns rows alone: one a frame while the +//! mark is in view, `{start_t, id, x, y, w, h}`, every row of one sighting +//! carrying the time it was first seen as `start_t`, and a new sighting +//! every `every` frames. Recipes 145, 146 and 152. + +use ffrwd_node::{Bound, Init, Input, Node, Out, Output, Result, Shape, Tick}; +use serde::Deserialize; +use stand_ins::{Rgba, Spot, Spotter}; + +#[derive(Deserialize)] +struct Params { + every: u64, +} + +struct SpotNode { + v: u32, + width: usize, + height: usize, + spotter: Spotter, +} + +impl Node for SpotNode { + const NAME: &'static str = "spot"; + const VERSION: &'static str = "0.1.0"; + const PARAMS_SCHEMA: &'static str = r#"{"type":"object","properties":{"every":{"type":"integer","minimum":1,"default":30}},"additionalProperties":false}"#; + type Params = Params; + + fn shape(_: &Params, _: &Bound) -> Result { + Ok(Shape::new() + .input(Input::video("v").clock().pixel_formats(&["rgba"])) + .output(Output::rows("spots").schema::())) + } + + fn init(params: Params, init: &Init) -> Result { + let v = init.stream("v")?; + let video = v.video_format().ok_or("`v` is a video input")?; + Ok(SpotNode { + v: v.id, + width: video.width as usize, + height: video.height as usize, + spotter: Spotter::new(params.every), + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + for frame in tick.frames(self.v) { + let bytes = tick.fetch(self.v, frame.index); + let picture = Rgba::new(&bytes, self.width, self.height)?; + let t = tick.time_base().seconds(frame.pts); + if let Some(spot) = self.spotter.see(t, &picture) { + out.row("spots", frame.pts, &spot)?; + } + } + Ok(()) + } +} + +ffrwd_node::export!(SpotNode); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Rational}; + use stand_ins::{Picture, Rect}; + + fn frame(x0: usize) -> Vec { + let mut picture = Picture::filled(64, 48, [200, 30, 30, 255]); + let mark = Rect { + x0, + y0: 8, + x1: x0 + 12, + y1: 20, + }; + picture.fill(mark, [128, 128, 128, 255]); + picture.data + } + + #[test] + fn a_row_a_frame_named_by_the_sighting() { + let v = BoundStream::video("v", 0, 64, 48, "rgba", Rational::new(1, 10)); + let mut spot = Harness::::new(r#"{"every":3}"#, vec![v]).unwrap(); + let mut rows = Vec::new(); + for n in 0..4 { + let tick = spot.tick(n).frame(0, n, frame(4 + n as usize)); + rows.extend(spot.process(&tick).unwrap().messages("spots")); + } + let read: Vec = rows + .iter() + .map(|(_, json)| ffrwd_node::parse(json).unwrap()) + .collect(); + assert_eq!( + rows.iter().map(|(pts, _)| *pts).collect::>(), + [0, 1, 2, 3] + ); + assert_eq!( + read.iter() + .map(|spot| (spot.start_t, spot.id)) + .collect::>(), + [(0.0, 0), (0.0, 0), (0.0, 0), (0.3, 1)] + ); + assert_eq!((read[2].x, read[2].y, read[2].w, read[2].h), (6, 8, 12, 12)); + } + + #[test] + fn no_mark_no_row() { + let v = BoundStream::video("v", 0, 64, 48, "rgba", Rational::new(1, 10)); + let mut spot = Harness::::new("", vec![v]).unwrap(); + let blank = Picture::filled(64, 48, [0, 0, 0, 255]).data; + let emitted = spot.process(&spot.tick(0).frame(0, 0, blank)).unwrap(); + assert!(emitted.items.is_empty()); + } +} diff --git a/sidecar/modules/stand-ins/Cargo.toml b/sidecar/modules/stand-ins/Cargo.toml new file mode 100644 index 0000000..5ef652e --- /dev/null +++ b/sidecar/modules/stand-ins/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "stand-ins" +version = "0.1.0" +edition = "2021" + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +ffrwd-frame = { git = "https://github.com/imbcmdth/ffrwd-frame", tag = "v0.1.0" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/stand-ins/src/font.rs b/sidecar/modules/stand-ins/src/font.rs new file mode 100644 index 0000000..109ae69 --- /dev/null +++ b/sidecar/modules/stand-ins/src/font.rs @@ -0,0 +1,593 @@ +//! A 5 by 9 bitmap font for printable ASCII: capitals and digits seven rows +//! tall, two rows below the baseline for descenders. Each row's five bits +//! run left to right from the highest. + +pub const WIDTH: usize = 5; +pub const HEIGHT: usize = 9; +/// Columns from one character's left edge to the next's. +pub const ADVANCE: usize = WIDTH + 1; + +const BOX: [u8; HEIGHT] = [ + 0b11111, 0b10001, 0b10001, 0b10001, 0b10001, 0b10001, 0b11111, 0b00000, 0b00000, +]; + +/// The rows of `c`'s glyph; a box for a character the font does not have. +pub fn glyph(c: char) -> [u8; HEIGHT] { + GLYPHS + .iter() + .find(|(known, _)| *known == c) + .map_or(BOX, |(_, rows)| *rows) +} + +const GLYPHS: [(char, [u8; HEIGHT]); 95] = [ + ( + ' ', + [ + 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '!', + [ + 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00000, 0b00100, 0b00000, 0b00000, + ], + ), + ( + '"', + [ + 0b01010, 0b01010, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '#', + [ + 0b01010, 0b01010, 0b11111, 0b01010, 0b11111, 0b01010, 0b01010, 0b00000, 0b00000, + ], + ), + ( + '$', + [ + 0b00100, 0b01111, 0b10100, 0b01110, 0b00101, 0b11110, 0b00100, 0b00000, 0b00000, + ], + ), + ( + '%', + [ + 0b11000, 0b11001, 0b00010, 0b00100, 0b01000, 0b10011, 0b00011, 0b00000, 0b00000, + ], + ), + ( + '&', + [ + 0b01100, 0b10010, 0b10100, 0b01000, 0b10101, 0b10010, 0b01101, 0b00000, 0b00000, + ], + ), + ( + '\'', + [ + 0b00100, 0b00100, 0b01000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '(', + [ + 0b00010, 0b00100, 0b01000, 0b01000, 0b01000, 0b00100, 0b00010, 0b00000, 0b00000, + ], + ), + ( + ')', + [ + 0b01000, 0b00100, 0b00010, 0b00010, 0b00010, 0b00100, 0b01000, 0b00000, 0b00000, + ], + ), + ( + '*', + [ + 0b00000, 0b00100, 0b10101, 0b01110, 0b10101, 0b00100, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '+', + [ + 0b00000, 0b00100, 0b00100, 0b11111, 0b00100, 0b00100, 0b00000, 0b00000, 0b00000, + ], + ), + ( + ',', + [ + 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b01100, 0b00100, 0b01000, 0b00000, + ], + ), + ( + '-', + [ + 0b00000, 0b00000, 0b00000, 0b11111, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '.', + [ + 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b01100, 0b01100, 0b00000, 0b00000, + ], + ), + ( + '/', + [ + 0b00000, 0b00001, 0b00010, 0b00100, 0b01000, 0b10000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '0', + [ + 0b01110, 0b10001, 0b10011, 0b10101, 0b11001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '1', + [ + 0b00100, 0b01100, 0b00100, 0b00100, 0b00100, 0b00100, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '2', + [ + 0b01110, 0b10001, 0b00001, 0b00010, 0b00100, 0b01000, 0b11111, 0b00000, 0b00000, + ], + ), + ( + '3', + [ + 0b11111, 0b00010, 0b00100, 0b00010, 0b00001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '4', + [ + 0b00010, 0b00110, 0b01010, 0b10010, 0b11111, 0b00010, 0b00010, 0b00000, 0b00000, + ], + ), + ( + '5', + [ + 0b11111, 0b10000, 0b11110, 0b00001, 0b00001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '6', + [ + 0b00110, 0b01000, 0b10000, 0b11110, 0b10001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '7', + [ + 0b11111, 0b00001, 0b00010, 0b00100, 0b01000, 0b01000, 0b01000, 0b00000, 0b00000, + ], + ), + ( + '8', + [ + 0b01110, 0b10001, 0b10001, 0b01110, 0b10001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '9', + [ + 0b01110, 0b10001, 0b10001, 0b01111, 0b00001, 0b00010, 0b01100, 0b00000, 0b00000, + ], + ), + ( + ':', + [ + 0b00000, 0b01100, 0b01100, 0b00000, 0b01100, 0b01100, 0b00000, 0b00000, 0b00000, + ], + ), + ( + ';', + [ + 0b00000, 0b01100, 0b01100, 0b00000, 0b01100, 0b00100, 0b01000, 0b00000, 0b00000, + ], + ), + ( + '<', + [ + 0b00010, 0b00100, 0b01000, 0b10000, 0b01000, 0b00100, 0b00010, 0b00000, 0b00000, + ], + ), + ( + '=', + [ + 0b00000, 0b00000, 0b11111, 0b00000, 0b11111, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '>', + [ + 0b01000, 0b00100, 0b00010, 0b00001, 0b00010, 0b00100, 0b01000, 0b00000, 0b00000, + ], + ), + ( + '?', + [ + 0b01110, 0b10001, 0b00001, 0b00010, 0b00100, 0b00000, 0b00100, 0b00000, 0b00000, + ], + ), + ( + '@', + [ + 0b01110, 0b10001, 0b00001, 0b01101, 0b10101, 0b10101, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'A', + [ + 0b01110, 0b10001, 0b10001, 0b11111, 0b10001, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'B', + [ + 0b11110, 0b10001, 0b10001, 0b11110, 0b10001, 0b10001, 0b11110, 0b00000, 0b00000, + ], + ), + ( + 'C', + [ + 0b01110, 0b10001, 0b10000, 0b10000, 0b10000, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'D', + [ + 0b11100, 0b10010, 0b10001, 0b10001, 0b10001, 0b10010, 0b11100, 0b00000, 0b00000, + ], + ), + ( + 'E', + [ + 0b11111, 0b10000, 0b10000, 0b11110, 0b10000, 0b10000, 0b11111, 0b00000, 0b00000, + ], + ), + ( + 'F', + [ + 0b11111, 0b10000, 0b10000, 0b11110, 0b10000, 0b10000, 0b10000, 0b00000, 0b00000, + ], + ), + ( + 'G', + [ + 0b01110, 0b10001, 0b10000, 0b10111, 0b10001, 0b10001, 0b01111, 0b00000, 0b00000, + ], + ), + ( + 'H', + [ + 0b10001, 0b10001, 0b10001, 0b11111, 0b10001, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'I', + [ + 0b01110, 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'J', + [ + 0b00111, 0b00010, 0b00010, 0b00010, 0b00010, 0b10010, 0b01100, 0b00000, 0b00000, + ], + ), + ( + 'K', + [ + 0b10001, 0b10010, 0b10100, 0b11000, 0b10100, 0b10010, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'L', + [ + 0b10000, 0b10000, 0b10000, 0b10000, 0b10000, 0b10000, 0b11111, 0b00000, 0b00000, + ], + ), + ( + 'M', + [ + 0b10001, 0b11011, 0b10101, 0b10101, 0b10001, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'N', + [ + 0b10001, 0b10001, 0b11001, 0b10101, 0b10011, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'O', + [ + 0b01110, 0b10001, 0b10001, 0b10001, 0b10001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'P', + [ + 0b11110, 0b10001, 0b10001, 0b11110, 0b10000, 0b10000, 0b10000, 0b00000, 0b00000, + ], + ), + ( + 'Q', + [ + 0b01110, 0b10001, 0b10001, 0b10001, 0b10101, 0b10010, 0b01101, 0b00000, 0b00000, + ], + ), + ( + 'R', + [ + 0b11110, 0b10001, 0b10001, 0b11110, 0b10100, 0b10010, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'S', + [ + 0b01111, 0b10000, 0b10000, 0b01110, 0b00001, 0b00001, 0b11110, 0b00000, 0b00000, + ], + ), + ( + 'T', + [ + 0b11111, 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00000, 0b00000, + ], + ), + ( + 'U', + [ + 0b10001, 0b10001, 0b10001, 0b10001, 0b10001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'V', + [ + 0b10001, 0b10001, 0b10001, 0b10001, 0b10001, 0b01010, 0b00100, 0b00000, 0b00000, + ], + ), + ( + 'W', + [ + 0b10001, 0b10001, 0b10001, 0b10101, 0b10101, 0b10101, 0b01010, 0b00000, 0b00000, + ], + ), + ( + 'X', + [ + 0b10001, 0b10001, 0b01010, 0b00100, 0b01010, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'Y', + [ + 0b10001, 0b10001, 0b01010, 0b00100, 0b00100, 0b00100, 0b00100, 0b00000, 0b00000, + ], + ), + ( + 'Z', + [ + 0b11111, 0b00001, 0b00010, 0b00100, 0b01000, 0b10000, 0b11111, 0b00000, 0b00000, + ], + ), + ( + '[', + [ + 0b01110, 0b01000, 0b01000, 0b01000, 0b01000, 0b01000, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '\\', + [ + 0b00000, 0b10000, 0b01000, 0b00100, 0b00010, 0b00001, 0b00000, 0b00000, 0b00000, + ], + ), + ( + ']', + [ + 0b01110, 0b00010, 0b00010, 0b00010, 0b00010, 0b00010, 0b01110, 0b00000, 0b00000, + ], + ), + ( + '^', + [ + 0b00100, 0b01010, 0b10001, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + '_', + [ + 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b11111, 0b00000, + ], + ), + ( + '`', + [ + 0b01000, 0b00100, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), + ( + 'a', + [ + 0b00000, 0b00000, 0b01110, 0b00001, 0b01111, 0b10001, 0b01111, 0b00000, 0b00000, + ], + ), + ( + 'b', + [ + 0b10000, 0b10000, 0b10110, 0b11001, 0b10001, 0b10001, 0b11110, 0b00000, 0b00000, + ], + ), + ( + 'c', + [ + 0b00000, 0b00000, 0b01110, 0b10000, 0b10000, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'd', + [ + 0b00001, 0b00001, 0b01101, 0b10011, 0b10001, 0b10001, 0b01111, 0b00000, 0b00000, + ], + ), + ( + 'e', + [ + 0b00000, 0b00000, 0b01110, 0b10001, 0b11111, 0b10000, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'f', + [ + 0b00110, 0b01001, 0b01000, 0b11100, 0b01000, 0b01000, 0b01000, 0b00000, 0b00000, + ], + ), + ( + 'g', + [ + 0b00000, 0b00000, 0b01111, 0b10001, 0b10001, 0b10001, 0b01111, 0b00001, 0b01110, + ], + ), + ( + 'h', + [ + 0b10000, 0b10000, 0b10110, 0b11001, 0b10001, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'i', + [ + 0b00100, 0b00000, 0b01100, 0b00100, 0b00100, 0b00100, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'j', + [ + 0b00010, 0b00000, 0b00110, 0b00010, 0b00010, 0b00010, 0b00010, 0b10010, 0b01100, + ], + ), + ( + 'k', + [ + 0b10000, 0b10000, 0b10010, 0b10100, 0b11000, 0b10100, 0b10010, 0b00000, 0b00000, + ], + ), + ( + 'l', + [ + 0b01100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'm', + [ + 0b00000, 0b00000, 0b11010, 0b10101, 0b10101, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'n', + [ + 0b00000, 0b00000, 0b10110, 0b11001, 0b10001, 0b10001, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'o', + [ + 0b00000, 0b00000, 0b01110, 0b10001, 0b10001, 0b10001, 0b01110, 0b00000, 0b00000, + ], + ), + ( + 'p', + [ + 0b00000, 0b00000, 0b11110, 0b10001, 0b10001, 0b10001, 0b11110, 0b10000, 0b10000, + ], + ), + ( + 'q', + [ + 0b00000, 0b00000, 0b01111, 0b10001, 0b10001, 0b10001, 0b01111, 0b00001, 0b00001, + ], + ), + ( + 'r', + [ + 0b00000, 0b00000, 0b10110, 0b11001, 0b10000, 0b10000, 0b10000, 0b00000, 0b00000, + ], + ), + ( + 's', + [ + 0b00000, 0b00000, 0b01111, 0b10000, 0b01110, 0b00001, 0b11110, 0b00000, 0b00000, + ], + ), + ( + 't', + [ + 0b01000, 0b01000, 0b11100, 0b01000, 0b01000, 0b01001, 0b00110, 0b00000, 0b00000, + ], + ), + ( + 'u', + [ + 0b00000, 0b00000, 0b10001, 0b10001, 0b10001, 0b10011, 0b01101, 0b00000, 0b00000, + ], + ), + ( + 'v', + [ + 0b00000, 0b00000, 0b10001, 0b10001, 0b10001, 0b01010, 0b00100, 0b00000, 0b00000, + ], + ), + ( + 'w', + [ + 0b00000, 0b00000, 0b10001, 0b10001, 0b10101, 0b10101, 0b01010, 0b00000, 0b00000, + ], + ), + ( + 'x', + [ + 0b00000, 0b00000, 0b10001, 0b01010, 0b00100, 0b01010, 0b10001, 0b00000, 0b00000, + ], + ), + ( + 'y', + [ + 0b00000, 0b00000, 0b10001, 0b10001, 0b10001, 0b10001, 0b01111, 0b00001, 0b01110, + ], + ), + ( + 'z', + [ + 0b00000, 0b00000, 0b11111, 0b00010, 0b00100, 0b01000, 0b11111, 0b00000, 0b00000, + ], + ), + ( + '{', + [ + 0b00010, 0b00100, 0b00100, 0b01000, 0b00100, 0b00100, 0b00010, 0b00000, 0b00000, + ], + ), + ( + '|', + [ + 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00100, 0b00000, 0b00000, + ], + ), + ( + '}', + [ + 0b01000, 0b00100, 0b00100, 0b00010, 0b00100, 0b00100, 0b01000, 0b00000, 0b00000, + ], + ), + ( + '~', + [ + 0b00000, 0b00000, 0b01000, 0b10101, 0b00010, 0b00000, 0b00000, 0b00000, 0b00000, + ], + ), +]; diff --git a/sidecar/modules/stand-ins/src/lib.rs b/sidecar/modules/stand-ins/src/lib.rs new file mode 100644 index 0000000..7128745 --- /dev/null +++ b/sidecar/modules/stand-ins/src/lib.rs @@ -0,0 +1,21 @@ +//! What the stand-in node modules share: the grey mark they track and the +//! few ways they draw on a picture. They are the modules cookbook recipes +//! 145 to 154 name, and the ffrwd-node SDK's examples. + +pub mod font; +mod mark; +mod picture; + +pub use ffrwd_frame::{Rect, Rgba}; +pub use mark::{find, Spot, Spotter}; +pub use picture::{text_size, Picture}; + +/// Six colours a drawing cycles through by id. +pub const PALETTE: [[u8; 4]; 6] = [ + [255, 64, 64, 255], + [64, 220, 64, 255], + [64, 128, 255, 255], + [255, 200, 0, 255], + [220, 64, 220, 255], + [0, 220, 220, 255], +]; diff --git a/sidecar/modules/stand-ins/src/mark.rs b/sidecar/modules/stand-ins/src/mark.rs new file mode 100644 index 0000000..1822869 --- /dev/null +++ b/sidecar/modules/stand-ins/src/mark.rs @@ -0,0 +1,160 @@ +use ffrwd_frame::{Rect, Rgba}; +use ffrwd_node::Spans; +use serde::{Deserialize, Serialize}; + +/// Fewer pixels than this are noise, not the mark. +const SMALLEST: usize = 32; + +fn grey(pixel: &[u8]) -> bool { + let (r, g, b) = (pixel[0], pixel[1], pixel[2]); + let (lo, hi) = (r.min(g).min(b), r.max(g).max(b)); + hi - lo < 24 && lo > 96 && hi < 160 +} + +/// The mark the stand-ins track: the box around the largest patch of mid +/// grey in the picture, which in ffmpeg's `testsrc2` is the grey shape that +/// grows and shrinks at the lower left. None when no patch is big enough. +pub fn find(frame: &Rgba) -> Option { + let (width, height) = (frame.width, frame.height); + let (pixels, _) = frame.data.as_chunks::<4>(); + let mut open: Vec = pixels.iter().map(|pixel| grey(pixel)).collect(); + let mut best: Option<(usize, Rect)> = None; + let mut stack = Vec::new(); + for start in 0..open.len() { + if !open[start] { + continue; + } + open[start] = false; + stack.push(start); + let mut count = 0; + let (x, y) = (start % width, start / width); + let mut rect = Rect { + x0: x, + y0: y, + x1: x + 1, + y1: y + 1, + }; + while let Some(at) = stack.pop() { + count += 1; + let (x, y) = (at % width, at / width); + rect.x0 = rect.x0.min(x); + rect.y0 = rect.y0.min(y); + rect.x1 = rect.x1.max(x + 1); + rect.y1 = rect.y1.max(y + 1); + let neighbours = [ + (x > 0).then(|| at - 1), + (x + 1 < width).then(|| at + 1), + (y > 0).then(|| at - width), + (y + 1 < height).then(|| at + width), + ]; + for next in neighbours.into_iter().flatten() { + if open[next] { + open[next] = false; + stack.push(next); + } + } + } + if count >= SMALLEST && best.is_none_or(|(most, _)| count > most) { + best = Some((count, rect)); + } + } + best.map(|(_, rect)| rect) +} + +/// One row of the mark: where it is on this frame, and the sighting it +/// belongs to. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct Spot { + /// When this sighting began, in seconds. + pub start_t: f64, + /// The sighting's number, from 0. + pub id: u64, + pub x: u32, + pub y: u32, + pub w: u32, + pub h: u32, +} + +/// The mark followed frame by frame: a sighting lasts while the mark stays +/// in view and at most `every` frames, so a mark always in view is a new +/// sighting every `every` frames. +pub struct Spotter { + spans: Spans<()>, +} + +impl Spotter { + pub fn new(every: u64) -> Spotter { + Spotter { + spans: Spans::new().longest(every), + } + } + + /// The frame at `t` seconds: its row, when the mark is in view. + pub fn see(&mut self, t: f64, frame: &Rgba) -> Option { + self.spans.tick(t); + let rect = find(frame)?; + let span = self.spans.see(()); + Some(Spot { + start_t: span.start_t, + id: span.number, + x: rect.x0 as u32, + y: rect.y0 as u32, + w: rect.width() as u32, + h: rect.height() as u32, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Picture; + + fn with_mark(rect: Rect) -> Picture { + let mut picture = Picture::filled(64, 48, [200, 30, 30, 255]); + picture.fill(rect, [128, 128, 128, 255]); + picture.fill( + Rect { + x0: 60, + y0: 0, + x1: 62, + y1: 2, + }, + [128, 128, 128, 255], + ); + picture + } + + #[test] + fn the_largest_grey_patch_is_the_mark() { + let rect = Rect { + x0: 10, + y0: 20, + x1: 26, + y1: 28, + }; + assert_eq!(find(&with_mark(rect).view()), Some(rect)); + assert_eq!( + find(&Picture::filled(64, 48, [200, 30, 30, 255]).view()), + None + ); + } + + #[test] + fn a_sighting_is_split_every_so_many_frames() { + let picture = with_mark(Rect { + x0: 4, + y0: 4, + x1: 20, + y1: 20, + }); + let mut spotter = Spotter::new(2); + let rows: Vec<(f64, u64)> = (0..5) + .map(|n| { + let spot = spotter.see(n as f64 / 10.0, &picture.view()).unwrap(); + (spot.start_t, spot.id) + }) + .collect(); + assert_eq!(rows, [(0.0, 0), (0.0, 0), (0.2, 1), (0.2, 1), (0.4, 2)]); + } +} diff --git a/sidecar/modules/stand-ins/src/picture.rs b/sidecar/modules/stand-ins/src/picture.rs new file mode 100644 index 0000000..40dbd6b --- /dev/null +++ b/sidecar/modules/stand-ins/src/picture.rs @@ -0,0 +1,232 @@ +use ffrwd_frame::{planes, Filter, Norm, Rect, Rgba}; + +use crate::font; + +/// What `planes` multiplies by to hand back eight-bit values unchanged: +/// scaled to 0..1, then divided by 1/255. +const UNCHANGED: Norm = Norm { + mean: [0.0; 3], + std: [1.0 / 255.0; 3], +}; + +/// An RGBA picture to draw on: four bytes a pixel, row after row, no +/// padding. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Picture { + pub data: Vec, + pub width: usize, + pub height: usize, +} + +impl Picture { + /// The bytes a host handed over for a `width` x `height` rgba frame. + pub fn new(data: Vec, width: usize, height: usize) -> Result { + Rgba::new(&data, width, height)?; + Ok(Picture { + data, + width, + height, + }) + } + + /// An opaque picture of one colour. + pub fn filled(width: usize, height: usize, colour: [u8; 4]) -> Picture { + Picture { + data: colour.repeat(width * height), + width, + height, + } + } + + pub fn view(&self) -> Rgba<'_> { + Rgba { + data: &self.data, + width: self.width, + height: self.height, + } + } + + fn clip(&self, rect: Rect) -> Rect { + let x1 = rect.x1.min(self.width); + let y1 = rect.y1.min(self.height); + Rect { + x0: rect.x0.min(x1), + y0: rect.y0.min(y1), + x1, + y1, + } + } + + /// `rect` painted over, alpha `colour[3]` of 255. + pub fn fill(&mut self, rect: Rect, colour: [u8; 4]) { + let rect = self.clip(rect); + let alpha = colour[3] as u32; + for y in rect.y0..rect.y1 { + let row = + &mut self.data[(y * self.width + rect.x0) * 4..(y * self.width + rect.x1) * 4]; + let (pixels, _) = row.as_chunks_mut::<4>(); + for pixel in pixels { + for channel in 0..3 { + let under = pixel[channel] as u32; + pixel[channel] = + ((colour[channel] as u32 * alpha + under * (255 - alpha) + 127) / 255) + as u8; + } + } + } + } + + /// `rect`'s edge, `thickness` pixels wide, drawn inside it. + pub fn outline(&mut self, rect: Rect, thickness: usize, colour: [u8; 4]) { + let rect = self.clip(rect); + let t = thickness + .min(rect.width().div_ceil(2)) + .min(rect.height().div_ceil(2)); + let band = |x0, y0, x1, y1| Rect { x0, y0, x1, y1 }; + self.fill(band(rect.x0, rect.y0, rect.x1, rect.y0 + t), colour); + self.fill(band(rect.x0, rect.y1 - t, rect.x1, rect.y1), colour); + self.fill(band(rect.x0, rect.y0, rect.x0 + t, rect.y1), colour); + self.fill(band(rect.x1 - t, rect.y0, rect.x1, rect.y1), colour); + } + + /// Every pixel `covered` names, one flag a pixel, darkened by `amount` + /// of its value: 0 leaves it, 1 makes it black. + pub fn darken(&mut self, covered: &[bool], amount: f64) { + let keep = ((1.0 - amount.clamp(0.0, 1.0)) * 256.0).round() as u32; + let (pixels, _) = self.data.as_chunks_mut::<4>(); + for (pixel, _) in pixels + .iter_mut() + .zip(covered) + .filter(|(_, covered)| **covered) + { + for channel in &mut pixel[..3] { + *channel = ((*channel as u32 * keep) >> 8) as u8; + } + } + } + + /// `other` copied in with its top left corner at `x`, `y`, cut to fit. + pub fn put(&mut self, x: usize, y: usize, other: &Picture) { + if x >= self.width || y >= self.height { + return; + } + let w = other.width.min(self.width - x); + for row in 0..other.height.min(self.height - y) { + let to = ((y + row) * self.width + x) * 4; + let from = row * other.width * 4; + self.data[to..to + w * 4].copy_from_slice(&other.data[from..from + w * 4]); + } + } + + /// The picture resized to `width` x `height` with Pillow's bilinear + /// filter, by way of ffrwd-frame. Alpha comes out opaque. + pub fn resized(&self, width: usize, height: usize) -> Picture { + let planar = planes( + &self.view(), + Rect::whole(self.width, self.height), + width, + height, + Filter::Bilinear, + UNCHANGED, + ); + let area = width * height; + let mut data = Vec::with_capacity(area * 4); + for at in 0..area { + for channel in 0..3 { + data.push(planar[channel * area + at].round().clamp(0.0, 255.0) as u8); + } + data.push(255); + } + Picture { + data, + width, + height, + } + } + + /// `text` in the bitmap font, each font pixel `scale` pixels square, + /// its first glyph's top left corner at `x`, `y`; whatever falls off the + /// picture is left out. + pub fn text(&mut self, x: i64, y: i64, scale: usize, text: &str, colour: [u8; 4]) { + let scale = scale.max(1); + let step = (font::ADVANCE * scale) as i64; + for (n, c) in text.chars().enumerate() { + let left = x + n as i64 * step; + if left >= self.width as i64 { + break; + } + if left + step <= 0 { + continue; + } + for (row, bits) in font::glyph(c).iter().enumerate() { + for column in 0..font::WIDTH { + if bits & (1 << (font::WIDTH - 1 - column)) == 0 { + continue; + } + let px = left + (column * scale) as i64; + let py = y + (row * scale) as i64; + if px + scale as i64 <= 0 || py + scale as i64 <= 0 { + continue; + } + let (x0, y0) = (px.max(0) as usize, py.max(0) as usize); + let x1 = (px + scale as i64).max(0) as usize; + let y1 = (py + scale as i64).max(0) as usize; + self.fill(Rect { x0, y0, x1, y1 }, colour); + } + } + } + } +} + +/// How wide and tall `text` stands in the bitmap font at `scale`. +pub fn text_size(text: &str, scale: usize) -> (usize, usize) { + let count = text.chars().count(); + let width = (count * font::ADVANCE).saturating_sub(1) * scale.max(1); + (width, font::HEIGHT * scale.max(1)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn fill_blends_by_alpha() { + let mut picture = Picture::filled(4, 4, [0, 0, 0, 255]); + picture.fill(Rect::whole(2, 2), [255, 255, 255, 255]); + picture.fill( + Rect { + x0: 2, + y0: 0, + x1: 9, + y1: 1, + }, + [200, 100, 0, 128], + ); + assert_eq!(&picture.data[..4], &[255, 255, 255, 255]); + assert_eq!(&picture.data[8..12], &[100, 50, 0, 255]); + assert_eq!(&picture.data[12..16], &[100, 50, 0, 255]); + assert_eq!(&picture.data[16 * 2..16 * 2 + 4], &[0, 0, 0, 255]); + } + + #[test] + fn resized_keeps_a_flat_colour() { + let picture = Picture::filled(8, 6, [10, 120, 250, 255]); + let half = picture.resized(4, 3); + assert_eq!((half.width, half.height), (4, 3)); + assert!(half + .data + .chunks(4) + .all(|pixel| pixel == [10, 120, 250, 255])); + } + + #[test] + fn text_lands_where_it_says() { + let mut picture = Picture::filled(20, 12, [0, 0, 0, 255]); + picture.text(1, 1, 1, "I", [255, 255, 255, 255]); + let lit = |x: usize, y: usize| picture.data[(y * 20 + x) * 4] == 255; + assert!(lit(2, 1) && lit(3, 1) && lit(4, 1)); + assert!(lit(3, 4) && !lit(2, 4)); + assert_eq!(text_size("Ii", 2), (22, 18)); + picture.text(-100, 1, 1, "far off", [255, 0, 0, 255]); + } +} diff --git a/sidecar/modules/ticker/Cargo.toml b/sidecar/modules/ticker/Cargo.toml new file mode 100644 index 0000000..37d01e3 --- /dev/null +++ b/sidecar/modules/ticker/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "ticker" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/ticker/src/lib.rs b/sidecar/modules/ticker/src/lib.rs new file mode 100644 index 0000000..311d0e4 --- /dev/null +++ b/sidecar/modules/ticker/src/lib.rs @@ -0,0 +1,137 @@ +//! A line of text crossing a `width` x `height` canvas from right to left, a +//! quarter of the canvas a second, at `fps`. A source: no inputs, a rate +//! clock, one relation row, and it never ends by itself. Recipe 154. + +use ffrwd_node::{Bound, Init, Node, Out, Output, Rational, Result, Shape, Tick}; +use serde::Deserialize; +use stand_ins::{text_size, Picture}; + +const BACKGROUND: [u8; 4] = [16, 24, 48, 255]; +const TEXT: [u8; 4] = [255, 255, 255, 255]; + +#[derive(Deserialize)] +struct Params { + text: String, + width: u32, + height: u32, + fps: f64, +} + +struct Ticker { + text: String, + width: usize, + height: usize, +} + +impl Ticker { + fn scale(&self) -> usize { + (self.height / 90).max(1) + } + + /// Where the text's left edge stands `seconds` in: it enters at the + /// right edge and comes round again once it has left at the left. + fn left(&self, seconds: f64) -> i64 { + let (wide, _) = text_size(&self.text, self.scale()); + let lap = (self.width + wide) as f64; + let travelled = (seconds * self.width as f64 / 4.0).rem_euclid(lap); + self.width as i64 - travelled.floor() as i64 + } +} + +impl Node for Ticker { + const NAME: &'static str = "ticker"; + const VERSION: &'static str = "0.1.0"; + const PARAMS_SCHEMA: &'static str = r#"{"type":"object","properties":{"text":{"type":"string"},"width":{"type":"integer","minimum":16,"maximum":8192,"default":1280},"height":{"type":"integer","minimum":16,"maximum":8192,"default":720},"fps":{"type":"number","exclusiveMinimum":0,"maximum":240,"default":30}},"required":["text"],"additionalProperties":false}"#; + type Params = Params; + + fn shape(params: &Params, _: &Bound) -> Result { + let (width, height) = (params.width, params.height); + Ok(Shape::new() + .rate(Rational::approximate(params.fps, 1001)) + .output( + Output::video("video") + .size(width, height) + .pixel_format("rgba") + .row(0), + ) + .relation_row(&format!(r#"{{"width":{width},"height":{height}}}"#)) + .bounded(false) + .pure()) + } + + fn init(params: Params, _: &Init) -> Result { + Ok(Ticker { + text: params.text, + width: params.width as usize, + height: params.height as usize, + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let mut canvas = Picture::filled(self.width, self.height, BACKGROUND); + let scale = self.scale(); + let (_, tall) = text_size(&self.text, scale); + let top = (self.height.saturating_sub(tall) / 2) as i64; + canvas.text(self.left(tick.seconds()), top, scale, &self.text, TEXT); + Ok(out.frame("video", tick.pts(), Some(1), canvas.data)?) + } +} + +ffrwd_node::export!(Ticker); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{Clock, Format, Payload, VideoFormat}; + + #[test] + fn a_source_at_its_rate() { + let ticker = Harness::::new(r#"{"text":"Nothing to see here"}"#, vec![]).unwrap(); + let shape = ticker.shape(); + assert_eq!(shape.clock, Some(Clock::Rate(Rational::new(30, 1)))); + assert!(!shape.bounded && shape.inputs.is_empty()); + assert_eq!(shape.relation, [r#"{"width":1280,"height":720}"#]); + assert!(matches!( + shape.outputs[0].format, + Some(Format::Video(VideoFormat { + width: 1280, + height: 720, + .. + })) + )); + assert!(Harness::::new("", vec![]).is_err()); + } + + #[test] + fn the_text_crosses_from_the_right() { + let ticker = Ticker { + text: "abc".to_owned(), + width: 400, + height: 90, + }; + assert_eq!(ticker.left(0.0), 400); + assert_eq!(ticker.left(1.0), 300); + assert_eq!(ticker.left(4.0), 0); + let lap = (400 + text_size("abc", 1).0) as f64 / 100.0; + assert!((ticker.left(lap + 1.0) - ticker.left(1.0)).abs() <= 1); + } + + #[test] + fn a_frame_a_tick() { + let mut ticker = + Harness::::new(r#"{"text":"hi","width":64,"height":32,"fps":10}"#, vec![]) + .unwrap(); + let emitted = ticker.process(&ticker.tick(25)).unwrap(); + let [Payload::Frame { + pts: 25, + duration: Some(1), + data, + }] = emitted.on("video")[..] + else { + panic!("no frame") + }; + assert_eq!(data.len(), 64 * 32 * 4); + assert!(data.chunks(4).any(|pixel| pixel == TEXT)); + } +} diff --git a/sidecar/modules/tile/Cargo.toml b/sidecar/modules/tile/Cargo.toml new file mode 100644 index 0000000..152c65e --- /dev/null +++ b/sidecar/modules/tile/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "tile" +version = "0.1.0" +edition = "2021" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +ffrwd-node = { git = "https://github.com/imbcmdth/ffrwd-node", tag = "v0.1.0" } +stand-ins = { path = "../stand-ins" } +serde = { version = "1.0.229", features = ["derive"] } diff --git a/sidecar/modules/tile/src/lib.rs b/sidecar/modules/tile/src/lib.rs new file mode 100644 index 0000000..6999d43 --- /dev/null +++ b/sidecar/modules/tile/src/lib.rs @@ -0,0 +1,234 @@ +//! Lays any number of pictures out in a grid of `columns`, on a `width` x +//! `height` canvas, each scaled to fit its cell. It ticks at the first +//! picture's rate, or at `fps` when the call gives one, and shows each +//! picture's newest frame, so the pictures need not share a rate or a start. +//! Recipe 150. + +use ffrwd_node::{Anchor, Bound, Init, Input, Node, Out, Output, Rational, Result, Shape, Tick}; +use serde::Deserialize; +use stand_ins::{Picture, Rect}; + +const BLACK: [u8; 4] = [0, 0, 0, 255]; + +#[derive(Deserialize)] +struct Params { + columns: usize, + fps: Option, + width: u32, + height: u32, +} + +struct Tiled { + id: u32, + width: usize, + height: usize, + at: Rect, +} + +struct Tile { + tiles: Vec, + width: usize, + height: usize, +} + +/// The cells of a grid of `count` pictures in `columns` across a `width` x +/// `height` canvas, in reading order. +fn cells(count: usize, columns: usize, width: usize, height: usize) -> Vec { + let columns = columns.max(1); + let rows = count.div_ceil(columns).max(1); + (0..count) + .map(|n| { + let (column, row) = (n % columns, n / columns); + Rect { + x0: column * width / columns, + y0: row * height / rows, + x1: (column + 1) * width / columns, + y1: (row + 1) * height / rows, + } + }) + .collect() +} + +/// The largest `width` x `height` picture that fits `cell` keeping its +/// shape, centred in it. +fn fit(width: usize, height: usize, cell: Rect) -> Rect { + let (cw, ch) = (cell.width(), cell.height()); + let (w, h) = if width * ch <= height * cw { + ((width * ch / height.max(1)).max(1), ch) + } else { + (cw, (height * cw / width.max(1)).max(1)) + }; + let x0 = cell.x0 + (cw - w.min(cw)) / 2; + let y0 = cell.y0 + (ch - h.min(ch)) / 2; + Rect { + x0, + y0, + x1: x0 + w, + y1: y0 + h, + } +} + +impl Node for Tile { + const NAME: &'static str = "tile"; + const VERSION: &'static str = "0.1.0"; + const PARAMS_SCHEMA: &'static str = r#"{"type":"object","properties":{"columns":{"type":"integer","minimum":1,"default":2},"fps":{"type":["number","null"],"exclusiveMinimum":0},"width":{"type":"integer","minimum":16,"default":1280},"height":{"type":"integer","minimum":16,"default":720}},"additionalProperties":false}"#; + type Params = Params; + + fn shape(params: &Params, _: &Bound) -> Result { + let shape = Shape::new() + .input( + Input::video("v") + .many() + .hold() + .anchor(Anchor::SharedClock) + .pixel_formats(&["rgba"]), + ) + .output( + Output::video("v") + .size(params.width, params.height) + .pixel_format("rgba"), + ) + .pure(); + Ok(match params.fps { + Some(fps) => shape.rate(Rational::approximate(fps, 1001)), + None => shape.rate_of("v"), + }) + } + + fn init(params: Params, init: &Init) -> Result { + let (width, height) = (params.width as usize, params.height as usize); + let streams = init.streams("v"); + let at = cells(streams.len(), params.columns, width, height); + let tiles = streams + .iter() + .zip(at) + .map(|(stream, cell)| { + let video = stream + .video_format() + .ok_or_else(|| format!("picture {} of `v` has no size", stream.id))?; + let (w, h) = (video.width as usize, video.height as usize); + Ok(Tiled { + id: stream.id, + width: w, + height: h, + at: fit(w, h, cell), + }) + }) + .collect::, String>>()?; + Ok(Tile { + tiles, + width, + height, + }) + } + + fn process(&mut self, tick: &Tick, out: &mut Out) -> Result<()> { + let mut canvas = Picture::filled(self.width, self.height, BLACK); + let mut shown = false; + for tile in &self.tiles { + let Some(frame) = tick.frame(tile.id) else { + continue; + }; + let picture = Picture::new(tick.fetch(tile.id, frame.index), tile.width, tile.height)?; + canvas.put( + tile.at.x0, + tile.at.y0, + &picture.resized(tile.at.width(), tile.at.height()), + ); + shown = true; + } + if !shown && tick.last() { + return Ok(()); + } + Ok(out.frame("v", tick.pts(), Some(1), canvas.data)?) + } +} + +ffrwd_node::export!(Tile); + +#[cfg(test)] +mod tests { + use super::*; + use ffrwd_node::mock::Harness; + use ffrwd_node::{BoundStream, Clock, Payload}; + + #[test] + fn a_grid_of_cells_in_reading_order() { + let grid = cells(3, 2, 1280, 720); + assert_eq!( + grid.iter() + .map(|cell| (cell.x0, cell.y0, cell.x1, cell.y1)) + .collect::>(), + [(0, 0, 640, 360), (640, 0, 1280, 360), (0, 360, 640, 720)] + ); + let row = cells(3, 3, 1280, 720); + assert_eq!((row[2].x0, row[2].x1, row[2].y1), (853, 1280, 720)); + } + + #[test] + fn a_picture_fits_its_cell_keeping_its_shape() { + let cell = Rect { + x0: 0, + y0: 0, + x1: 426, + y1: 720, + }; + let placed = fit(320, 240, cell); + assert_eq!((placed.width(), placed.height()), (426, 319)); + assert_eq!((placed.x0, placed.y0), (0, 200)); + let wide = Rect { + x0: 640, + y0: 0, + x1: 1280, + y1: 360, + }; + assert_eq!( + fit(320, 240, wide), + Rect { + x0: 720, + y0: 0, + x1: 1200, + y1: 360 + } + ); + } + + #[test] + fn the_rate_is_the_first_picture_s_unless_fps_says() { + let pictures = |count: u32| { + (0..count) + .map(|id| BoundStream::video("v", id, 32, 24, "rgba", Rational::new(1, 15360))) + .collect::>() + }; + let follows = Harness::::new("", pictures(2)).unwrap(); + assert_eq!(follows.shape().clock, Some(Clock::RateOf("v".to_owned()))); + let fixed = Harness::::new(r#"{"fps":29.97}"#, pictures(2)).unwrap(); + assert_eq!( + fixed.shape().clock, + Some(Clock::Rate(Rational::new(2997, 100))) + ); + } + + #[test] + fn each_picture_lands_in_its_cell() { + let bound = (0..3) + .map(|id| BoundStream::video("v", id, 16, 12, "rgba", Rational::new(1, 15))) + .collect(); + let mut tile = Harness::::new(r#"{"columns":2,"width":64,"height":48}"#, bound) + .unwrap() + .clock(Rational::new(1, 15)); + let red = Picture::filled(16, 12, [255, 0, 0, 255]).data; + let blue = Picture::filled(16, 12, [0, 0, 255, 255]).data; + let tick = tile.tick(5).frame(0, 77, red).frame(2, 3, blue); + let emitted = tile.process(&tick).unwrap(); + let [Payload::Frame { pts: 5, data, .. }] = emitted.on("v")[..] else { + panic!("no frame") + }; + let at = |x: usize, y: usize| &data[(y * 64 + x) * 4..(y * 64 + x) * 4 + 4]; + assert_eq!(at(16, 12), [255, 0, 0, 255]); + assert_eq!(at(48, 12), [0, 0, 0, 255]); + assert_eq!(at(16, 36), [0, 0, 255, 255]); + let ended = tile.process(&tile.tick(6).last()).unwrap(); + assert!(ended.items.is_empty()); + } +} From e3e687e9f905aad68c0db74f3f7c04fc50f59032 Mon Sep 17 00:00:00 2001 From: Jon-Carlos Rivera Date: Thu, 1 Oct 2026 18:15:53 -0700 Subject: [PATCH 05/58] feat(compiler): node networks lower to the sidecar's node spelling, with delays explained A region holding a node is a node network: pads bind ports by name, unlatched outputs are left off, every stream between the region and another process rides one NUT, a region takes several inputs and outputs, and the wire format of each edge comes from the ports it feeds, through a split where one stream feeds two nodes. Nodes with no inputs are sources, ended by their reader's -to; packets outputs and inputs are wired; ffrwd.merge_spans is the host's rowmerge with max_span; params past 4 KB go by -params-from; a hold port fed by a port number is listed under the command. explain's JSON gains "timing" and explain --delays prints each node's window in streaming words and each output's delay behind the source; compile sizes edge buffers from the same sums, warns past 512 MiB held (HELD_STREAM) and refuses a live lead that cannot be met (LIVE_LEAD). dialect.md carries the node world's rules. Recipes 145 to 154 are pinned from real compile output against the nine stand-ins and a node-aware sidecar; 151 no longer writes its rows as a track, and 153's prose names where the sound waits. Co-Authored-By: Claude Fable 5.1 --- cli/ffrwd/cli.py | 23 +- cli/ffrwd/compiler.py | 11 +- cli/ffrwd/emit.py | 102 +++++++- cli/ffrwd/execute.py | 105 +++++++- cli/ffrwd/functions.py | 5 +- cli/ffrwd/ir.py | 22 +- cli/ffrwd/lower.py | 202 +++++++++++---- cli/ffrwd/macros.py | 5 +- cli/ffrwd/processes.py | 486 +++++++++++++++++++++++++++++++---- cli/ffrwd/shapes.py | 22 +- cli/ffrwd/startup.py | 12 +- cli/ffrwd/timing.py | 112 ++++++-- cli/ffrwd/wasm.py | 87 ++++++- cli/tests/test_examples.py | 21 +- cli/tests/test_node_world.py | 413 ++++++++++++++++++++++++++++- docs/dialect.md | 99 +++++++ docs/examples.md | 111 ++++++-- 17 files changed, 1647 insertions(+), 191 deletions(-) diff --git a/cli/ffrwd/cli.py b/cli/ffrwd/cli.py index 525544a..aeddf5c 100644 --- a/cli/ffrwd/cli.py +++ b/cli/ffrwd/cli.py @@ -191,7 +191,19 @@ from pathlib import Path from typing import Any -from . import binaries, credentials, diagram, loudnorm, nn, redact, remote, show, store, wasm +from . import ( + binaries, + credentials, + diagram, + loudnorm, + nn, + redact, + remote, + show, + store, + timing, + wasm, +) from . import packages as packages_module from . import publish as publish_module from . import registry as registry_module @@ -541,6 +553,11 @@ def _build_parser() -> argparse.ArgumentParser: action="store_true", help="render the flowchart in the terminal (needs the diagram extra)", ) + explain_view.add_argument( + "--delays", + action="store_true", + help="print each node's window and how far behind the source each output runs", + ) validate_p = subparsers.add_parser("validate", help="check that a query compiles") _add_query_arguments(validate_p) _add_quiet_argument(validate_p) @@ -1474,6 +1491,10 @@ def _cmd_explain(args: argparse.Namespace, on_warning: OnWarning) -> int: return 1 graphs = compiled.graphs + if args.delays: + if compiled.timing is not None: + print(timing.summary(compiled.timing)) + return 0 if args.mermaid or args.diagram: text = diagram.render_diagram(graphs, compiled.plan) if args.mermaid: diff --git a/cli/ffrwd/compiler.py b/cli/ffrwd/compiler.py index 1602fc7..99b1eb6 100644 --- a/cli/ffrwd/compiler.py +++ b/cli/ffrwd/compiler.py @@ -706,7 +706,7 @@ def compile_all( for lateral in ready[0].laterals ], ) - timed = timing(ready[0], probes) + timed = timing(ready[0], probes, {m: a[2] for m, a in _module_anchors(res).items()}) if timed is not None: if _runs_live(res, probes, ready[0]): check_live_leads(ready[0], probes, _module_anchors(res)) @@ -715,20 +715,23 @@ def compile_all( span = _run_duration(ready, _probed_paths(res, probes)) stream_wasm = _stream_wasm(res) hosted = _hosted_wasm(res) + sourced = {ready[0].nodes[name].filter for name in ready[0].node_sources.values()} leaky = any(node.filter == LEAKY for node in ready[0].nodes.values()) - if not hosted and not ready[0].module_sources and not leaky: + if not hosted and not ready[0].module_sources and not leaky and not sourced: return Compiled( graphs=ready, default_timeout=budget, duration=span, timing=timed ) try: plan = partition( ready[0], - external=external_filters(*sorted({d.module for d in hosted.values()})), + external=external_filters( + *sorted({d.module for d in hosted.values()} | sourced) + ), probes=probes, pix_fmts=_wire_formats(stream_wasm, describes), shapes=_module_shapes(stream_wasm, describes), audio_wires=_audio_wires(stream_wasm, describes), - models=_nn_models(hosted, describes, packages), + models=_nn_models(hosted | _source_wasm(res), describes, packages), effects=_effect_grants(stream_wasm | _source_wasm(res), describes), anchors=res.input_anchors, ) diff --git a/cli/ffrwd/emit.py b/cli/ffrwd/emit.py index 7cc8d9d..90fd6b2 100644 --- a/cli/ffrwd/emit.py +++ b/cli/ffrwd/emit.py @@ -425,7 +425,8 @@ class OutputGroup: each takes an output stream index after every map of this group. `wire` is the stream format set when this group writes a pipe edge rather than a file; its codec and pixel format ride `options`, and what an audio - edge is conformed to is rendered off it. + edge is conformed to is rendered off it. `wires` is every stream's, in + map order, where the edge is one NUT carrying several. """ maps: list[OutputMap] @@ -437,6 +438,7 @@ class OutputGroup: metadata: int | None = None attachments: list[Attachment] = field(default_factory=list) wire: StreamFormat | None = None + wires: tuple[StreamFormat, ...] = () @dataclass @@ -924,7 +926,7 @@ def build_process_args( g: Graph, *, pipe_inputs: Sequence[tuple[str, str]] = (), - pipe_outputs: Sequence[tuple[str, StreamFormat]] = (), + pipe_outputs: Sequence[tuple[str, StreamFormat | Sequence[StreamFormat]]] = (), pipe_buffers: Sequence[EdgeBuffer | None] = (), pipe_live: Sequence[bool] = (), live: bool = False, @@ -1006,15 +1008,27 @@ def build_process_args( f"process writes {len(pipes)} pipes but {len(pipe_outputs)} were wired" ) groups = list(e.groups) - for slot, (index, (spelling, wire)) in enumerate(zip(pipes, pipe_outputs)): + for slot, (index, (spelling, carried)) in enumerate(zip(pipes, pipe_outputs)): group = groups[index] buffer = pipe_buffers[slot] if slot < len(pipe_buffers) else None live = pipe_live[slot] if slot < len(pipe_live) else False + wires: tuple[StreamFormat, ...] = ( + (carried,) + if isinstance(carried, VideoFormat | AudioFormat | DataFormat) + else tuple(carried) + ) + written = _bundle_options(wires, buffer, live=live) + # The container comes after every stream's codec, as one wire's does. + for key in (FIFO_FORMAT, QUEUE_SIZE, _FORMAT): + if key in written: + written[key] = written.pop(key) + wire = next((one for one in wires if isinstance(one, AudioFormat)), wires[0]) groups[index] = replace( group, path=spelling, wire=wire, - options={**_wire_options(wire, buffer, live=live), **group.options}, + wires=wires if len(wires) > 1 else (), + options={**written, **group.options}, ) return build_ffmpeg_args(replace(e, groups=groups)) @@ -1048,6 +1062,48 @@ def build_network_graph( return e.filter_complex, [m.target for group in e.groups for m in group.maps] +def build_node_network( + g: Graph, *, pipe_inputs: Sequence[str] = () +) -> tuple[str, list[list[str]]]: + """:func:`build_network_graph` for a region holding node modules. + + Its sinks are NUTs of several streams each, so the ``-map`` targets come + back grouped one list per sink, in sink order. + """ + slots = [index for index, path in enumerate(g.input_paths) if path == PIPE] + if len(slots) != len(pipe_inputs): + raise _internal( + f"network reads {len(slots)} pipes but {len(pipe_inputs)} were wired" + ) + paths = list(g.input_paths) + for slot, spelling in zip(slots, pipe_inputs): + paths[slot] = spelling + e = emit(replace(g, input_paths=paths), network=True) + return e.filter_complex, [[m.target for m in group.maps] for group in e.groups] + + +def _bundle_options( + wires: Sequence[StreamFormat], buffer: EdgeBuffer | None, *, live: bool +) -> dict[str, object]: + """The sink options writing every stream one edge carries. + + Streams of one kind share an option where they agree on it; where they + differ it is given per track, in map order, as a WITH list is. + """ + written: dict[str, object] = {} + kinds: dict[type, list[dict[str, object]]] = {} + for one in wires: + options = _wire_options(one, buffer, live=live) + written.update(options) + kinds.setdefault(type(one), []).append(options) + for tracks in kinds.values(): + for key in dict.fromkeys(key for options in tracks for key in options): + values = [options.get(key) for options in tracks] + if any(value != values[0] for value in values): + written[key] = values + return written + + def _render_conformance(group: OutputGroup) -> list[str]: """``-ar:``/``-ac:`` for each audio stream a wire edge constrains. @@ -1055,13 +1111,16 @@ def _render_conformance(group: OutputGroup) -> list[str]: and ahead of the codec that follows it. A module naming neither a rate nor a channel count renders nothing and the stream is left alone. """ - wire = group.wire - if not isinstance(wire, AudioFormat): + if not isinstance(group.wire, AudioFormat): return [] + audio = [one for one in group.wires if isinstance(one, AudioFormat)] or [group.wire] args: list[str] = [] + taken = 0 for index, mapping in enumerate(group.maps): if mapping.type != "audio": continue + wire = audio[min(taken, len(audio) - 1)] + taken += 1 if wire.required_rate is not None: args += [f"{SAMPLE_RATE_FLAG}:{index}", str(wire.required_rate)] if wire.required_channels is not None: @@ -1750,20 +1809,45 @@ def _render_chain( network: bool = False, ) -> str: head, tail = chain[0], chain[-1] + if network and (head.ports or head.out_ports): + return _render_node(head, g, labels, measure) prefix = "".join(f"[{_input_label(g, ref, labels, network)}]" for ref in head.inputs) body = ",".join(_render_filter(node, measure) for node in chain) suffix = "".join(f"[{labels[f'{tail.id}:{pad}']}]" for pad in range(pads[tail.id])) return f"{prefix}{body}{suffix}" +def _render_node(node: Node, g: Graph, labels: dict[str, str], measure: bool) -> str: + """A node module's chain: each pad names the port it binds. + + An output nothing in the network or its maps reads is left off, which + is how the node is told the query does not latch it. + """ + read = {_slot(ref) for other in g.nodes.values() for ref in other.inputs} + read |= {_slot(output.ref) for output in g.outputs if not is_src(output.ref)} + prefix = "".join( + f"[{port}={_input_label(g, ref, labels, True)}]" + for port, ref in zip(node.ports, node.inputs) + ) + suffix = "".join( + f"[{port}={labels[f'{node.id}:{pad}']}]" + for pad, port in enumerate(node.out_ports) + if f"{node.id}:{pad}" in read + ) + return f"{prefix}{_render_filter(node, measure)}{suffix}" + + def _input_label( g: Graph, ref: FrameRef, labels: dict[str, str], network: bool = False ) -> str: if is_src(ref): spec = _src_spec(g, ref) - # A network input carries one stream, so its per-type index says - # nothing and the subset grammar leaves it off. - return spec.rpartition(":")[0] if network else spec + # A network input's first stream of a kind is that kind alone; the + # subset grammar spells an index past it, which only a NUT carrying + # several streams of one kind has. + if network and spec.endswith(":0"): + return spec.rpartition(":")[0] + return spec slot = _slot(ref) label = labels.get(slot) if label is None: diff --git a/cli/ffrwd/execute.py b/cli/ffrwd/execute.py index 90e4cad..bf98226 100644 --- a/cli/ffrwd/execute.py +++ b/cli/ffrwd/execute.py @@ -146,6 +146,7 @@ from .errors import ErrorCode, FfrwdError from .ir import ( FEEDER_HOST, + PARAMS_FILE, STDERR_ROW, Lateral, LateralValue, @@ -166,6 +167,8 @@ StreamEdge, VideoFormat, encoded, + once_per_pipe, + pipe_key, ) from .relay import Relay, RelayEdge from .vars import substitute @@ -637,6 +640,10 @@ def _stop_player(player: subprocess.Popen[bytes]) -> None: # placeholders as they are, which is what a printed command shows. RowsNamer = Callable[[str], str] +# Writes one node's params, as JSON, to the file its placeholder stands for, +# and names that file. +ParamsNamer = Callable[[str, str], str] + # Renders one sidecar process as the argv that runs it, given the path each # stream it reads arrives on and the path each rows document it writes goes # to. The real one lands with the sidecar itself; until then a caller @@ -970,9 +977,10 @@ def _pipe_edges(plan: ProcessPlan) -> tuple[PipeEdge, ...]: The order is what pairs an edge with the ``pipe:`` slot it fills: a reading process's own inputs come first in its ``-i`` list, and a rows - track is one of those, where a frame edge is appended after them. + track is one of those, where a frame edge is appended after them. The + streams riding one NUT are one pipe, which the first of them stands for. """ - return (*plan.rows_edges, *plan.stream_edges) + return (*plan.rows_edges, *once_per_pipe(plan.stream_edges)) def plan_argv( @@ -982,6 +990,7 @@ def plan_argv( pipe_path: PipeNamer | None = None, rows_path: RowsNamer | None = None, apart: Callable[[PipeEdge], bool] | None = None, + params_path: ParamsNamer | None = None, ) -> dict[str, list[str]]: """The argv that runs each process of `plan`, keyed by process id. @@ -1001,6 +1010,10 @@ def plan_argv( `apart` says which edges a placed run cuts between nodes; each is named at both ends (:func:`wires`). + + `params_path` writes a node's params to a file and names it, for each + params placeholder; without it the placeholders stay, as a printed + command shows them. """ read: dict[PipeEdge, str] = {} write: dict[PipeEdge, str] = {} @@ -1011,17 +1024,22 @@ def plan_argv( write[wire.edge] = ( STDOUT if wire.write_stdio else _named(pipe_path, wire.edge, "write") ) + first = {pipe_key(edge): edge for edge in reversed(plan.stream_edges)} + for edge in plan.stream_edges: + if edge.nut: + read[edge] = read[first[edge.nut]] + write[edge] = write[first[edge.nut]] argv: dict[str, list[str]] = {} for process in plan.processes: - incoming = _once_per_ref( - [e for e in plan.stream_edges if e.target == process.id] - ) - outgoing = [e for e in plan.stream_edges if e.source == process.id] + incoming = _once_per_ref([e for e in plan.stream_edges if e.target == process.id]) + carried = [e for e in plan.stream_edges if e.source == process.id] + outgoing = once_per_pipe(carried) if isinstance(process, SidecarProcess): # A sidecar's reads are its pads, in the order its module takes # them, whatever order the startup walk put the edges in. incoming.sort(key=lambda edge: _pad_of(process, edge)) + incoming = once_per_pipe(incoming) argv[process.id] = _sidecar_args( process, sidecar_argv, @@ -1030,18 +1048,25 @@ def plan_argv( len(outgoing), ) continue + incoming = once_per_pipe(incoming) rows_in = _rows_inputs(process, plan) argv[process.id] = build_process_args( process.graph, pipe_inputs=[(read[edge], edge.container) for edge in rows_in] + [(read[edge], edge.format.container) for edge in incoming], - pipe_outputs=[(write[edge], edge.format) for edge in outgoing], + pipe_outputs=[ + ( + write[edge], + tuple(one.format for one in carried if pipe_key(one) == pipe_key(edge)), + ) + for edge in outgoing + ], pipe_buffers=[edge.buffer for edge in outgoing], pipe_live=[edge.live for edge in outgoing], live=any(edge.live for edge in incoming), copyts=all(keeps_clock(edge, plan) for edge in incoming), ) - return _resolve_rows_documents(argv, rows_path) + return _resolve_params_files(_resolve_rows_documents(argv, rows_path), plan, params_path) def _pad_of(process: SidecarProcess, edge: StreamEdge) -> int: @@ -1080,6 +1105,32 @@ def keeps_clock(edge: StreamEdge, plan: ProcessPlan) -> bool: ) +def _resolve_params_files( + argv: dict[str, list[str]], plan: ProcessPlan, params_path: ParamsNamer | None +) -> dict[str, list[str]]: + """Each node's params placeholder replaced by the file its params were + written to; unchanged without a namer.""" + if params_path is None: + return argv + graphs = { + process.id: process.graph + for process in plan.processes + if isinstance(process, SidecarProcess) and process.graph is not None + } + + def resolve(token: str) -> str: + name, sep, rest = token.partition("=") + if not sep or not rest.startswith(PARAMS_FILE): + return token + pid, _, node = rest[len(PARAMS_FILE) :].partition(":") + graph = graphs.get(pid) + if graph is None or node not in graph.nodes: + return token + return f"{name}={params_path(rest, json.dumps(graph.nodes[node].args, sort_keys=True))}" + + return {pid: [resolve(token) for token in args] for pid, args in argv.items()} + + def _resolve_rows_documents( argv: dict[str, list[str]], rows_path: RowsNamer | None ) -> dict[str, list[str]]: @@ -1118,7 +1169,12 @@ def _sidecar_writes( order it reads them, and then its rows documents. Everything else hands its frames on over one output: its stdout where the edge chains, named here only where it does not, the pipe the relay serves.""" - several = process.packet_source or process.packet_filter or process.data_filter + several = ( + process.packet_source + or process.packet_filter + or process.data_filter + or process.node_network + ) if several: streams = [write[edge] for edge in outgoing] else: @@ -1181,7 +1237,7 @@ def render_plan( run = plan_argv(plan, sidecar_argv=sidecar_argv, pipe_path=pipe_path or _placeholder_pipe) argv = {pid: redact.argv(words) for pid, words in run.items()} if _is_pipeline(plan) and not plan.feeder_edges: - return _render_pipeline(plan, argv) + return "\n".join([_render_pipeline(plan, argv), *_listen_lines(plan)]) return _render_listing(plan, argv) @@ -1234,6 +1290,7 @@ def _render_listing(plan: ProcessPlan, argv: Mapping[str, list[str]]) -> str: for index, process in enumerate(plan.processes, start=1) ] lines += _feeder_lines(plan) + lines += _listen_lines(plan) lines += _lateral_lines(plan) lines.append(_COURTESY_NOTE) return "\n".join(lines) @@ -1261,6 +1318,17 @@ def _feeder_lines(plan: ProcessPlan) -> list[str]: ] +def _listen_lines(plan: ProcessPlan) -> list[str]: + """One line per port a process listens on for an input a node holds: + whatever connects there is shown, and the query writes nothing to it.""" + return [ + f"# listens: {process.id} at {feeder_path(port)} for {name}({port_name})" + for process in plan.processes + if isinstance(process, SidecarProcess) + for port, name, port_name in process.listens + ] + + def _lateral_lines(plan: ProcessPlan) -> list[str]: """Each run-time lateral as a block of its own: the data stream it is started from, the connections its instances write, what binds each value @@ -1454,6 +1522,12 @@ def rows_path(placeholder: str) -> str: name = placeholder.rpartition(":")[2] or "0" return str(workspace() / f"rows-{name}.ndjson") + def params_path(placeholder: str, content: str) -> str: + # A node's params, written once where the run keeps its files. + path = workspace() / f"params-{len(list(workspace().glob('params-*')))}.json" + path.write_text(content, encoding="utf-8") + return str(path) + def pipe_path(edge: PipeEdge, side: Side) -> str: # Named here, made by the relay when the edge's stage starts. path = pipes.path(workspace(), str(len(named))) @@ -1465,6 +1539,7 @@ def pipe_path(edge: PipeEdge, side: Side) -> str: sidecar_argv=sidecar_argv, pipe_path=pipe_path, rows_path=rows_path, + params_path=params_path, ) assigned = wires(plan) terminal = terminal_member(plan) if work is not None else None @@ -1606,7 +1681,10 @@ def _sidecar_args( hint="pass sidecar_argv, which renders one sidecar process as argv", ) if streams > 1 and not ( - process.packet_source or process.packet_filter or process.data_filter + process.packet_source + or process.packet_filter + or process.data_filter + or process.node_network ): raise FfrwdError( ErrorCode.INTERNAL, @@ -1616,7 +1694,10 @@ def _sidecar_args( "spell a named pipe path", ) if len(reads) > 1 and not ( - process.packet_sink or process.packet_filter or process.data_filter + process.packet_sink + or process.packet_filter + or process.data_filter + or process.node_network ): raise FfrwdError( ErrorCode.INTERNAL, diff --git a/cli/ffrwd/functions.py b/cli/ffrwd/functions.py index 18698d5..d876cef 100644 --- a/cli/ffrwd/functions.py +++ b/cli/ffrwd/functions.py @@ -2113,8 +2113,11 @@ def _define_wasm( if found is None: raise return found - if declared.is_value or declared.is_codec or declared.is_sink or declared.is_packets: + if declared.is_value or declared.is_codec or declared.is_sink: return declared + if declared.is_packets: + # A node hands back the coded stream it was given, of that kind. + return replace(declared, outputs=(Parameter("", params[0].type),)) outputs = _node_outputs(node, name, identifier) return declared if outputs is None else replace(declared, outputs=outputs) diff --git a/cli/ffrwd/ir.py b/cli/ffrwd/ir.py index 9907725..42a3d29 100644 --- a/cli/ffrwd/ir.py +++ b/cli/ffrwd/ir.py @@ -110,6 +110,12 @@ ROWS_DOCUMENT = "ffrwd:rows:" +# Where a node's params go when they are too long for a command line, or not +# a flat list of values: ``ffrwd:params::``. A placeholder +# like a rows document's, which a run resolves to a file of the params. +PARAMS_FILE = "ffrwd:params:" + + def is_rows_document(path: str) -> bool: """True for a path that is one of those placeholders.""" return path.startswith(ROWS_DOCUMENT) @@ -1055,10 +1061,7 @@ def from_dict(cls, d: dict[str, object]) -> Graph: for alias, bounds in raw_input_trims.items(): assert isinstance(bounds, list) start, end = bounds - input_trims[str(alias)] = ( - float(start) if start is not None else None, - float(end) if end is not None else None, - ) + input_trims[str(alias)] = (_seconds(start), _seconds(end)) raw_input_options = d.get("input_options") input_options: dict[str, dict[str, object]] = {} @@ -1196,6 +1199,17 @@ def from_dict(cls, d: dict[str, object]) -> Graph: ) +def _seconds(value: object) -> float | None: + """A written bound as it was: a whole number stays one, so it reads back + the way it was written.""" + if value is None: + return None + if isinstance(value, int) and not isinstance(value, bool): + return value + assert isinstance(value, int | float | str) + return float(value) + + _MergeKey = tuple[str, tuple[tuple[str, object], ...]] diff --git a/cli/ffrwd/lower.py b/cli/ffrwd/lower.py index 624d465..f84a52c 100644 --- a/cli/ffrwd/lower.py +++ b/cli/ffrwd/lower.py @@ -425,7 +425,15 @@ from ffrwd.probe import probe as probe_one_path from ffrwd.processes import CLOCK_SIZE, COPY_CODEC, NUT, RAWVIDEO, ref_type from ffrwd.registry import DynamicFilter, FilterOption, Registry, SourceFilter -from ffrwd.shapes import InputPort, NodeShape, Shape, ShapeCache, row_mismatch +from ffrwd.shapes import ( + InputPort, + NodeShape, + OutputPort, + Shape, + ShapeCache, + node_shape, + row_mismatch, +) from ffrwd.sink import ( CODEC_PARAMS_FLAGS, COLOR_OPTIONS, @@ -3911,21 +3919,25 @@ def _node_source_probe(shape: NodeShape) -> ProbeResult: streams: list[StreamMeta] = [] by_row: dict[int, list[StreamMeta]] = {} for output in shape.outputs: - index = counted.get(output.kind, 0) - counted[output.kind] = index + 1 + kind = _output_kind(shape, output, {}) + index = counted.get(kind, 0) + counted[kind] = index + 1 found = output.format - video = found is not None and found.kind == "video" - audio = found is not None and found.kind == "audio" + coded = found is not None and found.kind == "packets" + video = found is not None and (found.kind == "video" or coded) and kind == "video" + audio = found is not None and (found.kind == "audio" or coded) and kind == "audio" stream = StreamMeta( - type=cast(StreamType, output.kind), + type=kind, index=index, metadata={}, width=found.width if found is not None and video else None, height=found.height if found is not None and video else None, - fps=fps if output.kind == "video" else None, + fps=fps if kind == "video" else None, sample_rate=found.sample_rate if found is not None and audio else None, - codec=_NODE_SOURCE_CODECS.get( - (found.sample_format if found is not None and audio else None) or output.kind + codec=found.codec + if found is not None and coded + else _NODE_SOURCE_CODECS.get( + (found.sample_format if found is not None and audio else None) or kind ), channels=found.channels if found is not None and audio else None, ) @@ -3964,6 +3976,29 @@ def _node_source_probe(shape: NodeShape) -> ProbeResult: } +def _output_kind( + shape: NodeShape, output: OutputPort, bound: Mapping[str, StreamType] +) -> StreamType: + """The kind of stream a node's output is in the graph. + + Coded packets are the kind they carry: what the coded stream says, or, + where the output follows an input's format, what is bound there. + """ + if output.kind != "packets": + return cast(StreamType, output.kind) + found = output.format + if found is not None and found.kind == "packets" and found.coded is not None: + return found.coded + follows = ( + found.port + if found is not None and found.kind == "like" + else shape.clock.port + if found is None and shape.clock.kind == "input" + else None + ) + return bound.get(follows or "", "video") + + def _text_or_none(value: object) -> str | None: return value if isinstance(value, str) else None @@ -8835,6 +8870,7 @@ def _add_node_source( lowering ends (:meth:`_place_node_sources`). A source that never ends reads as a live input does. """ + self._check_node_export(declared, described, inner, select) params = self._wasm_params(declared, described, call, inner, select, env, {}, first=0) shape = self._node_shape(declared, described, params, [], inner, select) required = next((port for port in shape.inputs if port.required), None) @@ -8848,21 +8884,11 @@ def _add_node_source( hint="a source reads nothing; a node reading a stream is called " "over that stream in the SELECT list", ) - packets = next((o for o in shape.outputs if o.kind == "packets"), None) - if packets is not None: - raise _error( - ErrorCode.UNSUPPORTED_SQL, - f"the module '{declared.module}' writes packets on its " - f"'{packets.name}' output, which a node source cannot hand on yet", - inner, - fallback=select, - hint="a node source writing coded packets is not wired yet", - ) result = _node_source_probe(shape) self.probes[alias] = result env.bindings[alias] = _InputBinding(alias=alias) self._bind_renditions(alias, join, env, select) - kinds = [cast(StreamType, output.kind) for output in shape.outputs] + kinds = [_output_kind(shape, output, {}) for output in shape.outputs] ref = self.ctx.node(declared.module, params, [], kinds) self.graph.nodes[ref].out_ports = [output.name for output in shape.outputs] self.graph.node_shapes[ref] = dict(shape.raw) @@ -10267,7 +10293,7 @@ def _split_where( if isinstance(env.bindings.get(alias), _RowBinding) or _reads_cte_value(alias, conjunct, env) } - if not rows: + if not rows or self._ends_node_source(conjunct, aliases): time_conjuncts.append(conjunct) continue if aliases - rows and len(rows) == 1 and self._is_row_window(conjunct, env): @@ -10288,6 +10314,21 @@ def _split_where( row_conjuncts.append(conjunct) return time_conjuncts, row_conjuncts, assertion_conjuncts + def _ends_node_source(self, conjunct: exp.Expr, aliases: set[str]) -> bool: + """Whether `conjunct` bounds the time of a node read in FROM: its + relation rows are renditions, and its time is the node's own.""" + parsed = _time_bounds(conjunct) + if parsed is None: + return False + column = parsed[0] + table = column.args.get("table") + return ( + table is not None + and aliases == {_fold(table)} + and _fold(table) in self.graph.node_sources + and _fold(column.this) == TIME_COLUMN + ) + def _check_row_window_seeks_a_file( self, conjunct: exp.Expr, where: exp.Where, env: _Env ) -> None: @@ -11610,7 +11651,9 @@ def _collect_trims( self._window_of(alias, low_node, high_node, env, row, select) for row in rows ] - if isinstance(env.bindings[alias], _InputBinding): + if isinstance(env.bindings[alias], _InputBinding) or ( + alias in self.graph.node_sources + ): if any( opt.name == "seek_end" for opt in self.res.input_options.get(alias, ()) @@ -15814,8 +15857,7 @@ def _node_output( "is not a stream", node, fallback=select, - hint=f"read one output off the call, {declared.name}(...){fields.split(',')[0]}: " - f"it writes {fields}" + hint=f"read one output off the call: it writes {fields}" if fields else declared.signature, ) @@ -15848,7 +15890,18 @@ def _lower_node_expr( call = _call_parts(base) assert call is not None # an Anonymous always splits output = self._node_output(declared, field_name, node, select) + if declared.is_packets: + self._check_packets_position(declared, base, select) + made = len(self.graph.nodes) instances, broadcast = self._node_instances(base, declared, call, env, select) + if declared.is_packets: + # Each new instance reads what the destination encodes, which the + # destination settles once its options are known. + self.packet_filter_calls.extend( + (instance.ref, declared, base, select) + for instance in instances + if instance.ref in list(self.graph.nodes)[made:] + ) streams = tuple( _Stream(ref=ref, type=kind) for ref, kind in ( @@ -15979,6 +16032,7 @@ def _node_instances( return found described = self._node_module(declared) assert described is not None # what `_node_call` checked + self._check_node_export(declared, described, base, select) written = self._node_written(declared, call, base, select) params_by_name = {param.name: param for param in declared.params} ports: dict[str, exp.Expr] = {} @@ -16043,6 +16097,25 @@ def _node_instances( self._node_calls[key] = (tuple(instances), length is not None) return self._node_calls[key] + def _check_node_export( + self, + declared: WasmFunction, + described: Described, + node: exp.Expr, + select: exp.Select, + ) -> None: + """The export a declaration names, against the one its module carries.""" + if not described.name or described.name == declared.export: + return + raise _error( + ErrorCode.UNSUPPORTED_SQL, + f"function '{declared.name}' names the export '{declared.export}', " + f"and '{declared.module}' exports '{described.name}'", + node, + fallback=select, + hint=f"a module carries one node; write '{described.name}' as the export", + ) + def _node_length( self, declared: WasmFunction, @@ -16114,10 +16187,9 @@ def _node_instance( name: self._held_port(declared, name, argument, base, select) for name, argument in numbers.items() } - shape = self._node_shape( - declared, described, {**shape_params, **held}, bound, base, select - ) + shape = self._node_shape(declared, described, shape_params, bound, base, select) self._check_node_ports(declared, shape, streams, numbers, base, select) + ports: dict[str, int] = {} for name, argument in numbers.items(): port = shape.input(name) hold = port.pairing.hold if port is not None else None @@ -16140,7 +16212,14 @@ def _node_instance( fallback=base, hint=f"write the port once: {declared.signature}", ) - params[hold.port_param] = held[name] + ports[hold.port_param] = held[name] + if ports: + # The port is one of the call's params, and the shape is asked + # for the params the call ends up with. + params.update(ports) + shape = self._node_shape( + declared, described, {**shape_params, **ports}, bound, base, select + ) inputs: list[FrameRef] = [] names: list[str] = [] for port in shape.inputs: @@ -16152,19 +16231,10 @@ def _node_instance( names.append(port.name) if port.kind == "data" and port.schema is not None: self._match_rows(declared, port, value, base, select) - kinds: list[StreamType] = [] - for output in shape.outputs: - if output.kind == "packets": - raise _error( - ErrorCode.UNSUPPORTED_SQL, - f"the module '{declared.module}' writes packets on its " - f"'{output.name}' output, which a node call cannot read yet", - base, - fallback=select, - hint="read its frames or its rows; a node writing coded " - "packets is not wired yet", - ) - kinds.append(cast(StreamType, output.kind)) + bound_kinds = { + name: value.streams[0].type for name, value in streams.items() if value.streams + } + kinds = [_output_kind(shape, output, bound_kinds) for output in shape.outputs] key = ( declared.module, declared.export, @@ -16259,7 +16329,8 @@ def _check_node_ports( fallback=select, hint=f"pass a stream for '{port.name}': {declared.signature}", ) - if _port_kind(param) != port.kind: + coded = port.kind == "packets" and _port_kind(param) in ("video", "audio") + if _port_kind(param) != port.kind and not coded: raise _error( ErrorCode.UNSUPPORTED_SQL, f"{declared.name}() declares '{port.name}' as {param.type}, and " @@ -16331,26 +16402,27 @@ def _node_pad( """ shape = instance.shape wanted = _port_kind(output) + made = self.graph.nodes[instance.ref].outputs if output.name: found = shape.output(output.name) if found is None: - made = ", ".join(f"'{o.name}'" for o in shape.outputs) or "none" + named = ", ".join(f"'{o.name}'" for o in shape.outputs) or "none" raise _error( ErrorCode.UNSUPPORTED_SQL, f"{declared.name}() returns the field '{output.name}', and the " f"module '{declared.module}' makes no output '{output.name}'", base, fallback=select, - hint=f"name the fields after the outputs it makes: {made}", + hint=f"name the fields after the outputs it makes: {named}", ) else: - kinds = [o for o in shape.outputs if o.kind == wanted] + kinds = [o for i, o in enumerate(shape.outputs) if made[i] == wanted] if len(shape.outputs) == 1: found = shape.outputs[0] elif len(kinds) == 1: found = kinds[0] else: - made = ", ".join(f"'{o.name}' ({o.kind})" for o in shape.outputs) + listed = ", ".join(f"'{o.name}' ({o.kind})" for o in shape.outputs) raise _error( ErrorCode.UNSUPPORTED_SQL, f"{declared.name}() returns {output.type}, and the module " @@ -16359,19 +16431,19 @@ def _node_pad( base, fallback=select, hint=f"name the one to read with RETURNS STRUCT( " - f", ...); it makes {made or 'nothing'}", + f", ...); it makes {listed or 'nothing'}", ) - if found.kind != wanted: + index = shape.outputs.index(found) + if made[index] != wanted: raise _error( ErrorCode.UNSUPPORTED_SQL, f"{declared.name}() returns {output.type}" + (f" as '{output.name}'" if output.name else "") - + f", and the module '{declared.module}' writes {found.kind} there", + + f", and the module '{declared.module}' writes {made[index]} there", base, fallback=select, - hint=f"declare it as {_PORT_TYPE_NAMES[found.kind]}", + hint=f"declare it as {_PORT_TYPE_NAMES[made[index]]}", ) - index = shape.outputs.index(found) ref = instance.ref if len(shape.outputs) == 1 else f"{instance.ref}:{index}" if found.kind == "data": schema = found.schema @@ -16381,7 +16453,7 @@ def _node_pad( self._data_schemas[ref] = (declared.name, schema) if output.annotation is not None: self._node_rows[ref] = (declared, output.annotation, base) - return ref, found.kind + return ref, made[index] def _node_rows_column(self, node: exp.Expr) -> bool: """Whether `node` is a node's rows, through any spans reduced off them.""" @@ -17445,11 +17517,23 @@ def _packet_filter_pads( already encoded without touching a picture. """ described = self.describes[declared.module] - inputs = self.graph.nodes[ref].inputs + filtered = self.graph.nodes[ref] + shape = node_shape(filtered.filter, self.graph.node_shapes[ref]) if ( + ref in self.graph.node_shapes + ) else None seen: dict[str, int] = {} pads: list[dict[str, object]] = [] - for input_ref in inputs: + for position, input_ref in enumerate(filtered.inputs): kind = ref_type(self.graph, input_ref) + port = shape.input(filtered.ports[position]) if shape is not None else None + if shape is not None and (port is None or port.kind != "packets"): + # A node's other inputs are not encoded for it: rows, or + # frames it reads as they are. + pads.append({}) + continue + if port is not None: + described = replace(described, video_codecs=port.accepts.codecs or None) + described = replace(described, audio_codecs=port.accepts.codecs) index = seen.get(kind, 0) seen[kind] = index + 1 if kind not in asked and self._copies_onto_sink(input_ref, kind, described): @@ -18079,6 +18163,16 @@ def _macro_options( fallback=node, hint=f"its signature is {macro.signature}", ) + written_names = {argument.name for argument in call.named} + missing = next((name for name in macro.required if name not in written_names), None) + if missing is not None: + raise _error( + ErrorCode.UDF_ARG_TYPE, + f"{call.display}() needs '{missing}'", + node, + fallback=select, + hint=f"write it by name: {macro.signature}", + ) per_row = any(_reads_row_column(argument.value, env) for argument in call.named) tuples = env.relation.tuples if per_row and env.relation is not None else [] cache: dict[int, dict[str, object]] = {} diff --git a/cli/ffrwd/macros.py b/cli/ffrwd/macros.py index b348435..2e67e3f 100644 --- a/cli/ffrwd/macros.py +++ b/cli/ffrwd/macros.py @@ -93,6 +93,8 @@ class Macro: options: tuple[str, ...] = () positive: tuple[str, ...] = () nonnegative: tuple[str, ...] = () + # The options a call has to write: what the node cannot do without. + required: tuple[str, ...] = () @property def signature(self) -> str: @@ -154,7 +156,7 @@ def _leaky(values: list[object], node: NodeBuilder, options: dict[str, object]) def _merge_spans(values: list[object], node: NodeBuilder, options: dict[str, object]) -> str: """The host's own rows node, grouping per-tick rows into spans by `start_t`.""" (rows,) = values - return node(ROWMERGE, {MERGE_SPANS: True, **options}, [str(rows)], ["data"]) + return node(ROWMERGE, dict(options), [str(rows)], ["data"]) _LEAKY_SOUND_HINT = ( @@ -212,6 +214,7 @@ def _merge_spans(values: list[object], node: NodeBuilder, options: dict[str, obj expand=_merge_spans, options=(MAX_SPAN,), positive=(MAX_SPAN,), + required=(MAX_SPAN,), ), "loudnorm2": Macro( name="loudnorm2", diff --git a/cli/ffrwd/processes.py b/cli/ffrwd/processes.py index 30b91c1..13af611 100644 --- a/cli/ffrwd/processes.py +++ b/cli/ffrwd/processes.py @@ -140,6 +140,8 @@ src_parts, ) from .probe import JSON_CODEC, ProbeResult, StreamMeta, is_url +from .shapes import NodeShape, node_shape +from .timing import Paths, paths_of __all__ = [ "COPY_CODEC", @@ -154,6 +156,9 @@ "PIPE_BUFFER_LIMIT", "RAWVIDEO", "SAFETY", + "SAMPLE_FMT_CODECS", + "WIRE_PIX_FMTS", + "WIRE_SAMPLE_FMTS", "CLOCK_SIZE", "AudioFormat", "DataFormat", @@ -188,7 +193,9 @@ "is_live_probe", "nothing_external", "check_spellable", + "once_per_pipe", "partition", + "pipe_key", ] # The container every raw stream edge is wrapped in, and the codecs inside it. @@ -205,6 +212,13 @@ # picks and the producing ffmpeg is told to write. DEFAULT_PIX_FMT = "yuv420p" +# The pixel formats a stream edge into or out of the sidecar can carry. +WIRE_PIX_FMTS: tuple[str, ...] = ("rgba", "yuv420p", "yuv422p", "yuv444p") + +# The sample formats one can carry, and the pcm each of them travels as. +WIRE_SAMPLE_FMTS: tuple[str, ...] = ("f32", "s16") +SAMPLE_FMT_CODECS: Mapping[str, str] = {"f32": "pcm_f32le", "s16": "pcm_s16le"} + # The width and height of the picture a data filter's CLOCK pad is handed: it # reads the pts and nothing else, so each frame is made as small as a frame # can usefully be before it crosses the pipe. @@ -690,6 +704,10 @@ class StreamEdge: `live` marks an edge the one reader of a live input writes. Its pictures reach the muxer as they come (:data:`PASSTHROUGH`), never duplicated or dropped to keep a constant rate, since the bound counts them one for one. + + `nut` names the one NUT this stream rides with every other stream between + the same two processes, where one end hosts node modules; "" for a stream + on a pipe of its own. """ source: str @@ -700,6 +718,7 @@ class StreamEdge: bound: int = 0 buffer: EdgeBuffer | None = None live: bool = False + nut: str = "" def to_dict(self) -> dict[str, object]: written: dict[str, object] = { @@ -717,6 +736,8 @@ def to_dict(self) -> dict[str, object]: written["buffer"] = self.buffer.to_dict() if self.live: written["live"] = True + if self.nut: + written["nut"] = self.nut return written @classmethod @@ -731,9 +752,28 @@ def from_dict(cls, d: Mapping[str, object]) -> StreamEdge: bound=_read_whole(d, "bound", 0), buffer=None if buffer is None else EdgeBuffer.from_dict(_read_object(d, "buffer")), live=d.get("live") is True, + nut=_read_text(d, "nut", ""), ) +def pipe_key(edge: StreamEdge) -> str: + """Which pipe `edge` rides: its NUT's, or one of its own.""" + return edge.nut or f"{edge.source}>{edge.target}:{edge.ref}" + + +def once_per_pipe(edges: Sequence[StreamEdge]) -> list[StreamEdge]: + """`edges` with one entry per pipe: the first edge of each NUT stands for it.""" + kept: list[StreamEdge] = [] + seen: set[str] = set() + for edge in edges: + key = pipe_key(edge) + if key in seen: + continue + seen.add(key) + kept.append(edge) + return kept + + @dataclass(frozen=True) class FileEdge: """A file handed from one process to another, ordering the two.""" @@ -1190,6 +1230,13 @@ class SidecarProcess: # (`color_range`, `color_primaries`, `color_trc`, `colorspace`), for the # fields the query settles: the NUT the codec reads carries none of it. color: tuple[tuple[str, str], ...] = () + # True for a region holding a node module: its pads name the ports they + # bind, and each of its edges is one NUT carrying every stream between it + # and the process at the far end. + node_network: bool = False + # The ports this region listens on for an input a node holds and the + # query left unbound: (port, the node's name in the network, its input). + listens: tuple[tuple[int, str, str], ...] = () @property def nodes(self) -> tuple[str, ...]: @@ -1211,6 +1258,8 @@ def network(self) -> bool: """ if self.graph is None: return False + if self.node_network: + return True if self.packet_sink or self.packet_source or self.packet_filter: return False if self.data_filter or self.codec: @@ -1267,6 +1316,10 @@ def to_dict(self) -> dict[str, object]: written["color"] = dict(self.color) if self.pads: written["pads"] = [None if p is None else p.to_dict() for p in self.pads] + if self.node_network: + written["node_network"] = True + if self.listens: + written["listens"] = [list(one) for one in self.listens] if self.network and self.graph is not None: written["graph"] = self.graph.to_dict() return written @@ -1313,6 +1366,12 @@ def from_dict(cls, d: Mapping[str, object]) -> SidecarProcess: codec=_read_text(d, "codec", ""), frame_rate=_read_text(d, "frame_rate", ""), color=tuple((flag, str(value)) for flag, value in _read_pairs(d, "color")), + node_network=d.get("node_network") is True, + listens=tuple( + (int(str(one[0])), str(one[1]), str(one[2])) + for one in _read_list(d, "listens") + if isinstance(one, list) and len(one) == 3 + ), ) @@ -1885,6 +1944,18 @@ def __init__( ) | frozenset( alias for alias, source in g.module_sources.items() if not source.bounded ) + self.node_shapes: dict[str, NodeShape] = { + name: node_shape(g.nodes[name].filter, raw) + for name, raw in g.node_shapes.items() + if name in g.nodes + } + # How late each node's refs run, counted once it is first asked. + self._paths: Paths | None = None + self.live |= frozenset( + alias + for alias, name in g.node_sources.items() + if name in self.node_shapes and not self.node_shapes[name].bounded + ) self.order = _topological(g) # A hosted node is external whoever asked: only the sidecar runs it. self.external = { @@ -2680,6 +2751,11 @@ def _region_writes(self, members: Sequence[str]) -> list[tuple[FrameRef, StreamT return written def _shape(self, name: str) -> ModuleShape: + found = self.node_shapes.get(name) + if found is not None: + # Its waits are counted in seconds (:mod:`ffrwd.timing`), not + # here in frames. + return ModuleShape(one_to_one=found.one_to_one, pure=found.pure) return self.shapes.get(self.g.nodes[name].filter, ModuleShape()) def _lookahead(self, members: Sequence[str], *, frames: bool = False) -> int: @@ -2746,6 +2822,8 @@ def _node_delay(self, name: str) -> int | None: # It drops pictures, but holds none: what it costs its path is # counted in time instead (:meth:`_late_seconds`). return 0 + if name in self.node_shapes: + return self._node_frames(name) if self.external.get(name, False): return self._frames_ahead(name) if self._shape(name).one_to_one else None if node.filter in SPLIT_FILTERS: @@ -2760,6 +2838,27 @@ def _node_delay(self, name: str) -> int | None: size = written if isinstance(written, int) and not isinstance(written, bool) else default return max(size - 1, 0) + def _node_frames(self, name: str) -> int | None: + """The frames node `name` holds past its clock: its window, the waits + its interval inputs add and its outputs' latency, at its pictures' rate.""" + if self._paths is None: + self._paths = paths_of(self.g, self.probes) + paths = self._paths + node = self.g.nodes[name] + shape = self.node_shapes[name] + ready = paths.ready(name)[0] + clock = shape.clock_input + refs = [ref for port, ref in zip(node.ports, node.inputs) if clock and port == clock.name] + clock_delay = paths.latest(refs) if refs else 0.0 + if ready is None or clock_delay is None: + return None + own = ready - clock_delay + max((o.latency for o in shape.outputs), default=0.0) + if own <= 0: + return 0 + video = next((ref for ref in node.inputs if ref_type(self.g, ref) == "video"), None) + rate = paths.rate(video, "video") if video is not None else None + return None if rate is None else math.ceil(own * rate) + def _node_delays(self, names: Sequence[str]) -> dict[str, int | None]: """Each node's delay from this process's own inputs, over its longest path.""" inside = set(names) @@ -3410,12 +3509,11 @@ def _check_lockstep(self) -> None: ): continue node = self.g.nodes[name] - if len(node.inputs) < 2: + refs = self._lockstep_refs(name) + if len(refs) < 2: continue - first = self._anchor(node.inputs[0]) - offender = next( - (ref for ref in node.inputs[1:] if self._anchor(ref) != first), None - ) + first = self._anchor(refs[0]) + offender = next((ref for ref in refs[1:] if self._anchor(ref) != first), None) if offender is None: continue raise FfrwdError( @@ -3427,6 +3525,28 @@ def _check_lockstep(self) -> None: "stream, through modules that declare one frame out per frame in", ) + def _lockstep_refs(self, name: str) -> list[FrameRef]: + """The streams of node `name` that pair with its clock frame for frame. + + Every input of an older module. A node pairs only its clock and its + lockstep inputs of the clock's own kind that way: sound against a + picture is cut to each tick by time, and held, interval and arrival + inputs pair by time as well. + """ + node = self.g.nodes[name] + shape = self.node_shapes.get(name) + if shape is None: + return list(node.inputs) + clock = shape.clock_input + if clock is None: + return [] + same = { + port.name + for port in shape.inputs + if port.pairing.kind == "lockstep" and port.kind == clock.kind + } + return [ref for port, ref in zip(node.ports, node.inputs) if port in same] + def _check_rows_reach(self) -> None: """That the rows a module reads can reach it. @@ -3504,6 +3624,7 @@ def run(self) -> ProcessPlan: color=self._codec_color(members), rows_in=self._region_rows_in(members), pads=self._region_pad_meta(members), + node_network=any(name in self.node_shapes for name in members), ) self.sidecars.append(sidecar) self.members[sidecar.id] = list(members) @@ -3549,9 +3670,14 @@ def run(self) -> ProcessPlan: demands.extend((process.id, ref, depth) for ref in self._consumed(process)) for sidecar in self.sidecars: + members = self.members[sidecar.id] + # A node region reads a source stream wherever a member does, so + # it reads every one at the region's own depth, and one feeder + # hands it all of them. + lowest = min(self.depth[member] for member in members) demands.extend( - (sidecar.id, ref, self.depth[reader]) - for ref, reader in self._region_reads(self.members[sidecar.id]) + (sidecar.id, ref, lowest if sidecar.node_network else self.depth[reader]) + for ref, reader in self._region_reads(members) ) while demands: @@ -3624,6 +3750,7 @@ def run(self) -> ProcessPlan: self._redirect_live_reads() self._add_rows_edges() self._add_rows_documents() + self._bundle_node_edges() self._bound_edges() self._mark_live_edges() self._check_handed_once() @@ -3639,6 +3766,20 @@ def run(self) -> ProcessPlan: ), ) + def _bundle_node_edges(self) -> None: + """Every stream between a node region and one other process, in one NUT. + + A node reads and writes its kinds side by side, so the streams one + process hands another travel interleaved on one pipe, and no stream of + it can wait on another pipe. + """ + nodes = {sidecar.id for sidecar in self.sidecars if sidecar.node_network} + if not nodes: + return + for index, edge in enumerate(self.edges): + if edge.source in nodes or edge.target in nodes: + self.edges[index] = replace(edge, nut=f"{edge.source}>{edge.target}") + def _add_data_read(self, source: str, target: str, ref: FrameRef, depth: int) -> None: """One more reader of the data stream `ref`, which sidecar `source` writes. @@ -3836,11 +3977,15 @@ def _region_rows( region, and each rows-bearing node is a document of its own. """ slots = self._module_slots(members, bindings) - return tuple( - RowsDocument(sink=self.g.rows_sinks[name], node=name, source=slots.get(name)) - for name in members - if name in self.g.rows_sinks - ) + found: list[RowsDocument] = [] + for name in members: + pads = [name] + [f"{name}:{pad}" for pad in range(len(self.g.nodes[name].outputs))] + for ref in pads: + if ref in self.g.rows_sinks: + found.append( + RowsDocument(sink=self.g.rows_sinks[ref], node=ref, source=slots.get(name)) + ) + return tuple(found) def _module_slots( self, members: Sequence[str], bindings: Sequence[ModuleBinding] @@ -3977,15 +4122,41 @@ def _wire_reader(self, target: str, reader: str | None) -> str | None: named = [ (name, wire) for name in self._behind_split(target, reader) - if (wire := ( - self.pix_fmts.get(self.g.nodes[name].filter), - self.audio_wires.get(self.g.nodes[name].filter), - )) != (None, None) + if (wire := self._named_wire(name, reader)) is not None ] if not named or any(wire != named[0][1] for _, wire in named): return reader return named[0][0] + def _named_wire(self, name: str, split: str | None) -> object: + """The wire module `name` names for what it reads off `split`, or None. + + A node's is what its port accepts; an older module's, the pixel + format and pcm it declared. + """ + node = self.g.nodes[name] + if name in self.node_shapes and split is not None: + ref = self.g.nodes[split].inputs[0] + position = self._read_position(node, ref) + if position is None: + return None + return self._node_input_wire(name, node.ports[position], ref) + wire = (self.pix_fmts.get(node.filter), self.audio_wires.get(node.filter)) + return None if wire == (None, None) else wire + + def _read_position(self, node: Node, ref: FrameRef) -> int | None: + """Where `node` reads `ref`: itself, or a pad of a split of it.""" + for position, read in enumerate(node.inputs): + producer = _ref_node(read) + if read == ref or ( + producer is not None + and producer in self.g.nodes + and self.g.nodes[producer].filter in SPLIT_FILTERS + and self.g.nodes[producer].inputs[0] == ref + ): + return position + return None + def _carries_annotations(self, ref: FrameRef, consumer: str | None) -> bool: """Whether this edge's frames travel with the producer's rows. @@ -4101,6 +4272,9 @@ def _format(self, ref: FrameRef, target: str | None = None) -> StreamFormat: # Messages cross as they are, whoever wrote them and whoever reads # them: copied into NUT, one packet each. return DataFormat() + node_wire = self._node_wire(ref, target) + if node_wire is not None: + return node_wire meta = self._origin_meta(ref) producer = _ref_node(ref) if producer is not None and ( @@ -4193,6 +4367,160 @@ def _format(self, ref: FrameRef, target: str | None = None) -> StreamFormat: timebase=_timebase(meta.fps) if meta else None, ) + def _node_wire(self, ref: FrameRef, target: str | None) -> StreamFormat | None: + """What an edge carries where a node writes it or reads it, else None. + + A node's output says its own format, or that it is an input's or the + clock input's; a node's input says what it accepts. Size and time base + are the stream's own, as on every other edge. + """ + producer = _ref_node(ref) + if producer is not None and producer in self.node_shapes: + return self._node_output_wire(producer, _ref_pad(ref), ref) + if target is not None and target in self.g.packet_filters: + return None # its destination settled what encodes for it + if target is not None and target in self.node_shapes: + node = self.g.nodes[target] + position = self._read_position(node, ref) + if position is not None: + return self._node_input_wire(target, node.ports[position], ref) + return None + + def _node_input_wire(self, name: str, port_name: str, ref: FrameRef) -> StreamFormat: + """What node `name` takes on its port `port_name`.""" + port = self.node_shapes[name].input(port_name) + meta = self._origin_meta(ref) + accepts = port.accepts if port is not None else None + if port is not None and port.kind == "packets": + return self._node_packets_wire(name, port_name, ref) + if ref_type(self.g, ref) == "audio": + formats = accepts.sample_formats if accepts is not None else () + sample = next((f for f in formats if f in WIRE_SAMPLE_FMTS), None) + if formats and sample is None: + raise FfrwdError( + ErrorCode.UNSUPPORTED_SQL, + f"the module '{self.g.nodes[name].filter}' takes " + f"{', '.join(formats)} on '{port_name}', and an edge carries " + f"{', '.join(WIRE_SAMPLE_FMTS)}", + hint="rebuild the module accepting one of those sample formats", + ) + return AudioFormat( + rate=meta.sample_rate if meta else None, + channels=meta.channels if meta else None, + codec=SAMPLE_FMT_CODECS[sample or WIRE_SAMPLE_FMTS[0]], + required_rate=accepts.sample_rates[0] if accepts and accepts.sample_rates else None, + required_channels=( + accepts.channel_counts[0] if accepts and accepts.channel_counts else None + ), + ) + formats = accepts.pixel_formats if accepts is not None else () + pix_fmt = next((f for f in formats if f in WIRE_PIX_FMTS), None) + if formats and pix_fmt is None: + raise FfrwdError( + ErrorCode.UNSUPPORTED_SQL, + f"the module '{self.g.nodes[name].filter}' takes {', '.join(formats)} " + f"on '{port_name}', and an edge carries {', '.join(WIRE_PIX_FMTS)}", + hint="rebuild the module accepting one of those pixel formats", + ) + size = self._picture_size(ref) + return VideoFormat( + pix_fmt=pix_fmt or DEFAULT_PIX_FMT, + width=size[0] if size else None, + height=size[1] if size else None, + timebase=_timebase(meta.fps) if meta else None, + ) + + def _node_packets_wire(self, name: str, port_name: str, ref: FrameRef) -> StreamFormat: + """A stream a node reads as coded packets: copied as it was coded. + + The module names the codecs it takes; a stream coded in another, or + not coded at all, is one this edge cannot hand it. + """ + port = self.node_shapes[name].input(port_name) + codecs = port.accepts.codecs if port is not None else () + producer = _ref_node(ref) + meta = self._origin_meta(ref) + coded = producer is None or ( + producer in self.node_shapes + and self.node_shapes[producer].outputs[_ref_pad(ref)].kind == "packets" + ) + codec = meta.codec if meta is not None and producer is None else None + if not coded or (codecs and codec is not None and codec not in codecs): + raise FfrwdError( + ErrorCode.UNSUPPORTED_SQL, + f"the module '{self.g.nodes[name].filter}' reads coded packets on " + f"'{port_name}', and {_named_ref(ref)} is " + + (f"coded as {codec}" if coded else "not coded"), + hint="hand it an input's own stream in a codec it takes" + + (f" ({', '.join(codecs)})" if codecs else ""), + ) + if ref_type(self.g, ref) == "audio": + return AudioFormat( + rate=meta.sample_rate if meta else None, + channels=meta.channels if meta else None, + codec=COPY_CODEC, + ) + return VideoFormat( + width=meta.width if meta else None, + height=meta.height if meta else None, + timebase=_timebase(meta.fps) if meta else None, + codec=COPY_CODEC, + ) + + def _node_output_wire(self, name: str, pad: int, ref: FrameRef) -> StreamFormat: + """What node `name` writes on its output `pad`.""" + shape = self.node_shapes[name] + node = self.g.nodes[name] + output = shape.outputs[pad] if pad < len(shape.outputs) else None + found = output.format if output is not None else None + if output is not None and output.kind == "packets": + # Coded: what the module writes crosses as it is, and is copied. + if node.outputs[pad] == "data": + return DataFormat() + if node.outputs[pad] == "audio": + return AudioFormat( + rate=found.sample_rate if found is not None else None, + channels=found.channels if found is not None else None, + codec=COPY_CODEC, + ) + return VideoFormat( + width=found.width if found is not None else None, + height=found.height if found is not None else None, + codec=COPY_CODEC, + ) + follows = ( + found.port + if found is not None and found.kind == "like" + else shape.clock.port + if found is None and shape.clock.kind == "input" + else None + ) + inherited: StreamFormat | None = None + if follows is not None and follows in node.ports: + read = node.inputs[node.ports.index(follows)] + inherited = self._format(read, name) + if node.outputs[pad] == "audio": + base = inherited if isinstance(inherited, AudioFormat) else AudioFormat() + sample = found.sample_format if found is not None else None + if found is not None and found.kind == "audio": + return AudioFormat( + rate=found.sample_rate, + channels=found.channels, + codec=SAMPLE_FMT_CODECS.get(sample or "", base.codec), + ) + return replace(base, codec=SAMPLE_FMT_CODECS.get(sample or "", base.codec)) + base_video = inherited if isinstance(inherited, VideoFormat) else VideoFormat() + if found is not None and found.kind == "video": + return VideoFormat( + pix_fmt=found.pixel_format or base_video.pix_fmt, + width=found.width, + height=found.height, + timebase=base_video.timebase, + ) + if found is not None and found.pixel_format: + return replace(base_video, pix_fmt=found.pixel_format) + return base_video + def _pix_fmt(self, ref: FrameRef, target: str | None) -> str: """The pixel format this edge carries. @@ -4312,25 +4640,60 @@ def substitute(ref: FrameRef) -> FrameRef: return keep, resized, substitute + def _pipe_bundles(self, process: _Pending) -> list[list[FrameRef]]: + """`process`'s pipe refs, those riding one NUT together, in pipe order.""" + nut_of = {edge.ref: edge.nut for edge in self.edges if edge.source == process.id} + bundles: list[list[FrameRef]] = [] + by_nut: dict[str, list[FrameRef]] = {} + for ref in process.pipes: + nut = nut_of.get(ref, "") + if not nut: + bundles.append([ref]) + continue + if nut not in by_nut: + by_nut[nut] = [] + bundles.append(by_nut[nut]) + by_nut[nut].append(ref) + return bundles + + def _read_aliases( + self, incoming: Sequence[StreamEdge], taken: set[str] + ) -> tuple[dict[FrameRef, str], dict[FrameRef, str], list[str]]: + """Each piped ref's input alias and the spec it is read under, and the + aliases in ``-i`` order: one per pipe, a NUT's streams counted per kind.""" + read_as: dict[FrameRef, str] = {} + alias_of: dict[FrameRef, str] = {} + order: list[str] = [] + by_pipe: dict[str, str] = {} + counted: dict[str, dict[str, int]] = {} + for edge in incoming: + if edge.ref in read_as: + continue + key = pipe_key(edge) + alias = by_pipe.get(key) + if alias is None: + alias = _unique_alias(edge.nut.replace(">", "_") or edge.ref, taken) + taken.add(alias) + by_pipe[key] = alias + counted[alias] = {} + order.append(alias) + marker = _marker(edge.format) + index = counted[alias].get(marker, 0) + counted[alias][marker] = index + 1 + alias_of[edge.ref] = alias + read_as[edge.ref] = f"src:{alias}:{marker}:{index}" + return read_as, alias_of, order + def _materialize(self, process: _Pending) -> FfmpegProcess: """`process` as a complete graph, its pipes now inputs and sinks.""" kept, resized, substitute = self._shrink_splits(process) incoming = [e for e in self.edges if e.target == process.id] - alias_of: dict[FrameRef, str] = {} - marker_of: dict[FrameRef, str] = {} taken = set(self.g.sources) - for edge in incoming: - if edge.ref in alias_of: - continue - alias = _unique_alias(edge.ref, taken) - taken.add(alias) - alias_of[edge.ref] = alias - marker_of[edge.ref] = _marker(edge.format) + read_as, alias_of, read_order = self._read_aliases(incoming, taken) def rewrite(ref: FrameRef) -> FrameRef: ref = substitute(ref) - alias = alias_of.get(ref) - return ref if alias is None else f"src:{alias}:{marker_of[ref]}:0" + return read_as.get(ref, ref) nodes: dict[str, Node] = {} for name in kept: @@ -4354,19 +4717,28 @@ def rewrite(ref: FrameRef) -> FrameRef: name=None, metadata={}, ) + for ref in bundle ], path=PIPE, ) - for ref in process.pipes + for bundle in self._pipe_bundles(process) ) paths, sources, trims, options = self._inputs(nodes, sinks) - for edge in incoming: - alias = alias_of[edge.ref] + for alias in read_order: if alias in sources: continue sources[alias] = len(paths) paths.append(PIPE) + # A node read in FROM ends where the query's own `WHERE .t` + # says: the reader takes that long of what it writes and closes. + for alias, name in self.g.node_sources.items(): + bounds = self.g.input_trims.get(alias) + if bounds is None: + continue + for ref, piped in alias_of.items(): + if _ref_node(ref) == name: + trims[piped] = bounds return FfmpegProcess( id=process.id, @@ -4396,16 +4768,17 @@ def _materialize_region(self, sidecar: SidecarProcess) -> SidecarProcess: if edge.target == sidecar.id and edge.ref not in seen: seen.add(edge.ref) incoming.append(edge) + # In the order its pads are read, which is the order the plan's argv + # writes its reads in. + incoming.sort( + key=lambda edge: sidecar.inputs.index(edge.ref) + if edge.ref in sidecar.inputs + else len(sidecar.inputs) + ) outgoing = [e for e in self.edges if e.source == sidecar.id] - alias_of: dict[FrameRef, str] = {} - marker_of: dict[FrameRef, str] = {} taken = set(self.g.sources) - for edge in incoming: - alias = _unique_alias(edge.ref, taken) - taken.add(alias) - alias_of[edge.ref] = alias - marker_of[edge.ref] = _marker(edge.format) + read_as, _, read_order = self._read_aliases(incoming, taken) names = {binding.path: binding.name for binding in sidecar.modules} dissolved: dict[str, FrameRef] = {} @@ -4413,9 +4786,7 @@ def _materialize_region(self, sidecar: SidecarProcess) -> SidecarProcess: def rewrite(ref: FrameRef) -> FrameRef: slot = ref if is_src(ref) else f"{_ref_node(ref)}:{_ref_pad(ref)}" ref = dissolved.get(slot, ref) - if ref in alias_of: - return f"src:{alias_of[ref]}:{marker_of[ref]}:0" - return ref + return read_as.get(ref, ref) nodes: dict[str, Node] = {} for name in members: # topological: a split precedes its readers @@ -4439,7 +4810,12 @@ def rewrite(ref: FrameRef) -> FrameRef: inputs=[rewrite(ref) for ref in node.inputs], outputs=list(node.outputs), reads_annotations=node.reads_annotations, + ports=list(node.ports), + out_ports=list(node.out_ports), ) + bundles: dict[str, list[StreamEdge]] = {} + for edge in outgoing: + bundles.setdefault(pipe_key(edge), []).append(edge) sinks = [ SinkUnit( outputs=[ @@ -4449,10 +4825,11 @@ def rewrite(ref: FrameRef) -> FrameRef: name=None, metadata={}, ) + for edge in bundle ], path=PIPE, ) - for edge in outgoing + for bundle in bundles.values() ] # The module whose rows leave is a sink of the region too: the network # string has to name the pad they were read off, even though its @@ -4494,17 +4871,38 @@ def rewrite(ref: FrameRef) -> FrameRef: ) return replace( sidecar, + listens=self._region_listens(members, names), reads_rows=any(e.annotations for e in self.edges if e.target == sidecar.id), writes_rows=any(e.annotations for e in self.edges if e.source == sidecar.id), rows_modules=self._rows_modules(sidecar, members), graph=Graph( - input_paths=[PIPE] * len(incoming), - sources={alias_of[e.ref]: index for index, e in enumerate(incoming)}, + input_paths=[PIPE] * len(read_order), + sources={alias: index for index, alias in enumerate(read_order)}, nodes=nodes, sinks=sinks, ), ) + def _region_listens( + self, members: Sequence[str], names: Mapping[str, str] + ) -> tuple[tuple[int, str, str], ...]: + """Each input a node of this region holds on a port of its own and the + query bound no stream to: the port its param carries, and the input.""" + found: list[tuple[int, str, str]] = [] + for name in members: + shape = self.node_shapes.get(name) + if shape is None: + continue + node = self.g.nodes[name] + for port in shape.inputs: + hold = port.pairing.hold + if hold is None or hold.port_param is None or port.name in node.ports: + continue + number = node.args.get(hold.port_param) + if isinstance(number, int) and not isinstance(number, bool): + found.append((number, names.get(node.filter, node.filter), port.name)) + return tuple(found) + def _rows_pad(self, name: str) -> str: """The node whose PAD the rows `name` writes were read off. @@ -4513,6 +4911,8 @@ def _rows_pad(self, name: str) -> str: and whose own output is not a pad at all. """ seen: set[str] = set() + if name not in self.g.nodes: + return name # a node's own pad: its rows leave on it while self.g.nodes[name].rows_only and name not in seen: seen.add(name) name = self.g.nodes[name].rows_inputs[0] diff --git a/cli/ffrwd/shapes.py b/cli/ffrwd/shapes.py index 4e6e820..cb710d2 100644 --- a/cli/ffrwd/shapes.py +++ b/cli/ffrwd/shapes.py @@ -136,9 +136,10 @@ class OutputFormat: """One arm of ``output-format``. `video`: width, height, `pixel_format`. `audio`: `sample_rate`, - `channels`, `sample_format`. `data`: `codec`. `packets`: `codec` and - `time_base`. `like`: `port`, with `pixel_format` or `sample_format` the - field it overrides. + `channels`, `sample_format`. `data`: `codec`. `packets`: the coded + stream, `codec`, `time_base`, what it carries as `coded` (video, audio or + data) with that kind's own fields, and `extradata` as hex. `like`: + `port`, with `pixel_format` or `sample_format` the field it overrides. """ kind: FormatKind @@ -151,6 +152,8 @@ class OutputFormat: codec: str | None = None time_base: tuple[int, int] | None = None port: str | None = None + coded: Literal["video", "audio", "data"] | None = None + extradata: str = "" @dataclass(frozen=True) @@ -415,8 +418,19 @@ def _output_format(value: object, module: str, port: str) -> OutputFormat | None sample_format=_text(body.get("sample_fmt")) or _text(body.get("sample_format")), ) if kind == "packets": + coded = body.get("format") + carried = coded if isinstance(coded, dict) else {} + coded_kind = _choice(carried.get("kind"), ("video", "audio", "data"), "") return OutputFormat( - "packets", codec=_text(body.get("codec")), time_base=_rational(body.get("time_base")) + "packets", + codec=_text(body.get("codec")), + time_base=_rational(body.get("time_base")), + coded=cast(Literal["video", "audio", "data"], coded_kind) if coded_kind else None, + width=_whole(carried.get("width")), + height=_whole(carried.get("height")), + sample_rate=_whole(carried.get("sample_rate")), + channels=_whole(carried.get("channels")), + extradata=_text(body.get("extradata")) or "", ) if kind == "like": return OutputFormat( diff --git a/cli/ffrwd/startup.py b/cli/ffrwd/startup.py index fa8ff3e..0017db9 100644 --- a/cli/ffrwd/startup.py +++ b/cli/ffrwd/startup.py @@ -66,6 +66,7 @@ RowsEdge, SidecarProcess, StreamEdge, + once_per_pipe, ) __all__ = ["Milestone", "arrange", "check", "relation", "stalled"] @@ -379,11 +380,12 @@ def _reads(plan: ProcessPlan, pid: str) -> list[StreamEdge]: def _writes(plan: ProcessPlan, pid: str) -> list[StreamEdge]: - return [e for e in plan.stream_edges if e.source == pid] + return once_per_pipe([e for e in plan.stream_edges if e.source == pid]) def _once_per_ref(edges: Sequence[StreamEdge]) -> list[StreamEdge]: - """One edge per ref: two edges of one ref share a single ``-i``.""" + """One edge per ref: two edges of one ref share a single ``-i``, and the + streams of one NUT one pipe.""" seen: set[str] = set() kept: list[StreamEdge] = [] for edge in edges: @@ -391,7 +393,7 @@ def _once_per_ref(edges: Sequence[StreamEdge]) -> list[StreamEdge]: continue seen.add(edge.ref) kept.append(edge) - return kept + return once_per_pipe(kept) # ---------------------------------------------------------------- reading a plan @@ -408,7 +410,7 @@ def _pipe_inputs(plan: ProcessPlan, pid: str) -> list[Edge]: def _pipe_outputs(plan: ProcessPlan, pid: str) -> list[Edge]: """What `pid` writes, in the order its outputs are rendered.""" - frames: list[Edge] = [e for e in plan.stream_edges if e.source == pid] + frames: list[Edge] = list(once_per_pipe([e for e in plan.stream_edges if e.source == pid])) rows: list[Edge] = [e for e in plan.rows_edges if e.source == pid] return frames + rows @@ -433,7 +435,7 @@ def _key(edge: Edge) -> Wire: if isinstance(edge, RowsEdge): return ("rows", edge.source, edge.target, edge.alias) if isinstance(edge, StreamEdge): - return ("stream", edge.source, edge.target, edge.ref) + return ("stream", edge.source, edge.target, edge.nut or edge.ref) return ("file", edge.source, edge.target) diff --git a/cli/ffrwd/timing.py b/cli/ffrwd/timing.py index 6aeb509..389ff3b 100644 --- a/cli/ffrwd/timing.py +++ b/cli/ffrwd/timing.py @@ -30,7 +30,6 @@ from .errors import ErrorCode, FfrwdError from .ir import ( MAX_SPAN, - MERGE_SPANS, ROWMERGE, FrameRef, Graph, @@ -45,8 +44,11 @@ "HOLD_LIMIT", "NodeTiming", "OutputTiming", + "Paths", "Timing", "check_live_leads", + "paths_of", + "summary", "timing", ] @@ -84,11 +86,14 @@ class NodeTiming: window: str waits: tuple[Wait, ...] delay: float | None + # What the query calls it. + called: str = "" def to_dict(self) -> dict[str, object]: return { "node": self.node, "module": self.module, + "called": self.called, "window": self.window, "inputs": [ { @@ -113,10 +118,18 @@ class OutputTiming: delay: float | None holds: float held: int | None = None + # Where it is written: the file, the stream's place in it and its kind; + # a rows file names no stream. + path: str = "" + index: int | None = None + kind: str = "" def to_dict(self) -> dict[str, object]: written: dict[str, object] = { "ref": self.ref, + "path": self.path, + "index": self.index, + "kind": self.kind, "delay": self.delay, "holds": self.holds, } @@ -137,7 +150,7 @@ def to_dict(self) -> dict[str, object]: } -class _Paths: +class Paths: """Delays over one graph, each ref's counted once.""" def __init__(self, graph: Graph, probes: Mapping[str, ProbeResult | None]) -> None: @@ -173,9 +186,9 @@ def _count(self, ref: FrameRef) -> float | None: if any(one is None for one in found): return None above = max((one for one in found if one is not None), default=0.0) - if node.filter == ROWMERGE and node.args.get(MERGE_SPANS): - span = node.args.get(MAX_SPAN) - return above + float(span) if isinstance(span, int | float) else None + span = node.args.get(MAX_SPAN) if node.filter == ROWMERGE else None + if isinstance(span, int | float) and not isinstance(span, bool): + return above + float(span) return above def ready(self, name: str) -> tuple[float | None, tuple[Wait, ...]]: @@ -221,6 +234,10 @@ def ready(self, name: str) -> tuple[float | None, tuple[Wait, ...]]: self._ready[name] = (total, tuple(waits)) return self._ready[name] + def latest(self, refs: list[FrameRef]) -> float | None: + """The latest of `refs`, or None where one of them has no size.""" + return self._latest(refs) + def _latest(self, refs: list[FrameRef]) -> float | None: found = [self.delay(ref) for ref in refs] if any(one is None for one in found): @@ -292,9 +309,22 @@ def rate(self, ref: FrameRef, kind: str) -> Fraction | None: shape = self.shapes.get(name) if shape is not None and shape.clock.rate is not None and kind == "video": return Fraction(*shape.clock.rate) + if shape is not None and shape.clock.kind == "rate-of" and kind == "video": + # The rate of the first stream bound to the port it names. + ticked = next( + (ref for port, ref in zip(node.ports, node.inputs) if port == shape.clock.port), + None, + ) + if ticked is not None: + return self.rate(ticked, kind) return self.rate(node.inputs[0], kind) if node.inputs else None +def paths_of(graph: Graph, probes: Mapping[str, ProbeResult | None]) -> Paths: + """How late each ref of `graph` runs, counted as it is asked for.""" + return Paths(graph, probes) + + def _fraction(fps: str | None) -> Fraction | None: numerator, _, denominator = (fps or "").partition("/") try: @@ -304,12 +334,20 @@ def _fraction(fps: str | None) -> Fraction | None: return found if found > 0 else None -def timing(graph: Graph, probes: Mapping[str, ProbeResult | None]) -> Timing | None: +def timing( + graph: Graph, + probes: Mapping[str, ProbeResult | None], + called: Mapping[str, str] | None = None, +) -> Timing | None: """Each node module's window and readiness, and each written stream's - delay; None for a graph with no node module in it.""" + delay; None for a graph with no node module in it. `called` names each + module path the way the query calls it.""" if not graph.node_shapes: return None - paths = _Paths(graph, probes) + paths = Paths(graph, probes) + names = called or {} + # A track minted from rows runs as late as the node writing them. + rows_of = {sink.alias: ref for ref, sink in graph.rows_sinks.items() if sink.alias} nodes: list[NodeTiming] = [] for name, shape in paths.shapes.items(): node = graph.nodes[name] @@ -322,25 +360,67 @@ def timing(graph: Graph, probes: Mapping[str, ProbeResult | None]) -> Timing | N window = window_words(clock, rate) elif shape.clock.kind == "rate" and shape.clock.rate is not None: window = f"rate {_rate_words(shape.clock.rate)}" + elif shape.clock.kind == "rate-of": + window = f"at the rate of {shape.clock.port}" else: window = shape.clock.kind ready, waits = paths.ready(name) - nodes.append(NodeTiming(name, node.filter, window, waits, ready)) + called_as = names.get(node.filter, node.filter) + nodes.append(NodeTiming(name, node.filter, window, waits, ready, called_as)) outputs: list[OutputTiming] = [] for unit in graph.sinks: - delays = [paths.delay(output.ref) for output in unit.outputs] + refs = [ + rows_of.get(src_parts(output.ref)[0], output.ref) if is_src(output.ref) + else output.ref + for output in unit.outputs + ] + delays = [paths.delay(ref) for ref in refs] latest = max((d for d in delays if d is not None), default=0.0) - for output, delay in zip(unit.outputs, delays): + for index, (output, delay) in enumerate(zip(unit.outputs, delays)): holds = 0.0 if delay is None else latest - delay per_second = paths.bytes_per_second(output.ref) if holds else None held = None if per_second is None else int(per_second * Fraction(holds)) - outputs.append(OutputTiming(output.ref, delay, holds, held)) - for ref in graph.rows_sinks: - if not any(output.ref == ref for output in outputs): - outputs.append(OutputTiming(ref, paths.delay(ref), 0.0)) + outputs.append( + OutputTiming( + output.ref, delay, holds, held, unit.path or "", index, output.type + ) + ) + for ref, sink in graph.rows_sinks.items(): + if sink.path: + outputs.append(OutputTiming(ref, paths.delay(ref), 0.0, path=sink.path, kind="rows")) return Timing(tuple(nodes), tuple(outputs)) +def summary(timed: Timing) -> str: + """What ``explain --delays`` prints: a line per node, then per output. + + A node says its window in streaming words and how each input it waits + for by interval is bounded; an output how far behind the source it runs, + and how long it waits for the latest stream written beside it. + """ + lines: list[str] = [] + for node in timed.nodes: + said = [node.window] + for wait in node.waits: + if wait.pairing != "interval": + continue + bound = "no bound" if wait.bound is None else f"at most {_seconds(wait.bound)}" + said.append(f"{wait.port} by interval, {bound}") + lines.append(f"{node.called}: {'; '.join(said)}") + for output in timed.outputs: + where = ( + f"{output.path} ({output.kind})" + if output.index is None + else f"{output.path} stream {output.index} ({output.kind})" + ) + late = "no known delay" if output.delay is None else ( + f"{_seconds(output.delay)} behind the source" + ) + waits = f", waits {_seconds(output.holds)}" if output.holds else "" + lines.append(f"{where}: {late}{waits}") + return "\n".join(lines) + + def _rate_words(rate: tuple[int, int]) -> str: num, den = rate return f"{num}/s" if den == 1 else f"{num}/{den}/s" @@ -358,7 +438,7 @@ def check_live_leads( past the bound is late for good. `anchors` maps a module path to where the query named it and the function's name. """ - paths = _Paths(graph, probes) + paths = Paths(graph, probes) for name, shape in paths.shapes.items(): node = graph.nodes[name] clock = shape.clock_input diff --git a/cli/ffrwd/wasm.py b/cli/ffrwd/wasm.py index 3adef73..c28acb2 100644 --- a/cli/ffrwd/wasm.py +++ b/cli/ffrwd/wasm.py @@ -50,20 +50,21 @@ import subprocess import threading from collections.abc import Callable, Mapping, Sequence -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from pathlib import Path from typing import Literal, Protocol, cast from . import binaries, nn, probe -from .emit import build_network_graph +from .emit import build_network_graph, build_node_network from .errors import ErrorCode, FfrwdError from .execute import STDIN, STDOUT -from .ir import RowsSink, StreamType +from .ir import PARAMS_FILE, PIPE, RowsSink, StreamType from .probe import ProbeResult, RenditionMeta, StreamMeta from .processes import ( NUT, - PCM_F32LE, - PCM_S16LE, + SAMPLE_FMT_CODECS, + WIRE_PIX_FMTS, + WIRE_SAMPLE_FMTS, AudioFormat, EffectGrant, ModelBinding, @@ -72,6 +73,7 @@ RowsDocument, SidecarProcess, ) +from .shapes import PARAMS_INLINE_LIMIT __all__ = [ "ANNOTATION_TYPES", @@ -181,8 +183,6 @@ # The package carrying the wit, whose version is the world's. WIT_PACKAGE = "ffrwd/wasm" -# The pixel formats a stream edge into or out of the sidecar can carry. -WIRE_PIX_FMTS: tuple[str, ...] = ("rgba", "yuv420p", "yuv422p", "yuv444p") # The coded video streams a stream edge can carry to a packet sink: the ones # the sidecar's NUT reader hands through untouched. @@ -251,8 +251,6 @@ # The sample formats one can carry, the pcm each of them travels as, and # the name ffmpeg's own options spell it by. -WIRE_SAMPLE_FMTS: tuple[str, ...] = ("f32", "s16") -SAMPLE_FMT_CODECS: Mapping[str, str] = {"f32": PCM_F32LE, "s16": PCM_S16LE} FFMPEG_SAMPLE_FMTS: Mapping[str, str] = {"f32": "flt", "s16": "s16"} # The JSON Schema types each declared annotation field type covers. `number` @@ -389,6 +387,9 @@ def _budget_hint(budget: float) -> str: # The sidecar's worker-thread cap. Unwritten, the sidecar sizes its own pool. _JOBS_FLAG = "-jobs" +# Where a node's params are read whole from a file: ``=``. +_PARAMS_FROM_FLAG = "-params-from" + # Which half of a codec package's module a run drives: "encode" or "decode". _CODEC_FLAG = "-codec" @@ -1982,7 +1983,10 @@ def _argv( "its outputs carries, one per output", ) argv = [binary] - if not process.packet_source: + # A source reads nothing: a packet source, or a network of nodes none of + # which is handed a stream. + reads_nothing = process.packet_source or (process.node_network and not process.inputs) + if not reads_nothing: for index, path in enumerate(reads or (STDIN,)): argv += ["-f", EDGE_FORMAT, "-i", path] meta: PadMeta | None = process.pads[index] if index < len(process.pads) else None @@ -2004,7 +2008,9 @@ def _argv( argv += [_FRAME_RATE_FLAG, process.frame_rate] for flag, value in process.color: argv += [f"-{flag}", value] - if process.network: + if process.node_network: + argv += _node_network_args(process, reads, writes) + elif process.network: argv += _network_args(process, writes) else: argv += ["-m", process.module] @@ -2184,6 +2190,65 @@ def _network_args(process: SidecarProcess, writes: Sequence[str] = ()) -> list[s return argv +def _node_network_args( + process: SidecarProcess, reads: Sequence[str], writes: Sequence[str] +) -> list[str]: + """The ``-m`` table, the network string and the outputs of a region + holding node modules. + + Each output is one NUT carrying every stream the plan takes to one + process, a ``-map`` per stream; each rows document an output of its own. + `reads` names its inputs in ``-i`` order and `writes` its stream outputs + then its documents; a printed command given none reads stdin and writes + stdout, then numbered pipes. + """ + graph = process.graph + if graph is None: # `network` is True for every node region + raise _reject( + f"process '{process.id}' hosts node modules and carries no graph", + hint="the plan was built without partitioning; recompile the query", + ) + wanted = graph.input_paths.count(PIPE) + given = list(reads)[:wanted] or [STDIN] * min(wanted, 1) + given += [f"pipe:{index}" for index in range(len(given), wanted)] + filed: list[str] = [] + nodes = dict(graph.nodes) + for name, node in graph.nodes.items(): + if not node.ports and not node.out_ports or not _params_filed(node.args): + continue + filed += [_PARAMS_FROM_FLAG, f"{node.filter}={PARAMS_FILE}{process.id}:{name}"] + nodes[name] = replace(node, args={}) + network, groups = build_node_network(replace(graph, nodes=nodes), pipe_inputs=given) + streams = len(groups) - len(process.rows) + paths = list(writes[:streams]) + paths += [STDOUT if not paths and index == 0 else f"pipe:{index + 1}" + for index in range(len(paths), streams)] + documents = writes[streams:] + argv: list[str] = [] + for binding in process.modules: + argv += ["-m", f"{binding.name}={binding.path}"] + argv += ["-filter_complex", network, *filed] + for targets, path in zip(groups[:streams], paths): + for target in targets: + argv += ["-map", target] + argv += ["-f", EDGE_FORMAT, path] + for index, (targets, document) in enumerate(zip(groups[streams:], process.rows)): + given_path = documents[index] if index < len(documents) else "" + path = document.sink.path or given_path or f"pipe:{streams + index + 1}" + for target in targets: + argv += ["-map", target] + argv += ["-f", document.sink.container, path] + return argv + + +def _params_filed(params: Mapping[str, object]) -> bool: + """Whether a node's params go in a file: too long for a command line, or + holding a list a filtergraph option cannot spell.""" + if any(not isinstance(value, str | int | float | bool) for value in params.values()): + return True + return len(json.dumps(params)) > PARAMS_INLINE_LIMIT + + def sidecar_argv( process: SidecarProcess, reads: Sequence[str] = (), diff --git a/cli/tests/test_examples.py b/cli/tests/test_examples.py index b590b16..324edad 100644 --- a/cli/tests/test_examples.py +++ b/cli/tests/test_examples.py @@ -201,8 +201,9 @@ def wrap_command(line: str, width: int = _WRAP_WIDTH) -> str: continuations. Any other line (a table/CSV row, a `$ ...` line) is returned unchanged. - Two shapes are wrapped: a whole `ffmpeg ...` command, and one numbered - member of the listing a plan with a named pipe prints (`3. sidecar: + Two shapes are wrapped: a whole command (`ffmpeg ...`, or `ffrwd-wasm + ...` where a node source starts the pipeline), and one numbered member + of the listing a plan with a named pipe prints (`3. sidecar: ffrwd-wasm ...`), whose argv is wrapped and whose `N. role: ` lead stays on the first line. @@ -218,7 +219,7 @@ def wrap_command(line: str, width: int = _WRAP_WIDTH) -> str: A token with no safe split point is left long. """ listed = _LISTING_RE.match(line) - if listed is None and not line.startswith("ffmpeg "): + if listed is None and not line.startswith(("ffmpeg ", "ffrwd-wasm ")): return line lead = listed.group("lead") if listed is not None else "" tokens = [_quote(token) for token in shlex.split(line[len(lead):])] @@ -325,11 +326,11 @@ def _shell_tokens(text: str) -> list[str]: def _assert_shlex_invariant(actual: str, expected: str) -> None: - """For a code block that wrapped a single `ffmpeg` line, prove the wrap kept + """For a code block that wrapped a single command line, prove the wrap kept the same shell command: the wrapped block text and the original unwrapped line must tokenize identically.""" actual_line = actual.rstrip("\n") - if "\n" in actual_line or not actual_line.startswith("ffmpeg "): + if "\n" in actual_line or not actual_line.startswith(("ffmpeg ", "ffrwd-wasm ")): return assert _shell_tokens(expected) == shlex.split(actual_line) @@ -511,6 +512,16 @@ def test_wrap_command_leaves_a_token_with_no_safe_split_point_long() -> None: assert shlex.split(wrapped.replace("\\\n", "")) == shlex.split(line) +def test_wrap_command_wraps_a_pipeline_a_node_source_starts() -> None: + line = ( + "ffrwd-wasm -m ticker=ticker.wasm -filter_complex 'ticker=fps=30[video=out0]' " + "-map '[out0]' -f nut pipe:1 | ffmpeg -f nut -i pipe:0 -map 0:v:0 ticker.mp4" + ) + wrapped = wrap_command(line, width=60) + assert len(wrapped.split("\n")) > 1 + assert _shell_tokens(wrapped) == shlex.split(line) + + def test_wrap_command_is_deterministic() -> None: line = ( "ffmpeg -i song.m4a -filter_complex " diff --git a/cli/tests/test_node_world.py b/cli/tests/test_node_world.py index d5fa13a..2d368ab 100644 --- a/cli/tests/test_node_world.py +++ b/cli/tests/test_node_world.py @@ -15,14 +15,17 @@ import pytest -from ffrwd import shapes +from ffrwd import shapes, wasm +from ffrwd.compiler import compile_all from ffrwd.errors import ErrorCode, FfrwdError +from ffrwd.execute import plan_argv from ffrwd.ir import Graph from ffrwd.lower import lower from ffrwd.parser import parse, resolve from ffrwd.probe import ProbeResult, StreamMeta from ffrwd.registry import Registry, load_reference -from ffrwd.timing import check_live_leads, timing +from ffrwd.timing import check_live_leads, summary, timing +from ffrwd.warnings import FfrwdWarning, WarningCode from ffrwd.wasm import WORLDS, Described SNAPSHOT_PATH = Path(__file__).resolve().parent / "data" / "reference_registry.json" @@ -169,7 +172,7 @@ def _inset(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, obje def _tile(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: hold = {"kind": "hold", "anchor": {"kind": "shared-clock"}, "lead": 0} - clock = ( + clock: dict[str, object] = ( {"kind": "rate", "rate": {"num": int(str(params["fps"])), "den": 1}} if "fps" in params else {"kind": "rate-of", "port": "v"} @@ -226,6 +229,7 @@ def _ticker(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, obj name: {"type": "number" if name != "text" else "string"} for name in ("text", "width", "height", "fps") }, + "sub.wasm": {"relay": {"type": "string"}}, } @@ -291,7 +295,7 @@ def _registry() -> Registry: return load_reference(SNAPSHOT_PATH) -def _probes() -> dict[str, ProbeResult | None]: +def _probes(rate: int = 48000) -> dict[str, ProbeResult | None]: def media() -> ProbeResult: return ProbeResult( streams=[ @@ -301,7 +305,7 @@ def media() -> ProbeResult: ), StreamMeta( type="audio", index=0, metadata={}, width=None, height=None, - fps=None, sample_rate=48000, codec="aac", channels=2, + fps=None, sample_rate=rate, codec="aac", channels=2, ), ] ) @@ -608,7 +612,7 @@ def test_spans_reduce_a_nodes_rows_into_a_rows_file() -> None: ) (spot,) = [node for node in graph.nodes.values() if node.filter == "spot.wasm"] (spans,) = [node for node in graph.nodes.values() if node.filter == "rowmerge"] - assert (spans.inputs, spans.args) == ([spot.id], {"merge_spans": True, "max_span": 10}) + assert (spans.inputs, spans.args) == ([spot.id], {"max_span": 10}) assert graph.rows_sinks[spans.id].path == "spots.ndjson" @@ -702,3 +706,400 @@ def burn(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object "burn() needs 'words' 1 s ahead of its clock, and the path feeding it runs 2 s behind" ) assert (caught.value.line, caught.value.col) == (3, 4) + + +# -- on the sidecar's command line ------------------------------------------- + + +def _plan_argv( + query: str, monkeypatch: pytest.MonkeyPatch, rate: int = 48000 +) -> dict[str, list[str]]: + """Each process of the compiled plan as the printed command shows it.""" + probes = _probes(rate) + monkeypatch.setattr( + "ffrwd.compiler.probe_path", lambda path, args=(), **kw: probes[path[0]] + ) + compiled = compile_all(_declared(query), describe=_node, shape=_Asked()) + assert compiled.plan is not None + return plan_argv( + compiled.plan, + sidecar_argv=wasm.shown_argv, + pipe_path=lambda edge, side: f"<{edge.source}-{edge.target} {side}>", + ) + + +def test_a_node_network_names_the_port_each_pad_binds(monkeypatch: pytest.MonkeyPatch) -> None: + argv = _plan_argv( + "COPY (SELECT ring(f.video[1], spot(f.video[1])) FROM input('f.mp4') f) " + "TO 'ringed.mp4'", + monkeypatch, + ) + sidecar = argv["sidecar0"] + assert sidecar[sidecar.index("-filter_complex") + 1] == ( + "[v=0:v]spot=every=30[spots=n1];[v=0:v][spots=n1]ring[v=out0]" + ) + assert sidecar[sidecar.index("-map") :] == ["-map", "[out0]", "-f", "nut", "pipe:1"] + + +def test_every_stream_one_process_hands_a_node_network_rides_one_nut( + monkeypatch: pytest.MonkeyPatch, +) -> None: + argv = _plan_argv( + "COPY (SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), f.audio[1] " + "FROM input('f.mp4') f) TO 'burned.mp4'", + monkeypatch, + ) + sidecar = argv["sidecar0"] + assert sidecar.count("-i") == 1 + assert sidecar[sidecar.index("-filter_complex") + 1] == ( + "[a=0:a]hear[cues=n1];[v=0:v][a=0:a:1][words=n1]burn[v=out0]" + ) + (feeder,) = [ + words for pid, words in argv.items() if pid.startswith("ffmpeg") and "nut" in words + and words[-1] == "pipe:1" + ] + assert feeder.count("-map") == 3 + assert feeder[-3:] == ["-f", "nut", "pipe:1"] + + +def _taking( + shaped: Callable[..., dict[str, object]], kind: str, accepts: Mapping[str, object] +) -> Callable[..., dict[str, object]]: + """`shaped` with every input of `kind` accepting `accepts`.""" + + def taking(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + shape = shaped(params, bound) + inputs = shape["inputs"] + assert isinstance(inputs, list) + for port in inputs: + if port["kind"] == kind: + port["accepts"] = dict(accepts) + return shape + + return taking + + +def _feeder(argv: Mapping[str, list[str]]) -> list[str]: + (feeder,) = [ + words for pid, words in argv.items() if pid.startswith("ffmpeg") and "nut" in words + and words[-1] == "pipe:1" + ] + return feeder + + +def test_a_stream_two_nodes_read_crosses_in_the_format_both_take( + monkeypatch: pytest.MonkeyPatch, +) -> None: + rgba = {"pixel_formats": ["rgba"]} + monkeypatch.setitem(SHAPES, "spot.wasm", _taking(_spot, "video", rgba)) + monkeypatch.setitem(SHAPES, "ring.wasm", _taking(_reader("spots", _ROWS), "video", rgba)) + feeder = _feeder(_plan_argv( + "COPY (SELECT ring(f.video[1], spot(f.video[1])) FROM input('f.mp4') f) " + "TO 'ringed.mp4'", + monkeypatch, + )) + assert feeder[feeder.index("-pix_fmt:0") + 1] == "rgba" + + +def test_each_stream_of_one_nut_is_conformed_to_the_port_it_feeds( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setitem(SHAPES, "hear.wasm", _taking( + _hear, "audio", {"sample_formats": ["f32"], "sample_rates": [48000]} + )) + monkeypatch.setitem(SHAPES, "burn.wasm", _taking(_burn, "audio", {"sample_formats": ["f32"]})) + feeder = _feeder(_plan_argv( + "COPY (SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), f.audio[1] " + "FROM input('f.mp4') f) TO 'burned.mp4'", + monkeypatch, + rate=44100, + )) + assert [word for word in feeder if word.startswith("-ar")] == ["-ar:0"] + assert feeder[feeder.index("-ar:0") + 1] == "48000" + + +def test_a_node_read_in_from_has_no_input_and_its_reader_ends_it( + monkeypatch: pytest.MonkeyPatch, +) -> None: + argv = _plan_argv( + "COPY (SELECT s.video[1] FROM ticker('Nothing to see here') s WHERE s.t < 10) " + "TO 'ticker.mp4'", + monkeypatch, + ) + assert "-i" not in argv["sidecar0"] + reader = argv["ffmpeg0"] + assert reader[reader.index("-to") + 1] == "10" + + +def test_a_node_source_whose_relation_is_its_renditions_ends_where_its_reader_says( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def ticker(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + shape = _ticker(params, bound) + outputs = shape["outputs"] + assert isinstance(outputs, list) + outputs[0]["row"] = 0 + shape["relation"] = [json.dumps({"width": 1280, "height": 720})] + return shape + + monkeypatch.setitem(SHAPES, "ticker.wasm", ticker) + argv = _plan_argv( + "COPY (SELECT s.video[1] FROM ticker('Nothing to see here') s " + "WHERE s.height = 720 AND s.t < 10) TO 'ticker.mp4'", + monkeypatch, + ) + reader = argv["ffmpeg0"] + assert reader[reader.index("-to") + 1] == "10" + + +def test_spans_are_the_hosts_rowmerge_with_its_span_written_as_rows( + monkeypatch: pytest.MonkeyPatch, +) -> None: + argv = _plan_argv( + "COPY (SELECT ffrwd.merge_spans(spot(f.video[1]), max_span => 10) " + "FROM input('f.mp4') f) TO 'spots.ndjson'", + monkeypatch, + ) + sidecar = argv["sidecar0"] + assert sidecar[sidecar.index("-filter_complex") + 1] == ( + "[v=0:v]spot=every=30[spots=n1];[n1]rowmerge=max_span=10[out0]" + ) + assert sidecar[-5:] == ["-map", "[out0]", "-f", "ndjson", "spots.ndjson"] + + +def test_spans_without_a_span_are_refused() -> None: + with pytest.raises(FfrwdError) as caught: + _lowered("COPY (SELECT ffrwd.merge_spans(spot(f.video[1])) FROM input('f.mp4') f) " + "TO 'spots.ndjson'") + assert caught.value.message == "ffrwd.merge_spans() needs 'max_span'" + + +def test_explain_delays_says_each_window_and_each_outputs_delay() -> None: + graph = _lowered( + "COPY (SELECT burn(f.video[1], f.audio[1], hear(f.audio[1])), f.audio[1]" + _FROM + ) + timed = timing(graph, _probes(), {"hear.wasm": "hear", "burn.wasm": "burn"}) + assert timed is not None + assert summary(timed).splitlines() == [ + "hear: tumbling 2 s", + "burn: per-frame; words by interval, no bound", + "out.mkv stream 0 (video): 2 s behind the source", + "out.mkv stream 1 (audio): 0 s behind the source, waits 2 s", + ] + + +# -- coded packets ----------------------------------------------------------- + + +def _subscribe(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + def coded(name: str, codec: str, row: int, carried: dict[str, object]) -> dict[str, object]: + return { + "name": name, + "kind": "packets", + "format": { + "kind": "packets", + "codec": codec, + "time_base": {"num": 1, "den": 90000}, + "format": carried, + "extradata": "", + "profile": None, + "level": None, + }, + "latency": 0, + "row": row, + } + + return _shape( + [], + [ + coded("hd", "h264", 0, {"kind": "video", "width": 1280, "height": 720}), + coded("hd_audio", "aac", 0, {"kind": "audio", "sample_rate": 48000, "channels": 2}), + coded("sd", "h264", 1, {"kind": "video", "width": 640, "height": 360}), + ], + {"kind": "self_clocked"}, + bounded=False, + relation=['{"name": "720p", "bandwidth": 3000000}', '{"name": "360p"}'], + ) + + +def _remux(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + coded = {**_clock("v"), "kind": "packets", "accepts": {"codecs": ["h264"]}} + return _shape( + [coded], + [{"name": "v", "kind": "packets", "format": {"kind": "like", "port": "v"}, "latency": 0}], + {"kind": "input", "port": "v"}, + ) + + +def test_a_node_source_writing_coded_packets_binds_one_row_per_rendition() -> None: + SHAPES["sub.wasm"] = _subscribe + try: + graph = _lowered( + "CREATE FUNCTION sub(relay text) RETURNS source AS 'sub.wasm', 'sub' LANGUAGE wasm;\n" + "COPY (SELECT v.video[1] FROM sub('r') v WHERE v.height = 720) TO 'out.mkv'", + {"sub.wasm": _node("sub.wasm")}, + ) + finally: + del SHAPES["sub.wasm"] + (sub,) = [node for node in graph.nodes.values() if node.filter == "sub.wasm"] + assert sub.outputs == ["video", "audio", "video"] + assert [output.ref for output in graph.sinks[0].outputs] == [f"{sub.id}:0"] + + +def test_a_node_reading_coded_packets_is_handed_the_stream_as_it_was_coded( + monkeypatch: pytest.MonkeyPatch, +) -> None: + SHAPES["remux.wasm"] = _remux + _DECLARATIONS["remux"] = ( + "CREATE FUNCTION remux(v video_stream) RETURNS video_stream " + "AS 'remux.wasm', 'remux' LANGUAGE wasm;" + ) + try: + argv = _plan_argv( + "COPY (SELECT remux(f.video[1]) FROM input('f.mp4') f) TO 'out.mkv'", monkeypatch + ) + finally: + del SHAPES["remux.wasm"] + del _DECLARATIONS["remux"] + feeder = argv["ffmpeg1"] + assert feeder[feeder.index("-c:0") + 1] == "copy" + reader = argv["ffmpeg0"] + assert reader[reader.index("-c:0") + 1] == "copy" + + +def test_params_too_long_for_a_command_line_are_read_from_a_file( + monkeypatch: pytest.MonkeyPatch, +) -> None: + long = "x" * (shapes.PARAMS_INLINE_LIMIT + 1) + argv = _plan_argv( + f"COPY (SELECT s.video[1] FROM ticker('{long}') s) TO 'ticker.mp4'", monkeypatch + ) + sidecar = argv["sidecar0"] + assert sidecar[sidecar.index("-filter_complex") + 1] == "ticker[v=out0]" + assert sidecar[sidecar.index("-params-from") + 1] == "ticker=ffrwd:params:sidecar0:n1" + + +def test_a_packets_function_over_a_node_hands_back_the_stream_still_coded( + monkeypatch: pytest.MonkeyPatch, +) -> None: + SHAPES["remux.wasm"] = _remux + _DECLARATIONS["remux"] = ( + "CREATE FUNCTION remux(v video_stream) RETURNS packets " + "AS 'remux.wasm', 'remux' LANGUAGE wasm;" + ) + try: + argv = _plan_argv( + "COPY (SELECT remux(f.video[1]) FROM input('f.mp4') f) TO 'out.mkv'", monkeypatch + ) + finally: + del SHAPES["remux.wasm"] + del _DECLARATIONS["remux"] + sidecar = argv["sidecar0"] + assert sidecar[sidecar.index("-filter_complex") + 1] == "[v=0:v]remux[v=out0]" + assert argv["ffmpeg1"][argv["ffmpeg1"].index("-c:0") + 1] == "copy" + + +def test_a_live_query_feeding_a_node_later_than_its_bound_is_refused_at_compile( + monkeypatch: pytest.MonkeyPatch, +) -> None: + bounded = {"kind": "interval", "latency": 1.0, "ahead": 0} + + def burn(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("v"), _input("words", "data", bounded, schema=_CUE)], + [_output("v", "video")], + {"kind": "input", "port": "v"}, + ) + + live = _probes()["f"] + monkeypatch.setattr("ffrwd.compiler.probe_path", lambda path, args=(), **kw: live) + SHAPES["burn.wasm"] = burn + try: + with pytest.raises(FfrwdError) as caught: + compile_all( + _declared( + "COPY (SELECT burn(f.video[1], words => hear(f.audio[1])) " + "FROM input('srt://127.0.0.1:9000') f) TO 'out.mkv'" + ), + describe=_node, + shape=_Asked(), + ) + finally: + SHAPES["burn.wasm"] = _burn + assert caught.value.code is ErrorCode.LIVE_LEAD + assert (caught.value.line, caught.value.col) == (2, 17) + + +def test_a_stream_waiting_long_beside_a_later_one_is_warned_about( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def hear(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + return _shape( + [_clock("a", "audio", window=48000 * 1500)], + [_output("cues", "data", schema=_CUE)], + {"kind": "input", "port": "a"}, + ) + + probes = _probes() + monkeypatch.setattr( + "ffrwd.compiler.probe_path", lambda path, args=(), **kw: probes[path[0]] + ) + said: list[FfrwdWarning] = [] + SHAPES["hear.wasm"] = hear + try: + compile_all( + _declared( + "COPY (SELECT burn(f.video[1], words => hear(f.audio[1])), f.audio[1] " + "FROM input('f.mp4') f) TO 'out.mkv'" + ), + describe=_node, + shape=_Asked(), + on_warning=said.append, + ) + finally: + SHAPES["hear.wasm"] = _hear + assert [warning.code for warning in said] == [WarningCode.HELD_STREAM] + + +def test_a_field_only_a_node_makes_is_refused_for_a_module_of_an_older_world() -> None: + old = Described(world=WORLDS[-1], name="matte", pixel_formats=("rgba",)) + error = _refused("COPY (SELECT matte(f.video[1]).mask" + _FROM, {"matte.wasm": old}) + assert error.message == ( + "'.mask' is the stream matte() was handed, and a stream is not read back off a struct" + ) + + +def _weave(params: Mapping[str, object], bound: Sequence[str]) -> dict[str, object]: + coded = {**_clock("v"), "kind": "packets", "accepts": {"codecs": ["h264"]}} + clip = _input("clip", "data", {"kind": "interval", "ahead": 0}, schema=_ROWS) + return _shape( + [coded, *([clip] if "clip" in bound else [])], + [{"name": "v", "kind": "packets", "format": {"kind": "like", "port": "v"}, "latency": 0}], + {"kind": "input", "port": "v"}, + ) + + +def test_a_node_reading_packets_reads_what_its_destination_encodes( + monkeypatch: pytest.MonkeyPatch, +) -> None: + SHAPES["weave.wasm"] = _weave + _DECLARATIONS["weave"] = ( + "CREATE FUNCTION weave(v video_stream, clip STRUCT(start_t number, id number, " + "x number, y number, w number, h number)[] DEFAULT NULL) RETURNS packets " + "AS 'weave.wasm', 'weave' LANGUAGE wasm;" + ) + try: + argv = _plan_argv( + "COPY (SELECT weave(f.video[1], clip => spot(f.video[1])) FROM input('f.mp4') f) " + "TO 'out.mkv' WITH (video_codec 'libx264', crf 20)", + monkeypatch, + ) + finally: + del SHAPES["weave.wasm"] + del _DECLARATIONS["weave"] + (weave,) = [words for words in argv.values() if "weave=weave.wasm" in words] + assert weave[weave.index("-filter_complex") + 1] == "[v=0:v][clip=1:d]weave[v=out0]" + (encoder,) = [words for words in argv.values() if "libx264" in words] + assert encoder[encoder.index("-c:0") + 1] == "libx264" + (writer,) = [words for words in argv.values() if "out.mkv" in words] + assert writer[writer.index("-c:0") + 1] == "copy" diff --git a/docs/dialect.md b/docs/dialect.md index b5be84e..a2f363a 100644 --- a/docs/dialect.md +++ b/docs/dialect.md @@ -33,12 +33,19 @@ function := CREATE FUNCTION name(param ptype [DEFAULT literal], ...) RETURNS rty AS 'module', 'export' LANGUAGE wasm | CREATE FUNCTION name(rows annotation) RETURNS annotation AS 'module', 'export' LANGUAGE wasm + | CREATE FUNCTION name(port ntype [DEFAULT NULL] | param vtype + [DEFAULT literal], ...) + RETURNS nrtype AS 'module', 'export' LANGUAGE wasm ptype := text | number | boolean | vector | _stream | chapter | cue | attachment | any of those with [] | STRUCT(field vtype, ...)[] rtype := text | number | boolean | vector | _stream | chapter | cue | attachment | any of those with [] | TABLE(col type, ...) + | STRUCT(name wstype, name annotation) wstype := video_stream | audio_stream | either of those with [] wrtype := wstype | sink | packets | STRUCT(name wstype, name annotation) +ntype := wstype | data_stream | annotation | any of those with [] +nrtype := wstype | data_stream | annotation | source + | STRUCT(name wstype | data_stream | annotation, ...) annotation := STRUCT(field vtype, ...)[] | cue[] vtype := text | number | boolean | vector select := [WITH cte (, cte)*] SELECT columns FROM from [WHERE pred] @@ -499,6 +506,98 @@ dest := 'path' | STDOUT | ( value-expression ) | sink(value, ...) `moq.subscribe`'s rows); read the stream as `.v[1]` there. A `RETURNS sink` reading several streams takes no annotation column. Recipe [144](examples.md#144-hand-one-modules-rows-to-a-module-reading-two-streams). +- A **node `LANGUAGE wasm` function** names a module exporting + `ffrwd:av@0.19.0`'s `node`, which its describe says (`"world": + "node-module"`). What a node reads and writes is its SHAPE for each + call: `ffrwd-wasm --shape` with the call's params and the names of the + inputs it binds, asked once per distinct module, params and bound + inputs. Its parameters are ports and values, in any order: a port is a + `video_stream`, an `audio_stream`, a `data_stream` or rows + (`STRUCT(...)[]`, `cue[]`), named as the module names its input; a + value is text, number, boolean or vector, as any module's. Arguments + fill the parameters by position and then by name, ports included: + `burn(f.video[1], words => hear(f.audio[1]))`. How each input pairs + with the node's clock is the module's to say, not the query's. Recipe + [148](examples.md#148-a-node-reads-the-picture-the-sound-and-the-words-at-once). + - `DEFAULT NULL` on any port makes it optional: a call that leaves it + off, or writes NULL, binds nothing there, and the shape is asked + without it. A port the module requires is refused left off. A port + the declaration names and the shape for these params has none of is + fine unbound and refused bound, naming it. Recipe + [149](examples.md#149-leave-an-input-out). + - `[]` on a port takes every stream the argument holds, in order: + `tile(ARRAY[a.video[1], b.video[1]])`. A port without it given an + array is one call per element, as a filter's is. Recipe + [150](examples.md#150-tile-any-number-of-pictures). + - An input the module holds on a port of its own (`hold` with a + `port_param`) is given a stream, or a port number: written in its + place, or by the param's name, `inset(v, port => 9100)`. Given a + number it binds nothing, and whatever connects to that port is shown. + - `RETURNS` a stream; rows alone (`STRUCT(...)[]`, `cue[]`), a data + stream the query reads while it runs; `STRUCT( , ...)`, + one field per output the module makes, read off the call + (`matte(v).mask`) or every field at once with `.*` in a WITH body; or + `source`, a node that reads nothing, called in FROM. A field names + the module's output of that name, and a lone stream or rows the + module's one output of that kind. Recipes + [145](examples.md#145-a-detector-returns-its-rows-and-the-picture-stays-where-it-was), + [151](examples.md#151-a-node-makes-a-matte-and-the-rows-that-go-with-it). + - A call over a node making a stream and the rows beside it hands a + reader both, the stream into one port and the rows into the next: + `ring(matte(v))` is `ring(matte(v).mask, matte(v).spots)`. + - Rows match by field, compared as the JSON schemas of the two ports: + every field the reading port names is in the producer's, with a type + it takes (`integer` is a `number`), and fields beyond those pass. A + missing or mistyped field is refused naming both. Recipe + [146](examples.md#146-a-reader-names-only-the-fields-it-reads). + - One call is one node wherever the query writes it: the same module, + arguments and params are one instance, and each of its outputs goes + to every reader. Recipe + [147](examples.md#147-one-call-however-many-places-read-it). + - Rows a COPY selects are what a module's rows always are there: the + rows of a `.ndjson` destination, a WebVTT track anywhere else. A + gather, `ARRAY(SELECT r FROM unnest() r WHERE ...)`, narrows + them on their way, as it narrows a module's annotation column. + - A node read in FROM binds its outputs as an input binds its streams, + `s.video[1]` the first picture, and its relation rows as renditions, + so `WHERE s.height = 720` picks one. `WHERE s.t < 10` (or `<=`) ends + it: its reader takes that much and closes. One that never ends makes + the query live. Recipe + [154](examples.md#154-a-page-with-no-inputs-is-a-source). + - A port reading coded packets is handed an input's own stream, + copied as it was coded, in a codec the module takes; an output + writing them is copied by whatever reads it. + - One region of a sidecar holds the nodes the query wires together, + and everything one process hands another travels as one NUT. A + signature only a node can carry (kinds mixed, a stream left out, a + value among ports) is refused for a module of an older world with + the refusal such a signature always had. +- **`ffrwd.merge_spans(, max_span => )`** turns rows + written once per tick into spans. Rows sharing a `start_t` are one + span, which keeps the last row's fields and ends at the last tick that + carried it plus that tick's length; a tick with no row for it is a gap + inside it. A span still open after `max_span` seconds is written as it + stands and goes on as a new one, so `max_span` is also how late a span + row may leave. It is the host's own node and runs in the sidecar + beside the rows' producer. Recipe + [152](examples.md#152-spans-from-the-rows-that-said-so-frame-by-frame). +- **What each node waits for.** A node's clock input reads a window + (per-frame, tumbling, hopping, sliding), and each output may leave late + by a latency it declares; an input paired by interval waits for its + producer, at most its own bound past the clock. Summed along each path, + those say how far behind the source every stream a query writes runs. + `ffrwd explain --delays` prints a line per node (its window, and each + interval input's bound) and per output (its delay, and how long it + waits for the latest stream written beside it); `explain` carries the + same under `timing`. A stream waiting more than 512 MiB of itself is + warned about (`HELD_STREAM`). A live query feeding an input later than + the bound its node set on it is refused as `LIVE_LEAD`. Recipe + [153](examples.md#153-see-what-each-node-waits-for). +- A **sql function returning a stream and its rows**, `RETURNS + STRUCT( , )`, selects both in its body, and a + call over it reads as both: handed to a node it fills the port it + stands in and the rows port after it, `ring(spotted(v))`; read off it, + `spotted(v).spots` is the rows; `.*` in a WITH body names both. - Trailing `;` allowed; `--` and `/* */` comments allowed. Unquoted identifiers fold to lowercase. View, CTE, and alias names share one flat namespace across the whole script. diff --git a/docs/examples.md b/docs/examples.md index 0c6cd3d..a3fdc65 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1142,7 +1142,14 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +ffmpeg -i tests/fixtures/testsrc.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f nut \ + pipe:1 | ffrwd-wasm -f nut -i pipe:0 -m \ + spot=../sidecar/modules/target/wasm32-wasip2/release/spot.wasm -m \ + ring=../sidecar/modules/target/wasm32-wasip2/release/ring.wasm -filter_complex \ + '[v=0:v]spot=every=30[spots=n1];[v=0:v][spots=n1]ring[v=out0]' -map '[out0]' -f nut \ + pipe:1 | ffmpeg -copyts -f nut -analyzeduration 0 -fpsprobesize 3 -i pipe:0 -map 0:v:0 \ + -c:0 libx264 -crf:0 20 ringed.mp4 ``` `spot` writes one row per frame for as long as the mark is in view, and every row of one mark carries the pts it was first seen at as `start_t`: the row for frame t says what is true at t, so a reader needs no look-ahead and a run split across workers agrees on the ids. The old spelling, a module returning `STRUCT(v video_stream, spots ...)` with the picture untouched, is what a package keeps when its own module has not moved: inside the sidecar the rows ride the frames exactly as before. A migrated package that wants the old reading back writes it in SQL - `CREATE FUNCTION spotted(v video_stream) RETURNS STRUCT(v video_stream, spots STRUCT(...)[]) AS $$ SELECT v, spot(v) AS spots $$ LANGUAGE sql` - and `ring(spotted(v))` reads the record as the stream and the rows, as a call over a two-part result always has. @@ -1165,13 +1172,22 @@ RETURNS video_stream COPY ( SELECT dim(f.video[1], - ARRAY(SELECT s FROM unnest(spot(f.video[1])) s WHERE s.w * s.h > 400)) + ARRAY(SELECT s FROM unnest(spot(f.video[1])) s WHERE s.w >= 20)) FROM input('tests/fixtures/testsrc.mp4') f ) TO 'dimmed.mp4' WITH (video_codec 'libx264', crf 20) ``` ``` - +$ ffrwd compile -f query.sql +ffmpeg -i tests/fixtures/testsrc.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f nut \ + pipe:1 | ffrwd-wasm -f nut -i pipe:0 -m \ + spot=../sidecar/modules/target/wasm32-wasip2/release/spot.wasm -m \ + dim=../sidecar/modules/target/wasm32-wasip2/release/dim.wasm -filter_complex \ + '[v=0:v]spot=every=30[spots=n1];'\ +'[n1]rowfilter=pred={"ge"\\:\[{"field"\\:"w"}\,{"lit"\\:20}\]}[n2];'\ +'[v=0:v][boxes=n2]dim=amount=0.5[v=out0]' -map '[out0]' -f nut pipe:1 | ffmpeg -copyts \ + -f nut -analyzeduration 0 -fpsprobesize 3 -i pipe:0 -map 0:v:0 -c:0 libx264 -crf:0 20 \ + dimmed.mp4 ``` A field the reader names that the producer's rows lack, or carries with another type, is still refused at compile time, naming both. The WHERE runs inside the sidecar as before, and it may test fields the reader never sees. @@ -1198,7 +1214,21 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +# named pipes: ffmpeg0 reads sidecar0, sidecar0; sidecar0 feeds ffmpeg0, ffmpeg0 +1. ffmpeg: ffmpeg -i tests/fixtures/av.mp4 -f webvtt -i \ + '' -f nut -analyzeduration 0 \ + -fpsprobesize 3 -i '' -map 2:v:0 -map 0:a:0 -map \ + 1:s:0 -c:2 copy -c:0 libx264 -crf:0 20 -c:1 copy heard.mkv +2. ffmpeg: ffmpeg -i tests/fixtures/av.mp4 -map 0:a:0 -map 0:v:0 -ar:0 48000 -c:0 \ + pcm_f32le -c:1 rawvideo -pix_fmt:1 rgba -f nut pipe:1 +3. sidecar: ffrwd-wasm -f nut -i pipe:0 -m \ + hear=../sidecar/modules/target/wasm32-wasip2/release/hear.wasm -m \ + burn=../sidecar/modules/target/wasm32-wasip2/release/burn.wasm -filter_complex \ + '[a=0:a]hear[cues=out1];[v=0:v][words=out1]burn[v=out0]' -map '[out0]' -f nut \ + '' -map '[out1]' -f webvtt \ + '' +# this listing is not a shell command -- run the plan with `ffrwd run` ``` Two calls whose arguments differ are still two nodes. The split is of a data edge, so what it costs is a second copy of each row, not a second run. @@ -1225,7 +1255,15 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +ffmpeg -i tests/fixtures/av.mp4 -filter_complex '[0:a:0]asplit=2[out0][out2]' -map \ + '[out0]' -map 0:v:0 -map '[out2]' -ar:0 48000 -c:0 pcm_f32le -c:2 pcm_f32le -c:1 \ + rawvideo -pix_fmt:1 rgba -f nut pipe:1 | ffrwd-wasm -f nut -i pipe:0 -m \ + hear=../sidecar/modules/target/wasm32-wasip2/release/hear.wasm -m \ + burn=../sidecar/modules/target/wasm32-wasip2/release/burn.wasm -filter_complex \ + '[a=0:a]hear[cues=n1];[v=0:v][a=0:a:1][words=n1]burn[v=out0]' -map '[out0]' -f nut \ + pipe:1 | ffmpeg -i tests/fixtures/av.mp4 -f nut -analyzeduration 0 -fpsprobesize 3 -i \ + pipe:0 -map 1:v:0 -map 0:a:0 -c:0 libx264 -crf:0 20 -c:1 aac burned.mp4 ``` `hear` works two seconds of sound at a time and says so, and a window's cues leave with the window. `burn` reads them by interval, so the host holds each picture until the window holding its time is done, and the picture leaves `burn` two seconds behind the sound that enters it. [Recipe 153](#153-see-what-each-node-waits-for) shows where that number is printed. @@ -1254,7 +1292,15 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +ffmpeg -i tests/fixtures/av.mp4 -map 0:v:0 -map 0:a:0 -c:0 rawvideo -pix_fmt:0 rgba -c:1 \ + pcm_f32le -f nut pipe:1 | ffrwd-wasm -f nut -i pipe:0 -m \ + burn=../sidecar/modules/target/wasm32-wasip2/release/burn.wasm -m \ + inset=../sidecar/modules/target/wasm32-wasip2/release/inset.wasm -filter_complex \ + '[v=0:v][a=0:a]burn[v=n1];[v=n1]inset=port=9100:lead=0.5[v=out0]' -map '[out0]' -f nut \ + pipe:1 | ffmpeg -i tests/fixtures/av.mp4 -f nut -analyzeduration 0 -fpsprobesize 3 -i \ + pipe:0 -map 1:v:0 -map 0:a:0 -c:0 libx264 -crf:0 20 -c:1 aac inset.mp4 +# listens: sidecar0 at tcp://127.0.0.1:9100 for inset(feed) ``` `feed` is a hold input: whatever connects to port 9100 is shown at the picture's pace from `lead` seconds after its first frame arrives, the last frame held while it runs late, and the picture alone again when it ends. The compile listing names the port and the process that owns it, as it does for a switch's feeders. An input with no `DEFAULT NULL` is required, and a call that leaves it off is refused. @@ -1278,14 +1324,29 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +# named pipes: sidecar0 reads ffmpeg1, ffmpeg2, ffmpeg3 +1. ffmpeg: ffmpeg -copyts -f nut -analyzeduration 0 -fpsprobesize 3 -i pipe:0 -map 0:v:0 \ + -c:0 libx264 -crf:0 20 tiled.mp4 +2. ffmpeg: ffmpeg -i tests/fixtures/av.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f \ + nut '' +3. ffmpeg: ffmpeg -i tests/fixtures/av2.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f \ + nut '' +4. ffmpeg: ffmpeg -i tests/fixtures/testsrc.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba \ + -f nut '' +5. sidecar: ffrwd-wasm -f nut -i '' -f nut \ + -i '' -f nut -i \ + '' -m \ + tile=../sidecar/modules/target/wasm32-wasip2/release/tile.wasm -filter_complex \ + '[v=0:v][v=1:v][v=2:v]tile=columns=3[v=out0]' -map '[out0]' -f nut pipe:1 +# this listing is not a shell command -- run the plan with `ffrwd run` ``` A bare array column broadcasts over a filter, one call per element; over a module port declared as an array it is the port's whole list, and a module that wants one call per element is called under `unnest`. `audio_stream[]` and a rows parameter with `[]` on the record work the same way. A port that takes several streams cannot be the module's clock, which is why `tile` keeps time itself; the rate it keeps is read off the first picture by the compiler, which knows every stream's rate before anything runs. ## 151. A node makes a matte and the rows that go with it -A module that produces two things returns a record naming both: `matte` makes a mask of the mark it finds and a row per mark, and both leave the one node. Read either field off the call, or every field at once with `.*` in a WITH body; however the fields are read, the call is one instance: +A module that produces two things returns a record naming both: `matte` makes a mask of the mark it finds and a row per mark, and both leave the one node. Read either field off the call, or every field at once with `.*` in a WITH body; however the fields are read, the call is one instance, and here `dim` reads both: ```pgsql CREATE FUNCTION matte(v video_stream, every number DEFAULT 30) @@ -1303,13 +1364,20 @@ RETURNS video_stream COPY ( WITH m AS (SELECT (matte(f.video[1])).* FROM input('tests/fixtures/testsrc.mp4') f) - SELECT dim(m.mask, m.spots), m.spots + SELECT dim(m.mask, m.spots) FROM m ) TO 'matte.mkv' WITH (video_codec 'ffv1') ``` ``` - +$ ffrwd compile -f query.sql +ffmpeg -i tests/fixtures/testsrc.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f nut \ + pipe:1 | ffrwd-wasm -f nut -i pipe:0 -m \ + matte=../sidecar/modules/target/wasm32-wasip2/release/matte.wasm -m \ + dim=../sidecar/modules/target/wasm32-wasip2/release/dim.wasm -filter_complex \ + '[v=0:v]matte=every=30[mask=n10][spots=n11];[v=n10][boxes=n11]dim=amount=0.5[v=out0]' \ + -map '[out0]' -f nut pipe:1 | ffmpeg -copyts -f nut -analyzeduration 0 -fpsprobesize 3 \ + -i pipe:0 -map 0:v:0 -c:0 ffv1 matte.mkv ``` Each field is an output port with a format and a time base of its own, declared by the module for the call's parameters. [Recipe 94](#94-blur-the-people-and-only-the-people)'s `segment` is this shape, and its rows no longer ride the map's frames: they are a data stream beside it, which is why `mask_select` can read them from a call `segment` is not part of. @@ -1331,7 +1399,12 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +ffmpeg -i tests/fixtures/testsrc.mp4 -map 0:v:0 -c:0 rawvideo -pix_fmt:0 rgba -f nut \ + pipe:1 | ffrwd-wasm -f nut -i pipe:0 -m \ + spot=../sidecar/modules/target/wasm32-wasip2/release/spot.wasm -filter_complex \ + '[v=0:v]spot=every=30[spots=n1];[n1]rowmerge=max_span=10[out0]' -map '[out0]' -f \ + ndjson spots.ndjson ``` The rows out carry `start_t` and `end_t` beside the fields in, one row per span, so `spots.ndjson` holds one line per mark rather than one per frame. Rows whose fields are `start_t` and `text` reduce to cues, and selecting them beside a picture writes a subtitle track. A span row leaves when its span ends, so a span is as late as it is long; `max_span` bounds that, and it is what a reader pairing by time waits for. A span still open after ten seconds is written as it stands and goes on as a new one. The reducer closes a span on its producer's progress, not on the next row, so the last span of a run ends where the rows did. @@ -1358,11 +1431,14 @@ COPY ( ``` ``` -$ ffrwd explain -f query.sql - +$ ffrwd explain --delays -f query.sql +hear: tumbling 2 s +burn: per-frame; words by interval, no bound +burned.mp4 stream 0 (video): 2 s behind the source +burned.mp4 stream 1 (audio): 0 s behind the source, waits 2 s ``` -`hear` is a tumbling window of 2 s, so its cues trail the sound by up to 2 s and nothing more; `burn`'s picture is 2 s behind the source, and the sound written beside it waits in its pipe for the same 2 s, which `compile` sizes. On a live input the same sums decide whether a query can run at all: a node that must act ahead of time (an ad decision that needs `announce_before_s`, a playout that needs `lead_s`) fed by a path later than that lead is refused at compile time as `LIVE_LEAD`, naming the node, the lead it needs and the delay of the path feeding it. A file run has no such rule, since nothing there is late. +`hear` is a tumbling window of 2 s, so its cues trail the sound by up to 2 s and nothing more; `burn`'s picture is 2 s behind the source, and the sound written beside it, read straight from the file, waits those 2 s at the muxer, which `compile` sizes. On a live input the same sums decide whether a query can run at all: a node that must act ahead of time (an ad decision that needs `announce_before_s`, a playout that needs `lead_s`) fed by a path later than that lead is refused at compile time as `LIVE_LEAD`, naming the node, the lead it needs and the delay of the path feeding it. A file run has no such rule, since nothing there is late. ## 154. A page with no inputs is a source @@ -1382,7 +1458,12 @@ COPY ( ``` ``` - +$ ffrwd compile -f query.sql +ffrwd-wasm -m ticker=../sidecar/modules/target/wasm32-wasip2/release/ticker.wasm \ + -filter_complex \ + 'ticker=text=Nothing\ to\ see\ here:width=1280:height=720:fps=30[video=out0]' -map \ + '[out0]' -f nut pipe:1 | ffmpeg -copyts -f nut -analyzeduration 0 -fpsprobesize 3 -to \ + 10 -i pipe:0 -map 0:v:0 -c:0 libx264 -crf:0 20 ticker.mp4 ``` In a file run the source runs as fast as its reader drains it; in a live run it is paced to the wall clock. `WHERE s.t < 10` ends it after ten seconds, as it would any source. A network source is the same shape with a clock of its own: it emits when it has something, and `shape` may reach the network at compile time to learn its outputs, as a manifest is probed. `ffrwd.blitz.compose` with no streams, a page that animates on its own, is this recipe's shape too. From cfa2128fc5ef31adfa28bb98e69bdb0c7660e20f Mon Sep 17 00:00:00 2001 From: Jon-Carlos Rivera Date: Thu, 1 Oct 2026 18:59:42 -0700 Subject: [PATCH 06/58] feat(sidecar): bind the node world, and run every older world through it mod world_0190 binds node-module, values-module and codec-module from wit/; node_world.rs is the node binding, the tick resource with frames fetched on demand, same resolved to the input's own buffer, and set-params refused when the shape changes. --describe reports a node module's world, and --shape prints the WIT's node-shape as JSON for a module's params and bound inputs, the adapted shape for a module of an older world (NODE-SHAPE.md). node.rs holds the shape, tick and emission types with the host's refusals and the progress rule; tick.rs assembles ticks: window and stride, lockstep with audio re-cut to a video clock, interval with ahead and latency, arrival, data and packets clocks, earlier rows. adapters.rs gives every older world a node shape and node_loop.rs runs them all through one loop; the five single-world run loops leave main.rs, each world's command-line setup kept word for word in legacy.rs. Frame modules run as nodes inside the existing lanes. The sidecar's suite and the cli's pass as before; NODE-CLI.md is the spelling for node calls that the compiler lowers to and the host reads next. The six things the adapters could not say in a shape are B1 to B6 in the plan's gap log. Co-Authored-By: Claude Fable 5.1 --- sidecar/NODE-CLI.md | 163 ++ sidecar/NODE-SHAPE.md | 342 ++++ sidecar/ffrwd-wasm/src/adapters.rs | 1465 ++++++++++++++++ sidecar/ffrwd-wasm/src/legacy.rs | 592 +++++++ sidecar/ffrwd-wasm/src/main.rs | 1228 ++------------ sidecar/ffrwd-wasm/src/network.rs | 5 +- sidecar/ffrwd-wasm/src/node_loop.rs | 574 +++++++ sidecar/ffrwd-wasm/src/rows_chain.rs | 43 +- sidecar/ffrwd-wasm/src/scheduler.rs | 141 +- sidecar/ffrwd-wasm/src/shape_json.rs | 281 ++++ sidecar/ffrwd-wasm/src/tick.rs | 1472 +++++++++++++++++ sidecar/ffrwd-wasm/tests/node_shape.rs | 164 ++ sidecar/modules/Cargo.lock | 9 + sidecar/modules/Cargo.toml | 1 + sidecar/modules/blur-boxes/src/lib.rs | 2 +- sidecar/modules/data-echo/src/lib.rs | 2 +- sidecar/modules/data-stamp/src/lib.rs | 2 +- sidecar/modules/embed-notes/src/lib.rs | 2 +- sidecar/modules/fauxlate/src/lib.rs | 2 +- sidecar/modules/feed-probe-audio/src/lib.rs | 2 +- sidecar/modules/feed-probe/src/lib.rs | 2 +- sidecar/modules/gpu-probe/src/lib.rs | 2 +- sidecar/modules/invert/src/lib.rs | 2 +- sidecar/modules/newest-borrow/src/lib.rs | 2 +- sidecar/modules/note-rows/src/lib.rs | 2 +- sidecar/modules/packet-head/src/lib.rs | 2 +- sidecar/modules/packet-keys/src/lib.rs | 2 +- sidecar/modules/packet-passthrough/src/lib.rs | 2 +- sidecar/modules/packet-sei/src/lib.rs | 2 +- sidecar/modules/packet-stats/src/lib.rs | 2 +- sidecar/modules/packet-tally/src/lib.rs | 2 +- sidecar/modules/pad-rows/src/lib.rs | 2 +- sidecar/modules/shape-probe/Cargo.toml | 13 + sidecar/modules/shape-probe/src/lib.rs | 316 ++++ sidecar/modules/source-replay-data/src/lib.rs | 2 +- sidecar/modules/source-replay-pair/src/lib.rs | 2 +- sidecar/modules/source-replay/src/lib.rs | 2 +- sidecar/runtime/src/lib.rs | 1 + sidecar/runtime/src/node.rs | 717 ++++++++ sidecar/runtime/src/runtime.rs | 564 +++++-- sidecar/runtime/src/runtime/node_world.rs | 885 ++++++++++ sidecar/runtime/tests/node.rs | 222 +++ 42 files changed, 7819 insertions(+), 1421 deletions(-) create mode 100644 sidecar/NODE-CLI.md create mode 100644 sidecar/NODE-SHAPE.md create mode 100644 sidecar/ffrwd-wasm/src/adapters.rs create mode 100644 sidecar/ffrwd-wasm/src/legacy.rs create mode 100644 sidecar/ffrwd-wasm/src/node_loop.rs create mode 100644 sidecar/ffrwd-wasm/src/shape_json.rs create mode 100644 sidecar/ffrwd-wasm/src/tick.rs create mode 100644 sidecar/ffrwd-wasm/tests/node_shape.rs create mode 100644 sidecar/modules/shape-probe/Cargo.toml create mode 100644 sidecar/modules/shape-probe/src/lib.rs create mode 100644 sidecar/runtime/src/node.rs create mode 100644 sidecar/runtime/src/runtime/node_world.rs create mode 100644 sidecar/runtime/tests/node.rs diff --git a/sidecar/NODE-CLI.md b/sidecar/NODE-CLI.md new file mode 100644 index 0000000..b233468 --- /dev/null +++ b/sidecar/NODE-CLI.md @@ -0,0 +1,163 @@ +# Node calls on the sidecar's command line + +How a compiled query hands the sidecar a network that holds node modules +(`ffrwd:av@0.19.0`). Everything an older module takes stays as it is: the +same `-m`, the same positional pads, the same outputs. A node module is +told apart by its export, and its pads say which port they bind. + +What a node's ports are named, which are required, and what kind each +carries come from `ffrwd-wasm --shape` (`NODE-SHAPE.md`). Port names in +the examples below are the stand-in modules' own. + +## Inputs + + -f nut -i + +One `-i` is one edge: one NUT, carrying every stream that crosses it, +video, audio, coded packets and data (a JSON stream, codec `json`) in any +mix, interleaved by time. A producer writes all of an edge's streams into +that one pipe, so no stream of it can wait on another pipe. + +A pad names a stream of an input by class and position, as ffmpeg's +stream specifiers do: + +| Pad | Stream | +|---|---| +| `[0:v]`, `[0:v:1]` | input 0's first, second video stream, raw or coded | +| `[0:a]`, `[0:a:1]` | its audio streams, raw or coded | +| `[0:d]`, `[0:d:1]` | its data streams | + +`[N:v]` is `[N:v:0]`. A raw stream binds a video or audio port, a coded +one a packets port, a data stream a data port. + +## Nodes + + -m = + -filter_complex ';;...' + -params-from = + +A chain is input pads, the node, output pads, as for any module. For a +node module every pad carries the port it binds before an `=`: + + [=]...==:=[=