Proposal: wasi-media

[kelnishi]

Motivation

While the wasi-gfx proposal provides standard interfaces for graphics rendering and windowing, there is currently no standard for handling the ingestion, compression, and packaging of audio and video streams.
wasi-media: a proposal for a standardized set of high-level WIT interfaces for media pipelines.
By shadowing existing W3C web standards (like WebCodecs, Media Capture, and Web Audio) where possible, a wasi-media would enable developers to write isomorphic media pipelines on top of wasm.

Scope

wasi-media API should be backend-agnostic and rely on robust capability negotiation to map guest requests to available hardware drivers or software fallbacks.
In Scope:

• Device enumeration and capture (webcams, microphones, desktop screen capture)
• Hardware-accelerated and software video/audio encoding and decoding
• Media container multiplexing and demultiplexing (e.g. mp4, mkv)
• Audio routing, mixing, and resampling
• Virtual loopback broadcasting (virtual webcam)

Out of Scope:

• Visual compositing, scaling, and 3D rendering (delegated to wasi-gfx)
• Window creation and UI event handling
• Direct access to GPU or other hardware

Use Cases

wasi-media workflows may include:

1. Basic Media Recording: Discover host devices (microphones, webcams), initiating a stream, passing raw frames to hardware encoders, and multiplexing the compressed chunks to a local file.
2. Stream Transcoding: A headless, server-side pipeline that demuxes an incoming RTSP or file stream, decodes the video, and re-encodes it to a different bitrate or resolution for distribution.
3. Real-Time Webcam Filters and Virtual Broadcasting: Acquiring a webcam source, passing the buffer to wasi-gfx or wasi-nn to apply a virtual overlay, and writing the composited frames back to the host OS as a virtual camera.
4. Non-Linear Video Editing (NLE): Loading disparate media clips, demuxing with precise seek-to-timestamp capabilities to generate frame previews, and rendering/muxing a finalized timeline to disk.
5. FFmpeg-style Processing: Stitching sequential rendered images into a video container, mixing multiple audio tracks with host-provided resampling, and muxing streams synthetically without relying on a real-time system clock.

Implementation Environments

The interfaces should be designed to be fulfilled by a variety of host embeddings:

• Web Browsers: Implemented via a lightweight JS shim that translates wasi-media calls directly to the standard WebCodecs, Media Capture, and Web audio APIs.
• Desktop/General Server Runtimes like Wasmtime: Backed by multimedia frameworks like GStreamer, FFmpeg (libavcodec/libavformat), or libwebrtc.
• Native Edge/Embedded: Wired directly to OS-level SDKs like AVFoundation (macOS), Media Foundation (Windows), or V4L2/VA-API (Linux) for native performance
• Virtualized Fallbacks: If a host lacks a specific hardware capability (like AV1 encoding), the component model allows linking a wasi-media import to a software-compiled wasm polyfill.

Draft WIT Definitions

1. Types (wasi:media/types.wit)
   Shared data structures representing raw buffers and compressed chunks.

package wasi:media@0.1.0-draft;

interface types {
    /// Import graphics abstract buffer for compositing.
    use wasi:gfx/graphics-context.{abstract-buffer}

    /// Represents standard video pixel formats for uncompressed frames.
    enum pixel-format {
        rgba8888,
        bgra8888,
        yuv420p,
        nv12,
    }

    /// Represents standard audio sample formats for uncompressed audio.
    enum sample-format {
        f32-planar,
        s16-interleaved,
    }

    /// An opaque handle to raw, uncompressed media data in memory.
    /// Designed to be interoperable with `wasi:gfx` abstract buffers.
    resource media-buffer {
        /// guest allocation of host-side buffers from bytes
        constructor(data: list<u8>);
        
        /// Factory method: guest allocation from floats:
        from-floats: static func(data: list<f32>) -> media-buffer;

        /// Zero-copy bridge from a graphics frame. 
        /// Uses a `borrow` of the abstract-buffer to access the underlying GPUTexture.
        from-gfx-buffer: static func(buf: borrow<abstract-buffer>) -> media-buffer;

        /// Returns the byte size of the underlying buffer.
        size: func() -> u64;
    }

    /// Represents a single, uncompressed frame of video.
    record video-frame {
        buffer: media-buffer,
        format: pixel-format,
        width: u32,
        height: u32,
        /// Presentation timestamp in microseconds.
        timestamp-us: u64,
    }

    /// Represents a chunk of uncompressed audio data.
    record audio-data {
        buffer: media-buffer,
        format: sample-format,
        sample-rate: u32,
        channels: u8,
        /// Presentation timestamp in microseconds.
        timestamp-us: u64,
    }

    /// Represents a packet of compressed media (audio or video).
    record encoded-chunk {
        data: list<u8>,
        /// Presentation timestamp in microseconds.
        timestamp-us: u64,
        /// The duration of the chunk in microseconds, if known.
        duration-us: option<u64>,
        /// Indicates if this chunk can be decoded independently.
        is-keyframe: bool,
    }
}

2. Codec (wasi:media/codec.wit)
   Hardware and software compression engine and capability negotiation.

interface codec {
    use types.{video-frame, audio-data, encoded-chunk};

    /// Supported video compression standards.
    enum video-codec-type {
        h264,
        hevc,
        av1,
        vp9,
    }

    /// Supported audio compression standards.
    enum audio-codec-type {
        aac,
        opus,
        mp3,
        flac,

        /// Uncompressed track types
        pcm-f32,
        pcm-s16
    }

    /// Indicates whether the host is utilizing hardware acceleration or a software fallback.
    enum codec-implementation {
        hardware,
        software,
    }

    /// Configuration parameters for initializing a video encoder.
    record video-encoder-config {
        codec: video-codec-type,
        width: u32,
        height: u32,
        bitrate: u32,
        framerate: u32,
        /// Optional parameters to declare incoming frame dimensions. 
        /// If different from target width/height, the host performs a native resize.
        source-width: option<u32>,
        source-height: option<u32>,
    }

    /// Configuration parameters for initializing an audio encoder.
    record audio-encoder-config {
        codec: audio-codec-type,
        sample-rate: u32,
        channels: u8,
        bitrate: u32,
    }

    /// The response from the host regarding requested codec capabilities.
    record encoder-support-result {
        supported: bool,
        implementation: codec-implementation,
    }

    /// Queries the host to check if a specific video configuration is supported.
    query-video-encoder-support: func(config: video-encoder-config) -> encoder-support-result;
    
    /// Queries the host to check if a specific audio configuration is supported.
    query-audio-encoder-support: func(config: audio-encoder-config) -> encoder-support-result;

    /// A resource representing an active video encoding session.
    resource video-encoder {
        /// Initializes the encoder. Traps if the configuration is unsupported.
        constructor(config: video-encoder-config);
        /// Submits a raw video frame for compression.
        encode: func(frame: video-frame, force-keyframe: bool);
        /// Retrieves compressed packets from the pipeline.
        get-encoded-chunks: func() -> list<encoded-chunk>;
        /// Forces the encoder to empty its internal buffers.
        flush: func();
    }

    /// A resource representing an active video decoding session.
    resource video-decoder {
        constructor(codec: video-codec-type);
        /// Submits a compressed packet for decoding.
        decode: func(chunk: encoded-chunk);
        /// Retrieves uncompressed video frames from the pipeline.
        get-decoded-frames: func() -> list<video-frame>;
        flush: func();
    }

    /// A resource representing an active audio encoding session.
    resource audio-encoder {
        constructor(config: audio-encoder-config);
        /// Submits raw audio data for compression.
        encode: func(data: audio-data);
        get-encoded-chunks: func() -> list<encoded-chunk>;
        flush: func();
    }

    /// A resource representing an active audio decoding session.
    resource audio-decoder {
        constructor(codec: audio-codec-type);
        /// Submits a compressed packet for decoding.
        decode: func(chunk: encoded-chunk);
        /// Retrieves uncompressed audio data from the pipeline.
        get-decoded-audio: func() -> list<audio-data>;
        flush: func();
    }
}

3. Capture (wasi:media/capture.wit)
   Hardware ingest and virtual loopback broadcasting.

interface capture {
    use types.{video-frame, audio-data};

    /// Categories of physical and virtual media devices.
    enum device-type {
        camera,
        microphone,
        screen,
        window,
        /// A virtual loopback device created by a Wasm guest.
        virtual-output,
    }

    /// Metadata describing an available capture device on the host.
    record device-info {
        id: string,
        name: string,
        type: device-type,
    }

    /// Request parameters for opening a media stream.
    record video-capture-constraints {
        device-id: option<string>,
        ideal-width: option<u32>,
        ideal-height: option<u32>,
        ideal-framerate: option<u32>,
    }

    /// Discovers available media capture devices on the host system.
    enumerate-devices: func() -> list<device-info>;

    /// A resource representing an active ingest stream from a physical device.
    resource media-stream {
        /// Opens a stream based on the provided constraints.
        constructor(constraints: video-capture-constraints);
        /// Pulls the latest video frame from the capture device buffer.
        read-video-frame: func() -> option<video-frame>;
        /// Pulls the latest audio data from the capture device buffer.
        read-audio-data: func() -> option<audio-data>;
        /// Closes the stream and releases the hardware.
        stop: func();
    }

    /// A resource that allows the Wasm component to present itself as a virtual device to the host OS.
    resource virtual-broadcaster {
        /// Registers a new virtual device with the host operating system.
        constructor(name: string, has-video: bool, has-audio: bool);
        /// Pushes a guest-generated video frame out to the host system as a virtual camera.
        write-video-frame: func(frame: video-frame);
        /// Pushes guest-generated audio out to the host system as a virtual microphone.
        write-audio-data: func(data: audio-data);
        /// Unregisters the virtual device.
        stop: func();
    }
}

4. Container (wasi:media/container.wit)
   Multiplexing and demultiplexing for streams and files.

interface container {
    use types.{encoded-chunk};

    /// Supported media packaging formats.
    enum container-format {
        mp4,
        mkv,
        webm,
        ts,
    }

    /// Metadata describing a specific stream within a multiplexed container.
    record track-info {
        id: u32,
        codec: string,
        is-video: bool,
    }

    /// A resource for packaging separate encoded tracks into a unified byte stream.
    resource muxer {
        /// Initializes a muxer for the target format.
        constructor(format: container-format);
        /// Registers a new video or audio track into the container header.
        add-track: func(info: track-info);
        /// Writes a compressed chunk to the multiplexed stream.
        write-chunk: func(track-id: u32, chunk: encoded-chunk);
        /// Retrieves the serialized container bytes to be written to disk or network.
        get-output-bytes: func() -> list<u8>;
        /// Finalizes the container's footers and metadata.
        finalize: func();
    }

    /// A resource for parsing a unified byte stream into separate encoded tracks.
    resource demuxer {
        /// Feeds raw container bytes (from a file or socket) into the parser.
        append-bytes: func(data: list<u8>);
        /// Reads the available tracks discovered in the container.
        get-tracks: func() -> list<track-info>;
        /// Pulls the next sequential compressed chunk for a specific track.
        read-chunk: func(track-id: u32) -> option<encoded-chunk>;
        /// Fast-forwards or rewinds the parser to the nearest keyframe at the target timestamp.
        seek-to-timestamp: func(time-us: u64) -> result<_, string>;
    }
}

5. Audio Mixer (wasi:media/audio-mixer.wit)
   Host-delegated audio resampling and channel mixing graph.

interface audio-mixer {
    use types.{audio-data};

    /// A resource representing a node-based audio processing and mixing graph.
    resource audio-context {
        /// Initializes the master output format. The host handles resampling to match this.
        constructor(master-sample-rate: u32, master-channels: u8);
        /// Registers an input track into the mix.
        add-source-track: func(track-id: u32);
        /// Submits raw audio data to a specific source track. 
        /// The host automatically resamples this data to match the master format.
        submit-audio: func(track-id: u32, data: audio-data);
        /// Applies gain (volume) adjustments to a specific track before mixing.
        set-track-gain: func(track-id: u32, gain: f32);
        /// Pulls the finalized, mixed audio data from the graph.
        read-master-output: func() -> option<audio-data>;
    }
}

6. Playback (wasi:media/playback.wit)
   Hardware playback of sampled media mirrors wasi-gfx surface.wit

interface playback {
    use types.{audio-data};
    use audio-mixer.{audio-context};

    /// Represents a physical audio output device on the host.
    record audio-output-info {
        id: string,
        name: string,
    }

    /// Queries the host for available speakers/headphones.
    enumerate-audio-outputs: func() -> list<audio-output-info>;

    /// Represents a connection to a physical output device.
    resource audio-player {
        /// Initializes playback on a specific device (or default if None).
        constructor(device-id: option<string>);

        /// Option A: Push/Enqueue raw audio data directly to the speaker.
        /// Useful for simple, unmixed playback (like an alert sound).
        enqueue-audio: func(data: audio-data);

        /// Option B: Link the speaker directly to the mixer's master output.
        /// The host will now automatically pull audio from the mixer graph 
        /// and play it out of the physical speakers in real-time.
        connect-mixer: func(context: audio-context);

        /// Playback state controls
        play: func();
        pause: func();
        set-system-volume: func(level: f32);
    }
}

7. Synchronization (wasi:media/clocks.wit)
   Centralized clock to prevent audio/video sync drift.

package wasi:media@0.1.0-draft;

interface clock {
    /// Defines what hardware or system mechanism drives the clock's progression.
    enum clock-source {
        /// Driven by the audio output hardware's buffer consumption rate.
        /// (Recommended master for media players).
        audio-playback,
        
        /// Driven by the display's VSync or surface presentation rate.
        /// (Recommended master for pure video/UI rendering or games).
        video-surface,
        
        /// Driven by the host OS monotonic system clock.
        /// (Recommended for headless environments or stream multiplexing).
        system-monotonic,
    }

    /// Represents a synchronized timeline for media playback and processing.
    resource media-clock {
        /// Initializes a new clock tied to a specific hardware or system source.
        constructor(source: clock-source);

        /// Starts or resumes the clock.
        start: func();

        /// Pauses the clock. The presentation time remains fixed until started again.
        pause: func();

        /// Sets the playback speed multiplier (e.g., 1.0 is normal, 2.0 is fast-forward, 
        /// -1.0 could theoretically be reverse if the demuxer supports it).
        set-rate: func(rate: f32);

        /// Seeks the clock's internal timeline to a specific timestamp in microseconds.
        /// Crucial for syncing the pipeline when a user scrubs an NLE timeline.
        seek: func(time-us: u64);

        /// Returns the current presentation time (PTS) in microseconds.
        /// Decoders and renderers constantly poll this to determine if a frame is due.
        now: func() -> u64;

        /// Blocks or yields guest execution until the clock reaches the target timestamp.
        /// This is the primary pacing mechanism. 
        wait-until: func(target-time-us: u64);
    }
}

[kelnishi]

Virtual webcams are kind of a nightmare, so I've struck them from the above use cases but theoretically should still be possible.

[guest271314]

Interesting. Non-trivial to test and maintain.

That capture group definition should have system audio and individual device audio, too; for the use case of capturing entire system audio "What-U-Hear"/speakers/headphones; and capturing specific devices, e.g., the output of a speech synthesis engine exclusively while other audio is being output to speakers.

Virtual devices are a thing.

[ValorZard]

Does this include WebRTC? If so, I’d like a datachannel api if possible.
If this does not include WebRTC, would a seperate proposal for that work?

[guest271314]

Does this include WebRTC? If so, I’d like a datachannel api if possible. If this does not include WebRTC, would a seperate proposal for that work?

libdatachannel has been compiled to WASM. Basically any networking API can be used to stream media. From fetch() to WebSocket to WebSocketStream to WebTransport to EventSource even.

[guest271314]

As always, media capture and playback capability depend on the specific device. Firefox took quite a while to start implementing Matroska support. Chromium proper does't support MP4 playback. MediaStreamTrack of kind audio is broken on Chrome; doesn't produce silence per W3C Media Capture and Streams. I'm not quite sure if Firefox has finally completed specification conformant implementation of HTML.captureStream(). Apple devices like H264 and AAC. And so forth. W3C Web Speech API still doesn't happen in the browser, and still doesn't have aa way to parse and handle a well-formed SSML document per the specification - and still no way to capture the speech output. So, having these media codecs and what's supposed to happen is one thing. What really happens on which device is a different matter, that requires constant testing.

[guest271314]

Now, what you can do in WASM world is build a stae machine that does what is says it does, in irrespective of devices doing this or that. You feed your data per specification, do what you do in WASM, and then the device holder is on their own to make it so on that specific device.