vod-transcode

A Rust worker that turns uploaded drone video into streamable video-on-demand. It consumes jobs from a RabbitMQ queue, pulls the source file from S3-compatible object storage, and writes back an HLS rendition, a subtitle track and a thumbnail.

Part of the ARU drone data platform — see Aru-Server, which publishes the jobs this worker consumes.

Transcoding is done with GStreamer through the gstreamer-rs bindings, writing directly to object storage via awss3hlssink rather than staging the output on local disk.

How it works

1. Declare a durable request queue (vod.transcode.req) and a durable response queue (vod.transcode.res).
2. Consume with prefetch 1 and manual acknowledgement, so one worker handles one job at a time and more replicas means more throughput.
3. For each job, build a GStreamer pipeline that decodes the source and tees it into several branches:
   • video — scaled and H.264-encoded into an HLS rendition
   • audio — encoded and muxed
   • subtitles — extracted to an .srt
   • thumbnail — a single frame to thumb.png
4. Publish a JSON response carrying the original metadata unchanged, a success flag, and the output paths.
5. Acknowledge the message only after the response is published, so a crash mid-job returns the work to the queue rather than losing it.

Output paths are derived from the input key: a job for vod/clip.mp4 produces vod/clip/index.m3u8, vod/clip/subtitle.srt and vod/clip/thumb.png.

Message shapes

Request:

{ "file": "vod/clip.mp4", "metadata": { "...": "opaque, echoed back" } }

Response:

{
  "metadata": { "...": "as sent" },
  "video": { "hls": "vod/clip", "srt": "vod/clip/subtitle.srt", "thumb": "vod/clip/thumb.png" },
  "success": true
}

metadata is passed through untouched so the caller can correlate a response with its request.

Requirements

• Rust 2021, stable toolchain
• GStreamer development libraries, including the bad plugin set — awss3hlssink lives in gst-plugins-bad
• A RabbitMQ instance and an S3-compatible object store

# macOS
brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad

# Debian / Ubuntu
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
                 gstreamer1.0-plugins-bad

Running it

cp .env.example .env     # adjust for your setup
docker compose up        # MinIO, RabbitMQ and the worker together

Or against your own infrastructure:

set -a && source .env && set +a
cargo run --release

test/ holds a small Node harness for publishing a job onto the queue and inspecting the response, which is the quickest way to exercise the worker by hand.

Configuration

Variable	Required	Purpose
S3_BUCKET	yes	bucket to read from and write to
S3_ENDPOINT	yes	object store endpoint URL
AWS_ACCESS_KEY_ID	no	defaults to minioadmin — see below
AWS_SECRET_ACCESS_KEY	no	defaults to minioadmin — see below
S3_PATH_STYLE	no	true for MinIO, false for AWS S3. Defaults to false
S3_REGION	no	defaults to ap-south-1
AMQP_ADDR	no	defaults to amqp://172.17.0.1:5672/%2f, the Docker bridge gateway
RUST_LOG	no	tracing filter. Defaults to info

S3_BUCKET and S3_ENDPOINT are read with .expect(), so the worker fails at startup if either is missing rather than running misconfigured.

A note on the credential defaults

The two AWS variables fall back to minioadmin / minioadmin, MinIO's documented defaults, so docker compose up works with no configuration. They are not Kesowa credentials and there is nothing to rotate.

But a silent fallback to a well-known credential is worth being deliberate about: a production deployment that forgets to set these does not fail, it quietly tries to authenticate with a public default. The bucket and endpoint fail loudly when unset; the credentials do not. Treat that asymmetry as a bug to fix rather than a feature — see CONTRIBUTING.md.

entrypoint.sh

A standalone gst-launch-1.0 pipeline kept for debugging — it transcodes a fixed /video.mp4 with hardcoded MinIO development credentials. It is not used by the worker and is not part of the deployed path. Useful for checking whether a GStreamer plugin problem is in the pipeline or in the Rust code around it.

Contributing

See CONTRIBUTING.md. Security reports go through SECURITY.md, not public issues.

License

Released under the MIT License. © Kesowa Infinite Ventures.

Note that GStreamer is LGPL, and the plugin sets vary: gst-plugins-bad and gst-plugins-ugly contain plugins under other licences, and some codecs carry patent obligations in some jurisdictions. x264enc, which this worker uses, is GPL. Linking dynamically keeps LGPL obligations satisfiable, but the plugin set you ship determines your actual obligations — audit it before distributing binaries. See CONTRIBUTING.md.