Grabby

Video ingress pipeline for collecting and extracting frames from YouTube.

search → frames

Search finds videos and records them in a SQLite pipeline DB. Frames pulls pending URLs from the DB, fetches each stream directly, and extracts JPEGs — no full download required. Everything is resumable and idempotent.

Quick start

# 1. Build the image
docker compose build

# 2. Add a grabby.yaml to your workdir (see format below)

# 3. Run the pipeline
docker compose run --rm grabby python search.py
docker compose run --rm grabby python frames.py --interval 20 --sleep 1

Output lands in $GRABBY_WORKDIR/{source}/{category}/ as JPEGs, with pipeline.db and grabby.yaml at the workdir root.

The repo is bind-mounted into the container, so code edits take effect immediately — only rebuild (docker compose build) when requirements.txt or the Dockerfile changes. If the long command gets old:

alias grabby='docker compose run --rm grabby'

Workdir layout

workdir/                      (or any dir set via GRABBY_WORKDIR)
  grabby.yaml                 ← you provide this
  pipeline.db                 ← created automatically
  shards/                     ← per-worker DB copies during a parallel run
  youtube/
    sports/                  ← frames land here
    auto/
    gym/

grabby.yaml format

search_terms:
  sports:
    - top sports highlights
    - best sports reactions
  auto:
    - top cars of the year
    - driving videos
  gym:
    - workouts
    - weight lifting technique

Categories become subdirectories. Terms are searched once and skipped on re-runs.

search_terms is a section rather than the whole file, so grabby.yaml has room for other ingress settings later without changing format.

Ingress config stops here: grabby.yaml decides what gets collected and what folder it lands in. What gets detected in those images is a separate question, answered downstream by samantics via its own coco-spec.yaml and samantics.yaml. A category here needs no counterpart there — collect hockey frames for months before deciding what to look for in them.

Scripts

search.py — find videos

Searches YouTube for each term and records discovered URLs in pipeline.db. Already-searched terms are skipped automatically.

# Search all terms in workdir/grabby.yaml
docker compose run --rm grabby python search.py

# Single explicit query (note: files URLs under the 'query' category, not your own)
docker compose run --rm grabby python search.py --query "best gym workouts" --limit 50

# Preview what a given limit would search, without hitting the network
docker compose run --rm grabby python search.py --limit 50 --dry-run

# Force a re-search of every term
docker compose run --rm grabby python search.py --rerun

Raising --limit

search_log records the --limit each term ran under, so raising it re-searches exactly the terms that need it. A term is re-searched when its previous run came back full — results_count == limit_used, meaning YouTube had more to give — and the new limit is higher. A term that returned fewer results than it asked for was exhausted, so a bigger limit buys nothing and it stays skipped.

$ docker compose run --rm grabby python search.py --limit 50 --dry-run
[dry-run] SEARCH [youtube/sports] 'top sports highlights' — was capped at 25, limit now 50
[dry-run] skip   [youtube/auto]   'driving videos' — exhausted at 18/25 result(s)

A term that returned zero results is always retried — yt-dlp runs with ignoreerrors, so a transient network failure looks like an empty result set and shouldn't be recorded as a settled answer.

Rows written before the limit was tracked have no limit_used. Their limit is inferred as max(25, results_count) — a run that returned 499 results can't have had a limit below 499, so old high-limit runs aren't mistaken for truncated ones. Adjust the floor with --assume-prior-limit N.

frames.py — extract frames

Reads pending URLs from pipeline.db and extracts JPEG frames directly from the YouTube stream — no full video download needed. Marks each URL done/failed in the DB so runs are resumable.

# Extract a frame every 20 seconds, 1s pause between videos
docker compose run --rm grabby python frames.py --interval 20 --sleep 1

# Specific timestamps
docker compose run --rm grabby python frames.py --timestamps 1m30s 2m45s 4m10s

# Filter to one category
docker compose run --rm grabby python frames.py --interval 20 --category gym

# Retry previously failed videos
docker compose run --rm grabby python frames.py --interval 20 --retry-failed

# Single URL (bypasses DB, useful for testing)
docker compose run --rm grabby python frames.py https://youtube.com/watch?v=abc123 --interval 30

download.py — download full videos (optional)

Downloads full MP4 files when you need the video itself rather than just frames. Not required for the frames pipeline.

# Download a single URL
docker compose run --rm grabby python download.py https://youtube.com/watch?v=abc123

# Clip: extract a time range only
docker compose run --rm grabby python download.py https://youtube.com/watch?v=abc123 --start 1m30s --end 2m45s

# Member-only / age-gated content: export cookies on the host first
./docker/export-cookies.sh firefox
docker compose run --rm grabby python download.py https://youtube.com/watch?v=abc123 \
    --cookies /workdir/cookies.txt

--browser reads the cookie store directly, which only works outside the container — inside, use --cookies with a file exported by docker/export-cookies.sh.

Running in parallel

Frame extraction is I/O-bound — yt-dlp resolves the stream, then ffmpeg seeks into it — so N workers scale nearly linearly. The three pieces that make that safe:

Shard the queue by id, not by category. --shard i/n keeps only rows where id % n == i. Category sharding sounds natural and doesn't work: whenever one category dominates the backlog it becomes the floor on total runtime. With a backlog that is 64% one category, splitting by category caps you at 1.6×, while id % 8 gives 8.0×.

One SQLite file per worker. SQLite is single-writer; N workers on one pipeline.db over a bind mount will corrupt it. shards.py copies the DB per worker (via SQLite's backup API, so a live -wal comes along) and folds the results back afterwards.

A circuit breaker. --max-consecutive-failures (default 10) stops a run rather than draining the queue into failed rows. A bot_check or rate_limit stops it immediately regardless — those mean YouTube is refusing you, not this video, and continuing just converts the backlog into failures at N× speed.

# 1. split the queue
docker compose run --rm grabby python shards.py prepare 8

# 2. run 8 workers (WORKERS must match --scale)
WORKERS=8 docker compose up --scale worker=8 worker

# 3. watch from another shell
docker compose run --rm grabby python shards.py status

# 4. fold the results back into pipeline.db
docker compose run --rm grabby python shards.py merge

Each replica claims a shard for itself. Compose gives a container no way to know which replica it is — --scale names the containers grabby-worker-1..N but leaves $HOSTNAME as the container ID, and all replicas share one service definition — so instead each worker races to mkdir shards/.claims/<i> and the winner owns shard <i>. --scale alone sets the width, with no per-worker config. shards.py prepare clears the claims; if a run dies hard and you restart without re-preparing, clear them with rm -rf <workdir>/shards/.claims.

Workers start WORKER_STAGGER seconds apart (default 3) so N of them don't resolve their first video in the same second. Extraction settings come from FRAMES_ARGS:

WORKERS=4 FRAMES_ARGS="--interval 30 --sleep 2" docker compose up --scale worker=4 worker

Merging is idempotent — a second merge claims nothing and inserts nothing — so re-running it after a partial failure is safe.

How many workers?

The limit isn't CPU, it's how much YouTube tolerates. Only one request per video is bot-check-sensitive (the yt-dlp metadata resolve); the ~10 range reads ffmpeg makes afterwards go to a CDN with a signed URL. So the number that matters is metadata resolutions per minute, which is workers ÷ seconds-per-video.

Note that --sleep is a much smaller lever than it looks: at ~115s per video, --sleep 1 is under 1% of the cycle. What paces you is the work itself.

Start at 4, watch shards.py status for bot_check / rate_limit tags, and go up from there. The failure tags exist so that decision is evidence rather than vibes.

The n challenge solving failed warning

WARNING: [youtube] <id>: n challenge solving failed: Some formats may be missing.
Ensure you have a supported JavaScript runtime and challenge solver script
distribution installed.

This is not a block. yt-dlp needs a JS runtime and the EJS solver scripts to decrypt YouTube's n parameter, and this one message covers every way either can be missing. Two separate causes bit this image:

The runtime was too old. Debian bookworm's nodejs package is 18.x, and yt-dlp declares MIN_SUPPORTED_VERSION = (22, 0, 0) for node (yt_dlp/utils/_jsruntime.py). Node 18 is detected, marked unsupported and skipped — indistinguishable from node being absent, since node --version works fine. The Dockerfile now installs a pinned Node 22 tarball instead of the distro package.

The solver scripts weren't in the image. yt-dlp vendors the solver core plus deno/bun lib variants, but not the generic yt.solver.lib.js that the node runtime needs. Without the yt-dlp-ejs package it is downloaded from GitHub on every container start — eight downloads per run here — and any failure surfaces only as this warning.

check-solver.py covers both, asking yt-dlp's own detection rather than looking for binaries — a runtime can be present, on PATH and perfectly runnable while yt-dlp still refuses it:

docker compose build --no-cache
docker compose run --rm grabby python check-solver.py

  [ok  ] yt-dlp — 2026.07.04
  [ok  ] node runtime — v22.20.0 at /usr/local/bin/node (yt-dlp needs >= 22.0.0)
  [ok  ] yt-dlp-ejs package — 0.8.0
  [ok  ] yt.solver.lib.min.js — 147561 bytes, hash accepted
  [ok  ] yt.solver.core.min.js — 6945 bytes, hash accepted

It exits non-zero if anything is missing, so it also works as a preflight before a long run. The hash check matters as much as the presence check: if yt-dlp and yt-dlp-ejs drift apart in version, the scripts are silently rejected and yt-dlp falls back to the GitHub download — the same failure with an extra step.

It matters more than a missing-formats warning suggests: an unsolved n parameter yields stream URLs that YouTube throttles or rejects, which comes back as forbidden (403) tags — so a broken solver and a pile of 403s are usually the same problem.

forbidden (403) on every frame of a video

Failed at 20.0s: ffmpeg error -- Server returned 403 Forbidden (access denied)
Failed at 40.0s: ffmpeg error -- Server returned 403 Forbidden (access denied)

Every seek failing, rather than the occasional one, is a different problem from a signed URL expiring mid-extraction.

A googlevideo URL is bound to the client that requested it. yt-dlp resolves it with its own User-Agent and cookie jar; ffmpeg then fetched it with Lavf/... and no cookies, and Google answered 403 for every byte range. get_direct_url now carries that context across as ffmpeg -headers / -cookies, the same way yt-dlp's own FFmpegFD downloader does.

This gets worse when you add cookies, which is counter-intuitive: an authenticated session hands out URLs tied to that session, so the anonymous ffmpeg fetch that used to work starts failing.

Failure tags

Every failure is recorded with a tag, so the reasons are countable afterwards instead of a wall of unique strings:

tag	meaning	stops the run
bot_check	"Sign in to confirm you're not a bot"	yes
rate_limit	HTTP 429	yes
forbidden	HTTP 403 — usually a signed URL expiring mid-extraction	no
age_gate, private, members_only, removed, geo, live	this video, not you	no
no_duration, no_format	metadata missing	no
other	unrecognised — worth reading	no

A video where every seek failed is recorded as failed, not as done with zero frames. Otherwise a block would look like success and the video would leave the queue for good.

Configuration

GRABBY_WORKDIR

All scripts resolve paths relative to a workdir. On the host it's whatever GRABBY_WORKDIR points at; compose bind-mounts that directory to /workdir inside the container, so container-side paths are always under /workdir.

.env file (recommended, gitignored):

GRABBY_WORKDIR=/path/to/workdir

Inline (one-off override):

GRABBY_WORKDIR=/tmp/test docker compose run --rm grabby python search.py

--workdir flag (explicit per-command, container path):

docker compose run --rm grabby python frames.py --workdir /workdir/other-dataset --interval 20

If GRABBY_WORKDIR is unset, compose falls back to ./workdir in the repo.

Other flags

All scripts accept --source (default: youtube) and --db (default: <workdir>/pipeline.db). Run any script with --help for the full list.

Pipeline DB

pipeline.db is a SQLite file at the workdir root. It tracks:

• search_log — which terms have been searched, with results_count and the limit_used (enables resume and limit-aware re-search)
• urls — every discovered URL with frames_status (pending/done/failed)
• frames — one row per extracted JPEG with its timestamp

Query it directly if you need to inspect state:

# How many frames per category?
python3 -c "
import sqlite3
conn = sqlite3.connect('pipeline.db')
for row in conn.execute('SELECT category, COUNT(*) FROM frames JOIN urls ON urls.id=frames.url_id GROUP BY category'):
    print(row)
"

---

Downstream

Grabby's output is designed to feed directly into samantics, an auto-labelling pipeline that deduplicates frames, runs SAM3 segmentation, and exports COCO-format annotations. Point GRABBY_WORKDIR in samantics at the same directory.