samantics

Auto-labelling pipeline for vision datasets. Takes raw frames from an ingress directory, deduplicates them, auto-labels with SAM3, curates the results, and exports COCO-format annotation JSON ready for training.

dedupe → label → curate

• dedupe — walks an ingress directory, runs dhash + ORB deduplication, copies accepted frames into a content-addressable image store
• label — runs SAM3 over accepted images, writes bounding boxes + RLE masks to labels.jsonl
• curate — keeps everything by default (opt-in filters), splits into train/val, exports coco/train.json + coco/val.json

Everything is resumable — pipeline.db tracks every image through all three stages so interrupted runs pick up where they left off.

---

Quick start

# 1. Copy .env.example and fill in your paths
cp .env.example .env

# 2. Build the Docker image
./docker/build.sh

# 3. Deduplicate frames from ingress
./docker/run.sh python dedupe.py

# 4. Auto-label (needs GPU)
./docker/run.sh python label.py

# 5. Curate and export COCO JSON
./docker/run.sh python curate.py

---

Configuration

All paths are set via environment variables. Copy .env.example to .env (gitignored) and edit:

DATASET_DIR=/path/to/my-dataset     # where pipeline.db, images/, labels.jsonl etc. live
GRABBY_WORKDIR=/path/to/ingress     # source frames (read-only in Docker)
MODEL_CACHE=/path/to/model-cache    # HuggingFace model cache (read-only in Docker)
HF_TOKEN=hf_...                     # optional, for gated models

DATASET_DIR is picked up automatically by all scripts. You can also pass --dataset-dir to any script to override it for a single run.

---

Dataset directory layout

{DATASET_DIR}/
  pipeline.db          ← SQLite pipeline state (WAL mode)
  images/              ← accepted frames, named {dhash}.jpg (content-addressable)
  labels.jsonl         ← SAM3 output: boxes (XYXY), category_ids, RLE masks, scores
  dataset.jsonl        ← curated subset of labels.jsonl
  coco/
    train.json         ← COCO format (XYWH boxes, RLE segmentation)
    val.json           ← COCO format
  coco-spec.yaml       ← category definitions (required by label.py and curate.py)
  samantics.yaml       ← labelling config: queries, always_query, thresholds

coco-spec.yaml and samantics.yaml are dataset-specific config that live alongside the data. Both are YAML for editing convenience; everything samantics writes — labels.jsonl, dataset.jsonl, coco/*.json — is still JSON.

coco-spec.yaml defines the category taxonomy. Supercategories group related classes — SAM3 queries are built per-supercategory. Category IDs can be any integers; leaving gaps between supercategories makes it easy to add new classes later.

info:
  year: 2025
  version: "1.0"            # quoted — bare 1.0 would parse as a float
  description: My Dataset
  contributor: you

licenses:
  - id: 1
    name: ""
    url: ""

categories:
  person:
    - id: 1
      name: person
      keypoints: [nose, left_eye, right_eye, left_shoulder, right_shoulder]
      skeleton:
        - [1, 2]
        - [1, 3]

  vehicle:
    - { id: 100, name: car }
    - { id: 101, name: truck }
    - { id: 102, name: airplane }

  animal:
    - { id: 200, name: dog }
    - { id: 201, name: cat }
    - { id: 202, name: bird }

  # Declared with no detections — labelled with the always-queries only.
  weather:

Categories nest under their supercategory; supercategory is re-attached on output, so coco/train.json gets the flat COCO list it always had. No images or annotations keys — those are filled in on output.

What the spec controls

Ingress (grabby) collects whatever its grabby.yaml says. The spec decides what labelling covers, and the three cases are distinct:

in the spec	what label.py does
declared with categories	queries those classes, plus any always-queried ones
declared, nothing under it	the always-queried supercategories only
not present at all	doesn't touch the images — they stay pending

The third case is why ingress and labelling can move independently: collect hockey frames for months before deciding what to detect in them. Categories the spec doesn't cover are reported at startup so they don't vanish silently:

Not in coco-spec.yaml, left unlabelled: boxing (3), cricket (2)
  5 image(s) stay pending. Declare the supercategory in the spec to label them.

Always-queried supercategories

always_query in samantics.yaml adds a supercategory to every image's query set, whatever its own category. There is no default — nothing is queried unless you ask for it, either in the config or with --always-supercategory (which overrides the config for that run).

In practice you almost always want people in the frame:

always_query:
  - person

With it unset, a supercategory declared with no categories has nothing at all to query, so label.py skips it and leaves those images pending:

Always queried: nothing (set always_query in samantics.yaml, or pass --always-supercategory person)
Declared with no detections: baseball, hockey -> queries: nothing
Skipping hockey (5 images): nothing to query — 'hockey' declares no categories
  and no --always-supercategory was given. Left pending.

Every run prints what it resolved to, so a missing setting shows up in the first few lines rather than in the annotation counts a day later:

Config:  /dataset/samantics.yaml
Always queried: person
Thresholds: score=0.5 iou=0.9 | model: facebook/sam3

Labelling is one-shot per image. An image labelled person-only under an empty supercategory is marked done and won't be revisited when you add classes for it. To pick those back up:

python label.py --relabel-supercategory hockey

Two YAML gotchas the loader guards against: a bare 1.0 or 2025-09-17 is a float and a date, not a string (quote them), and a category or supercategory literally named no, on, or yes parses as a boolean (quote it, or the loader will refuse the file). Duplicate category ids are rejected too.

No images or annotations keys — those are filled in on output.

Two YAML gotchas the loader guards against: a bare 1.0 or 2025-09-17 is a float and a date, not a string (quote them), and a category literally named no, on, or yes parses as a boolean (quote it, or the loader will refuse the file). Duplicate category ids are rejected too.

Converting an older coco-spec.json

python spec_convert.py coco-spec.json coco-spec.yaml

It converts both directions, nesting on the way in and flattening on the way out. If other tooling of yours reads the JSON form, keep the YAML as the source of truth and regenerate the JSON rather than hand-editing both:

python spec_convert.py coco-spec.yaml coco-spec.json

A supercategory declared with no categories is the one thing flat JSON can't represent; converting out and back drops it, and the converter warns when it does.

samantics.yaml is the labelling config: what to query, and how. It's optional — a dataset with no overrides needs no config at all.

# Text prompt sent to SAM3 per category. The category name is used by default;
# only list a category when the default gives poor results.
queries:
  pickleball:
    pickleball: ball
    pickleball_paddle: racket

# Queried on every image regardless of its own category. This is what gives
# supercategories with no detections of their own something to find.
always_query:
  - person

score_thresh: 0.5
iou_thresh: 0.9
model_id: facebook/sam3

CLI flags override anything here. The dividing line for what belongs in the file is does it change the output, or just how fast the run goes? — --batch-size and --cache-dir are properties of the machine you happen to be on, so they stay flags and are rejected if you put them in the config.

What was this image actually labelled with?

Every labelled image records the query set it was produced under. pipeline.db gets a label_configs table (hash → the full resolved config) and images.label_config pointing at it, so the question is a join rather than a guess.

The recorded unit is the query set for one supercategory, not the whole file. Adding a hockey class makes hockey images stale and leaves curling alone; changing score_thresh makes everything stale. Hashing the whole config would flag every image in the dataset for any edit.

label.py reports it on every run:

Labelled under a different config: curling (13277), pickleball (25460)
  Re-run with --relabel-stale to send them back to pending.

./docker/run.sh python label.py --relabel-stale

Images labelled before config tracking existed have no recorded config. They're reported as unknown and never counted as stale — assuming they match the current config, or that they don't, would both be wrong:

67462 image(s) labelled before config tracking — config unknown, not counted as stale.

To re-label those, target them explicitly with --relabel-supercategory <name>.

dedupe.py — deduplicate and ingest frames

Walks the ingress directory ({GRABBY_WORKDIR}/{source}/{category}/*.jpg), discovers all frames, and runs two-pass deduplication:

1. dhash — perceptual hash; images within a Hamming distance threshold are near-duplicates globally across the whole dataset
2. ORB — per-video feature matching; catches similar frames from the same video that dhash misses (slight motion, lighting changes)

Accepted images are copied to {DATASET_DIR}/images/{dhash}.jpg.

# Run everything (paths from .env)
./docker/run.sh python dedupe.py

# Filter to one category
./docker/run.sh python dedupe.py --category gym

# dhash only, no ORB (faster)
./docker/run.sh python dedupe.py --no-orb

# Dry run — count without writing
./docker/run.sh python dedupe.py --dry-run

label.py — SAM3 auto-labelling

Reads accepted images from pipeline.db, runs SAM3 inference in batches, appends results to labels.jsonl. Each image is marked done or failed in the DB so runs are resumable.

Requires coco-spec.yaml in DATASET_DIR. samantics.yaml is optional.

# Label all pending images
./docker/run.sh python label.py

# Filter to one category, retry previously failed
./docker/run.sh python label.py --category animal --retry-failed

# Custom batch size and confidence threshold
./docker/run.sh python label.py --batch-size 32 --score-thresh 0.6

# Person-only pass over a supercategory that has no detections defined yet
./docker/run.sh python label.py --category hockey

curate.py — curate, split, export COCO

Reads labels.jsonl, keeps every labeled image by default — empty and person-only frames are exported too, so they can train as background negatives downstream. Filtering is opt-in via --drop-empty / --drop-only-person. Splits into train/val and writes coco/train.json + coco/val.json. Updates curate_status in pipeline.db for each image.

# Standard run (85/15 split, nothing filtered)
./docker/run.sh python curate.py

# Custom split ratio and seed
./docker/run.sh python curate.py --train 80 --seed 1

# Keep only specific categories
./docker/run.sh python curate.py --categories animal vehicle

# Filter out empty and person-only frames (the old default behavior)
./docker/run.sh python curate.py --drop-empty --drop-only-person

stats.py — pipeline and annotation statistics

Prints a full picture of pipeline state and annotation counts. Useful for checking imports and dataset balance before training.

# Full report (pipeline DB + annotation stats from labels.jsonl)
./docker/run.sh python stats.py

# Stats from a specific file
./docker/run.sh python stats.py /path/to/labels.jsonl

# Include filtered-out images in counts
./docker/run.sh python stats.py --keep-empty --keep-only-person

split.py — re-split an existing JSONL

Splits any JSONL file into train/val (or train/val/test) by percentage. Useful for re-splitting without re-running the full curation step.

python split.py dataset.jsonl train.jsonl val.jsonl --train 80
python split.py dataset.jsonl train.jsonl val.jsonl test.jsonl --train 75 --val 15

to_coco.py — convert JSONL to COCO JSON

Converts any labels JSONL file to COCO annotation format. Useful for one-off conversions outside the main pipeline.

python to_coco.py labels.jsonl output.json
python to_coco.py labels.jsonl output.json --spec /path/to/coco-spec.yaml

import_existing.py — migrate from a previous pipeline

If you have data from an older separate-directory pipeline layout (01-selection/, 02-labels/, 03-curation/), import it into pipeline.db without re-running anything:

python import_existing.py --data-root /path/to/old-dataset

# Dry run first to verify counts
python import_existing.py --data-root /path/to/old-dataset --dry-run

Then run curate pointing at the existing labels file:

./docker/run.sh python curate.py --labels-jsonl /path/to/old-dataset/02-labels/labels.jsonl

---

Pipeline DB

pipeline.db is a SQLite database tracking every image through all three stages:

Stage	Column	Values
select	select_status	pending / accepted / rejected
label	label_status	pending / done / failed
curate	curate_status	pending / included / excluded

Useful queries:

# What failed labelling and why?
sqlite3 $DATASET_DIR/pipeline.db \
  "SELECT source_path, label_error FROM images WHERE label_status='failed' LIMIT 20"

# Which videos contributed the most accepted frames?
sqlite3 $DATASET_DIR/pipeline.db \
  "SELECT video_id, COUNT(*) n FROM images WHERE select_status='accepted'
   GROUP BY video_id ORDER BY n DESC LIMIT 10"

# Accepted but not yet labelled (work remaining before curate)
sqlite3 $DATASET_DIR/pipeline.db \
  "SELECT COUNT(*) FROM images WHERE select_status='accepted' AND label_status != 'done'"

---

Ingress layout

Frames are sourced from grabby, which organises output by source and category:

{GRABBY_WORKDIR}/
  {source}/              e.g. youtube/
    {category}/          e.g. gym/
      *.jpg              frames named {video_id}_{timestamp_ms}.jpg

Set GRABBY_WORKDIR in .env to point at grabby's output directory.

---

Docker

docker/run.sh loads .env, expands paths, and mounts:

Host path	Container path	Mode
DATASET_DIR	/dataset	read/write
GRABBY_WORKDIR	/grabby-workdir	read-only
MODEL_CACHE	/model-cache	read-only
project root	/workspace	read/write

./docker/build.sh                        # build image tagged 'samantics'
./docker/run.sh python label.py          # run a script
./docker/run.sh python stats.py          # check pipeline state
./docker/run.sh bash                     # interactive shell

GPU is used automatically if the nvidia Docker runtime is present.