Skip to content

feat(v1): run-2 tooling: image degradations, paired landmark error, pilot - #10

Merged
Mattral merged 2 commits into
mainfrom
feat/run2-pilot-tooling
Oct 8, 2026
Merged

Mattral merged 2 commits into
mainfrom
feat/run2-pilot-tooling

Conversation

@Mattral

@Mattral Mattral commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Tooling for run 2's pilot and design (no change to existing behaviour unless the new options are used).

  • worldgap.data.degradations.ImageDegradation: graded, deterministic image degradations (darken, downscale, blur, noise, the last seeded per frame). Each is a frozen, JSON-describable value, so a pre-registered condition can be re-created exactly. These are simulated degradations of real footage and must be labelled so.
  • extract_rollout_from_video(..., frame_transform=...): applies the transform to each decoded frame before MediaPipe and records it in metadata["frame_transform"].
  • paired_landmark_error(reference, degraded): for paired data (same video, one degraded), the distance between the degraded hand landmarks and the clean detection of the same frame, in units of the clean hand's size. A left/right label swap isn't counted as error, and frames lacking a hand on either side are skipped (dropout is measured separately). It measures tracking quality, which the presence mask (a world-model input) doesn't contain. That's why it's the candidate ground truth for comparing the gap score against a trivial dropout-counting baseline: with dropout as the ground truth, that baseline is the ground truth (ρ = 1).
  • scripts/run_v1_pilot.py: for one clean recording and a severity grid, reports dropout, landmark error and the trivial baselines (no_hand_fraction, presence_density), plus how well the dropout baseline alone ranks landmark error. By design it computes no gap scores or gap correlations, so severities can't be chosen by looking at the tool's performance.
  • v1_pilot/ is gitignored.

Tests: tests/test_degradations.py (12):

  • each degradation, and refusal of invalid ones;
  • the loader hook, which applies the transform and records it;
  • paired error: zero for identical rollouts, exact units, label-swap tolerance, skipped frames, unpaired shapes refused;
  • a pilot-script smoke test asserting no gap or Fréchet field appears in its output.

Verification

  • pytest -q: 175 passed
  • ruff check .: clean; the new script is 100755
  • CI green on 3.10 / 3.11 / 3.12

🤖 Generated with Claude Code

Mattral and others added 2 commits October 8, 2026 13:04
…ilot

ImageDegradation applies graded, deterministic darken/downscale/blur/
noise to real frames; extract_rollout_from_video takes frame_transform
and records it in metadata. paired_landmark_error measures distance from
the same frame's clean detection in hand sizes, a quality signal the
presence mask does not contain. run_v1_pilot.py reports dropout,
landmark error and trivial presence baselines per severity and computes
no gap scores by design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Mattral
Mattral merged commit ff54d98 into main Oct 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant