Air Canvas / Virtual Painter

Draw in the air with your index finger. A webcam tracks your hand with MediaPipe, and your strokes are painted onto a separate canvas layer that is composited over the mirrored camera feed in real time.

Gestures

The pen has a clutch: pointing moves the cursor, and ink only flows while you pinch. Lifting the pinch drops the stroke anchor, so separate letters are never joined by a travel line.

Hand state	Mode	What happens
Index finger extended	PEN UP	The cursor moves freely — nothing is painted. Reticle is a hollow white crosshair
... + pinch thumb (4) to index tip (8)	PEN DOWN	Index fingertip paints a smoothed line in the selected color. Reticle turns into a filled dot with a green ring
... or index tip tucked to the middle-finger base (9)	PEN DOWN	Same as the pinch — a second way to press the pen down
Index + middle up, held together	ERASE	A red circular eraser follows the midpoint of landmarks 8 and 12 and clears canvas pixels inside its radius
Anything else (fist, open hand, spread "peace" sign)	IDLE	Nothing is drawn or erased

Releasing the pinch lifts the pen, so the next pinch starts a brand new stroke. The clutch has hysteresis — a firm pinch to engage (PINCH_ENGAGE_RATIO), a clear release to disengage (PINCH_RELEASE_RATIO) — which stops the pen flickering on the threshold. The status bar shows a PINCH meter that fills as your fingers close and turns green the moment the pen engages.

Separating the fingers turns the eraser off, so a spread V-sign is a safe "parked" pose. Every proximity test is scaled by your palm size, so gestures behave the same close to or far from the camera.

Top HUD

A color bar spans the top of the window: Blue · Green · Red · Yellow · Clear. Hover a box to highlight it, then pinch to select that color; the last box wipes the canvas. A selection fires once per pinch, not once per frame, and strokes are never painted inside the header area.

The bottom status bar carries the current mode, the active colour, the eraser size, the PINCH meter and a live FPS read-out — green at 30 fps or better, amber from 20, red below. The session average is printed to the terminal on exit, so a run can be benchmarked without watching the HUD.

Performance

The loop was profiled and reworked to hold the camera's full frame rate on Apple Silicon. Measured on an M-series MacBook, 1280x720, MediaPipe 0.10.14, OpenCV 4.11, hand in frame, over an 8 s steady-state window:

Build	Steady-state FPS
Original	16.4
Optimized, but --no-threaded-capture	18.9
Optimized (default)	30.2 — the camera's own ceiling

The webcam tops out at 30 fps in every mode it offers, and cap.read() blocks for a full 33.4 ms waiting for it. The app is now camera-bound: the pipeline itself renders a frame in 15.3 ms, so there is roughly 2x headroom over what the sensor can deliver.

Per-frame pipeline cost at 1280x720, excluding the camera:

Stage	Before	After
Full loop, processing only	18.95 ms	15.33 ms
Full loop, including imshow	36.46 ms	30.64 ms
Canvas compositing	1.77 ms	0.65 ms
HUD (header + status bar)	1.60 ms	0.46 ms
cap.read()	33.4 ms, serial	overlapped

What changed

Camera reads moved to a background thread. This is the single biggest win — the two rows above that differ only in --no-threaded-capture are 18.9 vs 30.2 fps. A blocking cap.read() costs a whole sensor interval, and paying it inline makes capture latency and processing latency add up instead of overlap. OpenCV releases the GIL inside read(), so a plain thread really does overlap them. Only the newest frame is kept — no queue, no growing latency — and the main loop waits for a frame it has not seen yet, so the same capture is never processed twice and the FPS counter reports real throughput.

Compositing is premultiplied-alpha blending, in place, over a dirty rectangle. The old path converted the canvas to greyscale, thresholded it, inverted the mask, ran two bitwise_ands and an add — five full passes over 2.7 M pixels, allocating a new frame each time. Strokes are antialiased over black, so the canvas already holds colour x coverage; keeping the inverse coverage in a second layer, built by drawing the same primitives in black onto white, reduces the whole thing to

frame = frame * (1 - a) + layer

two SIMD passes confined to the bounding box that actually has ink, writing straight through a NumPy slice with no allocation and no copy. An untouched canvas costs a single branch.

That is also more correct than the code it replaced. The old binary mask stamped every antialiased edge pixel at its raw premultiplied value, which painted a thin dark outline around every stroke. Against a float-precision reference composite, the old path was off by up to 131/255 at the edges (mean 1.87); the new one is off by at most 2 (mean 0.02) — pure uint8 rounding.

The HUD blends only its own rows. Both the header and the status bar used to frame.copy() the whole 1280x720 image and addWeighted all of it just to tint a 90-pixel and a 46-pixel strip. They now blend their own slice against a prerendered strip, and the palette boxes and text metrics — which never change — are built once at startup instead of every frame.

Landmark maths avoids NumPy entirely. np.hypot on two scalars builds array machinery to compute one float. Distances are plain math.hypot on tuples, comparisons that only need an ordering use squared distances and skip the square root, the EMA keeps two floats instead of allocating a point, and the finger-extension state is computed once per frame and shared by both gesture tests rather than recomputed by each.

Inference runs on a pre-scaled frame (--infer-width, default 640) using buffers allocated once and reused, so the hot path performs no per-frame allocation and converts colour on a quarter of the pixels. MediaPipe returns normalised coordinates, so landmarks project straight back onto the full-resolution frame — the canvas and the webcam feed still render at native capture size.

Honest caveat on pre-scaling: it is not where the time goes on this stack. mp.solutions.hands resizes internally to the model's fixed input tensor, so hands.process() costs the same ~14 ms whether it is handed 1280x720 or 480x270 — measured flat within 5% across all four sizes. Pre-scaling still saves the full-resolution cvtColor and the larger buffer handoff, and it costs nothing, so it is on by default; but running --infer-width 0 gives back only a fraction of a millisecond. The frame rate came from the threaded capture and the compositing and HUD work, not from here.

The help overlay retires itself after 8 seconds. Ten antialiased putText calls cost ~1.2 ms per frame, which is a lot to spend forever on text the user has already read. h brings it back and then keeps manual control.

Still mp.solutions.hands. The Tasks API was deliberately left alone — see Why the classic Solutions API.

Reproducing the numbers

Run the app and read the FPS counter in the status bar, or check the line printed on exit:

$ python main.py
281 frames in 9.4s (29.8 FPS average)

Requirements

• Python 3.9 – 3.12. mediapipe==0.10.14 publishes no wheels for Python 3.13+, so a newer interpreter will fail to install it.
• A webcam.

Install

# From the project directory. Use a 3.9–3.12 interpreter:
python3.11 -m venv .venv          # or: ~/.pyenv/versions/3.11.10/bin/python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run

python main.py                 # default camera
python main.py --camera 1      # pick a different camera
python main.py --no-skeleton   # hide the hand landmark overlay
python main.py --width 960 --height 540

All options

Flag	Default	What it does
--camera N	0	Camera index
--width / --height	1280 / 720	Requested capture size (the camera may ignore it; the first frame wins)
--fps N	30	Capture rate requested from the camera
--infer-width N	640	Width the frame is scaled to before hand tracking. 0 tracks at full capture resolution. Drawing and display always stay at native resolution
--model-complexity {0,1}	1	MediaPipe hand model. 0 is ~2.4 ms/frame cheaper and slightly less precise
--alpha F	0.5	EMA smoothing, 0 < alpha <= 1; lower = smoother, laggier
--no-skeleton	off	Hide the landmark overlay
--no-threaded-capture	off	Read the camera inline instead of on a background thread. Costs ~11 fps — for debugging only

Keyboard controls

Key	Action
q / Esc	Quit
c	Clear the canvas
s	Save the canvas (strokes only, transparent-black background) as a timestamped PNG
[ / ]	Shrink / grow the eraser radius
h	Toggle the on-screen help overlay (it auto-hides after 8 s; pressing h takes over from then on)

How it works

 capture thread                  main loop
 ──────────────                  ─────────
 cap.read() ──► newest frame ──► flip(mirror) ─┬─► downscale to 640 ──► MediaPipe Hands
   (blocks ~33 ms,               (waits for a  │                              │
    overlapped)                   fresh frame) │                        21 landmarks
                                               │                              │
                                               │                    finger state ──► mode
                                               │                              │
                                               │        PEN DOWN: cv2.line(canvas, prev, cur, color)
                                               │        PEN UP:   prev_point = None (no ink)
                                               │        ERASE:    cv2.circle(canvas, mid, r, 0, -1)
                                               │                              │
                                               └─► composite_into(frame) ◄────┘
                                                          │
                                                    HUD ──► cv2.imshow

Landmarks come back normalised, so tracking at 640 wide costs nothing in accuracy of placement — the canvas and the displayed feed stay at full capture resolution.

• DrawingCanvas (main.py) owns an isolated H×W×3 uint8 array plus its inverse-coverage twin. The canvas is never drawn on by the camera and vice versa: cursors, the eraser visualizer, the HUD and the landmark skeleton go onto the mirrored frame, the strokes stay on the canvas, and the two meet only in composite_into.
• Compositing blends the canvas over the frame with the premultiplied-alpha over operator, in place, over the inked bounding box only — see Performance.
• PenClutch latches the pen-down state from the pinch distance, normalized by palm span, with separate engage/release thresholds for hysteresis.
• EmaFilter runs an exponential moving average on the tracked coordinates — smooth = alpha * current + (1 - alpha) * previous — keeping the running value in floats and rounding only on output so the residual never quantises away. The pen and the eraser own separate filters, so switching between them causes no glide, and the cursor is filtered on every frame the hand is up: by the time the pinch closes the average has already converged, so a stroke has no lag tail at its start.
• Stroke breaks happen on every pen lift, mode change, or lost hand (prev_point = None), and a jump larger than MAX_JUMP breaks the stroke instead of drawing a long streak across the screen after a tracking glitch.

Why the classic Solutions API

This uses mp.solutions.hands, not mp.tasks with a .task model file. The Tasks runner spins up a Metal/GPU delegate that crashes many local Python builds on macOS; the Solutions pipeline stays on a stable path there.

Tuning

All knobs live in the config block at the top of main.py:

Constant	Meaning
BRUSH_THICKNESS	Stroke width in pixels
ERASER_RADIUS	Starting eraser radius (also adjustable at runtime with [ / ])
ERASER_PROXIMITY_RATIO	How close index and middle must be, as a fraction of palm span
PINCH_ENGAGE_RATIO / PINCH_RELEASE_RATIO	Thumb-to-index pinch distance (as a fraction of palm span) that presses the pen down / lifts it
KNUCKLE_ENGAGE_RATIO / KNUCKLE_RELEASE_RATIO	Same, for the index-tip-to-middle-base trigger
EMA_ALPHA	Smoothing factor, 0 < alpha <= 1; 1 = raw and jittery, lower = smoother but laggier (also --alpha)
MAX_JUMP	Pixel distance above which a stroke is broken
HEADER_HEIGHT, STATUS_HEIGHT, PALETTE	HUD sizes and the color boxes
DETECTION_CONFIDENCE, TRACKING_CONFIDENCE	MediaPipe thresholds
INFERENCE_WIDTH	Width the frame is scaled to before tracking (also --infer-width)
MODEL_COMPLEXITY	MediaPipe hand model, 0 or 1 (also --model-complexity)
CAMERA_FPS	Capture rate requested from the camera (also --fps)
HELP_TIMEOUT	Seconds before the help overlay auto-hides; 0 disables the auto-hide
FPS_GOOD, FPS_OK	Thresholds for the green / amber / red FPS read-out

Troubleshooting

• "Could not open camera index 0" — on macOS, grant camera access to the app running Python (System Settings → Privacy & Security → Camera; approve Terminal / iTerm / VS Code). Then try --camera 1.
• pip install mediapipe==0.10.14 fails — you are almost certainly on Python 3.13+. Create the venv with a 3.9–3.12 interpreter.
• Choppy framerate — first check the FPS counter in the status bar. If it sits at the camera's rate (usually 30) the app is camera-bound and already as fast as the sensor allows. If it is below that, try --model-complexity 0 (the largest single saving, ~2.4 ms/frame), then --no-skeleton, then a smaller capture size (--width 960 --height 540). Lowering --infer-width is unlikely to help — see the caveat under Performance. Never run with --no-threaded-capture unless you are debugging; it costs about 11 fps.
• Strokes jitter — lower the smoothing factor toward 0.3 (--alpha 0.3). If drawing feels laggy instead, raise it toward 0.8.
• The pen won't go down — watch the PINCH meter in the status bar and pinch until it fills; if it never fills, raise PINCH_ENGAGE_RATIO slightly (e.g. 0.45).
• The pen drops mid-letter — your pinch is drifting open. Raise PINCH_RELEASE_RATIO toward 0.7 to widen the hysteresis band.