waveHome

waveHome is a Python-based gesture-controlled smart home prototype that uses computer vision and hand tracking to control virtual smart devices without voice commands.

Built with an ESP32-CAM, MediaPipe, and OpenCV, the project focuses on creating a more accessible smart-home experience for deaf, hard-of-hearing, and speech-disabled users through intuitive hand gestures.

The system currently controls a virtual smart lamp, but its architecture is designed for future integration with platforms like Google Home and Home Assistant.

---

Features

• Real-time video streaming from an ESP32-CAM or local webcam
• MediaPipe hand landmark detection
• Gesture recognition engine with sequence and motion tracking
• Live OpenCV overlays:
  • hand skeletons
  • bounding boxes
  • gesture labels
  • finger states
  • status panels
• Virtual smart lamp simulation
• Visual block-based workflow dashboard
• Rule-based automation system
• Modular smart-home action layer
• Future-ready Google Home bridge adapter

---

Demo Concepts

waveHome supports multiple gesture-driven smart-home interactions:

Action	Gesture
Toggle lamp	5 fingers -> fist -> 5 fingers -> fist
Increase brightness	fist -> thumb up
Decrease brightness	fist -> thumb down
Change color	fist -> peace sign + rotation
Party mode	fist -> horns -> fist

---

Architecture

ESP32-CAM / Webcam
        ↓
MediaPipe Hand Tracking
        ↓
Gesture Recognition Engine
        ↓
Rule Workflow System
        ↓
Virtual Smart Home Actions
        ↓
Google Home / Home Assistant Bridge

---

Project Structure

.
├── waveHome.py
└── wavehome/
    ├── app.py
    ├── camera.py
    ├── config.py
    ├── controller.py
    ├── drawing.py
    ├── geometry.py
    ├── gestures.py
    ├── model.py
    └── web/

---

Requirements

• Python 3.10+
• ESP32-CAM MJPEG stream (optional)
• Webcam (optional)

Python Dependencies

• opencv-python
• mediapipe
• numpy
• requests
• fastapi
• uvicorn
• pydantic

Install everything with:

python3 -m pip install opencv-python mediapipe numpy requests fastapi uvicorn pydantic

---

Configuration

Configuration is located in:

wavehome/config.py

Default camera stream:

CAMERA_URL = "http://esp32cam.local/stream"

---

Using a Local Webcam

USE_LOCAL_CAMERA = True
LOCAL_CAMERA_INDEX = None

When enabled, waveHome automatically scans for available webcams and ignores invalid or black-frame devices.

You can also manually select a camera:

LOCAL_CAMERA_INDEX = 0

---

Running waveHome

python3 waveHome.py

Press q to exit the OpenCV window.

On the first launch, the MediaPipe hand tracking model is downloaded automatically.

---

Gesture System

waveHome uses a hybrid gesture engine supporting:

• gesture sequences
• hold gestures
• motion gestures
• rotation tracking
• continuous value controls
• workflow states
• confirmation-based actions

---

Safety & False Trigger Protection

To reduce accidental activations, waveHome includes a command-mode system.

Activate command mode

both open palms held for 1 second

Fallback single-hand activation:

open palm held for 1.8 seconds

Once active, protected gestures become available.

---

Confirmation Workflow

Potentially dangerous actions can require confirmation:

both fists held
        ↓
pending confirmation
        ↓
two thumbs up = execute
thumb down = cancel

---

Workflow Dashboard

waveHome includes a browser-based visual workflow editor inspired by block programming systems.

Start the dashboard:

uvicorn wavehome.web.server:app --reload --host 127.0.0.1 --port 8080

Open:

http://127.0.0.1:8080/

---

Dashboard Features

The editor allows visual configuration of:

• trigger types
• gesture sequences
• hold/repeat behaviors
• motion tracking
• brightness controls
• color controls
• command mode logic
• cooldowns
• confirmation gestures
• smart-home actions

---

API Endpoints

GET  /api/health
GET  /api/capabilities
GET  /api/gestures
GET  /api/presets
GET  /api/rules
POST /api/rules/validate
PUT  /api/rules
POST /api/rules/reset

---

Rule Engine

Rules are stored in:

wavehome/rules/default_rules.json

Supported trigger types:

sequence
hold
repeat_hold
armed_hold
motion
value_control

Supported actions:

virtual_lamp.toggle
virtual_lamp.turn_on
virtual_lamp.turn_off
virtual_lamp.toggle_party
virtual_lamp.brightness_step
virtual_lamp.brightness_set
virtual_lamp.color_set
workflow.enter_command_mode
workflow.exit_command_mode
workflow.cancel

---

Example Interaction Flow

1. Hold both open palms
2. Command mode activates
3. Swipe to adjust brightness
4. Rotate peace sign to change color
5. Confirm global actions with two thumbs up

---

Future Smart Home Integration

waveHome already includes a provider abstraction layer for real smart-home integrations.

Supported abstract actions:

smart_home.set_power
smart_home.set_brightness
smart_home.set_color
smart_home.activate_scene

Future bridge targets include:

• Google Home
• Home Assistant
• Matter-compatible ecosystems

---

Google Home Bridge (Planned)

Enable bridge mode with:

export WAVEHOME_GOOGLE_HOME_ENABLED=true
export WAVEHOME_GOOGLE_HOME_BRIDGE_URL=http://127.0.0.1:9000
export WAVEHOME_GOOGLE_HOME_ACCESS_TOKEN=replace-with-token

The bridge adapter sends normalized commands to:

POST /commands

This layer is designed to later handle:

• OAuth
• permissions
• device discovery
• platform-specific APIs

---

Accessibility Goal

waveHome is designed as an accessibility-focused research prototype exploring alternative smart-home interaction methods for people who cannot reliably use voice assistants.

The long-term goal is to provide:

• non-verbal smart-home control
• safer gesture workflows
• customizable interaction logic
• low-cost hardware compatibility

---

Tech Stack

• Python
• OpenCV
• MediaPipe
• FastAPI
• ESP32-CAM
• NumPy
• Uvicorn