Figure 1: Player punching projected targets.

Beat Boxing is an immersive game that blends physical exercise with digital gaming. Developed at Hochschule Heilbronn, this project transforms a standard punching bag into a smart, interactive gaming surface.

Inspired by Beat Saber, but designed for real-world impact, players use colored boxing gloves to punch projected targets. Unlike VR, where you strike thin air, Beat Boxing provides genuine haptic feedback against a physical heavy bag.

The project operates as a closed-loop Mixed Reality system, divided into four distinct zones as illustrated below.

Figure 2: The data flow pipeline from physical input to visual output.

1. Zone 1: Physical Input (The Eyes): Two ESP32-CAM modules capture high-speed video of the player from a top-down perspective and stream it via MJPEG over WiFi.
2. Zone 2: Processing Layer (The Brain): Splitted into two Python backends. The first one processes the video feeds and calculates the real-time 3D coordinates ($X, Y, Z$). The second one records the acceleration data from the glove sensors.
3. Zone 3: Application Layer (The Game): All data from the processing layer is sent via UDP to the Unity Game Engine. Unity maps the virtual gloves to the physical space, spawns targets, and detects collisions and punching force.
4. Zone 4: Physical Output (The Display): A projector, connected to the PC via HDMI, maps the game visuals onto the curved surface of the punching bag. The player sees the targets on the bag and feels the impact when they punch.

This repository is organized into four main sub-projects. Please refer to their specific READMEs for detailed technical documentation.

To run the full system, you will need to set up the three modules in the following order:

Flash the ESP32-CAMs with the custom firmware to establish video streams.

• 👉 Go to: BeatBoxingEsp32 for flashing instructions and WiFi configuration.

Install the Python environment, calibrate your cameras, and tune the glove color tracking.

• 👉 Go to: BeatBoxingCameraTracking for dependency installation, stereo calibration, and color tuning steps.

Install the Python environment and calibrate the acceleration sensors on the gloves.

• 👉 Go to: BeatBoxingSensorTracking for dependency installation and acceleration sensor tuning setps.

Open the project in Unity, align the virtual projector, and start the game loop.

• 👉 Go to: BeatBoxingUnity for scene setup, projector alignment, and gameplay instructions.

University: Hochschule Heilbronn (HHN)
Course: Mixed Reality (SEM)
Team:

• NDXIII
• Kartoffelbauer