johnny mobile

A mecanum car that drives around after Johnny. Johnny is a fish.

The Johnny Mobile points a USB camera at Johnny's tank, finds Johnny with a YOLO26n model running on NCNN, smooths the motion with a Kalman filter, and shouts directions at an ESP32 over USB serial. The ESP32 does the actual driving. Johnny swims left, the car goes left.

Note

The low-level controller and all the wiring are Harry's work — @stellarbeing22. mecanum_firmware.ino, the PWM and ramping, the watchdog failsafe, and the sonar wiring on the ESP32 side all come from them. The vision half would be a very expensive paperweight without it.

flowchart LR
  J(("Johnny")) --> A

  subgraph PI["Raspberry Pi"]
    A["camera"] --> B["YOLO26n<br/>(NCNN)"]
    B --> C["ByteTrack<br/>ids"]
    C --> D["Kalman"]
    D --> E["arbiter"]
  end

  E -->|"FORWARD / LEFT / ..."| F

  subgraph ESP["ESP32"]
    F["serial"] --> G["PWM +<br/>ramping"]
    S["4x sonar"] -.->|"gate"| G
  end

  G --> H(("wheels"))

  classDef pi fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#0f172a
  classDef esp fill:#fed7aa,stroke:#ea580c,stroke-width:1.5px,color:#0f172a
  classDef out fill:#d1fae5,stroke:#059669,stroke-width:1.5px,color:#0f172a
  class A,B,C,D,E pi
  class F,G,S esp
  class H,J out

  style PI fill:#f1f5f9,stroke:#94a3b8,color:#0f172a
  style ESP fill:#fff7ed,stroke:#fdba74,color:#0f172a

Loading

File	What it does
tracker_drive.py	Spots Johnny, decides where to go, talks to the ESP32
mecanum_firmware.ino	ESP32 side: PWM, ramping, failsafe, not hitting things — by Harry
best_ncnn_model/	The Johnny detector. One class, and it is fish
roi_config.json	Which part of the frame is actually Johnny's tank

Getting it running

pip install -r requirements.txt

Flash mecanum_firmware.ino with the Arduino IDE (ESP32 core 3.x, though 2.x compiles fine). Wiring is in the comment at the top of the sketch.

Then:

python3 tracker_drive.py --calibrate            # click the 2 corners of Johnny's tank
python3 tracker_drive.py --headless             # go
python3 tracker_drive.py --headless --diagonals # go, but sideways too

No ESP32 plugged in? It still runs, just vision-only. Handy for checking the model can actually find Johnny before anything is able to roll away.

Two things that will absolutely bite you

1. Tell it which way the camera is facing. --axis-mode maps "Johnny moved right in the image" onto "the car drives forward." Get it backwards and the car will confidently drive perpendicular to Johnny.

• xy (default) — camera is rotated 90° from the chassis
• yx — camera faces the same way the car drives

Put it on blocks, run --verbose, watch the commands match what Johnny is doing. Then let it touch the floor.

2. The sonar ECHO pins are 5 V and the ESP32 is not 5 V tolerant. Wire ECHO straight to a GPIO and you kill the pin. Every ECHO line needs a divider — 1 kΩ from ECHO to the GPIO, 2 kΩ from GPIO to GND — or a proper level shifter. VCC still wants real 5 V; HC-SR04 gets flaky at 3.3 V.

Side	TRIG	ECHO
Front	32	34
Back	33	35
Left	25	36
Right	26	39

Not hitting things

Four HC-SR04s, one per side. Blocking is per-direction, not a panic stop — if something's in front, you can still back up, strafe, or reverse-diagonal away. Rotation is never blocked, since spinning closes no distance and it's how you get out when boxed in.

The gate zeroes the wheels but remembers the direction you asked for, so it just starts moving again when the obstacle clears. Nothing needs resending.

Two things worth knowing:

• A dead sensor looks exactly like a clear one. Silence is silence. SENSORS will report anything that's never answered since boot as FAULT, and tracker_drive.py runs it at startup and prints the reply. Actually read it.
• Obstacle stops don't use RAMPDECEL. That's 150 ms/step for smoothly following Johnny, which would take ~4.5 s to stop from 30 % duty. E-stops use ESTOPMS (2 ms/step) instead — 83–113 ms detect-to-zero-duty, about 3 cm of travel at 0.3 m/s. That's time to zero duty, not to actually motionless; mechanical coast happens on top.

Bench test it before you trust it

Wheels off the ground:

SAFETY:1
SENSORS          <- four distances, no FAULT
FORWARD          <- wheels spin
(hand in front)  <- [BLOCKED FRONT=..cm], wheels stop
(hand away)      <- [CLEAR], wheels resume
BACKWARD         <- still works with your hand in front

Full obstacle gating table

Obstacle	Blocks	Still allowed
Front	FORWARD, FL_DIAG, FR_DIAG	BACKWARD, both rear diagonals, strafes
Back	BACKWARD, BL_DIAG, BR_DIAG	FORWARD, both front diagonals, strafes
Left	LEFT, FL_DIAG, BL_DIAG	RIGHT, right diagonals, fore/aft
Right	RIGHT, FR_DIAG, BR_DIAG	LEFT, left diagonals, fore/aft

If the host dies

The watchdog is off at boot, so you can drive by hand from the Serial Monitor without it cutting out on you. tracker_drive.py arms it (750 ms) and keeps feeding it every 200 ms. If the script crashes, the USB pops out, or the Pi loses power, the wheels ramp to zero instead of driving off into the world. Any command re-arms it.

Serial protocol

Plain newline-terminated text. Set things once at startup, then one command per direction change plus a PING keepalive.

All commands

Command	Meaning
FORWARD BACKWARD LEFT RIGHT	translate (LEFT/RIGHT strafe)
FL_DIAG FR_DIAG BL_DIAG BR_DIAG	diagonal translation
ROTATE_L ROTATE_R	rotate in place (firmware only; tracker doesn't use it)
STOP	ramp to a halt
SETSPEED:NN	cruising duty, 0–100 %
RAMPMS:NN / RAMPDECEL:NN	accel / decel ramp step interval, ms
WATCHDOG:NN	stop if no serial for NN ms; 0 disarms
STOPDIST:NN	obstacle stop distance, cm
SAFETY:0 / SAFETY:1	obstacle stopping off / on
SENSORS	print the four sonar distances
ESTOPMS:NN	obstacle-stop ramp step interval, ms
PING	feed the watchdog, change nothing
?	help

Tuning

Flag	Default	What it does
--speed	30	cruising duty %
--ramp / --decel	30 / 150	accel / decel step interval, ms
--velocity-threshold	1.0	how fast Johnny has to move before the car chases, px/frame
--exit-ratio	0.6	fraction of that where it stops chasing
--min-dwell	0.15	minimum seconds in one direction before switching
--hold-seconds	1.0	keep going this long after losing Johnny
--conf	0.25	detection confidence threshold
--stop-distance	30	obstacle stop distance, cm

--velocity-threshold and --exit-ratio are a hysteresis band — it takes more speed to start moving than to keep moving. That plus --min-dwell is what keeps the car from twitching left-right-left on detector noise. Twitchy? Raise the threshold. Ignoring Johnny on a lazy day? Lower it.

Gotchas

• Velocities are pixels per frame, so your thresholds shift if the frame rate changes much. --hold-seconds is wall-clock and doesn't.
• GPIO 12 (rear-right LPWM) is an ESP32 strapping pin. If the board won't boot with the driver attached, check that first.
• PWM_FREQ_HZ is 20 kHz to stay out of hearing range. Drop it if your driver boards can't switch that fast.