Plexus Find Object

Zero-shot object search for the Plexus robot. Uses YOLO-World (open-vocabulary detection via CLIP text encoder) so any nameable object works. The robot autonomously searches for, approaches, and confirms arrival at user-specified objects using only natural language prompts — no training data needed.

How It Works

The system runs a continuous 5Hz control loop:

• MJPEG Streaming: Camera frames streamed via MJPEG from ESP32-CAM.
• YOLO-World Detection: Real-time object detection (~19ms inference) for user-specified target objects.
• Policy Control: A navigation policy processes detections and generates motor commands.
• WebSocket Control: Commands sent directly to the robot over a WebSocket connection.
• Search & Arrival: The robot searches the environment, approaches the target object when found, and confirms arrival.

Requirements

• Hardware:
  • Plexus robot (ESP32-CAM + TT motors + IR sensor)
  • WiFi connection to robot AP (default: 192.168.4.1)
• Software:
  • Python 3.10+
  • NVIDIA GPU with ≥4GB VRAM (CUDA)
  • Windows (required for WASD teleop window; Linux works in headless mode)
• Dependencies:
  • pip install -r requirements.txt (opencv-python, websocket-client, ultralytics, numpy)

Quick Start

git clone https://github.com/embrained/plexus-find-object.git
cd plexus-find-object
pip install -r requirements.txt
python find_object.py --goals "coffee cup" --policy reactive3 --patrol

Usage

Command Examples

• Single target:

  python find_object.py --goals "TV"

• Multiple targets with patrol:

  python find_object.py --goals "toy ball" "coffee cup" --patrol

• Policy selection:

  python find_object.py --goals "toy ball" --policy reactive3

• Custom confidence threshold:

  python find_object.py --goals "coffee cup" --confidence 0.25

• Headless mode:

  python find_object.py --goals "coffee cup" --headless

CLI Flags

• --goals: Target object prompts, e.g., --goals "TV" "bookshelf" (required)
• --policy: Navigation policy choice: reactive1, reactive2, or reactive3 (default: reactive1)
• --patrol: Enable continuous looping through target goals after arrival
• --robot-ip: Robot WiFi AP IP address (default: 192.168.4.1)
• --model: YOLO-World model checkpoint/variant (default: yolov8x-worldv2.pt)
• --confidence: Detection confidence threshold override (e.g. --confidence 0.25)
• --arrival-area: Bounding box area ratio threshold for confirming arrival (e.g. --arrival-area 0.10)
• --flip-mode: Camera frame flip mode (default: 0 for vertical flip)
• --headless: Run without display window
• --verbose: Enable debug logging output

Policies

• reactive1: Rotate in place. Scans by turning left/right (50/50) or standing still. Finds objects in immediate view without locomotion. Good for benchmarking detection.
• reactive2: Forward wander. Short bursts of forward motion (0.2–0.4s) and turns with stand-still pauses after turns. IR reversal on obstacles. Explores the environment but does not steer toward detected objects.
• reactive3: Forward wander + proportional approach. Same search as reactive2, but when a target is detected, steers proportionally toward it. Approaches until target covers >10% of camera FOV, then confirms for 3 seconds with gentle centering. The most capable policy.

Keyboard Controls

When the display window is focused:

• q — Quit
• t — Toggle WASD teleop mode (W/A/S/D keys control the robot)
• n — Skip to next goal
• p — Toggle patrol mode on/off

Prompt Tips

YOLO-World is sensitive to CLIP text prompts. Descriptive multi-word prompts work much better than broad single words:

• 'toy ball' works, 'ball' doesn't
• 'shelving unit' > 'bookshelf'
• 'drink can' > 'can'

Use the most specific, descriptive name for your target.

Project Structure

plexus-find-object/
├── find_object.py      # CLI entry point
├── navigator.py        # Core navigation engine
├── policies/
│   ├── reactive1.py    # Rotate-in-place search
│   ├── reactive2.py    # Forward wander search
│   └── reactive3.py    # Wander + approach search
├── requirements.txt
├── LICENSE
└── README.md

License

GPLv3