USR · Unmanned Spray Robot

Vision-guided crop and weed spraying, built around a mobile robot platform

What it does · Quick start · How it works · Hardware

USR is a research prototype that combines camera-based object detection, a PyQt desktop controller, serial communication, and ESP8266 firmware to aim a spray mechanism at detected targets.

✨ What it does

• Detects targets from a camera feed with custom YOLO checkpoints.
• Visualizes detections with bounding boxes, labels, coordinates, counters, and status information.
• Converts image coordinates into servo angles for a two-axis spray head.
• Triggers a spray cycle through the ESP8266 after a target is received.
• Controls the mobile platform through a separate PyQt joystick / motor-control interface.
• Supports camera calibration with chessboard images and includes dataset augmentation and annotation tools.

🧭 How it works

flowchart LR
    C[Camera] --> V[OpenCV frame capture]
    V --> D[YOLOv8 target detection]
    D --> G[Target center coordinates]
    G -->|x,y + newline| E[ESP8266 controller]
    E --> S[Pan / tilt servos]
    E --> P[Spray relay or pump]
    E -->|ESP-NOW trigger| M[Motor-control ESP8266]
    M --> W[Drive motor]
    U[PyQt5 desktop UI] --> D
    U --> E

Loading

The main firmware accepts newline-terminated coordinate messages in the form:

x,y

For the current arduino_code/arduino/arduino.ino implementation, camera coordinates are interpreted on a 640 × 480 frame, mapped to servo angles, and followed by a short spray-pump pulse. The special command runesp2 sends a one-byte ESP-NOW trigger to the second controller.

🗂️ Project layout

.
├── arduino_code/
│   ├── arduino/              # Main ESP8266 controller + ESP-NOW sender
│   ├── arduino/arduinoslave/ # ESP-NOW receiver + drive motor control
│   ├── servotest/             # Two-servo serial test
│   ├── servotest2/            # Servo mapping + spray output test
│   └── relaytest/             # Pump / relay timing test
├── model_train_code/          # Augmentation, mosaic, and annotation utilities
├── photos/
│   ├── camera calibration/    # Chessboard images for calibration
│   ├── usr-robot-prototype.jpeg # Photo of the assembled robot
│   └── USRlow.png             # Prototype photo used by the desktop launcher
├── python_code/
│   ├── main.py                # Desktop launcher: Drive / Spray
│   ├── spray.py               # Detector + spray control GUI
│   ├── car.py                 # Serial motor-control GUI
│   ├── detection.py           # Single-frame YOLO detection example
│   └── supporting/             # Camera, calibration, and UI helpers
├── *.pt                       # YOLO checkpoints
└── requirements.txt           # Core Python dependency pin

🚀 Quick start

1. Prepare the Python environment

From the repository root:

python -m venv .venv

Activate it with the command for your operating system:

# Windows PowerShell
.venv\\Scripts\\Activate.ps1

# Windows Command Prompt
.venv\\Scripts\\activate.bat

# Linux / macOS
source .venv/bin/activate

Install the pinned dependency and the packages used by the desktop interfaces:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install PyQt5 opencv-python pyserial psutil

For the dataset utilities, install their additional packages when needed:

python -m pip install albumentations tqdm

2. Connect the robot

Before launching the control software:

1. Flash the appropriate ESP8266 sketch from arduino_code/.
2. Connect the controller and camera to the computer.
3. Note the serial port assigned by the operating system.
4. Update any port constants in the selected Python entry point. The prototype currently contains example ports such as COM7, COM9, COM10, and COM11.
5. Keep the model checkpoint and photos/ paths available from the repository root; the current scripts use relative paths.

3. Launch the desktop controller

The top-level launcher presents the Drive and Spray modes:

python python_code/main.py

To open the more complete detector and spray panel directly:

python python_code/spray.py

The single-frame detector is useful for a quick camera/model smoke test:

python python_code/detection.py

🎛️ Desktop controls

Interface	Entry point	Purpose
Main launcher	python_code/main.py	Opens Drive or Spray mode.
Spray control	python_code/spray.py	Connects to a serial port, previews camera frames, runs YOLO inference, queues target centers, and sends them to the ESP8266.
Motor control	python_code/car.py	Sends speed and direction commands over serial.
Joystick prototype	python_code/car_control.py	Provides directional controls and a speed slider.
Detector prototype	python_code/spraycontrol.py	Displays a live annotated feed with crop / weed spray actions.

🧠 Model checkpoints

The repository includes several checkpoints from different training stages:

Checkpoint	Used by	Notes
june8.pt	main.py, gui.py, detection.py, spraycontrol.py	Primary checkpoint for several earlier detection flows.
novlast.pt	spray.py	Checkpoint used by the main spray-control panel.
python_code/last.pt	predict_from_live with serial.py	Checkpoint used by the legacy live-serial example.
may30.pt	—	Additional project checkpoint.
yolov8n.pt	—	Base YOLOv8 nano checkpoint included in the repository.

If you swap checkpoints, make sure the selected model’s class order matches the labels expected by the UI and firmware workflow.

🔌 Hardware & firmware

The firmware sketches target ESP8266 boards and use the Arduino IDE ecosystem.

Main controller

arduino_code/arduino/arduino.ino handles:

• X/Y servo positioning from camera coordinates;
• a spray output on D8;
• a pump / motor output on D1 using PWM;
• an ESP-NOW trigger for the second controller.

Slave controller

arduino_code/arduino/arduinoslave/arduinoslave.ino listens for the ESP-NOW trigger and runs the drive motor for the configured duration.

Bench-test sketches

Use the smaller sketches in arduino_code/servotest/, arduino_code/servotest2/, and arduino_code/relaytest/ to test actuators independently before connecting the full system.

Safety: Test with clean water, the spray system disabled, and wheels lifted clear of the ground until serial commands, servo limits, motor direction, and emergency stopping behavior have been verified.

📷 Camera calibration

The calibration helper uses the chessboard images in photos/camera calibration/ and the sample image in photos/:

python python_code/supporting/camera_calibration.py

The script is configured for an 8 × 6 internal-corner chessboard and a 4608 × 3456 source image. Adjust those values if your calibration target or camera changes.

🧪 Dataset tools

model_train_code/ contains utilities for preparing training data:

• augmentation.py — image-only augmentations;
• augmentation_annotation.py — image and YOLO annotation augmentation;
• mossaic_augmentation.py — interactive four-image mosaic builder;
• annotation_checker.py — annotation viewer/editor.

The augmentation scripts currently contain example Windows paths such as A:/.... Update the input and output directories before running them.

⚠️ Prototype notes

• This is an experimental robotics project, not a production agricultural safety system.
• The Python code is currently Windows-oriented because several flows use COM* serial-port names.
• Some files are alternate or legacy experiments; their model names, serial ports, and control behavior are not fully unified.
• requirements.txt contains the core Ultralytics pin, while the PyQt, OpenCV, serial, system-monitoring, and training-tool packages are installed separately above.
• There is currently no automated test suite. Validate each actuator and communication link on the bench before field operation.

🤝 Contributing

Improvements are welcome. A useful contribution usually includes:

1. a focused change;
2. updated wiring, protocol, or usage notes when behavior changes;
3. a bench-test procedure or short verification note;
4. no committed credentials, private datasets, or machine-specific paths.

📄 License

No license file is currently included in this repository. Add an explicit license before redistributing the code or hardware designs.

_{USR combines field hardware, computer vision, and embedded control into one research platform.}