Wall Following with Reinforcement Learning in ROS 2

This repository contains a ROS 2 + Gazebo project for autonomous right-wall following with the TurtleBot3 Burger using tabular Q-learning and SARSA.

The final system uses a manually discretized LiDAR state space, a compact action set, checkpointed training, quantitative reward logging, and a one-time test respawn strategy to demonstrate all required wall-following situations in the maze world.

Highlights

• ROS 2 Jazzy project built around the TurtleBot3 Burger in Gazebo
• Two learning agents: Q-learning and SARSA
• Final compact 27-state right-wall representation based on three LiDAR sectors
• Five-action motion policy: CRUISE, BANK_LEFT, BANK_RIGHT, HOOK_LEFT, HOOK_RIGHT
• Training checkpoints and reward CSV logs for both algorithms
• Comparison plots for reward, episode length, and summary metrics
• Legacy hand-authored Q-table baseline also included for reference

Environment

The maze world used for training and evaluation is shown below.

Final Comparison Snapshot

The figure below compares training reward over the first 550 episodes for both methods.

Using the shared 550-episode comparison window from the final runs:

Metric	Q-learning	SARSA
Average reward	937.86	1052.43
Best reward	3891.77	3927.00
Average episode length	172.41	175.17
Last-100 average reward	1354.85	1861.07
Last-100 average steps	198.67	227.85
Best wall-follow streak	92	98

SARSA achieved the stronger overall result in the final matched comparison, while Q-learning remained competitive and reached strong peak rewards.

Repository Layout

wall_following_ros2/
+-- checkpoints/                 # Saved training checkpoints for both algorithms
+-- config/
¦   +-- bridge_topics.yaml       # ROS <-> Gazebo bridge topics
¦   +-- generate_q_table.py      # Helper for legacy Q-table generation
¦   +-- q_table.yaml             # Legacy hand-authored Q-table
¦   +-- rl_params.yaml           # Main RL hyperparameters and spawn settings
+-- doc/
¦   +-- img/
+-- launch/
¦   +-- rl_wall_following.launch.py   # Main RL launch file
¦   +-- wall_following.launch.py      # Legacy Q-table launch
+-- logs/                        # Reward CSV files and generated comparison plots
+-- scripts/
¦   +-- q_learning_wall_follower.py
¦   +-- sarsa_wall_follower.py
¦   +-- rl_common.py             # Shared training/testing logic
¦   +-- plot_rewards.py          # Reward plotting helper
¦   +-- q_table_wall_follower.py # Legacy baseline controller
+-- worlds/
    +-- largemaze.world

Final RL Formulation

State Space

The final trained agents use a compact state representation:

• F: front distance bucket
• RF: right-front distance bucket
• RB: right-back distance bucket

Each variable is discretized into three bins, producing 27 total states.

LiDAR is divided into three sectors:

• front: 350° to 10°
• right-front: 300° to 340°
• right-back: 200° to 270°

Action Space

The final action set is:

• CRUISE
• BANK_LEFT
• BANK_RIGHT
• HOOK_LEFT
• HOOK_RIGHT

Key Training Parameters

Default tuned configuration in config/rl_params.yaml:

• alpha = 0.20
• gamma = 0.95
• epsilon_start = 0.55
• epsilon_decay = 0.992
• epsilon_min = 0.05
• max_episodes = 550
• max_steps_per_episode = 250
• target_distance = 0.60 m
• distance_band = 0.20 m

Requirements

This package assumes ROS 2 Jazzy and Gazebo with the TurtleBot3 simulation stack installed.

Main runtime dependencies include:

• ros-jazzy-rclpy
• ros-jazzy-ros-gz-bridge
• ros-jazzy-ros-gz-sim
• ros-jazzy-ros-gz-interfaces
• ros-jazzy-turtlebot3-gazebo
• ros-jazzy-nav2-bringup
• ros-jazzy-rviz2
• ros-jazzy-robot-state-publisher

Build

cd ~/ros2_ws_assignment2
source /opt/ros/jazzy/setup.bash
colcon build --packages-select wall_following_ros2
source install/setup.bash

If you use Conda, deactivate it first to avoid Python path conflicts:

conda deactivate 2>/dev/null || true
unset PYTHONPATH PYTHONHOME CONDA_PREFIX CONDA_DEFAULT_ENV

Run the RL Agents

Train Q-learning

ros2 launch wall_following_ros2 rl_wall_following.launch.py algorithm:=q_learning mode:=train

Train SARSA

ros2 launch wall_following_ros2 rl_wall_following.launch.py algorithm:=sarsa mode:=train

Resume training from latest checkpoint

ros2 launch wall_following_ros2 rl_wall_following.launch.py algorithm:=sarsa mode:=train resume:=true

Test latest Q-learning checkpoint

ros2 launch wall_following_ros2 rl_wall_following.launch.py algorithm:=q_learning mode:=test resume:=true

Test latest SARSA checkpoint

ros2 launch wall_following_ros2 rl_wall_following.launch.py algorithm:=sarsa mode:=test resume:=true

Outputs

Training writes:

• checkpoints to checkpoints/q_learning/ and checkpoints/sarsa/
• reward CSV logs to logs/q_learning_rewards.csv and logs/sarsa_rewards.csv

Generated comparison plots currently include:

• logs/training_reward_comparison_550.png
• logs/training_steps_comparison_550.png
• logs/algorithm_summary_comparison_550.png

Legacy Baseline

The repository also keeps the earlier hand-crafted Q-table controller for comparison:

ros2 launch wall_following_ros2 wall_following.launch.py

That controller is useful as a baseline, but the main project result is the RL-based right-wall-following system in rl_wall_following.launch.py.

Notes for Sharing

• The project is centered on the package at the root of this repository.
• Reward logs and checkpoints are included because they document the final training runs.
• If you want a lighter public repository, you can trim old checkpoints and keep only latest.yaml plus final plots.

Demo

Demo and report assets can be linked here:

• Video / drive folder: https://drive.google.com/drive/folders/1F6Jn4CsxW4kGzQGw0XMZueJe4_jBndT8?usp=sharing

Author

Himanshu Ranjan