Skip to content

About

End-to-end robot control based on generative diffusion model

Resources

Stars

1 star

Watchers

0 watching

Forks

 
 

Repository files navigation

Imitation learning with ROS 2

Object picking in IsaacSim Object picking in IsaacSim
Object picking with imitation learning Object picking when pose of object is randomized

The RoboImitate project supports imitation learning through a Diffusion Policy. This policy learns behavior using expert demonstrations. (Stay tuned for our upcoming YouTube presentation for more details!)

This repository allows you to:

🚀 Recent Model & Training Improvements

  1. Fixed Posterior Collapse (State Overfitting)
    • Impact: Critical
    • Details: Disconnected the observation.state (robot arm coordinates) from the Diffusion Policy's global conditioning. Previously, the network bypassed visual inputs and memorized coordinate transformations, causing the robot to blindly repeat the exact same movement. The policy is now strictly forced to rely on the camera image to detect and track the object.
  2. Fixed Simulation Randomization Bug
    • Impact: High
    • Details: Fixed a critical bug in pick_screwdriver.py (--sim mode) where an empty Twist() message caused the screwdriver to always spawn at the exact same fallback coordinates (0.35, 0.10). Added coordinate randomization that mirrors the training distribution, allowing inference tests to correctly evaluate visual tracking.
  3. Massive Training Speed & GPU VRAM Optimization
    • Impact: High
    • Details: Rewrote the training pipeline to load the entire dataset directly into GPU VRAM at startup (GPUDataset), eliminating CPU-GPU transfer bottlenecks. Enabled PyTorch hardware optimizations (TF32, cudnn.benchmark) and bfloat16 AMP (torch.autocast). This slashed VRAM usage by over 50% and increased training speed by >3x, allowing a massive batch size of 512.
  4. Learning Rate & Optimization Upgrades
    • Impact: Medium
    • Details: Implemented a dynamic sqrt-scaled learning rate based on batch size. Added a CosineAnnealingLR scheduler and gradient clipping (max_norm=10.0) to ensure stable convergence.
  5. Leveraged Pre-trained Vision Backbone
    • Impact: Medium
    • Details: Enabled IMAGENET1K_V1 pre-trained weights for the ResNet-18 vision backbone in config.py. This significantly reduces the training time and dataset size required to learn robust visual features compared to training from scratch.

Important

You need to have Docker installed. If you have an Nvidia GPU, you need to additionally follow this guide. Additionally, you need to install Isaac-Sim if you want to use simulation.

Note on Isaac Sim: We recommend using nvcr.io/nvidia/isaac-sim:4.2.0. Because this image does not have ROS 2 pre-installed, you will need to manually install ros-humble-ros-base inside the container as root and source it before running ./runapp.sh.

Install docker

sudo apt install git make curl
curl -sSL https://get.docker.com | sh && sudo usermod -aG docker $USER

Installation

  • Download our source code:
git clone https://github.com/MarijaGolubovic/robo_imitate.git && cd robo_imitate/docker
  • Build docker container
make build-pc run exec
  • Build ROS 2 packages
colcon build --symlink-install && source ./install/local_setup.bash

Model evaluation

Note

You can download pretrain model and aditional files from this link. Downloaded model and files you need to put inside folder imitation/outputs/train. If folder don't exist you need to create it.

  • Run Isaac-Sim Simulation

Inside the docker container run:

  • Run ROS 2 controller
ros2 launch xarm_bringup lite6_cartesian_launch.py rviz:=false sim:=true

If you want to visualize the robot set rviz to true.

  • Open another terminal and run docker
make exec
  • Run model inference inside docker
 cd src/robo_imitate && python3 ./imitation/pick_screwdriver --sim

Model training

Inside robo_imitate directory run follow commands:

docker build --build-arg UID=$(id -u) -t imitation .
docker run -v $(pwd)/imitation/:/docker/app/imitation:Z --gpus all -it -e DATA_PATH=imitation/data/sim_env_data.parquet -e BATCH_SIZE=32 -e EPOCH=50000 imitation

Tip

If you want to run model training inside docker, run this command inside the folder src/robo_imitate. Before that, you need to build the docker (see the Installation section for details).

python3 ./imitation/compute_stats --path imitation/data/sim_env_data.parquet
python3 ./imitation/train_script --path imitation/data/sim_env_data.parquet --batch_size 32 --epoch 50000

Acknowledgment

  • This project is done in collaboration with @SpesRobotics.
  • Thanks to LeRobot team for open sourcing LeRobot projects.
  • Thanks to Cheng Chi, Zhenjia Xu and colleagues for open sourcing Diffusion policy

Physical Implementation

Object picking physics env Object picking physics env
Physical robot picking Physical robot moving

About

End-to-end robot control based on generative diffusion model

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages