-
Notifications
You must be signed in to change notification settings - Fork 34
add isaaclab bimanual arm example #548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| <!-- | ||
| SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| # Bimanual Robot Example using IsaacLab | ||
|
|
||
| This directory contains minimal usage examples for teleoperating bimanual robot arms in IsaacLab. | ||
|
|
||
| ##UR bimanual | ||
|  | ||
|
|
||
|
|
||
| ## Prerequisites | ||
|
|
||
|
|
||
| ### 1.Isaac Teleop | ||
|
|
||
| Follow [Install Isaac Teleop](https://isaac-sim.github.io/IsaacLab/main/source/how-to/cloudxr_teleoperation.html#cloudxr-teleoperation) to install Isaac Teleop. | ||
|
|
||
| After installation, start the ClourXR server: | ||
|
|
||
| ```bash | ||
| python -m isaacteleop.cloudxr | ||
| ``` | ||
|
|
||
| ### 2.Isaac Lab | ||
|
|
||
| Follow [Install Isaac Lab](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html) to install Isaac Lab with virtual environment `env_isaaclab`. | ||
|
|
||
| After installation, activate the virtual environment: | ||
|
|
||
| ```bash | ||
| source <your_isaaclab_path>/env_isaaclab/bin/activate | ||
| ``` | ||
|
|
||
| ### 3. Install this repository to `env_isaaclab` | ||
|
|
||
| ```bash | ||
| cd examples/isaaclab_bimanual_robot | ||
| python -m pip install -e . | ||
|
|
||
| ## If you use uv: | ||
| # uv pip install -e . | ||
| ``` | ||
|
|
||
| ## How to run | ||
|
|
||
| 1. Source cloudxr env | ||
|
|
||
| ```bash | ||
| source ~/.cloudxr/run/cloudxr.env | ||
| ``` | ||
|
|
||
| 2. Run the task | ||
|
|
||
| List the two environments: | ||
|
|
||
| ```bash | ||
| python scripts/list_envs.py | ||
| ``` | ||
|
|
||
| And you should see: | ||
|
|
||
| ``` | ||
| +--------+------------------------------+---------------------------------+-----------------------------------------------------------------------------------------------+ | ||
| | S. No. | Task Name | Entry Point | Config | | ||
| +--------+------------------------------+---------------------------------+-----------------------------------------------------------------------------------------------+ | ||
| | 1 | Template-FlexivRizon-Play-v0 | isaaclab.envs:ManagerBasedRLEnv | bimanual.tasks.manager_based.flexiv_rizon.flexiv_rizon_bimanual_env:FlexivRizonBimanualEnvCfg | | ||
| | 2 | Template-UR10-Play-v0 | isaaclab.envs:ManagerBasedRLEnv | bimanual.tasks.manager_based.ur10.ur10_bimanual_env:UR10BimanualEnvCfg | | ||
| +--------+------------------------------+---------------------------------+-----------------------------------------------------------------------------------------------+ | ||
| ``` | ||
|
|
||
| To examine the task environment only (without XR or Teleop): | ||
|
|
||
| ```bash | ||
| python scripts/zero_agent.py --task Template-UR10-Play-v0 --num_envs=1 | ||
|
|
||
| python scripts/zero_agent.py --task Template-FlexivRizon-Play-v0 --num_envs=1 | ||
| ``` | ||
|
|
||
| Teleoperation with XR controllers: | ||
|
|
||
| ```bash | ||
| python scripts/teleop_se3_agent_bimanual_xr.py \ | ||
| --task Template-UR10-Play-v0 \ | ||
| --teleop_device motion_controllers \ | ||
| --num_envs 1 \ | ||
| --xr \ | ||
| --reverse_rotation_yz | ||
|
|
||
| python scripts/teleop_se3_agent_bimanual_xr.py \ | ||
| --task Template-FlexivRizon-Play-v0 \ | ||
| --teleop_device motion_controllers \ | ||
| --num_envs 1 \ | ||
| --xr \ | ||
| --sensitivity 0.5 \ | ||
| --enable_gripper | ||
| ``` | ||
|
|
||
| Connect you XR devices before running the task using the [Isaac Teleop Web client](https://nvidia.github.io/IsaacTeleop/client/). | ||
|
|
||
|
|
||
| In IsaacLab UI, click the `Start AR` button in the bottom right panel. | ||
|
|
||
| <img width="300" src="./images/start_xr.png" > | ||
|
|
||
| Key bindings: | ||
| - Press any button on the left controller to start teleoperation | ||
| - Move the left controller to control the left arm | ||
| - Move the right controller to control the right arm | ||
| - Press x or y buttons on the left controller to reset | ||
|
|
||
|
|
||
| Record data: | ||
|
|
||
| ```bash | ||
| python scripts/record_se3_agent_bimanual_xr.py \ | ||
| --task Template-UR10-Play-v0 \ | ||
| --teleop_device motion_controllers \ | ||
| --num_envs 1 \ | ||
| --xr \ | ||
| --dataset_file ./datasets/dataset.hdf5 | ||
|
|
||
| # or with flexiv rizon | ||
| python scripts/record_se3_agent_bimanual_xr.py \ | ||
| --task Template-FlexivRizon-Play-v0 \ | ||
| --teleop_device motion_controllers \ | ||
| --num_envs 1 \ | ||
| --xr \ | ||
| --sensitivity 0.5 \ | ||
| --enable_gripper \ | ||
| --dataset_file ./datasets/dataset_flexiv.hdf5 | ||
| ``` | ||
|
|
||
|  | ||
|
|
||
| Key bindings: | ||
| - Press any button on the left controller to start teleoperation | ||
| - Move the left controller to control the left arm | ||
| - Move the right controller to control the right arm | ||
| - Press x or y buttons on the left controller to reset | ||
| - Press a or b buttons on the right controller to save the trajectory and reset env | ||
|
|
||
| The dataset will be saved to `./datasets/` folder. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # Ignore everything in this directory | ||
| * | ||
| # Except this .gitignore file | ||
| !.gitignore |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| # Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: BSD-3-Clause | ||
|
|
||
| [project] | ||
| name = "bimanual" | ||
| version = "0.1.0" | ||
|
|
||
| [project.optional-dependencies] | ||
| data = [ | ||
| "lerobot", | ||
| ] | ||
|
|
||
| [tool.setuptools.packages.find] | ||
| where = ["source/"] | ||
|
|
||
|
|
||
|
|
||
| [tool.ruff] | ||
| line-length = 120 | ||
| target-version = "py310" | ||
|
|
||
| # Exclude directories | ||
| extend-exclude = [ | ||
| "logs", | ||
| "_isaac_sim", | ||
| ".vscode", | ||
| "_*", | ||
| ".git", | ||
| "assets", | ||
| "datasets", | ||
| ] | ||
|
|
||
| [tool.ruff.lint] | ||
| # Enable flake8 rules and other useful ones | ||
| select = [ | ||
| "E", # pycodestyle errors | ||
| "W", # pycodestyle warnings | ||
| "F", # pyflakes | ||
| "I", # isort | ||
| "UP", # pyupgrade | ||
| "C90", # mccabe complexity | ||
| # "D", # pydocstyle | ||
| "SIM", # flake8-simplify | ||
| "RET", # flake8-return | ||
| ] | ||
|
|
||
| # Ignore specific rules (matching your flake8 config) | ||
| ignore = [ | ||
| "E402", # Module level import not at top of file | ||
| "D401", # First line should be in imperative mood | ||
| "RET504", # Unnecessary variable assignment before return statement | ||
| "RET505", # Unnecessary elif after return statement | ||
| "SIM102", # Use a single if-statement instead of nested if-statements | ||
| "SIM103", # Return the negated condition directly | ||
| "SIM108", # Use ternary operator instead of if-else statement | ||
| "SIM117", # Merge with statements for context managers | ||
| "SIM118", # Use {key} in {dict} instead of {key} in {dict}.keys() | ||
| "UP006", # Use 'dict' instead of 'Dict' type annotation | ||
| "UP018", # Unnecessary `float` call (rewrite as a literal) | ||
| ] | ||
|
|
||
| [tool.ruff.lint.per-file-ignores] | ||
| "__init__.py" = ["F401"] # Allow unused imports in __init__.py files | ||
|
|
||
| [tool.ruff.lint.mccabe] | ||
| max-complexity = 30 | ||
|
|
||
| [tool.ruff.lint.pydocstyle] | ||
| convention = "google" | ||
|
|
||
| [tool.ruff.lint.isort] | ||
|
|
||
| # Custom import sections with separate sections for each Isaac Lab extension | ||
| section-order = [ | ||
| "future", | ||
| "standard-library", | ||
| "third-party", | ||
| # Group omniverse extensions separately since they are run-time dependencies | ||
| # which are pulled in by Isaac Lab extensions | ||
| "omniverse-extensions", | ||
| # Group Isaac Lab extensions together since they are all part of the Isaac Lab project | ||
| "isaaclab", | ||
| "isaaclab-contrib", | ||
| "isaaclab-rl", | ||
| "isaaclab-mimic", | ||
| "isaaclab-tasks", | ||
| "isaaclab-assets", | ||
| # First-party is reserved for project templates | ||
| "first-party", | ||
| "local-folder", | ||
| ] | ||
|
|
||
| [tool.ruff.lint.isort.sections] | ||
| # Define what belongs in each custom section | ||
|
|
||
| "omniverse-extensions" = [ | ||
| "isaacsim", | ||
| "omni", | ||
| "pxr", | ||
| "carb", | ||
| "usdrt", | ||
| "Semantics", | ||
| "curobo", | ||
| ] | ||
|
|
||
| "isaaclab" = ["isaaclab"] | ||
| "isaaclab-assets" = ["isaaclab_assets"] | ||
| "isaaclab-contrib" = ["isaaclab_contrib"] | ||
| "isaaclab-rl" = ["isaaclab_rl"] | ||
| "isaaclab-mimic" = ["isaaclab_mimic"] | ||
| "isaaclab-tasks" = ["isaaclab_tasks"] | ||
|
|
||
| [tool.ruff.format] | ||
|
|
||
| docstring-code-format = true | ||
|
|
||
| [tool.pyright] | ||
|
|
||
| include = ["source", "scripts"] | ||
| exclude = [ | ||
| "**/__pycache__", | ||
| "**/_isaac_sim", | ||
| "**/docs", | ||
| "**/logs", | ||
| ".git", | ||
| ".vscode", | ||
| ] | ||
|
|
||
| typeCheckingMode = "basic" | ||
| pythonVersion = "3.11" | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @yizhouzhao-nvidia IIRC, this this still using Isaac Lab 2.3? So this example is actually using the legacy CloudXR path in Isaac Lab rather than the new Isaac Teleop path. For the simple XR controller path this is probably find and we can actually land this as-is and then make a follow up PR to upgrade this to Isaac Lab 3.0. That PR can be used as a follow up example as well.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @rwiltz wdyt? |
||
| pythonPlatform = "Linux" | ||
| enableTypeIgnoreComments = true | ||
|
|
||
| # This is required as the CI pre-commit does not download the module (i.e. numpy, torch, prettytable) | ||
| # Therefore, we have to ignore missing imports | ||
| reportMissingImports = "none" | ||
| # This is required to ignore for type checks of modules with stubs missing. | ||
| reportMissingModuleSource = "none" # -> most common: prettytable in mdp managers | ||
|
|
||
| reportGeneralTypeIssues = "none" # -> raises 218 errors (usage of literal MISSING in dataclasses) | ||
| reportOptionalMemberAccess = "warning" # -> raises 8 errors | ||
| reportPrivateUsage = "warning" | ||
|
|
||
|
|
||
| [tool.codespell] | ||
| skip = '*.usd,*.usda,*.usdz,*.svg,*.png,_isaac_sim*,*.bib,*.css,*/_build' | ||
| quiet-level = 0 | ||
| # the world list should always have words in lower case | ||
| ignore-words-list = "haa,slq,collapsable,buss,reacher,thirdparty" | ||
|
|
||
|
|
||
| [tool.pytest.ini_options] | ||
|
|
||
| markers = [ | ||
| "isaacsim_ci: mark test to run in isaacsim ci", | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: BSD-3-Clause | ||
|
|
||
| """ | ||
| Script to print all the available environments in Isaac Lab. | ||
|
|
||
| The script iterates over all registered environments and stores the details in a table. | ||
| It prints the name of the environment, the entry point and the config file. | ||
|
|
||
| All the environments are registered in the `Bimanual` extension. They start | ||
| with `Template-` in their name. | ||
| """ | ||
|
|
||
| """Launch Isaac Sim Simulator first.""" | ||
|
|
||
| import argparse | ||
|
|
||
| from isaaclab.app import AppLauncher | ||
|
|
||
| # add argparse arguments | ||
| parser = argparse.ArgumentParser(description="List Isaac Lab environments.") | ||
| parser.add_argument("--keyword", type=str, default=None, help="Keyword to filter environments.") | ||
| # parse the arguments | ||
| args_cli = parser.parse_args() | ||
|
|
||
| # launch omniverse app | ||
| app_launcher = AppLauncher(headless=True) | ||
| simulation_app = app_launcher.app | ||
|
|
||
|
|
||
| """Rest everything follows.""" | ||
|
|
||
| import gymnasium as gym | ||
| from prettytable import PrettyTable | ||
|
|
||
| import bimanual.tasks # noqa: F401 | ||
|
|
||
|
|
||
| def main(): | ||
| """Print all environments registered in `Bimanual` extension.""" | ||
| # print all the available environments | ||
| table = PrettyTable(["S. No.", "Task Name", "Entry Point", "Config"]) | ||
| table.title = "Available Environments in Isaac Lab" | ||
| # set alignment of table columns | ||
| table.align["Task Name"] = "l" | ||
| table.align["Entry Point"] = "l" | ||
| table.align["Config"] = "l" | ||
|
|
||
| # count of environments | ||
| index = 0 | ||
| # acquire all Isaac environments names | ||
| for task_spec in gym.registry.values(): | ||
| if "Template-" in task_spec.id and (args_cli.keyword is None or args_cli.keyword in task_spec.id): | ||
| # add details to table | ||
| table.add_row([index + 1, task_spec.id, task_spec.entry_point, task_spec.kwargs["env_cfg_entry_point"]]) | ||
| # increment count | ||
| index += 1 | ||
|
|
||
| print(table) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| try: | ||
| # run the main function | ||
| main() | ||
| except Exception as e: | ||
| raise e | ||
| finally: | ||
| # close the app | ||
| simulation_app.close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: please rename to README.md so that it's consistent with the rest of the code base. thanks.