-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Update Compass tutorial doc #6134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
7e0e879
526f514
c2842d1
d5219ff
d77a606
8c4637a
8d660f7
5a8941f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,113 @@ | ||
| Training & Deploying COMPASS Navigation Policy with Real2Sim NuRec | ||
| ==================================================================== | ||
| Training and Deploying COMPASS Policies with NuRec Assets | ||
| ========================================================= | ||
|
|
||
| COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) trains and | ||
| deploys cross-embodiment navigation policies on NuRec Real2Sim assets — photoreal | ||
| Gaussian-splat reconstructions of real spaces — in Isaac Lab. | ||
| `COMPASS <https://arxiv.org/abs/2502.16372>`_ (Cross-embodiment Mobility Policy via Residual RL and | ||
| Skill Synthesis) trains navigation policies that can transfer across robot embodiments. Isaac Lab supports | ||
| COMPASS training with NuRec scene assets, so policies can be developed in reconstructed real-world | ||
| environments instead of relying only on hand-authored simulation scenes. | ||
|
|
||
| This tutorial now lives in the **COMPASS handbook**, next to the code it documents, where it | ||
| stays in sync with the NuRec support branch. | ||
| This page gives a short overview of the supported NuRec assets and example COMPASS training clips. For | ||
| installation, asset registration, training, evaluation, export, and ROS 2 deployment instructions, see the | ||
| `COMPASS handbook <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 CI Blocker — Broken link (404) This URL returns 404: The branch Please verify the correct branch name and path. If the branch hasn't been pushed yet, consider linking to a commit SHA or the repo root until it's available. |
||
|
|
||
| Where to find it | ||
| NuRec Assets for COMPASS | ||
| ------------------------ | ||
|
|
||
| NuRec assets matter for COMPASS because they allow navigation policies to train in simulation on realistic | ||
| geometry, visual appearance, and obstacle layouts. This, in turn, should reduce the sim-to-real gap between | ||
| training scenes and real operating environments and improve zero-shot transfer. They also make it possible | ||
| to evaluate policies in reconstructed simulation scenes that better reflect target deployment spaces. | ||
|
|
||
| Some sample assets compatible with COMPASS training are available in the | ||
| `PhysicalAI-Robotics-NuRec dataset on Hugging Face <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2>`_. | ||
|
|
||
| Real2Sim Assets | ||
| ~~~~~~~~~~~~~~~ | ||
|
|
||
| `NVIDIA Neural Reconstruction (NuRec) <https://docs.nvidia.com/nurec/index.html>`_ converts real-world | ||
| sensor data into digital simulation-ready assets for training and testing Physical AI agents. The Real2Sim | ||
| workflow converts stereo RGB captures into Isaac Sim-ready assets with aligned geometry, collision meshes, | ||
| and simulation-ready scene augmentation. The PhysicalAI-Robotics-NuRec dataset includes several robotics | ||
| scenes produced with this workflow; two examples are shown below. | ||
|
|
||
| .. list-table:: | ||
| :widths: 50 50 | ||
| :class: nurec-gif-grid | ||
|
|
||
| * - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/nurec_real2sim_galileo.gif | ||
| :alt: Galileo Real2Sim NuRec asset | ||
| :target: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/nova_carter-galileo | ||
| :class: nurec-gif-cover | ||
| :align: center | ||
|
|
||
| **Galileo** | ||
|
|
||
| | Embodiment: `Nova Carter [wheeled robot] <https://robotics.segway.com/nova-carter/>`_ | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 CI Blocker — codespell false positive codespell flags Check the repo's codespell config (likely |
||
| | Sensor Rig: 4 x `Stereo Camera <https://leopardimaging.com/wp-content/uploads/2024/07/LI-AR0234CS-STEREO-GMSL2-30_Datasheet_V1.8.pdf>`_ | ||
| - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/nurec_real2sim_living_room.gif | ||
| :alt: Living Room Real2Sim NuRec asset | ||
| :target: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/hand_hold-endeavor-livingroom | ||
| :class: nurec-gif-cover | ||
| :align: center | ||
|
|
||
| **Living Room** | ||
|
|
||
| | Embodiment: Hand-Held | ||
| | Sensor Rig: 1 x `Stereo Camera <https://leopardimaging.com/wp-content/uploads/2024/07/LI-AR0234CS-STEREO-GMSL2-30_Datasheet_V1.8.pdf>`_ | ||
|
|
||
| XGRIDS Assets | ||
| ~~~~~~~~~~~~~ | ||
|
|
||
| COMPASS also supports compatible assets reconstructed with XGRIDS. These assets are post-processed into the | ||
| NuRec format used by the COMPASS workflow, including mesh alignment and augmentation with simulation-ready | ||
| assets available in Isaac Sim. | ||
|
|
||
| .. list-table:: | ||
| :widths: 100 | ||
| :class: nurec-gif-grid nurec-gif-grid-single | ||
|
|
||
| * - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/xgrid_wormhole_with_sim_objects.gif | ||
| :alt: Wormhole XGRIDS reconstruction | ||
| :target: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/xgrid-wormhole | ||
| :class: nurec-gif-cover | ||
| :align: center | ||
|
|
||
| **Wormhole** | ||
|
|
||
| | Embodiment: Hand-Held | ||
| | Sensor Rig: `XGRIDS PortalCam <https://www.xgrids.com/intl/portalcamp>`_ [2 x fish-eye cameras, 2 x front cameras, 1 x LiDAR] | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Possible URL typo: The link target is Both URLs currently resolve (200), so this isn't a CI failure, but it may break if XGRIDS fixes the redirect in the future. |
||
|
|
||
| COMPASS Workflow | ||
| ---------------- | ||
|
|
||
| - **Repository:** `COMPASS on GitHub <https://github.com/NVlabs/COMPASS>`_ | ||
| - **Branch:** ``samc/support_nurec_assets_isaaclab_3.0`` (NuRec Real2Sim support) | ||
| - **Guide:** `docs/handbook/workflows/nurec_real2sim.md <https://github.com/NVlabs/COMPASS/blob/samc/support_nurec_assets_isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ in the COMPASS repository. | ||
| The complete COMPASS workflow is maintained in the COMPASS repository: | ||
|
|
||
| - **Repository:** `NVlabs/COMPASS <https://github.com/NVlabs/COMPASS>`_ | ||
| - **Branch:** ``real2sim/isaaclab_3.0`` | ||
| - **Guide:** `NuRec asset workflow <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ | ||
|
|
||
| The guide covers the COMPASS code path for training with NuRec assets generated by the Real2Sim workflow, | ||
| including Isaac Sim and Isaac Lab setup, COMPASS installation, asset download and registration, residual RL | ||
| specialist training, evaluation, ONNX and TensorRT export, and ROS 2 sim-to-real deployment. | ||
|
|
||
| Training Examples | ||
| ----------------- | ||
|
|
||
| The following clips show COMPASS training with different robot embodiments in NuRec scenes. | ||
|
|
||
| .. list-table:: | ||
| :widths: 50 50 | ||
| :class: nurec-gif-grid | ||
|
|
||
| .. code-block:: bash | ||
| * - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/compass_galileo_g1_training.gif | ||
| :alt: COMPASS training with a G1 robot in the Galileo scene | ||
| :class: nurec-gif-contain | ||
| :align: center | ||
|
|
||
| git clone https://github.com/NVlabs/COMPASS.git | ||
| cd COMPASS | ||
| git checkout samc/support_nurec_assets_isaaclab_3.0 | ||
| # Open docs/handbook/workflows/nurec_real2sim.md | ||
| **Galileo** with G1 | ||
| - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/compass_xgrid_wormhole_carter_training.gif | ||
| :alt: COMPASS training with a Carter robot in the XGRIDS Wormhole scene | ||
| :class: nurec-gif-contain | ||
| :align: center | ||
|
|
||
| The guide covers Isaac Sim / Isaac Lab installation, COMPASS setup, downloading and | ||
| registering NuRec Real2Sim scenes, training a residual RL specialist, evaluation, | ||
| ONNX / TensorRT export, and ROS2 / sim-to-real deployment. | ||
| **Wormhole** with Carter | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟢 Nit (robustness): The
white-space: nowraprule on the second line of single-grid captions could cause horizontal overflow on narrow viewports (mobile). Consider addingoverflow: hidden; text-overflow: ellipsis;or testing on small screens. Low priority since docs are primarily consumed on desktop.