-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Update COMPASS training doc #6135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
0272ec4
562e456
0cfb0c6
4d665a5
ddcc577
f69be8e
63a59e0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,27 +1,63 @@ | ||||||
| Training & Deploying COMPASS Navigation Policy with Real2Sim NuRec | ||||||
| ==================================================================== | ||||||
| Training and Deploying COMPASS Policies with NuRec Assets | ||||||
| ========================================================= | ||||||
|
|
||||||
| COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) trains and | ||||||
| deploys cross-embodiment navigation policies on NuRec Real2Sim assets — photoreal | ||||||
| Gaussian-splat reconstructions of real spaces — in Isaac Lab. | ||||||
| `COMPASS <https://arxiv.org/abs/2502.16372>`_ (Cross-embodiment Mobility Policy via Residual RL and | ||||||
| Skill Synthesis) trains navigation policies that can transfer across robot embodiments. Isaac Lab supports | ||||||
| COMPASS training with NuRec scene assets, so policies can be developed in reconstructed real-world | ||||||
| environments instead of relying only on hand-authored simulation scenes. | ||||||
|
|
||||||
| This tutorial now lives in the **COMPASS handbook**, next to the code it documents, where it | ||||||
| stays in sync with the NuRec support branch. | ||||||
| This page gives a short overview of the supported NuRec assets and example COMPASS training clips. For | ||||||
| installation, asset registration, training, evaluation, export, and ROS 2 deployment instructions, see the | ||||||
| `COMPASS handbook <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_. | ||||||
|
|
||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Critical: Broken link (404) This URL returns 404 — the branch The existing branch is
Suggested change
|
||||||
| Where to find it | ||||||
| ---------------- | ||||||
| NuRec Assets for COMPASS | ||||||
| ------------------------ | ||||||
|
|
||||||
| NuRec assets matter for COMPASS because they allow navigation policies to train in simulation on realistic | ||||||
| geometry, visual appearance, and obstacle layouts. This, in turn, should reduce the sim-to-real gap between | ||||||
| training scenes and real operating environments and improve zero-shot transfer. They also make it possible | ||||||
| to evaluate policies in reconstructed simulation scenes that better reflect target deployment spaces. | ||||||
|
|
||||||
| Some sample assets compatible with COMPASS training are available in the | ||||||
| `PhysicalAI-Robotics-NuRec dataset on Hugging Face <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2>`_. | ||||||
|
|
||||||
| Real2Sim Assets | ||||||
| ~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| `NVIDIA Neural Reconstruction (NuRec) <https://docs.nvidia.com/nurec/index.html>`_ converts real-world | ||||||
| sensor data into digital simulation-ready assets for training and testing Physical AI agents. The Real2Sim | ||||||
| workflow converts stereo RGB captures into Isaac Sim-ready assets with aligned geometry, collision meshes, | ||||||
| and simulation-ready scene augmentation. The PhysicalAI-Robotics-NuRec dataset includes several robotics | ||||||
| scenes produced with this workflow; two examples are: | ||||||
|
|
||||||
| - `Galileo <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/nova_carter-galileo>`_ — | ||||||
| Embodiment: `Nova Carter <https://robotics.segway.com/nova-carter/>`_, | ||||||
| Sensor Rig: 4 x `Stereo Camera <https://leopardimaging.com/wp-content/uploads/2024/07/LI-AR0234CS-STEREO-GMSL2-30_Datasheet_V1.8.pdf>`_ | ||||||
|
|
||||||
| - **Repository:** `COMPASS on GitHub <https://github.com/NVlabs/COMPASS>`_ | ||||||
| - **Branch:** ``samc/support_nurec_assets_isaaclab_3.0`` (NuRec Real2Sim support) | ||||||
| - **Guide:** `docs/handbook/workflows/nurec_real2sim.md <https://github.com/NVlabs/COMPASS/blob/samc/support_nurec_assets_isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ in the COMPASS repository. | ||||||
| - `Living Room <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/hand_hold-endeavor-livingroom>`_ — | ||||||
| Embodiment: Hand-Held, | ||||||
| Sensor Rig: 1 x `Stereo Camera <https://leopardimaging.com/wp-content/uploads/2024/07/LI-AR0234CS-STEREO-GMSL2-30_Datasheet_V1.8.pdf>`_ | ||||||
|
|
||||||
| XGRIDS Assets | ||||||
| ~~~~~~~~~~~~~ | ||||||
|
|
||||||
| COMPASS also supports compatible assets reconstructed with XGRIDS. These assets are post-processed into the | ||||||
| NuRec format used by the COMPASS workflow, including mesh alignment and augmentation with simulation-ready | ||||||
| assets available in Isaac Sim. | ||||||
|
|
||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Critical: Broken link (404) Same issue as line 12 — this link also needs to be updated to the correct branch.
Suggested change
|
||||||
| - `Wormhole <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/xgrid-wormhole>`_ — | ||||||
| Embodiment: Hand-Held, | ||||||
| Sensor Rig: `XGRIDS PortalCam <https://www.xgrids.com/intl/portalcamp>`_ [2 x fish-eye cameras, 2 x front cameras, 1 x LiDAR] | ||||||
|
|
||||||
| COMPASS Workflow | ||||||
| ---------------- | ||||||
|
|
||||||
| .. code-block:: bash | ||||||
| The complete COMPASS workflow is maintained in the COMPASS repository: | ||||||
|
|
||||||
| git clone https://github.com/NVlabs/COMPASS.git | ||||||
| cd COMPASS | ||||||
| git checkout samc/support_nurec_assets_isaaclab_3.0 | ||||||
| # Open docs/handbook/workflows/nurec_real2sim.md | ||||||
| - **Repository:** `NVlabs/COMPASS <https://github.com/NVlabs/COMPASS>`_ | ||||||
| - **Branch:** ``real2sim/isaaclab_3.0`` | ||||||
| - **Guide:** `NuRec asset workflow <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ | ||||||
|
|
||||||
| The guide covers Isaac Sim / Isaac Lab installation, COMPASS setup, downloading and | ||||||
| registering NuRec Real2Sim scenes, training a residual RL specialist, evaluation, | ||||||
| ONNX / TensorRT export, and ROS2 / sim-to-real deployment. | ||||||
| The guide covers the COMPASS code path for training with NuRec assets generated by the Real2Sim workflow, | ||||||
| including Isaac Sim and Isaac Lab setup, COMPASS installation, asset download and registration, residual RL | ||||||
| specialist training, evaluation, ONNX and TensorRT export, and ROS 2 sim-to-real deployment. | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 CI Blocker — Broken Link (404)
The
Check for Broken LinksCI job fails because this URL returns HTTP 404:Possible causes:
real2sim/isaaclab_3.0may not exist yet on the public repo (private or not yet pushed)Please verify the branch/path exist on the public
NVlabs/COMPASSrepo, or exclude this URL from the link checker config until the branch is published. This same URL appears again on line 87.