Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/source/_static/css/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,62 @@ a {
width: min(100%, 480px);
}

.nurec-gif-grid,
.nurec-gif-grid td {
border: none;
}

.nurec-gif-grid td {
vertical-align: top;
}

.nurec-gif-grid figure {
margin-left: auto;
margin-right: auto;
max-width: 360px;
}

figure.nurec-gif-cover img,
figure.nurec-gif-contain img {
aspect-ratio: 16 / 9;
display: block;
object-position: center;
width: 100%;
}

figure.nurec-gif-cover img {
object-fit: cover;
}

figure.nurec-gif-contain img {
object-fit: contain;
}

.nurec-gif-grid figcaption {
font-family: "NVIDIA Sans", "Inter", "Segoe UI", sans-serif;
font-size: 0.86rem;
line-height: 1.35;
}

.nurec-gif-grid figcaption p {
margin-bottom: 0.15rem;
}

.nurec-gif-grid figcaption .line-block {
margin-top: 0;
}

.nurec-gif-grid-single figcaption {
margin-left: 50%;
max-width: calc(100vw - 2rem);
transform: translateX(-50%);
width: max-content;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Nit (robustness): The white-space: nowrap rule on the second line of single-grid captions could cause horizontal overflow on narrow viewports (mobile). Consider adding overflow: hidden; text-overflow: ellipsis; or testing on small screens. Low priority since docs are primarily consumed on desktop.

.nurec-gif-grid-single figcaption .line-block .line:nth-child(2) {
white-space: nowrap;
}

/* Keep sphinx-copybutton visible on code blocks (not only on hover). */
button.copybtn {
opacity: 1;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,27 +1,113 @@
Training & Deploying COMPASS Navigation Policy with Real2Sim NuRec
====================================================================
Training and Deploying COMPASS Policies with NuRec Assets
=========================================================

COMPASS (Cross-embodiment Mobility Policy via Residual RL and Skill Synthesis) trains and
deploys cross-embodiment navigation policies on NuRec Real2Sim assets — photoreal
Gaussian-splat reconstructions of real spaces — in Isaac Lab.
`COMPASS <https://arxiv.org/abs/2502.16372>`_ (Cross-embodiment Mobility Policy via Residual RL and
Skill Synthesis) trains navigation policies that can transfer across robot embodiments. Isaac Lab supports
COMPASS training with NuRec scene assets, so policies can be developed in reconstructed real-world
environments instead of relying only on hand-authored simulation scenes.

This tutorial now lives in the **COMPASS handbook**, next to the code it documents, where it
stays in sync with the NuRec support branch.
This page gives a short overview of the supported NuRec assets and example COMPASS training clips. For
installation, asset registration, training, evaluation, export, and ROS 2 deployment instructions, see the
`COMPASS handbook <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 CI Blocker — Broken link (404)

This URL returns 404:

https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md

The branch real2sim/isaaclab_3.0 doesn't appear to exist on the COMPASS repo (or the path has changed). This also fails the "Check for Broken Links" CI job.

Please verify the correct branch name and path. If the branch hasn't been pushed yet, consider linking to a commit SHA or the repo root until it's available.


Where to find it
NuRec Assets for COMPASS
------------------------

NuRec assets matter for COMPASS because they allow navigation policies to train in simulation on realistic
geometry, visual appearance, and obstacle layouts. This, in turn, should reduce the sim-to-real gap between
training scenes and real operating environments and improve zero-shot transfer. They also make it possible
to evaluate policies in reconstructed simulation scenes that better reflect target deployment spaces.

Some sample assets compatible with COMPASS training are available in the
`PhysicalAI-Robotics-NuRec dataset on Hugging Face <https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2>`_.

Real2Sim Assets
~~~~~~~~~~~~~~~

`NVIDIA Neural Reconstruction (NuRec) <https://docs.nvidia.com/nurec/index.html>`_ converts real-world
sensor data into digital simulation-ready assets for training and testing Physical AI agents. The Real2Sim
workflow converts stereo RGB captures into Isaac Sim-ready assets with aligned geometry, collision meshes,
and simulation-ready scene augmentation. The PhysicalAI-Robotics-NuRec dataset includes several robotics
scenes produced with this workflow; two examples are shown below.

.. list-table::
:widths: 50 50
:class: nurec-gif-grid

* - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/nurec_real2sim_galileo.gif
:alt: Galileo Real2Sim NuRec asset
:target: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/nova_carter-galileo
:class: nurec-gif-cover
:align: center

**Galileo**

| Embodiment: `Nova Carter [wheeled robot] <https://robotics.segway.com/nova-carter/>`_

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 CI Blocker — codespell false positive

codespell flags segwaysegue here. Since this is the brand name Segway Robotics, you'll need to tell codespell to ignore it.

Check the repo's codespell config (likely .pre-commit-config.yaml or a .codespellrc/setup.cfg section) for an ignore-words-list and add segway there. Other Isaac Lab docs may already handle this — worth checking.

| Sensor Rig: 4 x `Stereo Camera <https://leopardimaging.com/wp-content/uploads/2024/07/LI-AR0234CS-STEREO-GMSL2-30_Datasheet_V1.8.pdf>`_
- .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/nurec_real2sim_living_room.gif
:alt: Living Room Real2Sim NuRec asset
:target: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/hand_hold-endeavor-livingroom
:class: nurec-gif-cover
:align: center

**Living Room**

| Embodiment: Hand-Held
| Sensor Rig: 1 x `Stereo Camera <https://leopardimaging.com/wp-content/uploads/2024/07/LI-AR0234CS-STEREO-GMSL2-30_Datasheet_V1.8.pdf>`_

XGRIDS Assets
~~~~~~~~~~~~~

COMPASS also supports compatible assets reconstructed with XGRIDS. These assets are post-processed into the
NuRec format used by the COMPASS workflow, including mesh alignment and augmentation with simulation-ready
assets available in Isaac Sim.

.. list-table::
:widths: 100
:class: nurec-gif-grid nurec-gif-grid-single

* - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/xgrid_wormhole_with_sim_objects.gif
:alt: Wormhole XGRIDS reconstruction
:target: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/v0.2/xgrid-wormhole
:class: nurec-gif-cover
:align: center

**Wormhole**

| Embodiment: Hand-Held
| Sensor Rig: `XGRIDS PortalCam <https://www.xgrids.com/intl/portalcamp>`_ [2 x fish-eye cameras, 2 x front cameras, 1 x LiDAR]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Possible URL typo: The link target is portalcamp (with trailing "p") but the product is called "PortalCam". The correct URL is likely:

https://www.xgrids.com/intl/portalcam

Both URLs currently resolve (200), so this isn't a CI failure, but it may break if XGRIDS fixes the redirect in the future.


COMPASS Workflow
----------------

- **Repository:** `COMPASS on GitHub <https://github.com/NVlabs/COMPASS>`_
- **Branch:** ``samc/support_nurec_assets_isaaclab_3.0`` (NuRec Real2Sim support)
- **Guide:** `docs/handbook/workflows/nurec_real2sim.md <https://github.com/NVlabs/COMPASS/blob/samc/support_nurec_assets_isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_ in the COMPASS repository.
The complete COMPASS workflow is maintained in the COMPASS repository:

- **Repository:** `NVlabs/COMPASS <https://github.com/NVlabs/COMPASS>`_
- **Branch:** ``real2sim/isaaclab_3.0``
- **Guide:** `NuRec asset workflow <https://github.com/NVlabs/COMPASS/blob/real2sim/isaaclab_3.0/docs/handbook/workflows/nurec_real2sim.md>`_

The guide covers the COMPASS code path for training with NuRec assets generated by the Real2Sim workflow,
including Isaac Sim and Isaac Lab setup, COMPASS installation, asset download and registration, residual RL
specialist training, evaluation, ONNX and TensorRT export, and ROS 2 sim-to-real deployment.

Training Examples
-----------------

The following clips show COMPASS training with different robot embodiments in NuRec scenes.

.. list-table::
:widths: 50 50
:class: nurec-gif-grid

.. code-block:: bash
* - .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/compass_galileo_g1_training.gif
:alt: COMPASS training with a G1 robot in the Galileo scene
:class: nurec-gif-contain
:align: center

git clone https://github.com/NVlabs/COMPASS.git
cd COMPASS
git checkout samc/support_nurec_assets_isaaclab_3.0
# Open docs/handbook/workflows/nurec_real2sim.md
**Galileo** with G1
- .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/compass_xgrid_wormhole_carter_training.gif
:alt: COMPASS training with a Carter robot in the XGRIDS Wormhole scene
:class: nurec-gif-contain
:align: center

The guide covers Isaac Sim / Isaac Lab installation, COMPASS setup, downloading and
registering NuRec Real2Sim scenes, training a residual RL specialist, evaluation,
ONNX / TensorRT export, and ROS2 / sim-to-real deployment.
**Wormhole** with Carter
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ reportPrivateUsage = "warning"
skip = '*.usd,*.usda,*.usdz,*.svg,*.png,_isaac_sim*,*.bib,*.css,*/_build'
quiet-level = 0
# the world list should always have words in lower case
ignore-words-list = "haa,slq,collapsable,buss,reacher,thirdparty"
ignore-words-list = "haa,slq,collapsable,buss,reacher,thirdparty,segway"


[tool.pytest.ini_options]
Expand Down
Loading