Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/source/references/retargeting/index.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0

Retargeting Interface
=====================

Expand Down Expand Up @@ -41,6 +44,15 @@ Available Retargeters
``hand_side`` (``"left"`` or ``"right"``), ``gripper_close_meters``, ``gripper_open_meters``,
and ``controller_threshold`` for trigger-based closing.

.. dropdown:: SO101ClutchRetargeter / SO101GripperRetargeter

Retargeters for the SO-101 5-DOF arm under full-pose SE3 IK. ``SO101ClutchRetargeter``
outputs a 7-D ``ee_pose`` like ``Se3AbsRetargeter`` but clutch-rebases controller position
around an origin captured on engage (no teleport) and composes a fixed orientation
calibration offset onto the grip orientation so the gripper pose follows the controller pose.
``SO101GripperRetargeter`` maps the trigger to a proportional jaw closedness in ``[0, 1]``.
See :doc:`so101` for the full setup.

.. dropdown:: DexHandRetargeter / DexBiManualRetargeter

Accurate hand tracking retargeter using the ``dex-retargeting`` library. It maps full hand
Expand Down Expand Up @@ -315,3 +327,4 @@ and :doc:`Contributing Guide <../../getting_started/contributing>` for details.
:caption: Retargeter setup guides

sharpa
so101
156 changes: 156 additions & 0 deletions docs/source/references/retargeting/so101.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0

Retargeters: SO-101 (5-DOF arm)
===============================

The SO-101 is a low-cost 5-DOF arm with a single-jaw gripper. The controller is meant to feel
like holding the **leader arm**: the gripper pose follows the controller pose directly. A
full SE3 pose is commanded and a single differential IK solves all 5 arm joints, tracking
position exactly and orientation best-effort (a 5-DOF arm is over-determined by one DOF on a
6-DOF pose). These two retargeters provide the pieces for comfortable XR controller
teleoperation of the arm (used by the Isaac Lab ``Isaac-Stack-Cube-SO101-IK-Abs-v0``
cube-stacking task):

* :class:`~isaacteleop.retargeters.SO101ClutchRetargeter` -- absolute EE **pose** (position +
orientation) with clutch-style position rebasing (no teleport on engage).
* :class:`~isaacteleop.retargeters.SO101GripperRetargeter` -- proportional (analog) jaw
closedness from the controller trigger.

Together they flatten (via :class:`~isaacteleop.retargeters.TensorReorderer`) into an 8-D action
``[pos_x, pos_y, pos_z, quat_x, quat_y, quat_z, quat_w, gripper]``.

At a glance
-----------

.. list-table::
:header-rows: 1
:widths: 28 22 50

* - Retargeter
- Output
- What it does
* - ``SO101ClutchRetargeter``
- 7-D ``ee_pose`` (xyz + xyzw quat)
- Same output contract as ``Se3AbsRetargeter``, but rebases controller motion around an
origin captured on engage: ``pos = home + scale * (p_ctrl - p0)``. The orientation is
the controller grip orientation composed with a fixed calibration offset; it drives the
SE3 IK.
* - ``SO101GripperRetargeter``
- 1 float ``gripper_command`` ``c`` in ``[0, 1]``
- Trigger -> jaw closedness (``0`` = open, ``1`` = closed), with a released-end deadzone.

Why a clutch
------------

``Se3AbsRetargeter`` maps the controller's absolute position straight to the EE target, so
engaging teleop teleports the arm to wherever the controller happens to be. The clutch instead
**re-arms** whenever teleop is not ``RUNNING`` and latches a controller origin ``p0`` on the
first ``RUNNING`` frame (the headset "Play"). From then on the EE position is driven by the
*delta* relative to ``p0``, so engaging with a steady controller does not move the arm. On the
latching frame ``p_ctrl == p0``, so the emitted position is exactly the home (no jump). The last
pose is held on a dropped frame.

The clutch keeps position-control IK (``use_relative_mode=False``): it emits an **absolute**
target, just rebased.

Frames and the home
^^^^^^^^^^^^^^^^^^^^

The controller stream reaching the clutch is already expressed in the robot **base** frame: the
Isaac Lab ``IsaacTeleopDevice`` rebases it upstream via its ``target_frame_prim_path`` (set to
the robot base), composing ``base_T_world`` onto the XR anchor before the controllers are
transformed. The clutch therefore applies the controller delta to the home directly, with no
world->base rotation of its own, and needs **no** live end-effector or base feed.

The ``home`` is the clutch's own running home: it is seeded on reset / first engage from the
static ``home_base_T_ee`` reset-origin (the gripper's pose in the base frame at the reset pose,
only its translation is used), and thereafter holds the last commanded pose so a mid-task
re-clutch resumes from where the arm was left.

.. note::

The fallback home, the position sign/scale knobs, and the orientation calibration offset
carry ``TODO(tune-in-sim)`` markers: the rebasing math is exact and unit-tested, but the
end-to-end controller->EE handedness and the neutral-controller -> neutral-gripper offset
should be confirmed in simulation.

Orientation calibration
-----------------------

The clutch emits the controller grip orientation composed with a fixed calibration offset:
``q_cmd = orientation_offset (x) q_grip`` (base-frame left multiply, renormalized). The offset
defaults to identity (passthrough); a non-identity offset maps a neutrally-held controller to a
sensible neutral gripper orientation for the SE3 IK. This single rotational offset replaces the
per-DOF roll/pitch calibration hacks of earlier revisions.

Gripper
-------

``SO101GripperRetargeter`` maps the analog trigger to a jaw **closedness** ``c`` in ``[0, 1]``
(``0`` = open, ``1`` = closed) with a small released-end deadzone, so a half-pressed trigger
leaves the jaw half-closed. Downstream, an order-locked ``JointPositionActionCfg`` applies the
affine ``joint = offset + scale * c`` mapping ``c`` onto the open/close joint angles. This is
deliberately independent of the shared :class:`~isaacteleop.retargeters.GripperRetargeter`'s
binary ``+1 = open`` / ``-1 = closed`` sign.

Use it from Python
------------------

.. code-block:: python

from isaacteleop.retargeters import (
SO101ClutchRetargeter,
SO101GripperRetargeter,
TensorReorderer,
)
from isaacteleop.retargeting_engine.deviceio_source_nodes import ControllersSource
from isaacteleop.retargeting_engine.interface import OutputCombiner, ValueInput
from isaacteleop.retargeting_engine.tensor_types import TransformMatrix

def build_so101_stack_pipeline():
controllers = ControllersSource(name="controllers")
world_T_anchor = ValueInput("world_T_anchor", TransformMatrix())
# The device rebases controller poses into the robot base frame upstream via
# target_frame_prim_path, so the clutch needs no live EE / base feed.
xformed = controllers.transformed(world_T_anchor.output(ValueInput.VALUE))

clutch = SO101ClutchRetargeter(name="ee_pose", input_device=ControllersSource.RIGHT)
connected_clutch = clutch.connect({
ControllersSource.RIGHT: xformed.output(ControllersSource.RIGHT),
})

gripper = SO101GripperRetargeter(name="gripper", input_device=ControllersSource.RIGHT)
connected_gripper = gripper.connect(
{ControllersSource.RIGHT: xformed.output(ControllersSource.RIGHT)}
)

# Keep all 7 pose names and pass the full pose (xyz + quat) plus gripper through.
ee_elements = ["pos_x", "pos_y", "pos_z", "quat_x", "quat_y", "quat_z", "quat_w"]
reorderer = TensorReorderer(
input_config={
"ee_pose": ee_elements,
"gripper_command": ["gripper_value"],
},
output_order=ee_elements + ["gripper_value"],
name="action_reorderer",
input_types={"ee_pose": "array", "gripper_command": "scalar"},
)
connected = reorderer.connect({
"ee_pose": connected_clutch.output("ee_pose"),
"gripper_command": connected_gripper.output("gripper_command"),
})
return OutputCombiner({"action": connected.output("output")})

See :ref:`isaac-teleop-pipeline-builder` for the general pipeline-builder pattern and
:doc:`index` for the full retargeting interface.

Validate
--------

The retargeters ship with sim-free unit tests (trigger/clutch math plus per-frame
``compute`` behavior):

.. code-block:: console

$ ctest --test-dir build -R retargeting_test_so101_retargeters --output-on-failure
1 change: 1 addition & 0 deletions src/core/python/pyproject.toml.in
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ packages = [
"isaacteleop.retargeting_engine.utilities",
"isaacteleop.retargeters",
"isaacteleop.retargeters.G1",
"isaacteleop.retargeters.SO101",
"isaacteleop.retargeting_engine_ui",
"isaacteleop.teleop_session_manager",
"isaacteleop.cloudxr",
Expand Down
Loading
Loading