feat: add generic joint-space teleop device (SO-101 leader)#660
Open
rwiltz wants to merge 5 commits into
Open
Conversation
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR introduces a generic joint-space device abstraction for Isaac Teleop, enabling developers to stream any robotic arm's joint state through a unified pipeline. It adds a FlatBuffers schema for joint-state snapshots, tracker implementations for live/replay modes, retargeting nodes supporting joint-mirroring and end-effector pose modes (with optional pinocchio-based FK), and a reference SO-101 leader arm plugin with synthetic trajectory backend. The feature includes comprehensive Python tests, a CLI example for offline/live retargeting, and documentation explaining the extension points. Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
rwiltz
force-pushed
the
rwiltz/so101-joint-space-device
branch
from
326291b to
da6e097
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/teleop/python/joint_space_device_example.py`:
- Around line 248-258: The check that computes varied using any(...) fails for
single-frame runs because the generator is empty; update the logic around varied
(used with actions and labels) to treat a single received frame as non-stale by
setting varied = True when len(actions) <= 1 (or when num_frames == 1) before
running the any(...) comparison, so the subsequent stale-stream exit is only
triggered for multi-frame cases where actions do not differ over time.
In `@src/core/replay_trackers/cpp/replay_joint_state_tracker_impl.cpp`:
- Around line 37-48: The update() method (ReplayJointStateTrackerImpl::update)
currently emits std::cerr every frame when mcap_viewers_->read(0) returns null,
causing spam at EOF/sparse streams; change this to either a lower-severity/no-op
log or emit it once on transition: replace the immediate std::cerr call in
ReplayJointStateTrackerImpl::update with a debug/trace-level log or implement a
small boolean member (e.g., warned_no_data_) that tracks whether the "no data"
condition was already reported and only logs the first time, and ensure
tracked_.data.reset() still runs when no record is present.
In `@src/retargeters/joint_space/joint_state_retargeter.py`:
- Around line 240-252: The reset branch in the block that handles
context.execution_events.reset (where _origin and _last_pose are cleared) does
not reset _last_gripper, causing stale gripper commands to be replayed when
joints are absent; update that same reset branch (the if
context.execution_events.reset ... block in joint_state_retargeter.py) to set
self._last_gripper back to the class's default initial gripper state (the same
value used when the object is constructed) so that when jin.is_none triggers the
hold-last path it emits the fresh/reset gripper value instead of the previous
session's command.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 12c11cce-bb3b-463a-b5df-527f43929f9a
📒 Files selected for processing (46)
CMakeLists.txtdocs/source/device/index.rstdocs/source/device/joint_space.rstdocs/source/references/retargeting/index.rstdocs/source/references/retargeting/joint_space.rstexamples/teleop/python/joint_space_device_example.pysrc/core/deviceio_base/cpp/inc/deviceio_base/joint_state_tracker_base.hppsrc/core/deviceio_trackers/cpp/CMakeLists.txtsrc/core/deviceio_trackers/cpp/inc/deviceio_trackers/joint_state_tracker.hppsrc/core/deviceio_trackers/cpp/joint_state_tracker.cppsrc/core/deviceio_trackers/python/deviceio_trackers_init.pysrc/core/deviceio_trackers/python/tracker_bindings.cppsrc/core/live_trackers/cpp/CMakeLists.txtsrc/core/live_trackers/cpp/inc/live_trackers/live_deviceio_factory.hppsrc/core/live_trackers/cpp/live_deviceio_factory.cppsrc/core/live_trackers/cpp/live_joint_state_tracker_impl.cppsrc/core/live_trackers/cpp/live_joint_state_tracker_impl.hppsrc/core/mcap/cpp/inc/mcap/recording_traits.hppsrc/core/python/deviceio_init.pysrc/core/python/pyproject.toml.insrc/core/python/requirements-retargeters.txtsrc/core/replay_trackers/cpp/CMakeLists.txtsrc/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hppsrc/core/replay_trackers/cpp/replay_deviceio_factory.cppsrc/core/replay_trackers/cpp/replay_joint_state_tracker_impl.cppsrc/core/replay_trackers/cpp/replay_joint_state_tracker_impl.hppsrc/core/retargeting_engine/python/deviceio_source_nodes/__init__.pysrc/core/retargeting_engine/python/deviceio_source_nodes/deviceio_tensor_types.pysrc/core/retargeting_engine/python/deviceio_source_nodes/joint_state_source.pysrc/core/retargeting_engine_tests/python/test_joint_state_retargeter.pysrc/core/retargeting_engine_tests/python/test_joint_state_source.pysrc/core/schema/fbs/joint_state.fbssrc/core/schema/python/CMakeLists.txtsrc/core/schema/python/joint_state_bindings.hsrc/core/schema/python/schema_init.pysrc/core/schema/python/schema_module.cppsrc/plugins/so101_leader/CMakeLists.txtsrc/plugins/so101_leader/README.mdsrc/plugins/so101_leader/main.cppsrc/plugins/so101_leader/plugin.yamlsrc/plugins/so101_leader/so101_leader_plugin.cppsrc/plugins/so101_leader/so101_leader_plugin.hppsrc/retargeters/CMakeLists.txtsrc/retargeters/__init__.pysrc/retargeters/joint_space/__init__.pysrc/retargeters/joint_space/joint_state_retargeter.py
rwiltz
force-pushed
the
rwiltz/so101-joint-space-device
branch
2 times, most recently
from
c04ff59 to
d2df06a
Compare
Add a reusable joint-space device path to Isaac Teleop: a name-keyed JointStateOutput FlatBuffer schema, a JointStateTracker with live and MCAP-replay backends (registered in the live/replay factories), a JointStateSource, and a dual-mode JointStateRetargeter -- joint mirror (name remap + per-joint affine) or URDF forward-kinematics EE pose. The SO-101 leader arm is the reference instance, with a so101_leader plugin that pushes JointStateOutput (synthetic backend; the real Feetech serial read is left as a marked seam). Includes the Python schema/tracker bindings, the joint_space_device example (live over OpenXR or offline), sim-free unit tests for both retargeter modes, and device + retargeting reference docs.
rwiltz
force-pushed
the
rwiltz/so101-joint-space-device
branch
from
d2df06a to
d838f56
Compare
jiwenc-nv
reviewed
Jun 11, 2026
| .. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| .. SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| Generic Joint-Space Device |
Collaborator
There was a problem hiding this comment.
is there a easier to follow term for this?
Collaborator
There was a problem hiding this comment.
why don't we call it leader follower arms?
| ROBOT_EE_POS_INPUT = "robot_ee_pos" | ||
|
|
||
| def __init__( | ||
| self, name: str, mode: str, config: JointStateRetargeterConfig |
Collaborator
There was a problem hiding this comment.
claude detected this: -
Constructor inverts the house convention JointStateRetargeter.init(self, name, mode, config) (joint_state_retargeter.py:2900); all 10 other config-bearing retargeters use (config, name). Fold mode into JointStateRetargeterConfig and restore (config, name) before the public example/tests lock the positional API.
maybe we should update AGENTS.md so that code generated in the future will be consistent.
| // Meaningful only when has_effort. Present for exoskeletons / bilateral force-feedback rigs. | ||
| effort: float (id: 3); | ||
|
|
||
| // Per-DOF validity. Defaults to true; set false when a single joint read is stale/missing. |
Collaborator
There was a problem hiding this comment.
looks like the retargeter is not using the valid field?
| // Per-frame state of a generic joint-space input device (leader arm, exoskeleton, glove, or | ||
| // any joint-encoder source), as name:value joint records. All fields are present when the | ||
| // parent Tracked/Record wrapper's data is non-null. | ||
| table JointStateOutput { |
Collaborator
There was a problem hiding this comment.
let's link about naming carefully, it makes or breaks the adoption i think.
why don't we call this GenericLeaderArmOutput?
Contributor
Author
There was a problem hiding this comment.
Because this works for leader arms and exoskeleton devices that operate in joint space
Wire the SO-101 leader plugin to real hardware so the joint-space pipeline can run end-to-end with a physical arm. Add FeetechBus, a minimal half-duplex client for the FEETECH SMS/STS bus servos (STS3215) that speaks the same wire protocol as the FEETECH SCServo SDK / LeRobot FeetechMotorsBus without an SDK dependency (POSIX termios only): disable torque so the leader can be back-driven, then read Present_Position each frame and convert ticks to radians with per-joint calibration. When no serial device path is given the plugin keeps its synthetic trajectory, so CI and the headless example still run hardware-free. A device path activates the live backend; calibration (servo id, sign, home tick) comes from an optional file and defaults to ids 1..6 / +1 / center 2048. The serial backend is POSIX-only; Windows compiles to a throwing stub and uses the synthetic fallback.
Read all six FEETECH servos in a single SYNC READ (instruction 0x82, one bus round-trip) instead of six sequential request/response pairs, matching LeRobot's sync_read and cutting per-frame latency. Add a `calibrate` subcommand that mirrors LeRobot's homing step: disable torque, prompt the operator to hold the zero pose, average a few sync reads, and print/write a calibration file (servo id, sign, home tick) in the format the plugin consumes. Runs off the serial bus only -- no OpenXR runtime required.
Extend the `calibrate` subcommand with a range-of-motion sweep that mirrors LeRobot's lerobot-calibrate: after the homing step, the operator moves every joint through its range while per-joint min/max ticks are recorded (ENTER to finish). The calibration file gains optional range_min/range_max columns; reads are clamped to that range, which guards against encoder-wrap spikes and out-of-range jitter. The command also prints the gripper's range endpoints in radians for the retargeter's gripper_open/gripper_close. Range columns are optional and default to the full 0..4095 (clamp no-op), so existing four-column calibration files and the no-file defaults are unchanged.
Make the SO-101 leader calibration interchangeable with LeRobot's. A
.json path is read/written in LeRobot's format ({id, drive_mode,
homing_offset, range_min, range_max} per joint); any other path stays
the plain-text format.
Mapping: range_min/range_max -> our range, the range midpoint ->
home_ticks (LeRobot's zero), drive_mode -> sign. LeRobot keeps each
joint's homing_offset in the servo EEPROM (its runtime normalization
ignores it), so on connect we read the live Homing_Offset (register 31,
sign-magnitude) and shift home/range by file-minus-servo to reconcile
the frames without writing to the servo. The calibrate subcommand emits
LeRobot JSON when the output path ends in .json.
Includes a small dependency-free JSON reader for the flat LeRobot schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a reusable joint-space device path to Isaac Teleop, intended as the
reference implementation for all future joint-space inputs (leader arms,
exoskeletons, ...). The SO-101 leader arm is the first concrete instance.
What's included
joint_state.fbs): name-keyedJointStateOutput(onename → positionentry per DOF) so any device works regardless of DOF countor ordering.
velocity/effort/ee_posereserved for future use.JointStateTrackerwith live (OpenXR) and MCAP-replaybackends, wired into the live/replay factories, plus Python schema + tracker
bindings.
JointStateSourceconverts the FlatBuffer snapshot into theretargeting engine's name-keyed tensor group.
JointStateRetargeter), dual-mode:joint— name remap + per-joint affine (lossless leader→follower mirror).No extra deps.
ee_pose— URDF forward kinematics (pinocchio) → 7-D EE pose + gripper, withoptional clutch to avoid jumps on engage.
so101_leader) that pushesJointStateOutput(syntheticbackend; the real Feetech serial read is a marked seam).
joint_space_device_example.py(live over OpenXR, or offline),sim-free unit tests for both retargeter modes + the source, and device +
retargeting reference docs.
Dependencies
pinocchio, now declared explicitly in the[retargeters]extra (pin>=4.0.0).jointmode needs nothing extra(pinocchio is imported lazily).
Test plan
main— clean.uvenv:uv pip install "isaacteleop[retargeters]"from thebuilt wheel → 24/24 unit tests pass (joint mode, EE mode incl. clutch,
source conversion).
so101_leaderplugin and consumesdata over the OpenXR runtime.
Summary by CodeRabbit
Release Notes
New Features
Documentation