Skip to content

Add Manus glove haptic feedback via XR_NVX1 push-tensor#568

Open
rwiltz wants to merge 3 commits into
mainfrom
rwiltz/manus-glove-haptics
Open

Add Manus glove haptic feedback via XR_NVX1 push-tensor#568
rwiltz wants to merge 3 commits into
mainfrom
rwiltz/manus-glove-haptics

Conversation

@rwiltz

@rwiltz rwiltz commented May 28, 2026

Copy link
Copy Markdown
Contributor

Stacks on the OpenXR controller haptics PR. Adds Manus Metagloves
Pro Haptic as the first glove integration on top of a vendor-neutral
haptic-glove adapter, with a Teleop→plugin transport that actually works
across processes.

Why

The Manus plugin runs in its own process. The earlier revision of this
PR tried to ferry haptic commands over XR_NV_opaque_data_channel, but
that extension is client → runtime only — the runtime cannot fan
those bytes out to a peer process plugin. The right transport is the
Tensor API (XR_NVX1_push_tensor producer + XR_NVX1_tensor_data
consumer), which the runtime fans across OpenXR sessions on the same
systemId by collection_id. The existing Generic3AxisPedalTracker
already uses this transport in the opposite (plugin → Teleop) direction.

What

Generic / vendor-neutral surface

  • Schema (src/core/schema/fbs/haptic_glove.fbs): new
    HapticGloveCommand { is_left: bool; powers: [float]; } wire payload
    plus the standard Tracked / Record wrappers required by
    SchemaTracker<RecordT, DataTableT>. MCAP recording off in v1.
  • New tracker pair (mirrors Generic3AxisPedalTracker pattern):
    • HapticGloveCommandPusherTracker (producer, pybind-exposed) →
      LiveHapticGloveCommandPusherTrackerImpl wraps core::SchemaPusher.
    • HapticGloveCommandReaderTracker (consumer, C++-only) →
      LiveHapticGloveCommandReaderTrackerImpl wraps
      core::SchemaTracker<HapticGloveCommandRecord, HapticGloveCommand>.
    • Replay impls present so trackers work under ReplaySession.
    • Both wired into live_deviceio_factory + replay_deviceio_factory;
      DeviceIOSession auto-aggregates the required extensions.
  • Python adapter (src/haptic_devices/glove.py):
    HapticGloveDevice(collection_id, num_fingers, max_payload_size, sides)
    queues per-side FingerPowerVector payloads;
    HapticGloveSource(name, device) drains the queue inside
    poll_tracker(session), encodes via flatc-generated Python,

Summary by CodeRabbit

Release Notes

  • New Features

    • Added haptic glove feedback support enabling cross-process haptic command delivery
    • Added hand pinch proximity-based haptic retargeting for glove finger control
    • Integrated haptic feedback with Manus glove device
  • Documentation

    • Updated haptic feedback examples with new hand pinch proximity demonstration and improved quick-start instructions

@rwiltz rwiltz requested review from jiwenc-nv and shaosu-nvidia May 28, 2026 16:17
@coderabbitai

coderabbitai Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 701dab63-e260-4e1d-9f4d-630b6edcdee1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • ✅ Review completed - (🔄 Check again to review again)
📝 Walkthrough

Walkthrough

This PR adds comprehensive haptic device feedback support to Isaac Teleop. It introduces abstract tracker contracts for pushing/reading haptic commands, live and replay implementations for OpenXR controller haptics and Manus glove integration, FlatBuffer serialization for glove commands, vendor-neutral device adapters (controller and glove), tactile-to-haptic retargeting mappers with gain/deadband/saturation shaping and EMA smoothing, a HapticSink component for integrating devices into the retargeting graph, two end-to-end example pipelines (hand pinch proximity and trigger-driven controller haptics), comprehensive test coverage, and build wiring to package Python bindings and examples.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch rwiltz/manus-glove-haptics

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/haptic_feedback/python/openxr_controller_haptic_example.py`:
- Around line 214-219: The monitoring dict currently maps
"haptic_left"/"haptic_right" directly from pulse_selectors which stays
unswapped, causing printed haptic sides to be wrong when cross-hand routing is
active; update the construction of monitoring (the monitoring variable) so that
when cross-hand mode is enabled you swap the pulse_selectors used for
"haptic_left" and "haptic_right" (mirroring what is done for trigger_selectors)
so the monitored haptic side matches the actual routed sink inputs.

In `@src/core/schema/fbs/CMakeLists.txt`:
- Around line 49-53: The staging directory copy step (add_custom_target
schema_fb_python using ${CMAKE_COMMAND} -E copy_directory from
${SCHEMA_PYTHON_OUTPUT_DIR} to ${SCHEMA_FB_STAGING_DIR}) can leave stale files
because copy_directory only overlays; change the target to explicitly remove or
clean ${SCHEMA_FB_STAGING_DIR} first (e.g., call ${CMAKE_COMMAND} -E
remove_directory ${SCHEMA_FB_STAGING_DIR} or a remove and re-create sequence)
before creating the directory and copying, ensuring removed/renamed generated
modules are not left behind.

In `@src/haptic_devices/glove.py`:
- Around line 121-136: In HapticGloveDevice.__init__, add a fail-fast
compatibility check between num_fingers and max_payload_size: compute the
minimum required payload (e.g., at least one payload unit per finger or using an
existing MIN_PAYLOAD_PER_FINGER constant) and raise ValueError if
max_payload_size is too small to accommodate num_fingers; reference the
constructor parameters num_fingers and max_payload_size (and introduce a
MIN_PAYLOAD_PER_FINGER constant if none exists) so misconfigured
HapticGloveDevice instances fail immediately rather than later in push/send
methods.

In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp`:
- Around line 174-181: In ManusTracker::apply_haptic_command, sanitize each
element of the powers array before clamping to [0,1]: for each index used to
populate the existing clamped std::array<float, 5> (the one passed to
CoreSdk_VibrateFingersForGlove), check std::isfinite(powers[i]) and replace
non-finite values with 0.0f (or another safe default) before calling std::clamp,
then pass the sanitized clamped array to CoreSdk_VibrateFingersForGlove; also
add `#include` <cmath> to the file to ensure std::isfinite is available.

In `@src/retargeters/tactile_retargeters.py`:
- Around line 408-427: The constructor currently allows non-positive dimensions
which can lead to runtime failures; add guard checks in the __init__ to validate
that rows, cols, and num_pads are positive integers (e.g., if rows <= 0 or cols
<= 0 or num_pads <= 0: raise ValueError with a clear message referencing the
class/name), and keep the existing reduction validation; apply the same
validation logic to the analogous constructor/block around the code that sets
self._rows/self._cols/self._num_pads/_reduction (the similar section referenced
at lines 490–495) so invalid heatmap dimensions fail fast.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c56d40e6-0e17-4153-82e1-a711d5acdeb2

📥 Commits

Reviewing files that changed from the base of the PR and between 1f0ab7a and 8140af4.

📒 Files selected for processing (64)
  • CMakeLists.txt
  • cmake/GenerateFlatBuffers.cmake
  • examples/haptic_feedback/CMakeLists.txt
  • examples/haptic_feedback/python/README.md
  • examples/haptic_feedback/python/hand_pinch_haptic_example.py
  • examples/haptic_feedback/python/openxr_controller_haptic_example.py
  • examples/haptic_feedback/python/pyproject.toml
  • src/core/AGENTS.md
  • src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_glove_command_pusher_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_glove_command_reader_tracker_base.hpp
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_trackers/cpp/controller_tracker.cpp
  • src/core/deviceio_trackers/cpp/haptic_glove_command_pusher_tracker.cpp
  • src/core/deviceio_trackers/cpp/haptic_glove_command_reader_tracker.cpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_glove_command_pusher_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_glove_command_reader_tracker.hpp
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/core/deviceio_trackers/python/tracker_bindings.cpp
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/live_trackers/cpp/inc/live_trackers/live_deviceio_factory.hpp
  • src/core/live_trackers/cpp/live_controller_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_controller_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_deviceio_factory.cpp
  • src/core/live_trackers/cpp/live_haptic_glove_command_pusher_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_haptic_glove_command_pusher_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_haptic_glove_command_reader_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_haptic_glove_command_reader_tracker_impl.hpp
  • src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp
  • src/core/python/CMakeLists.txt
  • src/core/python/pyproject.toml.in
  • src/core/python/requirements.txt
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hpp
  • src/core/replay_trackers/cpp/replay_controller_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_deviceio_factory.cpp
  • src/core/replay_trackers/cpp/replay_haptic_glove_command_pusher_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_haptic_glove_command_pusher_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_haptic_glove_command_reader_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_haptic_glove_command_reader_tracker_impl.hpp
  • src/core/retargeting_engine/python/deviceio_source_nodes/__init__.py
  • src/core/retargeting_engine/python/deviceio_source_nodes/haptic_sink.py
  • src/core/retargeting_engine/python/tensor_types/__init__.py
  • src/core/retargeting_engine/python/tensor_types/indices.py
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py
  • src/core/retargeting_engine_tests/python/test_haptic_devices.py
  • src/core/retargeting_engine_tests/python/test_haptic_glove.py
  • src/core/retargeting_engine_tests/python/test_haptic_sink.py
  • src/core/retargeting_engine_tests/python/test_tactile_retargeters.py
  • src/core/schema/fbs/CMakeLists.txt
  • src/core/schema/fbs/haptic_glove.fbs
  • src/haptic_devices/CMakeLists.txt
  • src/haptic_devices/__init__.py
  • src/haptic_devices/glove.py
  • src/haptic_devices/interface.py
  • src/haptic_devices/openxr_controller.py
  • src/plugins/manus/CMakeLists.txt
  • src/plugins/manus/core/manus_hand_tracking_plugin.cpp
  • src/plugins/manus/inc/core/manus_glove_collection.hpp
  • src/plugins/manus/inc/core/manus_hand_tracking_plugin.hpp
  • src/plugins/manus/tests/CMakeLists.txt
  • src/plugins/manus/tests/python/test_manus_glove_collection_id.py
  • src/retargeters/tactile_retargeters.py

Comment thread examples/haptic_feedback/python/openxr_controller_haptic_example.py Outdated
Comment thread src/core/schema/fbs/CMakeLists.txt Outdated
Comment thread src/haptic_devices/glove.py Outdated
Comment thread src/plugins/manus/core/manus_hand_tracking_plugin.cpp
Comment thread src/retargeters/tactile_retargeters.py
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from 8140af4 to a9b3541 Compare May 28, 2026 17:16
Comment thread src/plugins/manus/inc/core/manus_hand_tracking_plugin.hpp Outdated
Comment thread src/plugins/manus/core/manus_hand_tracking_plugin.cpp Outdated
Comment thread src/core/replay_trackers/cpp/replay_haptic_glove_command_pusher_tracker_impl.cpp Outdated
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from a9b3541 to 5f22738 Compare June 3, 2026 21:17
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from 5f22738 to 8e7deb6 Compare June 3, 2026 21:59
@rwiltz rwiltz requested review from aristarkhovNV and nv-jakob June 3, 2026 22:00
rwiltz added a commit that referenced this pull request Jun 3, 2026
Glove-path review fixes (CodeRabbit):

- Manus consumer: std::clamp passes NaN / +/-Inf through unchanged, so
  apply_haptic_command now guards each finger power with std::isfinite
  before clamping. A non-finite power can no longer reach
  CoreSdk_VibrateFingersForGlove (add <cmath>).
- TactileHeatmapToFingerPower / TactileHeatmapToWristPulse: validate
  rows/cols/num_pads >= 1 in __init__ so a misconfigured heatmap fails
  fast at construction instead of hitting an empty reduction at run
  time. Add a regression test for each.

The remaining CodeRabbit items no longer apply after the generic
push-device redesign: the flatc-Python schema staging target and the
per-vendor HapticGloveDevice payload-size check are both gone (the glove
path uses a fixed generous payload default and logs oversize pushes
once), and the cross-hand controller-example note is controller scope.
rwiltz added a commit that referenced this pull request Jun 4, 2026
Glove-path review fixes (CodeRabbit):

- Manus consumer: std::clamp passes NaN / +/-Inf through unchanged, so
  apply_haptic_command now guards each finger power with std::isfinite
  before clamping. A non-finite power can no longer reach
  CoreSdk_VibrateFingersForGlove (add <cmath>).
- TactileHeatmapToFingerPower / TactileHeatmapToWristPulse: validate
  rows/cols/num_pads >= 1 in __init__ so a misconfigured heatmap fails
  fast at construction instead of hitting an empty reduction at run
  time. Add a regression test for each.

The remaining CodeRabbit items no longer apply after the generic
push-device redesign: the flatc-Python schema staging target and the
per-vendor HapticGloveDevice payload-size check are both gone (the glove
path uses a fixed generous payload default and logs oversize pushes
once), and the cross-hand controller-example note is controller scope.
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from 73a9cb6 to 6009ec2 Compare June 4, 2026 03:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (1)
src/plugins/manus/core/manus_hand_tracking_plugin.cpp (1)

251-258: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Keep the tensor reader optional.

Unconditionally pushing m_haptic_reader into trackers makes XR_NVX1_tensor_data a hard requirement for OpenXRSession. On runtimes without that extension, initialize() drops into the catch path and disables all OpenXR-based Manus behavior, not just haptics. Gate reader creation on extension support/capability so hand injection and controller wrist fallback still come up when haptics are unavailable.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp` around lines 251 -
258, The code unconditionally creates and pushes m_haptic_reader
(core::HapticCommandReaderTracker with MANUS_GLOVE_COLLECTION_ID) into trackers
which forces XR_NVX1_tensor_data as a required OpenXR extension; change to make
the tensor reader optional by first checking whether the runtime advertises the
NVX1 tensor data capability/extension and only then creating m_haptic_reader and
pushing it into trackers — ensure when the extension is absent you skip creation
(leave m_haptic_reader null) so initialize() and OpenXRSession still succeed and
other Manus features (hand injection, controller wrist fallback) remain
available.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp`:
- Around line 31-34: The TensorPushTracker::push implementation must validate
payload size against max_payload_size() before calling into the impl; add an
explicit check in TensorPushTracker::push that if payload.size() >
max_payload_size() you reject the call (e.g., throw a std::invalid_argument or a
defined API error) with a clear message that includes payload.size() and
max_payload_size(), and only call static_cast<const
ITensorPushTrackerImpl&>(session.get_tracker_impl(*this)).push(payload) when the
size is valid.

In `@src/core/retargeting_engine_tests/python/test_haptic_devices.py`:
- Around line 228-231: Remove assertions that look for literal b"left"/b"right"
in the serialized payload and instead decode each serialized HapticCommand from
recorder.pushes; for each decoded HapticCommand (use the test's HapticCommand
decoder), assert the side flag (e.g., is_left or side bit) and that the
power/strength fields for the corresponding actuators are present/within
expected ranges; apply the same replacement for the similar checks at the other
block (lines 271-277) so tests validate the semantic side bit and power values
rather than the old string-based wire format.

In `@src/core/retargeting_engine_tests/python/test_haptic_sink.py`:
- Around line 120-128: The test test_connect_rejects_mismatched_upstream_type is
too broad by catching Exception; update the assertion to only accept the
mismatch-related errors (e.g. TypeError or AssertionError) raised by the
compatibility check in HapticSink.connect when connecting a
ControllerHapticPulse()-accepting _RecordingDevice to a ValueInput("upstream",
TactileVector(5)) output. Replace pytest.raises(Exception) with
pytest.raises((TypeError, AssertionError)) so only the expected type-mismatch
failures from HapticSink.connect (or the underlying TensorType checks) will make
the test pass.

In `@src/haptic_devices/controller.py`:
- Around line 67-77: The constructor __init__ currently accepts arbitrary
endpoint names into _endpoints while flush() only handles "left" and "right",
causing silent no-ops for unsupported endpoints; validate the provided endpoints
in __init__ against the supported set {"left","right"} and raise a clear
ValueError if any unsupported endpoint is present (mention the invalid names in
the message), or alternatively normalize inputs to the supported set; update
_endpoints, _pending and _error_logged initialization to only include validated
endpoints and ensure any code that references _endpoints or flush() (e.g. flush)
will operate only on the validated values.

---

Duplicate comments:
In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp`:
- Around line 251-258: The code unconditionally creates and pushes
m_haptic_reader (core::HapticCommandReaderTracker with
MANUS_GLOVE_COLLECTION_ID) into trackers which forces XR_NVX1_tensor_data as a
required OpenXR extension; change to make the tensor reader optional by first
checking whether the runtime advertises the NVX1 tensor data
capability/extension and only then creating m_haptic_reader and pushing it into
trackers — ensure when the extension is absent you skip creation (leave
m_haptic_reader null) so initialize() and OpenXRSession still succeed and other
Manus features (hand injection, controller wrist fallback) remain available.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 95694869-c854-472a-b5ea-39ca95cb1fc9

📥 Commits

Reviewing files that changed from the base of the PR and between 8140af4 and 6009ec2.

📒 Files selected for processing (71)
  • AGENTS.md
  • CMakeLists.txt
  • docs/source/device/haptic_feedback.rst
  • docs/source/device/index.rst
  • examples/haptic_feedback/CMakeLists.txt
  • examples/haptic_feedback/python/README.md
  • examples/haptic_feedback/python/controller_haptic_example.py
  • examples/haptic_feedback/python/hand_pinch_haptic_example.py
  • examples/haptic_feedback/python/pyproject.toml
  • src/core/AGENTS.md
  • src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/tensor_push_tracker_base.hpp
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_trackers/cpp/controller_tracker.cpp
  • src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/tensor_push_tracker.hpp
  • src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/core/deviceio_trackers/python/tracker_bindings.cpp
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/live_trackers/cpp/inc/live_trackers/live_deviceio_factory.hpp
  • src/core/live_trackers/cpp/live_controller_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_controller_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_deviceio_factory.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.hpp
  • src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp
  • src/core/python/CMakeLists.txt
  • src/core/python/deviceio_init.py
  • src/core/python/pyproject.toml.in
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hpp
  • src/core/replay_trackers/cpp/replay_controller_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_deviceio_factory.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.hpp
  • src/core/retargeting_engine/python/deviceio_source_nodes/__init__.py
  • src/core/retargeting_engine/python/deviceio_source_nodes/haptic_sink.py
  • src/core/retargeting_engine/python/deviceio_source_nodes/sink_interface.py
  • src/core/retargeting_engine/python/interface/retargeter_subgraph.py
  • src/core/retargeting_engine/python/tensor_types/__init__.py
  • src/core/retargeting_engine/python/tensor_types/indices.py
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py
  • src/core/retargeting_engine_tests/python/test_haptic_devices.py
  • src/core/retargeting_engine_tests/python/test_haptic_sink.py
  • src/core/retargeting_engine_tests/python/test_tactile_retargeters.py
  • src/core/schema/fbs/haptic_command.fbs
  • src/core/schema/python/CMakeLists.txt
  • src/core/schema/python/haptic_command_bindings.h
  • src/core/schema/python/schema_init.py
  • src/core/schema/python/schema_module.cpp
  • src/core/teleop_session_manager/python/config.py
  • src/core/teleop_session_manager/python/teleop_session.py
  • src/core/teleop_session_manager_tests/python/test_teleop_session.py
  • src/haptic_devices/CMakeLists.txt
  • src/haptic_devices/__init__.py
  • src/haptic_devices/controller.py
  • src/haptic_devices/glove.py
  • src/haptic_devices/interface.py
  • src/haptic_devices/push_tensor.py
  • src/plugins/manus/core/inc/manus/manus_glove_collection.hpp
  • src/plugins/manus/core/inc/manus/manus_hand_tracking_plugin.hpp
  • src/plugins/manus/core/manus_hand_tracking_plugin.cpp
  • src/retargeters/tactile_retargeters.py
✅ Files skipped from review due to trivial changes (11)
  • examples/haptic_feedback/CMakeLists.txt
  • src/plugins/manus/core/inc/manus/manus_glove_collection.hpp
  • docs/source/device/index.rst
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp
  • src/core/schema/fbs/haptic_command.fbs
  • src/core/AGENTS.md
  • src/core/deviceio_base/cpp/inc/deviceio_base/tensor_push_tracker_base.hpp
  • AGENTS.md
  • examples/haptic_feedback/python/pyproject.toml
  • examples/haptic_feedback/python/README.md
🚧 Files skipped from review as they are similar to previous changes (15)
  • src/core/python/CMakeLists.txt
  • src/core/python/pyproject.toml.in
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/retargeting_engine/python/tensor_types/indices.py
  • src/core/live_trackers/cpp/live_controller_tracker_impl.hpp
  • src/haptic_devices/CMakeLists.txt
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • CMakeLists.txt
  • src/core/retargeting_engine/python/tensor_types/init.py
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hpp
  • src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp
  • src/core/live_trackers/cpp/live_controller_tracker_impl.cpp
  • src/retargeters/tactile_retargeters.py

Comment on lines +31 to +34
void TensorPushTracker::push(const ITrackerSession& session, const std::vector<uint8_t>& payload) const
{
static_cast<const ITensorPushTrackerImpl&>(session.get_tracker_impl(*this)).push(payload);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Reject oversized payloads before dispatch.

push() documents payload.size() <= max_payload_size() but never enforces it. That lets callers pass an invalid buffer into the impl layer, where the failure mode becomes truncation/assert/runtime-specific behavior instead of a predictable API error.

Proposed fix
 void TensorPushTracker::push(const ITrackerSession& session, const std::vector<uint8_t>& payload) const
 {
+    if (payload.size() > max_payload_size_)
+    {
+        throw std::invalid_argument("TensorPushTracker: payload size exceeds max_payload_size");
+    }
     static_cast<const ITensorPushTrackerImpl&>(session.get_tracker_impl(*this)).push(payload);
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp` around lines 31 - 34,
The TensorPushTracker::push implementation must validate payload size against
max_payload_size() before calling into the impl; add an explicit check in
TensorPushTracker::push that if payload.size() > max_payload_size() you reject
the call (e.g., throw a std::invalid_argument or a defined API error) with a
clear message that includes payload.size() and max_payload_size(), and only call
static_cast<const
ITensorPushTrackerImpl&>(session.get_tracker_impl(*this)).push(payload) when the
size is valid.

Comment thread src/core/retargeting_engine_tests/python/test_haptic_devices.py
Comment thread src/core/retargeting_engine_tests/python/test_haptic_sink.py
Comment thread src/haptic_devices/controller.py
rwiltz added a commit that referenced this pull request Jun 4, 2026
Glove-path review fixes (CodeRabbit):

- Manus consumer: std::clamp passes NaN / +/-Inf through unchanged, so
  apply_haptic_command now guards each finger power with std::isfinite
  before clamping. A non-finite power can no longer reach
  CoreSdk_VibrateFingersForGlove (add <cmath>).
- TactileHeatmapToFingerPower / TactileHeatmapToWristPulse: validate
  rows/cols/num_pads >= 1 in __init__ so a misconfigured heatmap fails
  fast at construction instead of hitting an empty reduction at run
  time. Add a regression test for each.

The remaining CodeRabbit items no longer apply after the generic
push-device redesign: the flatc-Python schema staging target and the
per-vendor HapticGloveDevice payload-size check are both gone (the glove
path uses a fixed generous payload default and logs oversize pushes
once), and the cross-hand controller-example note is controller scope.
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from 6009ec2 to 424cc3d Compare June 4, 2026 15:55
rwiltz added a commit that referenced this pull request Jun 4, 2026
Glove-path review fixes (CodeRabbit):

- Manus consumer: std::clamp passes NaN / +/-Inf through unchanged, so
  apply_haptic_command now guards each finger power with std::isfinite
  before clamping. A non-finite power can no longer reach
  CoreSdk_VibrateFingersForGlove (add <cmath>).
- TactileHeatmapToFingerPower / TactileHeatmapToWristPulse: validate
  rows/cols/num_pads >= 1 in __init__ so a misconfigured heatmap fails
  fast at construction instead of hitting an empty reduction at run
  time. Add a regression test for each.

The remaining CodeRabbit items no longer apply after the generic
push-device redesign: the flatc-Python schema staging target and the
per-vendor HapticGloveDevice payload-size check are both gone (the glove
path uses a fixed generous payload default and logs oversize pushes
once), and the cross-hand controller-example note is controller scope.
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from 424cc3d to 1420c89 Compare June 4, 2026 16:29
rwiltz added 3 commits June 4, 2026 18:03
Build the cross-process device-output path as a vendor-neutral layer so
any out-of-process haptic device (glove, exoskeleton) reuses one
transport and one wire schema instead of a per-vendor tracker and
FlatBuffer.

Core transport (C++):
- HapticCommand schema (endpoint: string, values: [float]).
- TensorPushTracker: generic producer that pushes opaque payloads over
  XR_NVX1_push_tensor; HapticCommandReaderTracker: the paired consumer.
- Base/facade/live/replay impls, factory wiring, and pybind bindings,
  plus a pack_haptic_command() encoder on the schema module.

Python device + retargeting:
- PushTensorHapticDevice (IHapticDevice) serializes per-endpoint values
  into a HapticCommand and pushes them; haptic_glove_device() binds it to
  FingerPowerVector.
- FingerPowerVector / FingerIndex tensor types and the FingerPower tactile
  retargeters (TactileVector/Heatmap -> FingerPower, FingerPower -> pulse).

Reference integrations:
- Manus plugin consumes HapticCommand via HapticCommandReaderTracker and
  drives CoreSdk_VibrateFingersForGlove.
- hand_pinch_haptic_example.py: pinch proximity -> per-finger vibration.

OpenXR vocabulary stays in the live/runtime layer; device adapters,
schemas, and retargeters remain vendor-neutral.
Production review pass over the glove device-output path:

- Run clang-format on the push/reader trackers, factories, and the Manus
  plugin. CI runs clang-format-14 and rejects unformatted C++ (pre-commit
  does not check it), so these files needed formatting.
- Replace the Manus literal finger count with a kManusFingerCount constant
  so the actuator count reads as a parameter a different glove would change.
- Log the replay-mode push drop once instead of every frame.
- Drop version-referencing comment framing (no "v1" labels; describe the
  current behavior directly).
Glove-path review fixes (CodeRabbit):

- Manus consumer: std::clamp passes NaN / +/-Inf through unchanged, so
  apply_haptic_command now guards each finger power with std::isfinite
  before clamping. A non-finite power can no longer reach
  CoreSdk_VibrateFingersForGlove (add <cmath>).
- TactileHeatmapToFingerPower / TactileHeatmapToWristPulse: validate
  rows/cols/num_pads >= 1 in __init__ so a misconfigured heatmap fails
  fast at construction instead of hitting an empty reduction at run
  time. Add a regression test for each.

The remaining CodeRabbit items no longer apply after the generic
push-device redesign: the flatc-Python schema staging target and the
per-vendor HapticGloveDevice payload-size check are both gone (the glove
path uses a fixed generous payload default and logs oversize pushes
once), and the cross-hand controller-example note is controller scope.
@rwiltz rwiltz force-pushed the rwiltz/manus-glove-haptics branch from 1420c89 to af8ac30 Compare June 4, 2026 22:05

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (2)
src/plugins/manus/core/manus_hand_tracking_plugin.cpp (1)

251-259: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make haptic reader registration conditional to avoid hard-failing OpenXR on runtimes without tensor-data support.

Registering m_haptic_reader unconditionally turns haptics into a mandatory extension dependency for session creation. That can disable otherwise-working hand/controller injection on runtimes that don’t advertise the tensor-data extension.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp` around lines 251 -
259, The haptic reader (m_haptic_reader created with
core::HapticCommandReaderTracker and pushed into trackers) is registered
unconditionally which makes XR_NVX1_tensor_data a hard session dependency;
change this to only create and register m_haptic_reader if the runtime
advertises tensor-data support (check the session/instance extension list for
"XR_NVX1_tensor_data" or use the project's extension-query helper), otherwise
leave m_haptic_reader null and do not push it into trackers; also update any
later code that dereferences m_haptic_reader to guard against null before use
and emit a debug/log message when haptics are skipped.
src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp (1)

31-34: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Enforce payload-size bounds before forwarding to the impl.

Line 31 accepts arbitrary payloads, but max_payload_size_ is never enforced before dispatch on Line 33. Oversized payloads should be rejected here to keep API behavior deterministic.

Suggested fix
 void TensorPushTracker::push(const ITrackerSession& session, const std::vector<uint8_t>& payload) const
 {
+    if (payload.size() > max_payload_size_)
+    {
+        throw std::invalid_argument("TensorPushTracker: payload size exceeds max_payload_size");
+    }
     static_cast<const ITensorPushTrackerImpl&>(session.get_tracker_impl(*this)).push(payload);
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp` around lines 31 - 34,
TensorPushTracker::push currently forwards any payload without checking
max_payload_size_; add a bounds check at the start of TensorPushTracker::push
that compares payload.size() against the member max_payload_size_ and rejects
oversized payloads (e.g., by throwing a clear exception such as
std::length_error or std::invalid_argument with a message including
payload.size() and max_payload_size_). If within bounds, continue to call
session.get_tracker_impl(*this).push(payload) as before; ensure the check is
performed before the static_cast/forward so the impl never receives an oversized
buffer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/retargeters/tactile_retargeters.py`:
- Around line 669-685: TactileVectorToControllerPulse.__init__ accepts
num_taxels with no validation which can cause invalid shapes later; add an
explicit check in __init__ that num_taxels is an int >= 1 (and optionally raise
TypeError if not int) and raise a clear ValueError for bad values; update any
related attributes (self._num_taxels) only after validation so
input_spec/reshape logic uses a validated value (refer to
TactileVectorToControllerPulse.__init__, self._num_taxels, input_spec and
reshape paths).
- Around line 789-809: TactileHeatmapToControllerPulse.__init__ currently allows
non-positive rows, cols, and num_pads; add early validation in the constructor
(TactileHeatmapToControllerPulse.__init__) to raise ValueError if rows <= 0,
cols <= 0, or num_pads <= 0 with a clear message including the parameter name
and the provided value so invalid heatmap dimensions are rejected immediately;
keep the existing reduction validation and then assign self._rows, self._cols,
and self._num_pads only after these checks.

---

Duplicate comments:
In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp`:
- Around line 31-34: TensorPushTracker::push currently forwards any payload
without checking max_payload_size_; add a bounds check at the start of
TensorPushTracker::push that compares payload.size() against the member
max_payload_size_ and rejects oversized payloads (e.g., by throwing a clear
exception such as std::length_error or std::invalid_argument with a message
including payload.size() and max_payload_size_). If within bounds, continue to
call session.get_tracker_impl(*this).push(payload) as before; ensure the check
is performed before the static_cast/forward so the impl never receives an
oversized buffer.

In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp`:
- Around line 251-259: The haptic reader (m_haptic_reader created with
core::HapticCommandReaderTracker and pushed into trackers) is registered
unconditionally which makes XR_NVX1_tensor_data a hard session dependency;
change this to only create and register m_haptic_reader if the runtime
advertises tensor-data support (check the session/instance extension list for
"XR_NVX1_tensor_data" or use the project's extension-query helper), otherwise
leave m_haptic_reader null and do not push it into trackers; also update any
later code that dereferences m_haptic_reader to guard against null before use
and emit a debug/log message when haptics are skipped.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 56096cf1-c154-4d5d-ad7e-95cddbae375a

📥 Commits

Reviewing files that changed from the base of the PR and between 8140af4 and af8ac30.

📒 Files selected for processing (42)
  • examples/haptic_feedback/python/README.md
  • examples/haptic_feedback/python/hand_pinch_haptic_example.py
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/tensor_push_tracker_base.hpp
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/tensor_push_tracker.hpp
  • src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/core/deviceio_trackers/python/tracker_bindings.cpp
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/live_trackers/cpp/inc/live_trackers/live_deviceio_factory.hpp
  • src/core/live_trackers/cpp/live_deviceio_factory.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.hpp
  • src/core/python/deviceio_init.py
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hpp
  • src/core/replay_trackers/cpp/replay_deviceio_factory.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.hpp
  • src/core/retargeting_engine/python/tensor_types/__init__.py
  • src/core/retargeting_engine/python/tensor_types/indices.py
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py
  • src/core/retargeting_engine_tests/python/test_haptic_devices.py
  • src/core/retargeting_engine_tests/python/test_tactile_retargeters.py
  • src/core/schema/fbs/haptic_command.fbs
  • src/core/schema/python/CMakeLists.txt
  • src/core/schema/python/haptic_command_bindings.h
  • src/core/schema/python/schema_init.py
  • src/core/schema/python/schema_module.cpp
  • src/haptic_devices/glove.py
  • src/haptic_devices/push_tensor.py
  • src/plugins/manus/core/inc/manus/manus_glove_collection.hpp
  • src/plugins/manus/core/inc/manus/manus_hand_tracking_plugin.hpp
  • src/plugins/manus/core/manus_hand_tracking_plugin.cpp
  • src/retargeters/tactile_retargeters.py
✅ Files skipped from review due to trivial changes (5)
  • src/core/schema/python/CMakeLists.txt
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/haptic_devices/glove.py
  • examples/haptic_feedback/python/README.md
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/retargeting_engine/python/tensor_types/init.py

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 2

♻️ Duplicate comments (2)
src/plugins/manus/core/manus_hand_tracking_plugin.cpp (1)

251-259: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make haptic reader registration conditional to avoid hard-failing OpenXR on runtimes without tensor-data support.

Registering m_haptic_reader unconditionally turns haptics into a mandatory extension dependency for session creation. That can disable otherwise-working hand/controller injection on runtimes that don’t advertise the tensor-data extension.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp` around lines 251 -
259, The haptic reader (m_haptic_reader created with
core::HapticCommandReaderTracker and pushed into trackers) is registered
unconditionally which makes XR_NVX1_tensor_data a hard session dependency;
change this to only create and register m_haptic_reader if the runtime
advertises tensor-data support (check the session/instance extension list for
"XR_NVX1_tensor_data" or use the project's extension-query helper), otherwise
leave m_haptic_reader null and do not push it into trackers; also update any
later code that dereferences m_haptic_reader to guard against null before use
and emit a debug/log message when haptics are skipped.
src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp (1)

31-34: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Enforce payload-size bounds before forwarding to the impl.

Line 31 accepts arbitrary payloads, but max_payload_size_ is never enforced before dispatch on Line 33. Oversized payloads should be rejected here to keep API behavior deterministic.

Suggested fix
 void TensorPushTracker::push(const ITrackerSession& session, const std::vector<uint8_t>& payload) const
 {
+    if (payload.size() > max_payload_size_)
+    {
+        throw std::invalid_argument("TensorPushTracker: payload size exceeds max_payload_size");
+    }
     static_cast<const ITensorPushTrackerImpl&>(session.get_tracker_impl(*this)).push(payload);
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp` around lines 31 - 34,
TensorPushTracker::push currently forwards any payload without checking
max_payload_size_; add a bounds check at the start of TensorPushTracker::push
that compares payload.size() against the member max_payload_size_ and rejects
oversized payloads (e.g., by throwing a clear exception such as
std::length_error or std::invalid_argument with a message including
payload.size() and max_payload_size_). If within bounds, continue to call
session.get_tracker_impl(*this).push(payload) as before; ensure the check is
performed before the static_cast/forward so the impl never receives an oversized
buffer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/retargeters/tactile_retargeters.py`:
- Around line 669-685: TactileVectorToControllerPulse.__init__ accepts
num_taxels with no validation which can cause invalid shapes later; add an
explicit check in __init__ that num_taxels is an int >= 1 (and optionally raise
TypeError if not int) and raise a clear ValueError for bad values; update any
related attributes (self._num_taxels) only after validation so
input_spec/reshape logic uses a validated value (refer to
TactileVectorToControllerPulse.__init__, self._num_taxels, input_spec and
reshape paths).
- Around line 789-809: TactileHeatmapToControllerPulse.__init__ currently allows
non-positive rows, cols, and num_pads; add early validation in the constructor
(TactileHeatmapToControllerPulse.__init__) to raise ValueError if rows <= 0,
cols <= 0, or num_pads <= 0 with a clear message including the parameter name
and the provided value so invalid heatmap dimensions are rejected immediately;
keep the existing reduction validation and then assign self._rows, self._cols,
and self._num_pads only after these checks.

---

Duplicate comments:
In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp`:
- Around line 31-34: TensorPushTracker::push currently forwards any payload
without checking max_payload_size_; add a bounds check at the start of
TensorPushTracker::push that compares payload.size() against the member
max_payload_size_ and rejects oversized payloads (e.g., by throwing a clear
exception such as std::length_error or std::invalid_argument with a message
including payload.size() and max_payload_size_). If within bounds, continue to
call session.get_tracker_impl(*this).push(payload) as before; ensure the check
is performed before the static_cast/forward so the impl never receives an
oversized buffer.

In `@src/plugins/manus/core/manus_hand_tracking_plugin.cpp`:
- Around line 251-259: The haptic reader (m_haptic_reader created with
core::HapticCommandReaderTracker and pushed into trackers) is registered
unconditionally which makes XR_NVX1_tensor_data a hard session dependency;
change this to only create and register m_haptic_reader if the runtime
advertises tensor-data support (check the session/instance extension list for
"XR_NVX1_tensor_data" or use the project's extension-query helper), otherwise
leave m_haptic_reader null and do not push it into trackers; also update any
later code that dereferences m_haptic_reader to guard against null before use
and emit a debug/log message when haptics are skipped.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 56096cf1-c154-4d5d-ad7e-95cddbae375a

📥 Commits

Reviewing files that changed from the base of the PR and between 8140af4 and af8ac30.

📒 Files selected for processing (42)
  • examples/haptic_feedback/python/README.md
  • examples/haptic_feedback/python/hand_pinch_haptic_example.py
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/tensor_push_tracker_base.hpp
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/tensor_push_tracker.hpp
  • src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/core/deviceio_trackers/python/tracker_bindings.cpp
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/live_trackers/cpp/inc/live_trackers/live_deviceio_factory.hpp
  • src/core/live_trackers/cpp/live_deviceio_factory.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.hpp
  • src/core/python/deviceio_init.py
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hpp
  • src/core/replay_trackers/cpp/replay_deviceio_factory.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.hpp
  • src/core/retargeting_engine/python/tensor_types/__init__.py
  • src/core/retargeting_engine/python/tensor_types/indices.py
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py
  • src/core/retargeting_engine_tests/python/test_haptic_devices.py
  • src/core/retargeting_engine_tests/python/test_tactile_retargeters.py
  • src/core/schema/fbs/haptic_command.fbs
  • src/core/schema/python/CMakeLists.txt
  • src/core/schema/python/haptic_command_bindings.h
  • src/core/schema/python/schema_init.py
  • src/core/schema/python/schema_module.cpp
  • src/haptic_devices/glove.py
  • src/haptic_devices/push_tensor.py
  • src/plugins/manus/core/inc/manus/manus_glove_collection.hpp
  • src/plugins/manus/core/inc/manus/manus_hand_tracking_plugin.hpp
  • src/plugins/manus/core/manus_hand_tracking_plugin.cpp
  • src/retargeters/tactile_retargeters.py
✅ Files skipped from review due to trivial changes (5)
  • src/core/schema/python/CMakeLists.txt
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/haptic_devices/glove.py
  • examples/haptic_feedback/python/README.md
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/retargeting_engine/python/tensor_types/init.py
🛑 Comments failed to post (2)
src/retargeters/tactile_retargeters.py (2)

669-685: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Validate num_taxels in TactileVectorToControllerPulse.__init__.

num_taxels is accepted without a lower-bound check, so invalid values can slip into input_spec and reshape paths instead of failing fast.

Proposed fix
     def __init__(
         self,
         name: str,
         num_taxels: int,
         reduction: Literal["max", "mean", "sum"] = "max",
         gain: float = 1.0,
         deadband: float = 0.0,
         saturation: float = 1.0,
         frequency_hz: float = 0.0,
         duration_s: float = 0.0,
     ) -> None:
+        if num_taxels < 1:
+            raise ValueError(
+                f"TactileVectorToControllerPulse '{name}' requires num_taxels >= 1, got {num_taxels}"
+            )
         if reduction not in ("max", "mean", "sum"):
             raise ValueError(
                 f"TactileVectorToControllerPulse '{name}': unknown reduction '{reduction}'"
             )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def __init__(
        self,
        name: str,
        num_taxels: int,
        reduction: Literal["max", "mean", "sum"] = "max",
        gain: float = 1.0,
        deadband: float = 0.0,
        saturation: float = 1.0,
        frequency_hz: float = 0.0,
        duration_s: float = 0.0,
    ) -> None:
        if num_taxels < 1:
            raise ValueError(
                f"TactileVectorToControllerPulse '{name}' requires num_taxels >= 1, got {num_taxels}"
            )
        if reduction not in ("max", "mean", "sum"):
            raise ValueError(
                f"TactileVectorToControllerPulse '{name}': unknown reduction '{reduction}'"
            )
        self._num_taxels = num_taxels
        self._reduction = reduction
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/retargeters/tactile_retargeters.py` around lines 669 - 685,
TactileVectorToControllerPulse.__init__ accepts num_taxels with no validation
which can cause invalid shapes later; add an explicit check in __init__ that
num_taxels is an int >= 1 (and optionally raise TypeError if not int) and raise
a clear ValueError for bad values; update any related attributes
(self._num_taxels) only after validation so input_spec/reshape logic uses a
validated value (refer to TactileVectorToControllerPulse.__init__,
self._num_taxels, input_spec and reshape paths).

789-809: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fail fast on non-positive heatmap dimensions in TactileHeatmapToControllerPulse.__init__.

This constructor currently accepts invalid rows, cols, and num_pads, unlike the other heatmap mappers. That can defer bad configuration into runtime behavior instead of rejecting early.

Proposed fix
     def __init__(
         self,
         name: str,
         rows: int,
         cols: int,
         num_pads: int = 1,
         reduction: Literal["max", "mean", "sum"] = "max",
         gain: float = 1.0,
         deadband: float = 0.0,
         saturation: float = 1.0,
         frequency_hz: float = 0.0,
         duration_s: float = 0.0,
     ) -> None:
+        if rows < 1 or cols < 1 or num_pads < 1:
+            raise ValueError(
+                f"TactileHeatmapToControllerPulse '{name}' requires rows/cols/num_pads >= 1, "
+                f"got rows={rows}, cols={cols}, num_pads={num_pads}"
+            )
         if reduction not in ("max", "mean", "sum"):
             raise ValueError(
                 f"TactileHeatmapToControllerPulse '{name}': unknown reduction '{reduction}'"
             )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/retargeters/tactile_retargeters.py` around lines 789 - 809,
TactileHeatmapToControllerPulse.__init__ currently allows non-positive rows,
cols, and num_pads; add early validation in the constructor
(TactileHeatmapToControllerPulse.__init__) to raise ValueError if rows <= 0,
cols <= 0, or num_pads <= 0 with a clear message including the parameter name
and the provided value so invalid heatmap dimensions are rejected immediately;
keep the existing reduction validation and then assign self._rows, self._cols,
and self._num_pads only after these checks.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp (1)

31-34: ⚠️ Potential issue | 🔴 Critical

Payload size validation still missing.

The documented precondition payload.size() <= max_payload_size() is not enforced before forwarding to the impl layer. This was flagged in a previous review and remains unaddressed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp` around lines 31 - 34,
TensorPushTracker::push currently forwards the payload to the impl without
checking the documented precondition; add a size check before calling into
ITensorPushTrackerImpl::push: if payload.size() > max_payload_size() then handle
the error (e.g., throw std::length_error or assert/log and return) to prevent
oversized payloads from reaching session.get_tracker_impl(*this).push(payload).
Ensure you reference TensorPushTracker::push, max_payload_size(),
ITrackerSession, and ITensorPushTrackerImpl::push when implementing the check so
the validation sits in the public-facing push wrapper.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp`:
- Around line 31-34: TensorPushTracker::push currently forwards the payload to
the impl without checking the documented precondition; add a size check before
calling into ITensorPushTrackerImpl::push: if payload.size() >
max_payload_size() then handle the error (e.g., throw std::length_error or
assert/log and return) to prevent oversized payloads from reaching
session.get_tracker_impl(*this).push(payload). Ensure you reference
TensorPushTracker::push, max_payload_size(), ITrackerSession, and
ITensorPushTrackerImpl::push when implementing the check so the validation sits
in the public-facing push wrapper.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 674df2b7-6c6b-4b1a-b19b-0b1691066433

📥 Commits

Reviewing files that changed from the base of the PR and between 8140af4 and af8ac30.

📒 Files selected for processing (42)
  • examples/haptic_feedback/python/README.md
  • examples/haptic_feedback/python/hand_pinch_haptic_example.py
  • src/core/deviceio_base/cpp/inc/deviceio_base/haptic_command_reader_tracker_base.hpp
  • src/core/deviceio_base/cpp/inc/deviceio_base/tensor_push_tracker_base.hpp
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_trackers/cpp/haptic_command_reader_tracker.cpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/haptic_command_reader_tracker.hpp
  • src/core/deviceio_trackers/cpp/inc/deviceio_trackers/tensor_push_tracker.hpp
  • src/core/deviceio_trackers/cpp/tensor_push_tracker.cpp
  • src/core/deviceio_trackers/python/deviceio_trackers_init.py
  • src/core/deviceio_trackers/python/tracker_bindings.cpp
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/live_trackers/cpp/inc/live_trackers/live_deviceio_factory.hpp
  • src/core/live_trackers/cpp/live_deviceio_factory.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_haptic_command_reader_tracker_impl.hpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.cpp
  • src/core/live_trackers/cpp/live_tensor_push_tracker_impl.hpp
  • src/core/python/deviceio_init.py
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hpp
  • src/core/replay_trackers/cpp/replay_deviceio_factory.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_haptic_command_reader_tracker_impl.hpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.cpp
  • src/core/replay_trackers/cpp/replay_tensor_push_tracker_impl.hpp
  • src/core/retargeting_engine/python/tensor_types/__init__.py
  • src/core/retargeting_engine/python/tensor_types/indices.py
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py
  • src/core/retargeting_engine_tests/python/test_haptic_devices.py
  • src/core/retargeting_engine_tests/python/test_tactile_retargeters.py
  • src/core/schema/fbs/haptic_command.fbs
  • src/core/schema/python/CMakeLists.txt
  • src/core/schema/python/haptic_command_bindings.h
  • src/core/schema/python/schema_init.py
  • src/core/schema/python/schema_module.cpp
  • src/haptic_devices/glove.py
  • src/haptic_devices/push_tensor.py
  • src/plugins/manus/core/inc/manus/manus_glove_collection.hpp
  • src/plugins/manus/core/inc/manus/manus_hand_tracking_plugin.hpp
  • src/plugins/manus/core/manus_hand_tracking_plugin.cpp
  • src/retargeters/tactile_retargeters.py
✅ Files skipped from review due to trivial changes (1)
  • src/plugins/manus/core/inc/manus/manus_glove_collection.hpp
🚧 Files skipped from review as they are similar to previous changes (6)
  • src/core/replay_trackers/cpp/CMakeLists.txt
  • src/core/replay_trackers/cpp/inc/replay_trackers/replay_deviceio_factory.hpp
  • src/core/live_trackers/cpp/CMakeLists.txt
  • src/core/deviceio_trackers/cpp/CMakeLists.txt
  • src/core/retargeting_engine/python/tensor_types/init.py
  • src/core/retargeting_engine/python/tensor_types/tactile_types.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants