Add MediaPipe Hand & Gesture Detector block (JdeRobot/VisualCircuit#470) - #17
Add MediaPipe Hand & Gesture Detector block (JdeRobot/VisualCircuit#470)#17prawnsgupta wants to merge 8 commits into
Conversation
|
@prawnsgupta Thank you for opening the PR. The video is very interesting. When I try to add your block using "File → Add as Block", I'm unable to add it. Did you try adding the block using the .vc3 file you shared? Also, was that .vc3 file generated using "File → Save Block"? |
|
The import failure has been root-caused: the previously shared The block has now been regenerated correctly:
The corrected file is pushed to this branch at UtilityThe HandGestureDetector block detects and recognizes hand gestures in real-time video using MediaPipe's hand landmark detection. It can output either an annotated image with hand landmarks drawn, or the raw landmark/gesture data as an array for downstream processing. This enables gesture-based control flows, hand pose estimation pipelines, and interactive gesture recognition applications. Expected Input and OutputInputs:
Outputs:
Parameters:
Supported Models: MediaPipe Hand Landmarker (detects up to 2 hands per frame with confidence scoring) A short screen recording of the block importing and in use is attached below: screen-recording-2026-07-24-101409_WAatBaQR.mp4Two runtime requirements remain outside the scope of the
Input on the preferred approach for these two items would be appreciated, and the follow-up will be pushed accordingly. |
|
Hello @prawnsgupta It is working now. Great work! Below is an image of it working. Regarding MediaPipe, you can either integrate it into the existing backend or, if you prefer, feel free to create PR for the required changes in the VisualCircuit backend and add the corresponding Note: It is entirely your choice whether to create a PR for adding the MediaPipe library and task file to the VisualCircuit backend and frontend. This change is somewhat time-consuming, as it requires a good understanding of VisualCircuit's data flow and backend architecture. If you would like to work on it, please let us know. Otherwise, we can handle those changes on our side. Before merging, there are two things that would be great to have:
Again, great job thinking outside the box! Using MediaPipe opens up many possibilities for future computer vision tasks. |
Review feedback from BkPankaj on PR JdeRobot#17: the block only handed back a picture, so nothing downstream could act on what the hand was doing, and it had no icon. OutputType now takes a third value, gesture, which shares a 2x5 array. Row 0 is the left hand and row 1 the right, columns are thumb, index, middle, ring and pinky, and a 1 means that finger is held up. A hand that is not in frame stays zeroed so the array keeps the same shape every frame and the wire does not get resized. Checked the 2D array survives the shared memory wire, read_array rebuilds it from the stored dim and shape so it comes back as 2x5. The image and landmarks outputs behave exactly as before. Also tightened the code to match the style of the other blocks, dropped the commentary that was not earning its place, and added a hand icon so the block is recognisable on the canvas.
|
Hey @prawnsgupta , Thank you for making the changes. I see that you added the Output Type parameter to the block. However, don't you think that if a user wants both the image output and the finger count, they would need to use two Gesture blocks? That would result in unnecessary duplicate processing. Instead, you could create two outputs—one for the image and another for the values. What do you think about this approach? |
|
Hey, @BkPankaj Agreed, and it is a good point to have raised before this went in. The parameter based The block has been restructured accordingly. It now exposes two output pins,
The change is pushed to this branch, and the block has been re-tested through Thanks for the thorough review on this one, the block is in better shape for it. If there are any further changes you'd like to see, do let me know. If everything looks good, I'd appreciate it if you could go ahead and merge the PR. |
|
@prawnsgupta Thank you for making the changes. I am encountering errors, and the camera stops working after they occur. I have attached screenshots of the error messages along with the circuit image for reference. 
|
The previous .vc3 was hand-authored from the legacy block-library JSON template and was missing the editor object that loadPackage() needs (editor.layers[0]/[1].models), so "Add as block" failed with "Cannot read properties of undefined (reading 'layers')". This file was produced by File > Save Block in the running app, then had one duplicate Out output block + duplicate wire/link entries removed (ticking "Code -> Out" a second time in the Save Block dialog creates a brand-new Output block+link rather than reusing the existing one). Verified: imports via Add as block with no error, and the resulting package block exposes exactly one Img/Enable input and one Out output.
Review feedback from BkPankaj on PR JdeRobot#17: the block only handed back a picture, so nothing downstream could act on what the hand was doing, and it had no icon. OutputType now takes a third value, gesture, which shares a 2x5 array. Row 0 is the left hand and row 1 the right, columns are thumb, index, middle, ring and pinky, and a 1 means that finger is held up. A hand that is not in frame stays zeroed so the array keeps the same shape every frame and the wire does not get resized. Checked the 2D array survives the shared memory wire, read_array rebuilds it from the stored dim and shape so it comes back as 2x5. The image and landmarks outputs behave exactly as before. Also tightened the code to match the style of the other blocks, dropped the commentary that was not earning its place, and added a hand icon so the block is recognisable on the canvas.
BkPankaj pointed out that switching image vs finger data through the OutputType parameter meant anyone who wanted both would need two copies of the block on the canvas, running MediaPipe twice on the same feed for no reason. The block now exposes two output pins, Image and Gesture, both filled in from the same detection pass, so OutputType is gone. Checked with a mocked detector that a single frame only calls detect() once and still produces both outputs, and that stays true across repeated frames. MirroredInput is unchanged.
The previous commit updated design.graph.blocks with the new Image and Gesture outputs but missed that the code node under editor.layers keeps its own copy of the same code/ports/params. That copy was never touched across the last two commits, so it still had the very first version of the code with the old OutputType parameter in it. Anyone double clicking into the block to look at it would have seen that instead of what actually runs. Copied code, ports and params from the design block over to its editor-layer twin so both agree. Nothing else in the file changed.
VisualCircuit builds each block's outputs dict from the wires it finds in the
circuit (backend/staticfiles/synthesis/main.py), so a pin the user leaves
unconnected is simply not there at runtime. This block wrote to Image and
Gesture unconditionally, so a circuit that only needed one of them died on the
first frame with
InvalidOutputNameException: Gesture is not declared in outputs
and took the video feed down with it. Camera -> HandGestureDetector ->
ScreenViewer, the most obvious way to try the block, hits this every time.
Work out once which pins are connected and only write to those. The drawing is
skipped too when nothing is reading the picture, since that frame would just be
thrown away.
While here, move synchronise() inside the loop. It was sitting after it, so the
block never yielded and spun the detector as fast as the CPU allowed instead of
at the circuit's frequency.
The block was written before the marketplace validator landed, so package category and tags were still empty and CI stopped at level 1 with "You must provide a Category before publishing". Category is one of the six the Edit Project Info dialog offers; Computer Vision is the fitting one here.
19bd771 to
5f9f1c2
Compare
2 participants
Implements the Hand & Gesture Detector block approved in JdeRobot/VisualCircuit#470 (thanks @BkPankaj for the go-ahead).
What it does: runs MediaPipe's Hand Landmarker on the input image, detects up to two hands (21 landmarks each), and estimates a simple gesture (extended-finger count). Fills the gap in VisualCircuit's CV blocks (FaceDetector/ObjectDetector exist, no hand/landmark block) and pairs with MotorDriver/Teleoperator for gesture-driven robotics.
Img)OutputType):image(annotated frame with 21-point skeleton + gesture label) orlandmarks(flat array of 21 (x,y) landmarks + finger count)OutputType(image/landmarks),MirroredInput(yes/no for pre-mirrored feeds)Implementation notes:
mediapipe.tasks.vision.HandLandmarker) — the legacymp.solutions.handsAPI was removed in 0.10.30+, so this is the durable choice.Needs a maintainer decision:
mediapipe(declared in the block'sdependencies).hand_landmarker.task(~7.6 MB) — should it live underutils/models/hand_landmarker/alongside the Haar/YOLO models?Demo video below.
Hand.Gesture.Demo.Video.mp4