Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketches
Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketches
This research paper introduces Sketch-MoMa, a teleoperation system that interprets hand-drawn sketches superimposed on observation images to control mobile manipulators. By leveraging Vision-Language Models (VLMs), the system infers drawn shapes and low-level tasks, enabling precise motion planning for operations like grasping and rotation. The approach is validated with seven tasks and five sketch shapes, showing competitive usability compared to existing 2D interfaces in a user study involving fourteen participants.
The authors validate Sketch-MoMa using state-of-the-art Vision-Language Models across seven tasks and five sketch shapes, demonstrating competitive usability in a user experiment with fourteen participants.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Control
- TeleoperatedOpen source S3
Reported numbers
tasks
7
View original evidence
he robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usabil
Open source S4sketch shapes
5
View original evidence
he robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usabil
Open source S4
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
modalities to set the sketches' semantics. This requires complex operations for users and leads to decreasing usability. In this paper, we propose Sketch-MoMa, a teleoperation system using the user-given hand-drawn sketches as instructions to control a robot. We use Vision-Language Models (VLMs) to understand the user-given sketches superimposed on an observation image and infer drawn shapes and low-level tasks of t
Open source S3
ity of our approach compared with the existing 2D interface through a user experiment with 14 participants.
Open source S5
he robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usabil
Open source S4
Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketches
Open source S1
Source：arXiv Robotics — research abstracts · arxiv.org