PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation
PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation
This paper presents PolyUMI, an open-source platform for scalable visual-tactile-audio demonstration collection and robot deployment. It features a lightweight, wireless handheld gripper that records synchronized sensory data. The paper also introduces VisTA, a token-level multimodal policy that integrates information across sensors and time to predict contact-aware robot actions. Experiments show that tactile and audio signals provide valuable task-relevant information beyond vision, and VisTA is competitive with or outperforms existing multimodal policies.
The paper introduces PolyUMI, an open-source platform for collecting visual-tactile-audio data for robot manipulation, and VisTA, a multimodal policy that integrates heterogeneous sensory signals. The work highlights the importance of tactile and audio information in tasks like object inference and slip control, showing that these modalities provide task-relevant data beyond vision.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation
Open source S1
rioception, limiting access to contact information that is difficult to infer visually. We present PolyUMI, an open-source platform for scalable visual--tactile--audio demonstration collection and robot deployment. Its lightweight, wireless handheld gripper records synchronized wrist-camera, optical tactile, contact-audio, and proprioceptive observations without requiring a tethered workstation. The same sensing fing
Open source S3
er can be transferred to the robot end effector, preserving the sensing geometry between demonstration collection and policy execution. To effectively use these heterogeneous observations, we further introduce VisTA, a token-level multimodal policy that integrates information across sensors and time to predict contact-aware robot actions. Experiments spanning object inference, slip control, and contact-rich manipulat
Open source S4
ion show that touch and audio reveal task-relevant information beyond vision and that VisTA is competitive with or outperforms existing multimodal policies. Together, PolyUMI and VisTA provide an accessible pipeline for collecting multimodal demonstrations and learning policies that perceive physical interaction beyond vision. Project Page: https://polyumi-vista.github.io
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org