VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
This research presents VT-MUSE, a framework for learning unified sequential representations from visual and tactile data to enhance robotic manipulation. It addresses limitations in existing methods by capturing fine-grained cross-modal dependencies and temporal evolution of contact through a two-stage process involving modality-specific encoders and a conditional variational latent model. The learned representation is integrated into a Transformer policy, achieving significant performance improvements over baselines on simulation benchmarks and demonstrating substantial gains in real-world experiments.
The paper introduces VT-MUSE, a two-stage representation learning framework that jointly adapts visual and tactile encoders via cross-modal temporal alignment to improve manipulation performance on both simulation benchmarks and real-world experiments.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
isual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves sub
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org