MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning
MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning
This research introduces MDF++, a visuo-tactile world-action model designed to learn physically grounded, control-relevant representations for robot manipulation. Unlike existing world-action models that rely on frozen pretrained encoders, MDF++ employs an end-to-end learning strategy with a diffusion backbone and a multi-task learning approach to leverage cross-modal correlations. The authors evaluate the model on contact-rich tasks such as real-world bolt screwing and nut mating, reporting performance improvements of up to 50% compared to models built on frozen representations.
The paper proposes MDF++, a visuo-tactile world-action model that uses end-to-end representation learning with a diffusion backbone to avoid feature collapse associated with naive joint training, achieving up to 50% improvement over models using frozen pretrained encoders on contact-rich tasks like bolt screwing and nut mating.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
percent improvement over frozen pretrained WAMs
≤50
View original evidence
correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.
Open source S4
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.
Open source S4
ed for robot control. In this paper, we tackle the problem of learning physicallygrounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal
Open source S3
Source：arXiv Robotics — research abstracts · arxiv.org