DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
This research introduces DynaFLIP, a dynamics-aware multimodal pre-training framework designed to enhance robot manipulation by integrating motion understanding directly into the perception stage. The authors construct image-language-3D flow triplets from heterogeneous human and robot videos to train an image-only encoder. By minimizing simplex volume in a shared hyperspherical space alongside cosine regularization and contrastive objectives, DynaFLIP aligns modalities effectively while avoiding geometric ambiguity. Validated across diverse simulation and real-world setups, the framework achieves up to +22.5% performance gains under out-of-distribution scenarios compared to baselines, demonstrating that encoding how the world changes under action improves robot generalization.
The paper introduces DynaFLIP, a multimodal pre-training framework that integrates image, language, and 3D flow data to improve robot manipulation by encoding motion dynamics upstream in the perception stage.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
LAs. We validate this across diverse simulation and real-world setups, with gains reaching +22.5% under out-of-distribution scenarios. Our results suggest that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.
Open source S5
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Open source S1
arXiv:2605.30350v2 Announce Type: replace Abstract: Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recognition or vision-language alignment, leaving motion understanding to downstream policies. We introduce DynaFLIP, a dynamics-aware multimodal pre-training framework that
Open source S2
pushes motion understanding upstream into perception. We construct image-language-3D flow triplets from heterogeneous human and robot videos, and use these triplets as training-time supervision to shape an image-only encoder. Our key idea is to encourage the three modalities to span a small simplex volume in the shared hyperspherical space -- a smaller simplex volume indicating stronger alignment. To avoid the geome
Open source S3
tric ambiguity and trivial collapse of naive volume minimization, we combine simplex-volume minimization with a cosine regularizer and a contrastive objective. Our analyses show that DynaFLIP focuses on control-relevant regions critical for manipulation. The resulting dynamics-aware representations serve as reusable visual backbones and consistently outperform baselines across diverse downstream policies, including V
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org