DreamX: Video Diffusion Model is a Universal Latent Action Learner
DreamX: Video Diffusion Model is a Universal Latent Action Learner
This research presents DreamX, a framework that utilizes a pretrained video diffusion model to extract compact latent actions from video data. By compressing latent video frames by 300x and leveraging the model's generative prior, DreamX disentangles action dynamics from visual appearance. The resulting shared latent space facilitates any-to-any video transfer and zero-shot action transfer across diverse embodiments (human and robot) and environments. On the LARY-Bench benchmark, DreamX achieves state-of-the-art performance among latent action models. Additionally, the method enables policy learning in a compact latent action space, reducing computational costs for video-diffusion-based policies.
The paper introduces DreamX, a method that repurposes a pretrained video diffusion model to learn universal latent actions disentangled from visual appearance, achieving state-of-the-art performance on LARY-Bench and enabling zero-shot transfer across embodiments.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
latent video frames compressed to tokens
4
View original evidence
limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior,
Open source S3compression ratio for latent video frames
300
View original evidence
limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior,
Open source S3latent action tokens per frame
4
View original evidence
limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior,
Open source S3compressed latent video frame token count
4
View original evidence
limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior,
Open source S3compression factor for latent actions
300
View original evidence
limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior,
Open source S3
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior,
Open source S3
these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org