Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling

This research presents GenSight, a unified embodied world model that integrates future video generation with dense perception of geometric, semantic, and motion structures into a single pretrained video diffusion transformer. By using a shared DiT backbone and lightweight task modulation, GenSight predicts all modalities within the same architecture and supports both multi-step future video generation with aligned dense predictions and single-step dense perception from observed videos. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both tasks, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.