Robot-Factored World Models via Robot Rendering

This paper introduces Robot-Factored World Models (RoFacto), which decouple action realization and robot rendering from scene-response prediction. By rolling command sequences through the robot's own controller and kinematics to generate nominal trajectories, and rendering these trajectories using the robot URDF, RoFacto factors out geometry, kinematics, and appearance into explicit rendered geometry. The model uses paired end-effector and scene depth for geometric cues regarding contact and occlusion, while combining camera-aware static RGB context with rendered robot geometry to form a shared visual interface across viewpoints and embodiments. Experiments demonstrate that this rendered interface outperforms vector-conditioned baselines, supports transfer to unseen robot embodiments, and enables the generation of robot manipulation videos from retargeted human demonstrations.