Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation

This paper introduces 3D World Simulator, an action-conditioned multimodal world modeling framework that uses a hybrid particle-voxel autoencoder to compress point clouds into compact 3D voxel latents. The system jointly predicts appearance, geometry, and tactile observations from a unified 3D representation, enabling geometrically consistent rendering from arbitrary viewpoints and integration of spatially aligned tactile signals. Experiments demonstrate interactive, long-horizon prediction at 50 Hz using wrist-mounted and novel camera views, scalable data generation for policy training and evaluation, and model-based planning with unseen objects and robot configurations on large datasets containing diverse objects.