Physical Diffusion Policy via MPM-Guided 3D Gaussian Simulation

This paper introduces a physics-aware diffusion policy that augments visual imitation learning with object physical understanding. The framework recovers material properties from visual demonstrations by coupling 3D Gaussian Splatting with differentiable Material Point Method simulation, enabling accurate physical parameter estimation. These estimated physics conditions are integrated into a material-aware Mixture-of-Experts diffusion policy for adaptive behavior generation across different physical regimes. A physics memory bank is introduced for efficient retrieval without online simulation optimization. Experiments on simulated and real-world objects demonstrate improved manipulation success and generalization over existing diffusion-based approaches, highlighting the critical role of incorporating physical knowledge for scalable robot manipulation.