TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation

This research presents TactileLab, a scalable multimodal tactile simulation framework built on IsaacLab that introduces simulated tactile flow—an optical-flow-based representation—to efficiently estimate dense tangential motion at contact interfaces without relying on computationally expensive soft-body physics. By combining this flow field with geometric features from a contact depth field and a novel policy architecture for processing proprioceptive and tactile signals, the authors demonstrate efficient large-scale reinforcement learning for contact-rich tasks. The framework includes a real-to-sim tactile translation method to map real sensor streams to simulated fields, validated across diverse tactile sensor designs, enabling successful sim-to-real deployment on dual-arm and dexterous in-hand manipulation tasks.