Benchmarking Real-World Robotic Manipulation through Stochastic Challenges

This research paper proposes a new benchmarking paradigm for real-world robotic manipulation called stochastic challenges. It addresses the bottleneck of reproducibility and comparability in interactive tasks by using reproducible real-world scenes combined with random task samples that mimic hidden test sets. The authors demonstrate this approach by comparing two different robots tested in two different labs on a novel mobile manipulation benchmark, showing how success rates over random samples allow for system-level innovation and fair comparison between platforms.