When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
Simulation benchmarks penalize robot policies for failures often caused by physics engine artifacts. This work identifies a geometric threshold bug in collision detection that causes objects to be ejected unrealistically. The authors present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo, which flags episodes with such artifacts. Correcting the underlying engine parameter eliminates these artifacts, reducing false failures from 12% to 0% and increasing success rates on five container tasks from 89% to 98%. The authors recommend rerunning benchmarks on PhysX 5.3.1 or earlier with a repaired engine or reporting success rate ranges alongside artifact flags.
The authors identify a specific geometric threshold bug in physics engines (e.g., PhysX) that causes false failure rates in simulation benchmarks, propose a plugin to audit and flag these artifacts, and report that correcting the engine parameter reduces artifact-induced failures from 12% to 0% while raising success rates on five container tasks from 89% to 98%.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S2
Reported numbers
Reported success rate
89%
View original evidence
l below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate toge
Open source S5Reported success rate
98%
View original evidence
l below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate toge
Open source S5failure count
12
View original evidence
l below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate toge
Open source S5penetration distance
1–2
View original evidence
ne simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fal
Open source S4
- paper: not_reported
What remains unknown
Not established in the collected evidence: Control, Data origin.
Reported performance applies to the described task. It does not establish general autonomy or deployment readiness.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2610.41857v1 Announce Type: new Abstract: Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, a
Open source S2
l below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate toge
Open source S5
ther with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.
Open source S6
ne simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fal
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org