MolmoSpaces-CommonSense: Evaluating Commonsense in Robotic Manipulation Policies
MolmoSpaces-CommonSense: Evaluating Commonsense in Robotic Manipulation Policies
This paper introduces MolmoSpaces-CommonSense, a benchmark for evaluating commonsense reasoning in robotic manipulation policies. The authors address limitations in existing simulation benchmarks by incorporating tasks requiring reasoning about occlusion, containment, support relations, and hidden objects. Evaluations of several zero-shot generalist manipulation policies reveal consistent failures: success rates drop by over 50% as distractors increase, policies act on incorrect objects 30% of the time in cluttered settings, and affordance-sensitive grasping remains unreliable with only about 50% correct object part selection even with specific language prompts. The study suggests existing benchmarks may overestimate zero-shot real-world readiness.
The authors introduce MolmoSpaces-CommonSense, a new benchmark evaluating commonsense reasoning in robotic manipulation policies by testing occlusion, containment, and support relations; results show success rates drop over 50% in cluttered scenes and affordance-sensitive grasping remains unreliable at about 50% correct part selection.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
Reported success rate
>50%
Reported zero-shot result; the excerpt does not specify an evaluation protocol.
View original evidence
ffordances, and hidden objects. These evaluations require policies not only to recognize relevant scene structure but also to execute correct actions in them. We evaluate several zero-shot generalist manipulation policies. Across model types and architectures, we observe consistent failures in commonsense reasoning: success rates in cluttered scenes drop by over 50% as distractors increase, policies act on incorrect
Open source S4
- code: Reported pending Open source S6
- paper: not_reported
What remains unknown
Not established in the collected evidence: Environment, Control, Data origin.
Reported performance applies to the described task. It does not establish general autonomy or deployment readiness.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
ffordances, and hidden objects. These evaluations require policies not only to recognize relevant scene structure but also to execute correct actions in them. We evaluate several zero-shot generalist manipulation policies. Across model types and architectures, we observe consistent failures in commonsense reasoning: success rates in cluttered scenes drop by over 50% as distractors increase, policies act on incorrect
Open source S4
objects in cluttered settings 30% of the time, and affordance-sensitive grasping remains unreliable even with highly specific language prompts, with correct object part selection happening only about 50% of the time. Further, VLM-based success estimation struggles to reliably evaluate rollouts in these settings, suggesting that the challenge extends beyond control to the broader perception-reasoning-action stack. Our
Open source S5
results suggest that existing benchmarks may substantially overestimate zero-shot real-world readiness. Code and policy leaderboard link at https://github.com/allenai/molmospaces.
Open source S6
Source：arXiv Robotics — research abstracts · arxiv.org