MESA: An Evaluation Framework for Compositional, Semantic, and Spatial Generalization in Robotics
MESA: An Evaluation Framework for Compositional, Semantic, and Spatial Generalization in Robotics
This paper introduces MESA, a new evaluation framework designed to precisely measure language-conditioned policy generalization in robotics. It offers five evaluation suites that isolate in-distribution performance and four axes of generalization: spatial configuration, object instance, object category, and subtask composition. The authors also release MESA-Gen, a pipeline for generating tasks and demonstrations, instantiated on two platforms as the single-arm MESA suite and BiMESA, the first large-scale benchmark for bimanual generalization. Evaluating seven baseline methods including diffusion policies, VLM-based approaches, and pretrained VLAs, the study finds the strongest model achieves 52% on MESA and 49% on BiMESA, providing meaningful signals for future progress while maintaining reproducibility beyond simulation.
The authors introduce MESA, a new evaluation framework comprising five suites to measure language-conditioned policy generalization across spatial, object instance, category, and subtask composition axes, validated on single-arm and bimanual platforms with seven baseline methods showing top performance at 52% and 49% respectively.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
e strongest model achieves only 52% on MESA and 49% on BiMESA, providing meaningful signal while leaving ample room for progress. MESA rankings are broadly consistent with RoboArena scores, suggesting that MESA captures method-level differences relevant beyond simulation. Project website: https://pairlab.github.io/MESA.
Open source S5
To populate these suites at scale, we develop MESA-Gen, a pipeline for task and demonstration generation built by improving MimicGen. We instantiate the framework on two platforms and release both: the single-arm MESA suite, and BiMESA, the first large-scale benchmark for bimanual generalization. We evaluate seven baseline methods spanning diffusion policies, VLM-based approaches, and pretrained VLAs, finding that th
Open source S4
MESA: An Evaluation Framework for Compositional, Semantic, and Spatial Generalization in Robotics
Open source S1
arXiv:2611.05567v1 Announce Type: new Abstract: Despite growing interest in foundation models for robotic manipulation, systematic evaluation of their capabilities remains elusive. Existing benchmarks are either narrow and saturated, offering little signal on SOTA methods, broadly scoped, providing diversity at the cost of controlled experimentation, or require training on large, messy open-source datasets. We introd
Open source S2
uce MESA, an evaluation framework designed for precise, reproducible measurement of language-conditioned policy generalization. MESA provides five evaluation suites that isolate in-distribution performance and four distinct axes of generalization: spatial configuration, object instance, object category, and subtask composition, enabling researchers to diagnose where and why policies fail, not just whether they fail.
Open source S3
Source：arXiv Robotics — research abstracts · arxiv.org