Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
This paper presents Evo-DexBench, a new benchmark for evaluating generalization in dexterous manipulation policies. The dataset includes 30 tasks organized into six functional families, each with three everyday scenes and 1,248 multimodal demonstrations collected via VR teleoperation using simulated Shadow Hands with 123 taxels. The benchmark follows the Core-Gen principle, where core tasks provide source knowledge for novel target tasks with few demonstrations. Experiments show that joint training on core and generation tasks improves performance by up to 38% compared to sequential fine-tuning, and functionally matched core data yields greater gains than mismatched data. The authors release the dataset and benchmark to support further evaluation of dexterous manipulation policies.
The authors introduce Evo-DexBench, a benchmark designed to evaluate generalization in dexterous manipulation by controlling source-task relatedness and limiting target-task exposure; experiments on 30 tasks across six functional families show that joint Core + Gen training improves success rates over sequential fine-tuning, with the best-performing policy achieving only 24.4% average success.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
trajectories
1,248
Trajectory count
View original evidence
s and target-task exposure. It comprises 30 tasks in six functional families, three everyday scenes per task, and 1,248 multimodal demonstrations collected via VR teleoperation with simulated Shadow Hands equipped with 123 taxels. Experiments show that, among six representative policies, even the best-performing method achieves an average success rate of only 24.4%, highlighting the challenges existing methods face i
Open source S4Reported success rate
24.4%
Reported trials
View original evidence
s and target-task exposure. It comprises 30 tasks in six functional families, three everyday scenes per task, and 1,248 multimodal demonstrations collected via VR teleoperation with simulated Shadow Hands equipped with 123 taxels. Experiments show that, among six representative policies, even the best-performing method achieves an average success rate of only 24.4%, highlighting the challenges existing methods face i
Open source S4robots
6
View original evidence
s and target-task exposure. It comprises 30 tasks in six functional families, three everyday scenes per task, and 1,248 multimodal demonstrations collected via VR teleoperation with simulated Shadow Hands equipped with 123 taxels. Experiments show that, among six representative policies, even the best-performing method achieves an average success rate of only 24.4%, highlighting the challenges existing methods face i
Open source S4trajectories
16
Trajectory count
View original evidence
n transferring functional interaction knowledge. Further analysis shows that, across the two tasks, joint Core + Gen training improves performance by 12% and 38%, respectively, over training on Core followed by fine-tuning on Gen, indicating that joint training makes better use of Core knowledge. With matched Core data, 8 Gen demonstrations achieve performance comparable to training on 16--32 Gen demonstrations alone
Open source S5trajectories
32
Trajectory count
View original evidence
n transferring functional interaction knowledge. Further analysis shows that, across the two tasks, joint Core + Gen training improves performance by 12% and 38%, respectively, over training on Core followed by fine-tuning on Gen, indicating that joint training makes better use of Core knowledge. With matched Core data, 8 Gen demonstrations achieve performance comparable to training on 16--32 Gen demonstrations alone
Open source S5
- paper: not_reported
- dataset: Reported pending Open source S6
- code: not_reported
- weights: not_reported
- dataset: not_reported
What remains unknown
Not established in the collected evidence: Environment, Control, Data origin.
Dataset licensing and commercial-use rights are not established here.
Reported performance applies to the described task. It does not establish general autonomy or deployment readiness.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
s and target-task exposure. It comprises 30 tasks in six functional families, three everyday scenes per task, and 1,248 multimodal demonstrations collected via VR teleoperation with simulated Shadow Hands equipped with 123 taxels. Experiments show that, among six representative policies, even the best-performing method achieves an average success rate of only 24.4%, highlighting the challenges existing methods face i
Open source S4
n transferring functional interaction knowledge. Further analysis shows that, across the two tasks, joint Core + Gen training improves performance by 12% and 38%, respectively, over training on Core followed by fine-tuning on Gen, indicating that joint training makes better use of Core knowledge. With matched Core data, 8 Gen demonstrations achieve performance comparable to training on 16--32 Gen demonstrations alone
Open source S5
. On all three tested Gen targets, functionally matched Core data yields greater gains in success rate than mismatched data. We release the Evo-DexBench dataset and benchmark to support the evaluation of dexterous manipulation policies' generalization across related tasks.
Open source S6
Implications for data suppliers
RoboSignal interpretation and collection questions, not statements of buyer demand.
- Confirm the required data type and collection setting with the buyer; this source does not establish a complete collection specification.
- Compare the reported units and scope before using these quantities in a budget. Recording hours, sensor-hours and trajectories are different measures.
- Validate demand and acceptance criteria with a buyer before scaling. Publication, popularity and a research result do not establish a purchase commitment.
Source：arXiv Robotics — research abstracts · arxiv.org