HandsOn: A Benchmark for Dexterous Manipulation with Cross-Embodiment and Task Generalization

Editor | Chen Chen, Clio

The robot-operated 'exam' now needs a new one tailored for the dexterous hand.

Past robotic operations benchmark, mostly focused on grippers. With first-person (ego) human operation data continuously accumulating, how humans grasp, use tools, and complete fine contact is being recorded on an increasingly large scale. Dexterous hands, with structures similar to human hands, have the potential to leverage this data for larger-scale learning and become an important direction for robotic operations.

However, the dexterous hand keeps evolving, but the 'exam' has not kept up.

Dexterous hands have a large number of active joints, tasks are full of contact, often require bilateral coordination, and must maintain the results of previous actions in longer action sequences; it is difficult to determine success or failure, and it is also difficult to locate where the failure occurs. The benchmarks of the gripper era cannot directly answer these questions, while systems and reliable evaluation frameworks specifically for dexterous hands are still lacking.

Data and models are moving forward, yet there is a lack of a measuring stick to gauge progress.

Recently, the team led by Ding Ning from Tsinghua University, in collaboration with Beijing University of Posts and Telecommunications, Peking University, and researchers from Nature Will, have jointly constructed a composable simulation platform and evaluation benchmark for cross-body dexterous operations named HandsOn, aiming to fill this gap.

HandsOn includes 66 manually verified tasks and 18,000 human demonstrations Teleoperation, covering evaluations around tool use, fine contact, bimanual collaboration, and task transfer. The platform supports 5 types of robotic arms, 6 types of dexterous hands, and approximately 200 types of rigid, jointed, and deformable object assets, which can be combined in randomized scenarios.

HandsOn includes 18,000 selected simulated teleoperation demonstrations, with tasks divided into Core-50 and Transfer-16.

This benchmark's results are not easy to achieve: the team evaluated Diffusion strategy、vision-language-action models、World Action Model and different approaches. The results showed that, in the Core-50 tasks where training demonstrations were provided, the best-performing multi-task strategy achieved a success rate of 29.1%. Facing Transfer-16, where no target task training demonstrations were provided, the average success rate of the 8 trained multi-task strategies was below 1%.

In addition, GPT-6, which was not fine-tuned on the benchmark data, controlled the palm and fingers through move_eef an interface, achieving success rates of 5.2% for Core-50 and 1.9% for Transfer-16 in 660 tests.

The team also further tracked aspects such as grasping, contact, bilateral coordination, and maintaining the target state, discovering that robots often complete the first half of an action but fail at critical moments.

The related paper, code, and data resources have been released.

• Paper arXiv: https://arxiv.org/abs/2610.26765

• Code GitHub: https://github.com/TsinghuaC3I/HandsOn

• Data Hugging Face: https://huggingface.co/datasets/Cyan666/HandsOn-Datasets

Why do dexterous hands need a new evaluation benchmark?

The challenges of dexterous hands are not just having more fingers than grippers, but also in controlling the constantly changing contact relationships during operation.

After picking up a mouse, a robot needs to avoid triggering the click too early; when setting two knobs, it needs to keep them both at the target position simultaneously; when placing a box lid, it also needs to maintain the original alignment after releasing it. These tasks all require the robot to coordinate finger movements, judge contact states, and ensure that previously completed actions are not disrupted by subsequent actions.

This also changes the way robot operations are evaluated. For grippers, whether the grasp is successful and whether the object is placed at the designated location are often important criteria. For dexterous hands, simply grasping an object is not enough. The robot also needs to truly use tools, complete fine operations, and maintain the correct object state throughout multi-step tasks. A single action making progress does not mean the entire task is successful.

The accumulation of first-person human operation data provides new opportunities for dexterous hands to learn these skills. However, even after increasing the amount of demonstration data, two questions still need to be answered: can the model stably reproduce learned operations? And can it transfer existing skills to new objects, task combinations, or deformable objects?

Existing benchmarks for dexterous operations have already covered different robot bodies, functional operations, and bilateral coordination. However, related research still has room for improvement in terms of the scale of human demonstration data, the number of unseen tasks specifically reserved for evaluation, and the diagnosis of operation failures.

HandsOn's design starts from here, separating the measurement of execution capability and generalization capability, dividing the 66 tasks into two sets of evaluations: Core-50 and Transfer-16.

Core-50 provides demonstration data for strategy learning and evaluates strategy performance through a new round of independent execution tests; Transfer-16 is specifically used for generalization capability evaluation and does not include the demonstration data of the target task in the training set.

HandsOn suite: Core-50 contains demonstration content; Transfer-16 is solely for evaluation purposes.

This design not only tests whether a robot can complete tasks with demonstration, but also further examines whether existing skills can be transferred to new operational conditions. Combined with staged diagnostics, researchers can also analyze whether failures occur during contact establishment, functional operation, bimanual coordination, target state maintenance, or subsequent step connections.

Core-50: Can it execute tasks stably after learning?

Core-50 contains 50 tasks that provide training demonstrations, covering abilities such as functional understanding, numerical input, spatial relations, joint operations, precise contact, and bimanual coordination.

These categories correspond to specific life scenarios. When operating scissors, the robot needs to lift them off the table, complete three opening and closing motions while maintaining a continuous grasp; when setting a timer, it needs to input the specified minutes and seconds, then withdraw its hands after starting; unfolding a folding fan requires holding the outer fan bones with both hands, unfolding it to the target angle, and maintaining it. Making coffee further links steps such as taking the cup, closing the door, receiving the coffee, and placing it, examining the connection between actions.

Making coffee: multiple sub-goals linked together

Take the cup from the cabinet, close the door, receive the coffee, and place it on the tray.

Washing the bottle: order and bimanual division of labor

One hand stabilizes the bottle, while the other hand sequentially washes the designated areas.

Unfolding a folding fan: bimanual coordination of joint movements

Hold the outer fan bones with both hands, unfold it to the target angle, and maintain it.

Setting a timer: numerical goals and button operations

Input the specified minutes and seconds, then start and withdraw the hands.

Suspended Scissors: Grasping and Functional Actions

Lift the scissors off the table, maintain the grasp, and complete three opening and closing motions.

Trigger Drill: Contact and Functional Activation

Maintain contact between the drill bit and the wooden block, while pressing the trigger for half a second.

Six task examples in Core-50. Each task has a clear goal, action constraints, and completion conditions; all images are original simulation scenes.

Task design unfolds along three dimensions.

• The first dimension is interaction type, including basic operations, functional use, and contact-intensive operations.

• The second dimension is task structure, including single-stage operations, repeated operations, and long-term tasks.

• The third dimension is object type, covering rigid bodies, objects with rotating or sliding joints, and deformable objects.

In Core-50, the three types of interactions include 17, 18, and 15 tasks respectively. The three dimensions cross-combine, enabling the benchmark to cover different difficulty operational scenarios.

Transfer-16: Can it still complete tasks with different objects and tasks?

Transfer-16 contains 16 evaluation tasks that do not provide target task training demonstrations, further examining whether the model can transfer existing skills. Among them, 7 tasks examine asset or relational transfer, 6 tasks examine skill composition, and the remaining 3 involve operations on deformable objects.

Asset or relational transfer requires the model to face new objects or target relationships, find the correct operational object, and complete the task. Skill composition requires the model to connect already learned actions in the correct sequence. Deformable object operations introduce items such as towels, sponges, and pillows, requiring the model to handle shape changes of objects during operations.

For example, in the task of passing a block and then covering a box, the robot needs to first complete the bimanual transfer, then place the block into the box, and finally align and close the box lid. Each local action may have already appeared in other tasks, but the model still needs to determine whether the current step is completed and whether the result of the previous step can be maintained.

The requirement for continuous folding of a towel is more complex. The robot needs to continuously complete two folds and let the towel remain stably on the table after release. Deformable objects change shape with the position of grasping and the way of applying force, making the operation process more unpredictable.

Audio Mixer Knob: Asset Transfer

Find the corresponding knob on a new device, adjust it to the target, and maintain it.

Handover Block Box: Skill Composition

Complete the bimanual handover, release into the box, then align the box lid and close it.

Continuous Folding Towel: Flexible Object Manipulation

Fold twice continuously, and let the towel remain stably on the table after release.

Compressing a pillow for packaging: deformation and bimanual handling

Squeeze, lift, and place into a paper box, and maintain stability after release.

Representative tasks in Transfer-16, demonstrating asset transfer, skill composition, and operations on deformable objects.

It can be seen that Core-50 mainly tests learning and execution capabilities on tasks with demonstrations, while Transfer-16 further tests whether existing capabilities can be transferred and combined.

Let failures be accurately recorded

To ensure the reliability of the evaluation, HandsOn has created task cards for each task, specifying the goal, standard initial state, expected final state, allowed and prohibited actions, and evidence for determining whether the task is completed.

Completion conditions can be determined based on object pose, joint states, or image information. The research team also conducted manual operation verification for tasks and set up an independent review process to check whether the success labels are consistent with the actual physical states.

In multi-step tasks, the evaluation also records the progress of sequentially completed stages. For example, in a block transfer task, it distinguishes between both hands contacting an object simultaneously, and the receiving hand already being able to independently support the object. In skill combination tasks, it checks whether prerequisite steps are completed and whether existing results are maintained.

This allows researchers to distinguish between two scenarios: whether the model has not completed the task at all, or whether it has completed part of the task but failed in subsequent steps.

What did the robot learn after 18,000 demonstrations?

Unlike gripper data, demonstrations for dexterous hands require recording fine-grained actions at the finger level. HandsOn adopts a human teleoperation approach, converting human hand operations into action demonstrations for dexterous hands, which also aligns with the development direction of first-person human operation data.

Around these tasks, HandsOn collected approximately 56 hours of human operation records, involving at least 10 operators and over 1,050 recording sessions. Operators wore MANUS data gloves and used wrist-tracking devices to control the simulated robot, with the system synchronously saving image, state, action, and language information.

Out of 18,000 demonstrations, 15,178 were used for training and 2,822 for validation. Each interaction includes three streams of images from the front and wrist perspectives: the front view presents the table layout and hand positions, while the wrist view provides a local observation close to the fingers and objects. For tasks requiring precise contact, the local view can supplement information about how fingers approach, grasp, and manipulate objects.

After the data scale increases, how will the success rate change? The research team selected two tasks: 'putting a ball into a bowl' and 'lifting scissors and opening and closing them', and conducted single-task experiments using π0.5, increasing the number of training demonstrations sequentially to 10, 50, 100, and 200, and finally using the full training pool.

π0.5 single-task demonstration scale experiment, complete training pools were respectively 438 and 439 examples. Each checkpoint trained 20,000 updates and tested 100 episodes; the shaded area represents the 95% Wilson interval. A single training seed was used, and the interval does not include uncertainty from the training seed or subset selection.

The results show that the success rate of the ball-throwing task increased from 7% to 83%, and the scissors task increased from 1% to 44%. Both tasks showed continuous improvement across five test scales. The complete training pools respectively contained 438 and 439 demonstrations, with each checkpoint trained 20,000 updates and measured performance through 100 episodes of new closed-loop execution.

These results provide an intuitive reference for data gains. As the number of demonstrations increases, the model achieves higher success rates in both tasks; at the same time, there remains a significant gap between the scissors operation and the ball-throwing task, highlighting the difficulty of combining stable grasping with tool-functional actions. The experiment used a single training seed, and the reported 95% Wilson interval reflects uncertainty in the test episodes. Variations caused by the training seed and subset selection still require further validation.

Data can also be expanded visually. HandsOn supports high-fidelity re-rendering of the same operation, changing the desktop material, lighting, room background, and slight camera offsets, while preserving the original state and action supervision. Researchers can thus generate different observations around the same interaction, checking the strategy's adaptability to appearance changes.

In terms of robot configuration, HandsOn establishes simulation correspondences around the real ROKAE robotic arm and the first-generation Wuji Hand. Each hand contains 20 active joints. Kinematics, joint limits, asset scale, camera calibration, and state and action coordinates are all included in the correspondence records. The main camera and wrist camera work together, integrating the overall layout and hand details into the same dataset and evaluation process.

Simulation versus real aircraft perspective in a toy airplane scenario, with wrist image magnification displayed.

The platform supports 5 robotic arms, 6 dexterous hands, and approximately 200 assets. This data collection and quantitative evaluation uniformly adopted the ROKAE—Wuji and PhysX configuration. The team also provided video demonstrations of real-world execution using the simulated demonstration adaptation strategy. Under the condition of no target-task real-world demonstrations, the videos qualitatively show contact, manipulation, and pose maintenance. Regarding real-world execution performance, one can currently observe it through these videos, with the evidence scope differing from the quantitative evaluations in simulation.

29.1% success rate, where is the difficulty in two-handed coordination?

HandsOn evaluated 10 methods, including 5 vision-language-action models (VLA), 3 world-action models (WAM), diffusion policies trained separately by task, and zero-shot control referencing GPT-6. The first 9 methods used HandsOn demonstrations for training, with 8 of them using multi-task training.

On Core-50, the 9 trained methods collectively completed nearly 45,000 effective tests. The highest task average success rate from the multi-task strategy was achieved by OpenWAM, at 29.1%. The differences between tasks are quite apparent: grasping, manipulation, and some joint operations are relatively easy, while tasks involving using tools to act on the third object or completing precise coordination remain difficult.

The results of the transfer evaluation are even more severe. On Transfer-16, the average task success rate of the 8 multi-task trained strategies was all below 1%, and the single-task Diffusion Policy did not participate in this part of the evaluation. Facing new assets, skill combinations, and deformable objects, the ability to convert existing training experience into complete task success remains limited.

When complete success is rare, stage progress provides an entry point for further analysis. The dual knob task is an example: each hand operates a knob, adjusting both sides to the target state, and satisfying conditions such as holding and releasing. In 800 attempts by 8 multi-task models, 219 attempts had both sides reaching the target individually, 38 attempts achieved simultaneous alignment, and finally 5 attempts met the full conditions.

The gap mainly appears in how the two local results can jointly hold. A common situation is that the hand that arrives first continues to turn, while the other hand has not yet completed, and the existing results have already changed.

The block transfer demonstrates the temporal requirements between the two hands. In 367 rounds where both hands grasped the block together, 188 rounds achieved independent takeover by the receiving hand; in the remaining 179 rounds, 175 rounds resulted in the block falling afterward. The timing of when the giving hand releases needs to be coordinated with the receiving hand's support state.

Dual knobs total 800 rounds; among the 367 shared contact rounds in block handoffs, 188 rounds achieved independent takeover by the receiving hand.

Long tasks add requirements for sequence. In the combined task requiring first covering the kettle and then covering the box, 340 out of 400 attempts moved the box cover first. These records provide trackable issues for subsequent research: how the model confirms the completion of the current step, how to maintain existing results, and when to initiate the next operation.

The study also checks whether actions change with the language goal by keeping the scene intact and replacing the target object in the instruction. Some checkpoints still have limited responses to object names, indicating that target selection also requires ongoing validation. A single task failure may involve contact control, or it may occur earlier during the target understanding and action selection phase.

The zero-shot testing of GPT-6 provides another set of observations. It reads three RGB images, robot state, task instruction, and execution feedback in each round, and specifies the hand pose and up to 40 finger joint targets via move_eef.

Without HandsOn demonstration training or fine-tuning, the model tested 10 times on each of the 66 tasks, achieving 26 Core-50 successes and 3 Transfer-16 successes, corresponding to success rates of 5.2% and 1.9% respectively.

Action records show that the model can propose some plans related to operational prerequisites. For example, when folding a towel, it once proposed to first slightly lift it to confirm that the towel edge moves with the hand, and then continue folding. The round ultimately failed, highlighting the difficulty in translating the plan into contact and motion control.

For robots moving toward everyday environments, these details are all related to practical use. As task configurations, demonstration data, and evaluation methods become public, researchers can test new methods under common settings and analyze the sources of success rate changes along the execution process. For robots, this benchmark provides a set of experimental conditions for continuous comparison as they gradually master daily operations with their hands.

Conclusion: Letting capability progress withstand practical testing

The hand is one of the most intricate organs of humans. People use their hands to use tools, reshape the environment, and accumulate experience about objects and the world through repeated operations.

To enter human living and working spaces, robots also need to integrate perception, judgment, and action into continuous physical interaction. For the robotics industry, the maturity of hand capabilities will determine how much actual work general models can take on.

The robotics industry is gradually entering a stage that requires accumulating common evidence. While technical demonstrations show the possibility of capabilities, large-scale applications require this capability to withstand repeated execution, scene changes, and long-term operation testing.

In this process, public benchmarks play a role in connecting research results with application needs. As model capabilities improve, evaluations also need to absorb new tasks and constraints from real-world applications, so that score growth corresponds to broader and more stable operational capabilities.

The long-term value of HandsOn will also gradually become apparent as researchers participate: new methods drive performance improvements, failure cases bring new research questions, and application needs push test standards to continuously improve. For the industry, such accumulation helps form clearer capability boundaries, providing a more solid basis for research investment and implementation expectations.