538 Hours of 'Extremely Messy' Human Data, Real Robot Takes Less Than an Hour: Hong Kong University Teaches Robots to Unpack Packages, Use Hot Glue Guns, and Make Cold Drinks

If the training videos shown to the robot come from chaotic households, busy factories, and pharmacies, with the camera constantly shaking violently as people move around, could it still learn fine-grained operations?

A recent work by the MMLab team at the University of Hong Kong shows that

These data that look ``not clean'' can also be used to train robots.

Recently, the team proposed EgoWild2Dex, directly using 538.9 hours of uncurated real human first-person operation data to train robots. With less than 1 hour of real device data per downstream task, the robot is able to continuously complete complex dual-arm dexterous long-sequence tasks such as unboxing, using a hot melt glue gun to fix a figurine, and making a cold drink, achieving an average success rate of 96.7%.

Rather than first curating the data

Cleaning

In contrast to laboratory settings, EgoWild2Dex directly utilizes natural data that retains realistic factors such as viewpoint jitter, complex backgrounds, and long-tail object distributions.

EgoWild2Dex learns from first-person human experiences in the real world and transfers these experiences to dual-arm dexterous robots, enabling long-term complex operations.

Currently, the research paper has been made public on arXiv, and the project homepage is now online. The dataset, model, and code are planned to be released later.

Project Homepage: https://mmlab.hk/egowild2dex

Paper link: https://arxiv.org/pdf/2609.23755

Robot data is expensive, but human experience is everywhere.

Robot learning has long faced a real-world problem:

real robot data is too expensive.

To enable robots to learn complex operations, often requires data collectors to repeatedly perform teleoperation on real robots, collect expert data, and process failed trajectories. Once the task involves dual-arm coordination, tool dexterous use, fine contact, and long-time sequence operations, the data collection cost will increase further. However, humans perform a vast amount of operations in the real world every day: unpacking packages, organizing goods, assembling parts, picking up tools, making drinks... These daily behaviors themselves are ready-made operational experiences.

Therefore, an increasing number of studies have begun to try to utilize

first-person (Egocentric) data

to help robots learn. However, many previous datasets still focused on pre-designed tasks and were collected in relatively controlled environments. But data from controlled environments still has a significant gap compared to the scenarios robots encounter during actual deployment.

EgoWild2Dex takes a more direct approach: it no longer requires people to manually clean the data, but instead directly enters the real environment to record actions that humans naturally perform.

The research team collected natural human operations in scenarios such as homes, factories, pharmacies, hospitals, supermarkets, and parcel stations. These videos did not deliberately maintain a stable viewpoint, nor were background distractions minimized. When people walk, look down, or turn around, the camera viewpoint changes dramatically. The data shows that in EgoWild, the cumulative average head rotation angle per second reaches

15.93° , and each frame image on average contains 34.78 different object instances.

According to traditional robot data collection standards, this dataset is indeed quite

``Dirty``

But the 'dirty' here does not refer to annotation errors or poor data quality, but rather refers to it Preserved the inherent complexity of the real world.

The seemingly obstructive ``noise`` to model learning, How to become a key resource for robot training?

The first obstacle in using human data to train robots is the perspective difference.

between humans and robots. Human first-person cameras are typically mounted on the head. When people perform tasks, they constantly move around, turn their heads, and look down, resulting in very dynamic visuals. In contrast, robot cameras often have a more stable perspective. If we directly mix these two types of data for training, the model would actually have to deal with two very different visual distributions at the same time.

To address this, EgoWild2Dex proposes

GeoFormer（Geometric Transformer）

a lightweight, differentiable geometric transformation module.

In simple terms, GeoFormer will transform ``the view seen by a person when moving'' into a perspective closer to a robot's camera.

GeoFormer learns the geometric relationships between the human first-person perspective and the robot's perspective, and dynamically transforms the view to reduce the visual differences between human data and robot data. This means that humans do not need to deliberately keep the camera stable when collecting data to adapt to the robot. Compared to traditional Project + Inpaint workflows that rely on precise camera calibration, 3D reprojection, and image inpainting, GeoFormer relies less on precise calibration and has improved processing speed

21.9

times, and better visual alignment results.

The perspective alignment and training process of GeoFormer. (a) Perspective alignment: predicts the geometric relationship between the robot's perspective and the human first-person perspective, and aligns human observations to the robot's perspective; (b) Geometric transformation learning: optimizes transformation parameters through multi-level predictions and uses a discriminator to supervise the generation of more natural results; (c) VLA training: aligns human data and robot data for strategy training

GeoFormer aligns dynamic human first-person observations to a perspective closer to the robot's camera, thereby reducing the observational differences between humans and robots

After perspective alignment, it is still necessary to address the differences between human actions and robot actions.

EgoWild2Dex uses action retargeting and inverse kinematics to map three types of data to a unified robot-native action space: (1) human first-person data collected in natural environments (Ego); (2) glove demonstration data collected from the robot's perspective (Glove); (3) real robot demonstration data (Robot). After retargeting, the three types of data can be represented using a unified robot action representation and participate in strategy training together.

Map open-scene human first-person data, task-related human first-person data, and glove demonstration data from the robot's perspective to a unified robot-native action space

After completing the alignment of vision and action spaces, EgoWild2Dex adopts a progressive human-robot joint training paradigm (Progressive Human-Robot Training). The entire training is divided into three steps: first, pretrain with large-scale human data, then add task-related data, and finally fine-tune with a small amount of real robot data.

1. Open pre-training (Human-to-Robot Learning)

: First, the model pretrains using large-scale In-the-Wild human first-person data. This stage does not require all data to come from tasks the robot will eventually perform. The VLA model thus encounters more scenes, objects, and ways of operation, learning general visual and operational priors.

2. Human-Robot Co-training

）

: Next, the model further incorporates Ego data, glove demonstration data, and a small amount of real robot data related to downstream tasks. With a unified action space, the broad human experience begins to establish correspondence with specific robot tasks.

3. Robot-Domain Refinement

: Finally, the model is fine-tuned using a small amount of real robot demonstrations and trajectory recovery. This stage is mainly used to further adapt to physical contact, dynamics, and execution errors in real robots. The amount of real robot data used for each downstream task is less than 1 hour.

Ego data, glove data, and real robot data

The research team also released simultaneously

Large-scale ``Extremely Dirty'' First-Person Human Perspective Dataset EgoWild

The dataset contains

538.9 hours

of operational data, covering approximately

179,000 operation segments , 126,000 independent task descriptions and 1,282 object categories. The term ``dirty'' here does not refer to incorrectly labeled or low-quality data, but rather to data that does not deliberately avoid complex factors present in real-world environments. EgoWild directly collects data in scenarios such as homes, factories, pharmacies, hospitals, supermarkets, and parcel stations, preserving camera motion, complex backgrounds, long-tail object distributions, and natural human operational workflows.

EgoWild not only provides rich language annotation, but also includes

full-body human pose, hand joint

and other fine-grained annotations, and divides operational behaviors into

atomic actions, action commands, and skill commands

three levels. In addition, the data processing pipeline has been specifically optimized for hand action recognition, human sliding steps, and other issues, providing the model with higher quality and more complete supervisory information.

EgoWild records natural human operations in real environments such as homes, factories, pharmacies, hospitals, and supermarkets, retaining dynamic perspectives, complex backgrounds, and diversified operation processes

Experimental validation

In real-device experiments, the research team tested three complex long-sequence dual-arm operation tasks:

Use a utility knife to cut open the tape and unbox it.

as well as

Using a shovel to break ice and make ice drinks

The model achieved

96.7%

average task success rate

and demonstrated the ability to respond to external disturbances and resume task execution in experiments. In the unboxing and delivering a package task, facing objects not seen during training, the research team also tested the model's zero-shot generalization ability when encountering objects not seen during training, with an average success rate of 33.3%.

In addition, the study transferred this framework from the Suanling Piper robotic arm to another robotic arm, Tianji Marvin Pro, demonstrating the feasibility of transferring the method to different robotic bodies.

continuously completing multiple subtasks such as grasping tools, cutting tape, opening the box lid, and retrieving objects

coordinated bimanual operation of a glue gun and decorative items, completing continuous fine operations such as alignment, gluing, grasping, and fixation

completing multi-stage bimanual operations such as scooping ice, pouring ice, holding a cup, aligning with the spout, and catching water

facing objects inside the box not seen during training, the robot could still attempt to grasp and retrieve them

after task execution was disturbed (fingers stuck, humans putting flowers back), the robot could adjust its actions based on the current state and continue with subsequent operations

when the decorative item being held was taken away, the robot could re-adjust its hand and arm and complete the grasp again

facing interference where the cup was taken away by humans, the robot could re-grasp the cup and continue with the task

EgoWild2Dex further transferred from Piper robotic arm to Tianji Marvin Pro

Summary and Outlook

The most noteworthy aspect of EgoWild2Dex, perhaps, is not the success rate of a specific task, but its expansion of data sources for robots:

In addition to expensive real robot demonstrations, unscripted human daily operations can also become data sources for robot training.

Of course, from first-person human videos to stable executable robot strategies, there are still multiple differences such as perspective, body, and dynamics in between. However, if such data can be more effectively utilized, the scale of data available for robots is expected to expand significantly.

EgoWild2Dex is just an initial exploration in this direction. We look forward to future robots that can truly learn from human daily experiences, continuously adapt, accumulate, and evolve in complex and changing real-world environments.

© THE END

For republication, please contact this official account for authorization

For submissions or media inquiries: [email protected]