10 Hours of Data + One Human Video = ? SkeLAM's Zero-Shot Cross-Embodiment Motion Transfer

Based on the pre-trained Cosmos-Predict2.5-2B, SkeLAM is trained using approximately 10 hours of robot interaction data. In the human-to-robot transfer experiment, when the input is replaced with human demonstrations, the robotic arm in the generated video still exhibits motion directions corresponding to the demonstrations. During transfer, all model parameters are frozen, and no additional fine-tuning is performed for these demonstrations.

This work comes from institutions such as the Harbin Institute of Technology (Shenzhen) and the Show Lab at the National University of Singapore. The team uses a robotic skeleton to supervise latent actions in videos, allowing the motion latent vector to more accurately represent the robot's motion, while another latent vector preserves the visual changes such as object responses. Real control commands thus have a dedicated motion representation as an alignment target.

In the DROID evaluation, compared to DreamDojo, SkeLAM reduces the end-effector trajectory error by 19.5% and the perceptual distance LPIPS by 11.1%.

Paper title: Skeleton-Grounded Latent Action Model for Robot World Models

Project homepage:

https://aopolin-lv.github.io/SkeLAM/assets/SkeLAM.pdf

Project homepage and video: aopolin-lv.github.io/SkeLAM

Human video as prompt:

Zero-shot cross-embodiment motion transfer

The input for this experiment is a segment of UniSkill human demonstration video, as well as a robot scene image. The RGB encoder extracts latent actions from the demonstration, and the world model generates subsequent video in the given robot scene based on this.

When a human pushes a towel downward, the generated robotic arm moves downward accordingly; when the human pulls paper upward, the robotic arm also lifts. Although the appearance, structure, and scene of the human hand and robotic arm are different, the motion direction from the demonstration is still reflected in the generated results. In contrast, DreamDojo shows smaller end-effector displacement in these two examples.

Fig. 1 | Two examples of zero-shot transfer: pushing a towel downward and pulling paper upward. From top to bottom, each group shows the human demonstration, DreamDojo, and SkeLAM, with circles indicating the end-effector positions. The robot images are the model-generated results.

Human demonstrations here serve as the motion context during inference: changing the demonstration allows the generated motion to be altered in these examples without updating the model parameters. This is a specific form of in-context human-to-robot motion transfer, and also provides qualitative evidence for zero-shot cross-embodiment motion transfer.

Skeleton supervision, Making latent actions easier to align with real control Learning actions from video first requires handling the issue of mixed robotic arm motion and object responses. When the robotic arm moves forward, it might push away a cup or might not touch anything at all. The control commands can be the same, but the changes in the video are different. If these changes are all encoded into the same latent action, the real control command needs to correspond to a target that varies with the scene.

SkeLAM constrains the motion components within the skeleton. The skeleton records the joint positions and connection relationships of the robot, providing a reference for motion in the video; another residual representation preserves the supplementary visual dynamics. Together, they support video reconstruction and also provide conditions for the world model.

Fig. 2 | Overview of the SkeLAM method. The skeleton supervises the motion representation in the video, with motion and residual components respectively providing conditions for the world model, and the action adapter maps robot commands to the motion representation.

Giving the latent action a clear motion reference

The same robotic arm motion, under different contact states, may result in different object responses. These changes help in video reconstruction, but when the latent action includes both motion and interaction results, the robot control command needs to correspond to a target that varies with the scene.

SkeLAM uses the skeleton to constrain the motion components. The skeleton preserves the joint positions and connection relationships of the robot, allowing a more direct description of how the robot moves; another residual representation preserves the supplementary visual dynamics. Together, they support video reconstruction and also provide conditions for the world model.

Specifically, the skeleton decoder supervises the motion components extracted from the RGB video, enabling it to predict the corresponding skeleton changes. Cross-modal reconstruction exchange further requires: replacing the motion components in RGB reconstruction with the corresponding skeleton representation, retaining the original residual, and still requiring the model to reconstruct the same segment of visual changes. In this way, the correspondence between the video and the skeleton representation will directly affect the reconstruction results.

In order to apply this representation for robot control, SkeLAM trained a lightweight action adapter that maps real robot actions to motion components, and then fine-tunes the world model using the adapter's predicted conditions. During inference, the model generates future videos based on the current frame and robot actions, with residual inputs set to zero.

Are the generated results closer to real motion?

In the DROID evaluation, compared to DreamDojo, SkeLAM's end-point trajectory error ADE decreased from 51.11 to 41.15 pixels, and the visual distance LPIPS decreased from 0.1473 to 0.1310. The former measures the deviation between the generated motion and the reference trajectory, while the latter reflects the perceptual difference between the generated image and the real video.

The following comparison demonstrates how these differences appear in the video. SkeLAM is closer to the curved trajectory in the reference video and better preserves the desktop layout.

Figure 3 | Generated results under the same initial frame and robot actions. The leftmost column is the real video, the second column is SkeLAM; the last row overlays the end-effector motion trajectory, facilitating comparison of the generated motion with the reference motion.

Make more human videos resources for robot learning

Along this direction, large-scale human operation videos, especially first-person videos, are expected to become sources of motion for robot world models. If stable transferable action representations can be extracted from them, there will be an opportunity to use more human demonstrations for video generation in robot scenarios, expanding the interactive processes available for model learning, and further exploring their value in robot policy training and evaluation.

To move from these generated results to real robots, more complex contacts and longer operation processes need to be tested, and the actual help of the generated videos for policy learning and closed-loop execution needs to be assessed.

The full paper, introduction video, and more visual results are available on the project homepage.

Author Introduction

Qi Lv: First author of this paper, from Harbin Institute of Technology (Shenzhen) and National University of Singapore, with research involving robot manipulation, action representation, and world models.

Jiayi Lin: From Harbin Institute of Technology (Shenzhen), with research involving robot manipulation, action representation, and world models.

Yuxin Jiang: PhD student at National University of Singapore and A*STAR, with research directions including generative models, video world models, and their applications in embodied intelligence.

Mike Shou Zheng: Corresponding author of this paper, assistant professor at National University of Singapore, with research directions including video understanding and multimodal intelligence.

© THE END

For republication, please contact this official account for authorization

For submissions or media inquiries: [email protected]