Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
This paper introduces RoboTryOn, a method that treats human-to-robot translation as virtual try-on to convert human videos into target robot videos while preserving task context. By autoregressively translating keyframes and using an inverse dynamics model with depth features for action inference, the approach enables policy learning from synthetic data. Experiments show that policies trained with this synthetic data improve average success rates on novel bimanual manipulation tasks from 2% to 60% without real robot demonstrations of those tasks, matching teleoperated demonstration performance at approximately one-third the collection cost.
The authors present RoboTryOn, a method that synthesizes dexterous robot demonstrations from human videos using a virtual try-on approach to bridge visual and action gaps for high-DoF robots, reporting improved success rates on novel tasks without real-world demonstrations of those specific tasks.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Data origin
- Synthetic dataOpen source S5
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.
Open source S5
Implications for data suppliers
RoboSignal interpretation and collection questions, not statements of buyer demand.
- Ask which physical-world evaluation and real-data validation the buyer requires for simulated data.
- Validate demand and acceptance criteria with a buyer before scaling. Publication, popularity and a research result do not establish a purchase commitment.
Source：arXiv Robotics — research abstracts · arxiv.org