VREgo: Learning Robot Manipulation from Egocentric Demonstrations Using Consumer VR Headsets
VREgo: Consumer VR Headsets for Learning Robot Manipulation from Egocentric Demonstrations
This research presents VREgo, a framework for training robot manipulation policies using demonstrations recorded on consumer VR headsets like the PICO 4 Ultra. The system utilizes on-device tracking and stereo localization to capture head and hand movements, adapting HumanEgo preprocessing techniques to estimate grasp thresholds and update object positions. Evaluated in simulation on two tabletop tasks, VREgo trained with 40 demonstrations from a single operator achieved success rates of 67%, 74%, and 60% across three seeds for cube placement, compared to Project Aria's 73%, 78%, and 0%. For a stacking task, the baseline achieved 94% success. The authors address issues like premature gripper opening by adapting trajectories and adding execution rules, ultimately reaching 93.8% success across noise seeds. The code is available at https://vrego.chuange.org/.
The study demonstrates that manipulation policies trained on consumer VR headset recordings (PICO 4 Ultra) can achieve performance comparable to high-end research hardware (Project Aria) on specific tabletop tasks, though the authors note limitations in open-source gripper behavior and replanning delays that were mitigated through rule adaptation.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S3
Reported numbers
training demonstrations
40
View original evidence
positions after release, and propagate segmentation masks between keyframes. We evaluate policies in simulation on two tabletop tasks recorded by one operator. For cube placement, we compare PICO and Project Aria recordings of the same 41 layouts. With 40 training demonstrations, the PICO policies placed the cube on the tray in 67, 74, and 60 of 100 evaluation layouts across three training seeds. The corresponding Ar
Open source S3evaluation layouts
100
View original evidence
positions after release, and propagate segmentation masks between keyframes. We evaluate policies in simulation on two tabletop tasks recorded by one operator. For cube placement, we compare PICO and Project Aria recordings of the same 41 layouts. With 40 training demonstrations, the PICO policies placed the cube on the tray in 67, 74, and 60 of 100 evaluation layouts across three training seeds. The corresponding Ar
Open source S3
- code: not_reported
- weights: not_reported
- dataset: not_reported
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
positions after release, and propagate segmentation masks between keyframes. We evaluate policies in simulation on two tabletop tasks recorded by one operator. For cube placement, we compare PICO and Project Aria recordings of the same 41 layouts. With 40 training demonstrations, the PICO policies placed the cube on the tray in 67, 74, and 60 of 100 evaluation layouts across three training seeds. The corresponding Ar
Open source S3
arXiv:2611.07272v1 Announce Type: new Abstract: VREgo trains manipulation policies from demonstrations recorded on consumer VR headsets; we instantiate it with EgoKit on one such headset, the PICO 4 Ultra. On-device head and hand tracking and stereo object localisation allow the operator to begin a demonstration at the table. We adapt HumanEgo's preprocessing to estimate grasp thresholds per recording, update object
Open source S2
ia results were 73, 78, and 0. For a two-stage stacking task recorded with the PICO alone, the baseline placed the block on the tray in 94 of 100 layouts and met the end-of-episode criterion for both placements in two. In one diagnostic episode, repeated replanning deferred release beyond the eight executed steps. Demonstration replays also showed that the open gripper could displace the cube before grasping it. Afte
Open source S4
r adapting the training trajectories and adding task-specific execution rules that use simulator object positions, the system met both placement criteria in 93.8% of episodes across four evaluation noise seeds. The same 100 layouts informed rule development and evaluation. VREgo is available at https://vrego.chuange.org/.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org