Wearable system captures human activity data for robot training
Wearable system captures human activity data for robot training
Researchers from the Shanghai AI Laboratory, Zhejiang University, and Shanghai Jiao Tong University have developed a wearable system to capture human activity data for robot training. The system combines a head-mounted camera, motion trackers, and sensor-equipped gloves to record egocentric video, body motion, finger articulation, and tactile data. The resulting dataset, EgoExplorer, includes 100 hours of recordings and a movement-reconstruction pipeline to estimate whole-body motion. The study shows improvements in body-position error reduction and hand stability compared to baselines, though deployment on physical robots was not tested.
Source: Tech Xplore — Robotics · Read original article ↗
Article text · Original source · English
Humanoid robots, robotic systems with a body shape that resembles that of humans, could tackle various chores and everyday manual tasks. To successfully complete these tasks, however, the robots should be able to reliably coordinate their movements, posture and how they touch or grasp different objects.
Human demonstrations of how to complete a task could offer valuable guidance that humanoids could follow. However, collecting human demonstration data that could be helpful for training robotics algorithms can be challenging.
Researchers at the Shanghai AI Laboratory, Zhejiang University and Shanghai Jiao Tong University recently compiled a dataset containing 100 hours of human activity recorded with a new wearable system. The approach used to create the dataset, presented in an arXiv preprint, combines video from a wearer's viewpoint with reconstructed body movements, recorded finger movements, available tactile data and descriptions of individual task steps.
"Learning humanoid loco-manipulation from human demonstrations rests on observing how locomotion, posture, and object interaction evolve together as tasks unfold," Yunsong Zhou, Xuekun Jiang and their colleagues wrote in their paper. "Scaling collection across diverse environments, in turn, favors lightweight, portable capture systems. Yet, existing datasets provide only partial views of this coordination."
Recording data while people complete tasks
Zhou, Jiang and their colleagues developed a new wearable system that can capture data while humans are completing different activities. This system combines a head-mounted GoPro HERO13 camera with five VIVE Ultimate Trackers, sensor-equipped MANUS Metagloves or Wuji gloves, and custom 3D-printed mounts and straps.
The head-mounted camera records video from a wearer's viewpoint. The five trackers, one placed on a wearer's head, one at the back of each hand and one at each ankle, track their own position and orientation.
The sensor-equipped gloves can record the movement of a wearer's fingers, with the Wuji gloves also recording tactile data. Finally, the mounts and straps were used to secure the equipment.
Using this wearable system, the researchers recorded 100 hours of data while human wearers completed a wide range of activities. The different data collected were combined into a new dataset called EgoExplorer.
"With an emphasis on professional activities, its continuous recordings align egocentric video with reconstructed whole-body motion, fine-grained finger articulation, and tactile observations, accompanied by step-level descriptions," wrote the authors.
In their paper, Zhou, Jiang and their colleagues also introduced a movement-reconstruction pipeline called Calibrate-Complete-Constrain. This is a sequence of processing steps that allows them to turn the data recorded by the wearable system into estimates of whole-body movement.
"To achieve this coverage with sparse body sensing, we develop the Calibrate-Complete-Constrain pipeline," wrote Zhou, Jiang and their colleagues. "Concretely, reusable calibration fixes participant geometry and sensor attachments before motion inference to disentangle mounting offsets from articulation."
The software developed by the researchers estimates how joints moved when they were not directly tracked by the sensors. It achieves this using previously learned patterns of movement. Finally, the software checks that the reconstructed movements match the data collected by sensors, remain smooth over time and respect the body shape of wearers.
From human demonstrations to robot simulations
The team compared its movement-reconstruction method to some previously introduced approaches and found that it reduced body-position errors against tracker-fitted reference measurements. In preliminary learning experiments, jointly generating video, movement and touch data was found to produce better results than generating each separately.
The researchers also trained a controller for a simulated Unitree G1 humanoid robot to follow movements adapted from two of the human demonstration recordings they collected. The simulated robot followed both movement sequences without falling, although this did not demonstrate precise object interaction or good performance on unfamiliar tasks. Moreover, the study did not test deployment on physical robots.
"Against tracker-fitted references, the method reduces body-point error by 71.9% relative to the strongest baseline; matched evaluations also show gains in hand stability and contact estimation," wrote the authors. "Preliminary studies explore multimodal generation and conditioning alongside humanoid retargeting with whole-body control in simulation."
In the future, the EgoExplorer dataset could be used to train other robot controllers in simulation. Eventually, controllers trained using these recordings could help to improve the ability of humanoid robots to complete tasks in different real-world environments.
Written for you by our author Ingrid Fadelli, edited by Gaby Clark, and fact-checked and reviewed by Robert Egan—this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive. If this reporting matters to you, please consider a donation (especially monthly). You'll get an ad-free account as a thank-you.
Publication details
Yunsong Zhou et al., EgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ, arXiv (2026). DOI: 10.48550/arxiv.2610.18872
Journal information: arXiv
Who's behind this story?
Ingrid Fadelli
Freelance journalist with BSc Psychology and MA International Journalism. Covers AI, robotics, neuroscience, and astrophysics since 2018. Full profile →
Gaby Clark
MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile →
Robert Egan
Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →
© 2026 Science X Network
Citation: Wearable system captures human activity data for robot training (2026, October 24) retrieved 24 October 2026 from https://techxplore.com/news/2026-10-wearable-captures-human-robot.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
hours of human activity recorded
100 hours
View original evidence
lenging. Researchers at the Shanghai AI Laboratory, Zhejiang University and Shanghai Jiao Tong University recently compiled a dataset containing 100 hours of human activity recorded with a new wearable system. The approach used to create the dataset, presented in an arXiv preprint , combines video from a wearer's viewpoint with reconstructed body movements, recorded finger movements, available tactile data and descri
Open source S4number of trackers used
5
View original evidence
of this coordination." Overview of EgoExplorer. Credit: arXiv (2026). DOI: 10.48550/arxiv.2610.18872 Recording data while people complete tasks Zhou, Jiang and their colleagues developed a new wearable system that can capture data while humans are completing different activities. This system combines a head-mounted GoPro HERO13 camera with five VIVE Ultimate Trackers, sensor-equipped MANUS Metagloves or Wuji gloves,
Open source S6number of gloves used
5
View original evidence
of this coordination." Overview of EgoExplorer. Credit: arXiv (2026). DOI: 10.48550/arxiv.2610.18872 Recording data while people complete tasks Zhou, Jiang and their colleagues developed a new wearable system that can capture data while humans are completing different activities. This system combines a head-mounted GoPro HERO13 camera with five VIVE Ultimate Trackers, sensor-equipped MANUS Metagloves or Wuji gloves,
Open source S6number of mounts and straps used
1
View original evidence
and custom 3D-printed mounts and straps. The head-mounted camera records video from a wearer's viewpoint. The five trackers, one placed on a wearer's head, one at the back of each hand and one at each ankle, track their own position and orientation. The sensor-equipped gloves can record the movement of a wearer's fingers, with the Wuji gloves also recording tactile data. Finally, the mounts and straps were used to s
Open source S7number of human demonstration recordings used
2
View original evidence
xperiments, jointly generating video, movement and touch data was found to produce better results than generating each separately. The researchers also trained a controller for a simulated Unitree G1 humanoid robot to follow movements adapted from two of the human demonstration recordings they collected. The simulated robot followed both movement sequences without falling, although this did not demonstrate precise ob
Open source S12
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
What remains unknown
Dataset licensing and commercial-use rights are not established here.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
lenging. Researchers at the Shanghai AI Laboratory, Zhejiang University and Shanghai Jiao Tong University recently compiled a dataset containing 100 hours of human activity recorded with a new wearable system. The approach used to create the dataset, presented in an arXiv preprint , combines video from a wearer's viewpoint with reconstructed body movements, recorded finger movements, available tactile data and descri
Open source S4
of this coordination." Overview of EgoExplorer. Credit: arXiv (2026). DOI: 10.48550/arxiv.2610.18872 Recording data while people complete tasks Zhou, Jiang and their colleagues developed a new wearable system that can capture data while humans are completing different activities. This system combines a head-mounted GoPro HERO13 camera with five VIVE Ultimate Trackers, sensor-equipped MANUS Metagloves or Wuji gloves,
Open source S6
and custom 3D-printed mounts and straps. The head-mounted camera records video from a wearer's viewpoint. The five trackers, one placed on a wearer's head, one at the back of each hand and one at each ankle, track their own position and orientation. The sensor-equipped gloves can record the movement of a wearer's fingers, with the Wuji gloves also recording tactile data. Finally, the mounts and straps were used to s
Open source S7
ecure the equipment. Using this wearable system, the researchers recorded 100 hours of data while human wearers completed a wide range of activities. The different data collected were combined into a new dataset called EgoExplorer. "With an emphasis on professional activities, its continuous recordings align egocentric video with reconstructed whole-body motion, fine-grained finger articulation, and tactile observati
Open source S8
ject interaction or good performance on unfamiliar tasks. Moreover, the study did not test deployment on physical robots. "Against tracker-fitted references, the method reduces body-point error by 71.9% relative to the strongest baseline; matched evaluations also show gains in hand stability and contact estimation," wrote the authors. "Preliminary studies explore multimodal generation and conditioning alongside human
Open source S13
xperiments, jointly generating video, movement and touch data was found to produce better results than generating each separately. The researchers also trained a controller for a simulated Unitree G1 humanoid robot to follow movements adapted from two of the human demonstration recordings they collected. The simulated robot followed both movement sequences without falling, although this did not demonstrate precise ob
Open source S12
Implications for data suppliers
RoboSignal interpretation and collection questions, not statements of buyer demand.
- Confirm the required data type and collection setting with the buyer; this source does not establish a complete collection specification.
- Compare the reported units and scope before using these quantities in a budget. Recording hours, sensor-hours and trajectories are different measures.
- Validate demand and acceptance criteria with a buyer before scaling. Publication, popularity and a research result do not establish a purchase commitment.
Source：Tech Xplore — Robotics · techxplore.com