MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding
Overview
This paper introduces MotorSense, the first large-scale dataset pairing egocentric video with bimanual surface electromyography (EMG) and IMU signals across 100 hours of unscripted manipulation. The authors argue that EMG provides a dense, occlusion-free signal for capturing finger movement, hand configuration, and contact forces, offering a scalable alternative to manual annotation. Using MotorSense, they demonstrate sensorimotor alignment via cross-modal pre-training between video and EMG, which improves performance on action recognition, 3D hand reconstruction, and 3D contact reconstruction compared to baseline self-supervised methods. Additionally, the dataset supports generative applications where EMG signals condition a video diffusion model for fine-grained hand manipulation videos.
Generated from attributed reports · Updated 2 hours ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · hours of unscripted manipulation: 100 hours · Basis not reported
“IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producin”
Exact source · revision 1Source owner not reported
Reported quantity · action-centric computer vision tasks: 4 other · Basis not reported
“to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.”
Exact source · revision 1Source owner not reported
Reported quantity · popular video datasets: 8 other · Basis not reported
“to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.”
Exact source · revision 1Source owner not reported
Reported quantity · downstream tasks: 3 other · Basis not reported
“g an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalMotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding
This paper introduces MotorSense, the first large-scale dataset pairing egocentric video with bimanual surface electromyography (EMG) and IMU signals across 100 hours of unscripted manipulation. The authors argue that EMG provides a dense, occlusion-free signal for capturing finger movement, hand configuration, and contact forces, offering a scalable alternative to manual annotation. Using MotorSense, they demonstrate sensorimotor alignment via cross-modal pre-training between video and EMG, which improves performance on action recognition, 3D hand reconstruction, and 3D contact reconstruction compared to baseline self-supervised methods. Additionally, the dataset supports generative applications where EMG signals condition a video diffusion model for fine-grained hand manipulation videos.
Event coverage history
There is not enough continuous observation data to show a trend.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).