TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

Pick up a raw egg with your fingers, applying just the right amount of force: you need to prevent it from slipping without crushing it. Twist a damp towel, and the force applied by your fingertips must also adjust continuously as the water is squeezed out. For humans, these actions are almost instinctive. However, if relying solely on vision, a camera can capture the trajectory of the hand but cannot detect the position of finger contact or the pressure applied.

In recent years, first-person human video has rapidly expanded to tens of thousands of hours in academia, and an increasing number of works have begun to see the benefits brought by the growth in data scale. Visual-tactile data, however, is still far from reaching such a scale: previously published human visual-tactile datasets mostly range from several hours to dozens of hours. If simply merging multiple small datasets, the tactile sensors, data collection processes, and data formats differ across different institutions. Although the data volume has increased, the hardware and processes have also changed, making it difficult to determine exactly how much contribution 'more data' has made.

Recently, researchers from 15 institutions including Texas A&M University, Google DeepMind, Carnegie Mellon University, Stanford University, University of California, Berkeley, Yale University, Microsoft, NVIDIA, University of Liverpool, Meta, University of Washington, Northwestern University, Sony, Georgia Institute of Technology, and Overfit Lab (a data provider) released TouchScale A dataset built under a unified sensor and unified acquisition and synchronization process500-hour human vision-tactile dataset.

The team hopes to answer a key question: if instead of concatenating data from different sources, they build within a single sensing and data collection system high-quality high quality First-person vision-tactile data has truly expanded to hundreds of hours. Will it, like large-scale video, also show scaling gains as data scale increases? These can be further translated into robotic manipulation capabilities?What does TouchScale bring?

What does TouchScale bring?

1.500-hour unified wearable device acquisition TouchScale contains 500 hours of human hand visual-tactile synchronized data, all collected by the same wearable system, simultaneously recording head RGB-D, dual wrist RGB, and full hand tactile signals.

2. Human 'touch sense' can be transferred to robots Without human action labels, nor mapping human hand actions to robots, using only human visual-tactile data for mid-training, the average success rate of robots performing 4 real-world contact tasks rose from 22.5% to 57.5%.

3. Hundreds of hours of human visual-tactile data begin to show clear Scaling gains TouchScale used for mid-training, real-world operations become stronger overall as data scale increases; in independent zero-shot tactile prediction experiments, the same upward trend was also observed. Scaling gains extend from perception all the way to robot control.

• Title: TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning
• Paper link: https://arxiv.org/abs/2610.10288
• Project homepage: https://touch-scale.github.io/
• Dataset address: https://huggingface.co/datasets/2077AIDataFoundation/TouchScale

500 hours of tactile data, the challenge is not just about ``collecting a lot''

For TouchScale, the real challenge is not simply stacking hours to 500, but continuously expanding under the same set of sensors, the same synchronization and acquisition process. Previously published human vision-tactile datasets are mostly under 30 hours; if multiple small datasets are directly concatenated, different hardware and acquisition processes will vary along with the scale of data, making it difficult to isolate the impact of scale itself.

TouchScale therefore fixes the hardware at the acquisition end. A head-mounted RGB-D camera records the overall operation process and scene depth, while RGB cameras on both wrists record the detailed interaction between the hand and object at close range; full-hand tactile gloves are worn on both hands, each containing 880 tactile sensor units (taxel), covering the five fingers and palm, used to record the normal force during contact.

In order to ensure the quality of large-scale tactile data, TouchScale controls data quality from both hardware and algorithm sides. At the acquisition end, high-precision, low-noise flexible skin tactile sensors are used to stably record pressure changes of fingers and palms during interaction; after acquisition, a VLM-assisted automatic quality inspection process is applied, using only the wrist video to determine grasping, releasing, and participating fingers, and cross-comparing with actual tactile responses. Obvious sensor failures are directly filtered out, while uncertain samples such as single-finger missing, severe occlusion, or insufficient visual evidence automatically enter manual review.

What was actually collected in 500 hours?

TouchScale includes approximately 87,000 interaction recordings, about 2,000 task descriptions, and over 1,500 objects, covering 9 types of acquisition environments, over 800 scene configurations, and completed by about 20 data collectors. Tasks include both daily operations and more granular contact and tool usage. For example, pouring water from a kettle into a wide-mouth bottle requires continuous adjustment of the container's orientation; rolling up a notebook charging cable and securing it with a Velcro fastener involves hand coordination and continuous contact; sealing the top seam of a paper box with tape includes actions such as grasping the tool, positioning, and continuous pressing. Different tasks, objects, and execution methods also bring different hand-object poses and tactile contact patterns.

Compared with existing public human vision-tactile data, TouchScale not only expands the scale to 500 hours, but also maintains a unified set of sensors and acquisition process, and provides full-hand tactile acquisition for both hands, with each hand containing 880 taxels. This provides the conditions for studying the impact of data scale itself.

From Perception to Real Machine: What Does 500 Hours of Data Bring?

Same 16 hours, better cross-sensor generalization

The first set of experiments examines zero-shot cross-sensor tactile prediction. The model predicts hand tactile based on first-person view and wrist video, then directly tests on EgoTactile collected by another set of tactile gloves, without using target data fine-tuning. Since the sensor layouts of different gloves differ, results are mapped to a unified 12 hand anatomical regions during evaluation.

First, keep the training data volume at around 16 hours: the model trained on the TouchScale subset achieved a contact IoU of 0.181, which is higher than EgoTouch's 0.134. This comparison eliminates the factor of “more data,” indicating that under the same training volume, TouchScale is also more beneficial for cross-sensor generalization.

Tactile supervision can also help vision in reverse

The second experiment used tactile data as a supervisory signal for the visual encoder, then transferred the learned visual representations to the action recognition task. Researchers pre-trained using TouchScale, OpenTouch, FEEL, and EgoTouch, while keeping the same initialization and training budget.

On the three benchmarks MECCANO, Something-Something V2, and Ego-Exo4D, the visual encoder pre-trained with TouchScale achieved the highest accuracy in both linear probing and end-to-end fine-tuning settings compared to those pre-trained with OpenTouch, FEEL, and EgoTouch.

Finally, see the real device: Can humans' 'touch sense' be used by robots?

Real robot experiments use xArm6 robotic arm and BrainCo Revo 2 dexterous hand, setting up four contact-intensive tasks: soft-hard object sorting, bottle cap removal, test tube transfer, and whiteboard wiping. Each task uses 50 robot demonstrations and undergoes 20 tests.

The control setup is simple: both versions start from the same N0-VTLA checkpoint, use visual and tactile input, and employ exactly the same robot demonstrations for post-training. The only difference is that one version undergoes TouchScale mid-training before post-training.

TouchScale mid-training does not use human action labels, nor does it map human hand trajectories into robot actions. It adopts the future tactile prediction objective from N0-VTLA, allowing the model to predict subsequent tactile changes based on current visual, linguistic, and tactile inputs. In other words, what the robot learns from human data is not 'how the hand should move,' but 'how contact will occur.'

After joining TouchScale mid-training, the average success rate of the four tasks increased from 22.5% to 57.5%, with all four tasks seeing improvements:

• Soft-hard object sorting: 10% → 60%
• Bottle cap removal: 40% → 70%
• Test tube transfer: 30% → 60%
• Whiteboard erasing: 10% → 40%

Can the effect continue to improve with more data?

What's truly exciting is that as the scale of TouchScale data continues to grow, both the model and the robots are still improving.

In the tactile prediction task, the model and evaluation setup remain unchanged, with only the training data being increased. After TouchScale expanded from 10% to 100%, the cIoU increased from 0.311 to 0.383. In other words, the more human visual-tactile interactions the model has seen, the more accurate its contact prediction can be when faced with a tactile sensor it has never seen before.

And this trend hasn't stopped at the perception level. The real-world demonstration data used by the robot remains unchanged, and with only the addition of TouchScale mid-training human data, the average success rate of the four tasks increased from 22.5% to 57.5%.

In conclusion: Tactile has reached the starting point of Scaling

What TouchScale wants to answer is, under the premise of consistent sensors and data collection process, what can be achieved by simply expanding the scale of human visual-tactile data. Now it seems the answer is already quite clear: as the data volume increases, tactile prediction across sensors becomes more accurate overall, and robots trained with TouchScale mid-training also gain stronger real-world operational capabilities.

This means that touch is no longer just an expensive, scattered add-on modality in robotic systems. It can also be scaled for collection and learning, just like first-person video, and further transformed into physical interaction experience needed by robots. More importantly, TouchScale demonstrates not just a larger dataset, but a new path for data expansion: when the 'sense of touch' in human operations is systematically recorded, robots can gain transferable physical interaction experience from these unlabeled human experiences.

Of course, this path has just begun. The current real-world experiments only cover one robot platform and four contact-intensive tasks. There are still many questions worth further study about how human tactile experience is transformed into better robot control. But TouchScale has already sent a clear signal: the scaling path that vision has already taken is now also being taken by touch.

Author Introduction

The co-first authors of this paper are Dayou Li, a Ph.D. student at Texas A&M University, Hao Wang, a senior research engineer at Google DeepMind, Qianqian Yang, a Ph.D. student at Carnegie Mellon University, and Zihao Zhu, a Ph.D. student at Texas A&M University. The corresponding author is Zhiwen Fan, an assistant professor at Texas A&M University, whose research focuses on machine learning and embodied intelligence.

This article is from WeChat Official Account“Machine Heart” , 36kr published with authorization.