Ant Group's LingBot-Touch: An Open-Source Tactile Foundation Model for Robotic Manipulation

Jin Lei, from Afni Temple

Qubit | Official WeChat Account QbitAI

It's way too different.

Clearly gave

Robot

installed

touch

, how did it become dumber...

△

Image generated by AI

The matter is like this.

Recently, a team conducted an experiment, they

Sharpa Wave dexterous hand

installed tactile sensors on all five fingers, the task was to let it zip up a bag.

The motion model they chose was

π0.5, under the condition of relying only on the camera, the success rate reached 39%.

Then the team fed the readings of the fingertip sensors to it as they were; logically speaking, the robot gained another sense, and zipping the zipper should have been more effortless.

But guess what.

AUV (Aiyouwei)... the success rate directly

Dropped to 15%!

And this is not an isolated case.

Similarly, with the same combination of Sharpa Wave dexterous hand and π0.5, after connecting the original tactile sensors to do

handover box

For the task, the success rate dropped from 57% to 48%.

Even if the dexterous hand was replaced with XHand, the average success rate across the 6 tasks dropped from 49.71% to 48%.

So, what is the actual reason why the robot became clumsy despite having an additional sense?

The answer is, the problem lies in the matter of reading the values.

Because tactile data is not quite the same as other data, and the robot's fingers are not in contact with objects most of the time, the readings are almost a large area of “0”; even when they begin to contact an object, the signals are sparse and noisy.

In addition, different sensors each perform their own duties: visual tactile sensing is responsible for the deformation of the soft rubber on the fingertip, magnetic tactile sensing is responsible for measuring the magnetic field, and piezoresistive tactile sensing is responsible for recording resistance... the data are not all the same.

△

How three types of tactile sensors perceive contact, and what are the trade-offs for each

For example, for

gripping a bottle cap

For this task, the three sensors can provide three completely different numbers, the action model

not only is it incomprehensible, but it also has to be used forcefully... it can be said that it is doing more harm than good.

Then how should this situation be broken?

Ant Group Lingbo

Provided a solution—adding a “translation” between the sensor and the action model.

(Note: The experiment above was done by Ant Lingbo.)

Simply put, it is first translating the data read by various sensors into a type that the action model can understand, such as where the finger touched, how much force was used, which way the force deviated, and whether there was any slipping, and then handing it over.

Still that π0.5, still that zipper task, after such operations, the success rate directly soared to

59%, even 20 percentage points higher than not using touch at all; the success rate of the task of handing over a box also returned from 48% to 63%.

And this “translation” is what Ant Lingbo is today

The latest open-source

tactile pre-trained foundation model LingBot-Touch.

With its support, the action of unscrewing a bottle cap that we just mentioned becomes very smooth:

One must know that the place where it is very easy to fail when unscrewing a bottle cap is precisely that small area where the fingertip and the bottle cap stick together: if squeezed slightly looser, the fingertip will slip on the cap surface, and even if the wrist turns half a turn, the cap remains motionless. And this scene is exactly what the camera cannot see clearly.

So, how exactly is this “translation” of LingBot-Touch achieved?

Dexterous hand needs a translation

Let's first clearly explain the work done by LingBot-Touch.

Simply put, it is responsible for receiving signals from three types of sensors: magnetic tactile, visual tactile, and piezoresistive tactile, and then

uniformly encoding them into a type of tactile Token, from which it reads the contact position, the magnitude of force, and how the direction of force changes over time.

Its position is perfectly caught in the middle, connecting different sensors and dexterous hands downwards, and handing over these Tokens to different motion models upwards.

In this way, no matter how diverse the signals from various sensors are, they can all be uniformly understood when they reach the motion model, which can be described as different paths leading to the same destination.

△

LingBot-Touch connects multiple tactile sensors with real dexterous operations

Moreover, the lightweight version of LingBot-Touch

LingBot-Touch-Tiny

Only about

5.32 million parameters, and currently the technical reports, code, and model weights have all been open-sourced.

So, how was LingBot-Touch trained?

Overall, the Ant Lingbo team mainly divided it into three steps.

The first step is to unify the coordinates.

The first challenge the team encountered was that the number and arrangement of contact points of different sensors were simply too different. For example, some dexterous hands have 116 contact points on the thumb alone, while some fabric gloves have only a small 3×3 array.

If they were matched strictly by number, they certainly would not align.

For this reason, the team first mapped all the contact points of the sensors into the same standardized fingertip space, and then assigned an “address” to each Token, which is jointly composed of time, finger, and three-dimensional position.

Addresses will enter the model through Rotary Position Embedding (RoPE). In this way, the model no longer requires a fixed number of contact points, and it can still be read even if a different layout is used.

On this basis, the three types of sensors each have a lightweight entry responsible for converting raw readings into Tokens; behind the entries, they share the same tactile Transformer backbone.

In other words, the differences between sensors are handled by their respective entries, while the patterns of the contact itself are placed in the backbone to be learned together.

△

LingBot-Touch overall architecture, three types of sensors with separate entries and a shared backbone

The second step is to make a prediction in the feature space.

After unifying the coordinates, the team encountered the second pit, which is that large area of “0”s we mentioned at the beginning.

If you let the model restore the original readings, it will quickly find a shortcut; as long as it learns the “no touch” background well, the loss can be reduced very low. But in this way, although the score looks good, not much of the truly useful contact information is learned.

So LingBot-Touch chose the JEPA route.

Specifically, during training, the model randomly masks

90%

the contact points, and they are masked for the entire duration; then, the model must predict the representations of the masked parts in the feature space based on the remaining 10% of changes, without needing to restore readings point by point.

Under such operations, the effect was immediate; using the same 1000 hours of data for training, the error of LingBot-Touch in judging the contact center is 0.90 millimeters, while VAE and MAE are both above 3 millimeters.

For example, using a segment of touch to find the corresponding video frame, the top-5 recall rate of LingBot-Touch is 32.07%, while the other two methods are both less than 6%.

△

The same data, comparison of the effects of feature space prediction and restoration of original readings

The third step is expanding the data to prevent the model from rote memorization.

The pre-training data of LingBot-Touch has a total of

4983 hours

、approximately

1.33 billion frames. Two-thirds of these come from people wearing haptic gloves doing work, and the remaining one-third come from teleoperating the Sharpa Wave dual-arm robot.

The reason for using gloves in large quantities is that it does not require a robot, and one person can record several hours in a normal environment in a day. Moreover, these data only labeled task categories such as on, off, assembly, and pouring water, and not a single frame labeled the magnitude of force.

△

Composition of pre-training data, covering three types of sensing principles

However, even if enough data is accumulated, the team still has to guard against the model “slacking off.”

Because there are only three sensor layouts in the pre-training, the model will likely memorize these three sets of contact positions directly and then be at a loss when it encounters a fourth layout.

Therefore, during training, the team will randomly change the sampling density of the contacts and then slightly move the position of each contact, forcing the model to learn the underlying patterns of contact itself.

After these three steps are completed, it is time to move on to real-machine testing.

The tests covered two models of dexterous hands, among which

High degrees of freedom

Sharpa Wave does

20 tasks

tasks, low degrees of freedom XHand does 6 tasks, a total of 26 tasks.

There are four action models participating in the test, which are Diffusion Policy, π0.5, LingBot-VLA, and T-Rex. Each model must be tested under three settings: without tactile, directly feeding raw signals, and connecting to LingBot-Touch.

As a result, on both hands, the group connecting to LingBot-Touch had the highest average success rate for all four models.

Compared to without tactile, the success rates of T-Rex, π0.5, and LingBot-VLA increased by 23.18, 21.81, and 24.81 percentage points respectively, while the cost incurred was only an increase in the number of parameters of 0.13% to 0.16%.

△

Average success rates of the four action models under the three settings on the two dexterous hands

Take inserting a straw with one hand as an example; the difficulty of this task lies in that one moment of piercing the seal. As the straw presses against the transparent seal and pushes down, the resistance becomes greater; once the film breaks, the hand will suddenly loosen.

It is difficult for the camera to distinguish whether the film is being pressed inward or has already been pierced, but the fingertips can. If the force is released one step too slowly, the straw will go directly to the bottom of the cup.

Of course, what best illustrates the problem is the XHand set of experiments.

The magnetic tactile sensors used by XHand are models that never appeared in the pre-training data; furthermore, the magnetic tactile data in the pre-training all came from human-worn gloves. When connecting to XHand, LingBot-Touch did not change even a single parameter.

Even so, the average success rate of π0.5 on the 6 tasks still rose from 48.00% when directly feeding the original signal to 74.17%.

Among them, for the task of separating stacked cups, the success rate more than doubled from 32.5% to reach 72.5%.

This task requires the dexterous hand to only take away the topmost cup; it must use one hand to hold the entire stack of cups, while the other hand pinches the cup rim and pulls outward. If the resistance suddenly becomes smaller while the fingertip is still firmly pressed against the cup rim, it indicates that the cup has already loosened and the force must be retracted immediately; if the fingertip itself slips on the cup rim, then it should be regripped.

These two situations have completely different tactile sensations.

In other words, the experience accumulated by human hands helped the robot understand a hand it had never touched before.

In addition to this, the team also compared LingBot-Touch with the previous tactile foundation model FTP-1.

When judging the contact center on Sharpa Wave, the error of FTP-1 was 2.19 millimeters, while LingBot-Touch was 0.12 millimeters; on XHand, the error also dropped from 5.89 millimeters to 0.16 millimeters.

It should be noted that FTP-1 was equipped with specific adapters on these two hands respectively, while LingBot-Touch used the same frozen encoder.

△

Comparison with FTP-1 on two types of dexterous hands

Additionally, LingBot-Touch's

Scaling Law

has also been preliminarily revealed.

The Tiny version begins to saturate at around 4000 hours, and when switched to the larger capacity Small version, the effect continues to rise up to 5000 hours; the success rate of real-machine tasks also steadily improves with the increase of pre-training data.

△

The impact of pre-training data volume and model scale on performance

Completing the puzzle of touch

After looking at the technical details, if we put LingBot-Touch back into the entire technical roadmap of Ant Lingbo, we can see its position more clearly.

Before this, Lingbo's general brain has already laid out three capabilities: spatial perception has LingBot-Vision and LingBot-Depth 2.0, environmental interaction has LingBot-Video and LingBot-World 2.0, and dexterous manipulation has LingBot-VLA 2.0 and LingBot-VA 2.0.

It can be seen that these few capabilities are mostly solving

before touching

matters, such as where the object is, where the hand should reach, and what to do next.

However, once the finger touches the object, the information that vision can provide is reduced, for example, the place where the fingertip is pressed against the bottle cap will be blocked by the hand, and after the key is inserted into the keyhole, nothing can be seen at all.

What LingBot-Touch fills in is exactly this part. To summarize in one sentence, it is

visual pipe before contact, tactile pipe after contact.

Unlocking with both hands is a typical example.

One hand holds the padlock, while the other hand holds the key and aligns it with the keyhole; this step mainly relies on the eyes; but once the key is pushed in, it might push the lock crooked, get stuck halfway, and then you have to pull it back a bit, change the angle, and try again.

Once the key entered the keyhole, the camera basically could no longer help, and the dexterous hand could only rely on the resistance transmitted back to the fingertips from the key to make judgments.

Seeing this, perhaps some friends will ask, why does Lingbo make tactile sense into a separate basic model?

The reason is that the control method for each hand is different, but they all must understand contact.

If the matter of “understanding contact” is taken out separately and made into a reusable layer, then the experience accumulated by one piece of hardware can become the starting point for the next robot.

This approach is actually in the same vein as what Lingbo did previously with LingBot-VLA.

VLA studies how the same brain drives robots of different configurations, which is "one brain, multiple machines"; here at Touch, the problem is pushed one step further, becoming whether the contact experience gathered from different hands and different sensors can be learned together and then applied to new robots.

And the previous set of results from XHand can be considered as providing a preliminary answer.

Several design choices of LingBot-Touch were also back-calculated from this goal. Using gloves to collect data is because gloves do not depend on the robot body, allowing the data scale to be expanded; designing the entry based on sensing principles rather than product models is to leave room for new sensors in the future.

This is also what Lingbo has always emphasized

Embodied native, that is, starting from how the robot perceives and acts in the physical world to decide how the model should be designed and what data should be used for training.

Currently, LingBot-Touch has been integrated with LingBot-VLA and LingBot-Depth to improve general brain-hand-eye coordination and the ability to complete delicate operations.

In terms of deployment, LingBo is also conducting commercial tests with hardware partners such as Leju and Tihuo, as well as scene partners like Guoda Pharmacy and Longsheng, in retail sorting, logistics sorting, and industrial scenes.

The stronger the large model, the more embodied native cannot be absent

After talking about LingBot-Touch itself, we will pull the perspective back a bit.

After GPT-6 Astra was released in September this year, the RoboDojo team had it run 42 simulation operation tasks under zero-shot conditions, and as a result, its comprehensive score ranked first among 43 entries under evaluation.

To be honest, this matter made the embodied circle quite panicked for a moment.

But in the same report, there is another set of numbers. GPT-6 Astra's success rate in the precision dimension was only 4%, while the best-performing public policy was 20.42%; for tasks requiring precise contact such as insertion and tower building, it was almost completely defeated, and real-machine testing was also halted due to safety issues.

The report attributed this gap to a lack of physical common sense, namely the understanding of contact, force, and collision.

And just today, Anthropic released Claude Fable 5.2, pushing the progress bar of large model robot control forward another step.

According to the internal test results released by Anthropic's official blog, Fable 5.2 reached an average success rate of

64.9% in RoboDojo simulation tests, which is more than double GPT-6 Astra (29.8%) under the same test framework, and also exceeds most AI systems specifically built for controlling robots.

In the official demonstration, the robot controlled by Fable 5.2 can also solve a Rubik's Cube, twist a bottle cap, and paint. However, what RoboDojo tests is controlling a simulated dual-arm robot by looking at camera footage.

From Fable 5.1 to Fable 5.2, over the two months in between, the success rate in the same set of tests rose from 27.6% to 64.9%. The general large model's abilities in understanding tasks, reasoning and planning, and even directly performing operations are becoming visibly stronger.

But in our view, this does not mean that embodied models should step aside and only focus on “doing.”

High scores in simulation and steadily finishing the job in the real world are separated by the generalization of the physical world. Change a hand, change a material, change a scene never seen before, can the robot still do the job well? This question cannot be bypassed; it must be tackled by the brain that grew up from the physical world itself.

CEO of Ant Intelligence Wave

Zhu Xing's judgment is also the same.

He believes that the progress of these digital world large language models, such as GPT-6 Astra and Claude Fable 5.2, indeed makes the complex task reasoning and physical operation capabilities of robots stronger, and Lingbo is also actively absorbing them; however, to truly complete the operation loop, "the hard bone of physical operation generalization still cannot be bypassed," and to a large extent still relies on the breakthroughs of the embodied foundation models themselves.

In his view, general large models and embodied foundation models are closer

1+1>2

in relationship, both breaking through individually and promoting each other, jointly accelerating the maturity of the embodied intelligence industry.

LingBot-Touch is Lingbo's attempt to gnaw at this hard bone.

Touch is a modality native to the physical world. There are massive amounts of text, images, and videos on the internet, yet almost no webpage records exactly how many Newtons of force were used when a thumb pinched a bottle cap. This kind of experience can only be accumulated bit by bit in real interactions, and it is precisely one of the foundations of the physical world's generalization.

For example, the Dexterous Hand

Touching mahjong tiles relies on this kind of experience.

When the two-dot and three-dot tiles are picked up, they are both circles; to distinguish them, one must feel how many circles there are in total and where each is arranged, and all these clues are hidden in the shallow carvings under the fingertips.

In the past, tactile sensing was always fragmented, with each sensor having a set of data and a set of models, and changing a hand meant starting over from scratch. However, LingBot-Touch proves that tactile sensing can also have a unified representation.

If we push it one step further, in this LingBot-Touch method—namely, unifying coordinates, separating entries, sharing the main trunk, and making predictions in the feature space—actually, no single link belongs solely to touch.

This line of thinking also runs through Ant Lingbo's entire embodied native route. According to the plan, Lingbo will continuously explore the upper limits of physical operation generalization, create a robot industry general brain across configurations, and push more physical modalities such as vision, depth, touch, and even temperature to be aligned in a unified model. As for what other sensing methods robots will have in the future, no one can say for sure now, but at least, there is already a path that can be referenced.

This is exactly what Ant Lingbo wants to do: create a layer of intelligence between hardware and scenes, converting the physical experience accumulated from different bodies and different sensors into shareable general capabilities.

Moreover, Ant Lingbo provides this layer of capability in an open-source, decomposable manner, and will continue to open-source it to the community while simultaneously advancing commercialization.

LingBot-Touch can be used independently, connected to action models from other companies like π0.5 and Diffusion Policy, and it is equally effective; sensor manufacturers can continue following their own technical routes, and complete machine manufacturers do not need to change sensors for a specific model.

Audi disclosed a figure earlier this year: the automation ratio of wire harness production and assembly across the entire industry is still less than 10%. This type of work involving edge contact and adjustment is exactly where dexterous hands have been slow to take over.

Looking at China, there are many types of dexterous hands and tactile sensors in China; in the past, this was a problem of non-interoperable data; but once experience can be shared, this diversity will instead become an advantage in terms of data.

In short, just because a robot hand has signals does not mean it will necessarily use them.

As Zhu Xing said, “the quality of modal input largely determines the quality of action output,” without good input, there is no good output.

The faster general large models run, the more the embodied native brain cannot fall behind. That ability for robots to perform tasks steadily and well in the physical world must, after all, rely on the embodied native brain, trained bit by bit in the real world.

Code link:

https://technology.robbyant.com/lingbot-touch

Technical Report:

https://github.com/Robbyant/lingbot-touch/tree/main/paper

GitHub:

https://github.com/Robbyant/lingbot-touch

Hugging Face:

https://huggingface.co/collections/robbyant/lingbot-touch

ModelScope Community:

https://www.modelscope.cn/collections/Robbyant/LingBot-Touch

One-click triple action

"Like","Share","Heart"

Welcome to leave your thoughts in the comment section!

—

End

—

🌟 Light up the star rating 🌟

Frontier advances in technology every day