Fysiverse-UniWAM: A Unified World Action Model for Navigation and Manipulation Without Large-Scale Robot Pre-Training

Editor: Chen Chen

Robots are no longer just able to know where to go, but also know where to stop, what direction to face the target, and even after stopping, can directly start working.

It sounds obvious, but for robots, this is actually a long-standing challenge.

For example, letting a robot clean up cups on the table. Being able to walk to the table doesn't mean it can pick up the cup. If the base stops slightly further away, the robotic arm may not reach; if the direction is slightly off, the subsequent grasping may also be affected.

How can robots not only reach the destination, but also stop accurately, pick things up, and even perform more complex dexterous operations?

Recently, a domestic team has provided a new and interesting approach.

Fysics AI, in collaboration with the Cognitive and Intelligent Technology Laboratory (CITLab) at Fudan University, proposed a Unified World Action Model Fysiverse-UniWAM, attempting to use a single model to integrate autonomous navigation, mobile manipulation, and dexterous operations at fixed workstations three types of originally relatively independent capabilities.

More notably, this model did not follow the traditional route of starting with large-scale robot pre-training.

The team started from pre-trained video world models, achieved joint learning of navigation and manipulation through shared backbone networks, and proposed MAP (Manipulation Anchor Pose, operation anchor pose), building a dataset called MAP-Data with over 1.5 million trajectories, accumulating approximately 7,500 hours.

What is the actual performance of this different technical route?

The results were somewhat surprising.

In the MAP-Bench benchmark test, Fysiverse-UniWAM achieved an average position error of only 9.85 centimeters, and an average orientation error of 2.30°compared to the strongest external baseline, it was reduced by 30.1% and 44.0%.

What's more interesting is that the team directly deployed the model to 4 types of robot platforms, conducting coverage of autonomous navigation, mobile operation, and fixed workstation operation 24 real robot tests.

Among them, In 13 navigation tasks, the position error was all leading, and in 4 mobile operation tasks, the success rate was all first; in complex fixed workstation operations, the overall performance is also close to the large-scale robot pre-training π₀.₅.

Don't rush to look at the technical principles, let's directly see the real robot performance.

24 real robot tests, how strong is a single model?

Don't rush to see the technical principles, let's go straight to the real robot.

Even when changing the target, scene, or robot platform, the model can still complete the corresponding navigation tasks.

Researchers moved the robot to a position far from the target, changed the starting point and orientation, and the robot could then autonomously plan the approach route based on the current environment, adjust the chassis position and direction, and finally accurately dock.

Even when changing the target, scene, or robot platform, the model can still complete the corresponding navigation tasks.

The significance of accurate stopping is to create suitable conditions for the subsequent robotic arm operations.

In 13 MAP navigation real robot tests, UniWAM's position error was first in all tasks, and the orientation error achieved the best in 11 of the tasks.

In mobile manipulation tasks, the robot needs to autonomously approach the target, adjust the base, and then control the robotic arm to complete grasping and placing.

For example, from moving to the table edge, to reaching out to pick up the cup and store it, the entire process is autonomously completed by the model.

In the case of color block classification, the robot needs to identify the target color, coordinate the base and robotic arm, and place the block in the corresponding tray.

These tasks truly test the ability to connect navigation and manipulation, rather than just completing a single action.

When switching to color block classification, the robot needs to identify the target color, coordinate the chassis and robotic arm, and place the block into the corresponding tray.

These tasks truly test the ability to connect navigation and operation, rather than just completing a single action.

Item-by-item evaluation results of 4 mobile manipulation tasks in Table 7 of the technical report UniWAM-Scaled achieved first place in all 4 mobile operation tasks, with an average success rate of 80.0%, leading the current strongest robot pre-training baseline π₀.₅ by 17.5 percentage points.

Evaluation results for 4 mobile operation tasks in Table 7 of the technical report

First is a

Pick Up Anything one-take interaction demonstration

First is a segment As the human hand moves to different positions in three-dimensional space, the robotic arm continuously perceives the hand position and adjusts the delivery action accordingly, completing precise delivery. Wherever the hand reaches, the robotic arm will deliver the object there..

Even more challenging operations are not a problem.

When human hands move to different positions in 3D space, the robotic arm will continuously perceive the hand position and adjust the delivery action accordingly to complete precise delivery. Wherever the hand reaches, the robotic arm will deliver the object there.

In the task, the robot needs to coordinate both arms, precisely grasp, and place layer by layer, ultimately building a three-layer cup tower.

In High-Precision Test Tube Insertion In the task, the robot needs to control the position and orientation of the robotic arm's end-effector to complete precise alignment and insertion, which imposes higher requirements on spatial precision.

In And In the task, the robot needs to control the position and orientation of the robotic arm's end-effector to complete fine alignment and insertion, which imposes higher requirements on spatial precision.

. Long-range coffee making In the task, the robot also needs to continuously complete multiple operation steps, posing higher challenges for long-term task planning, action coordination, and execution stability.

From generalization grasping, dual-arm collaboration to fine insertion, Fysiverse-UniWAM demonstrates rich dexterous operation capabilities.

In 7 real robot evaluations of fixed workstation tasks, the model exceeded π₀.₅ in 3 tasks, with an overall success rate of 68.9%, only lagging π₀.₅ by 2.5 percentage points.

Success rate comparison for 7 fixed workstation operation tasks in Table 7 of the technical report

Overall, looking at the 24 real robot tasks, Table 6 of the technical report summarizes the overall performance of three types of tasks: UniWAM-Scaled achieved the best performance in MAP navigation and mobile operation docking accuracy, ranked first in mobile operation success rate, and second in fixed workstation operation success rate.

Table 6 of the technical report shows the overall evaluation results of the 24 real robot tasks

A unified model that has not undergone large-scale robot pre-training can not only lead in navigation and mobile operation, but also handle such rich dexterous operations. What is the technical secret behind this?

Without relying on large-scale robot pre-training, what path did UniWAM take?

In recent years, large-scale robot pre-training, represented by π₀.₅, GR00T, has become an important technical route for embodied intelligence.

However, real robot data is not easy to obtain. Especially complete mobile operation trajectories, which require continuous collection of the entire process from movement, docking to robotic arm task execution, are costly and complex.

More importantly, navigation and operation themselves have significant differences: the former focuses on environmental layout and chassis movement, while the latter focuses on close-range object interaction and fine control of the robotic arm.

So, can navigation and operation learn together in a single model without relying on massive complete robot operation trajectories?

To this end, the team split the problem into two levels.

The first question: How does the robot know where the suitable navigation endpoint for operation is?

The second question: Since navigation and operation data and actions are different, how can they learn together in a single model?

The corresponding answers are precisely MAP and Mixed-Stream.

Challenges faced by mobile operation and Fysiverse-UniWAM's solutions.

MAP: Let the navigation endpoint become the starting point for operation

Traditional semantic navigation typically only requires the robot to arrive near the target, but for mobile operations, 'arrival' is far from sufficient.

The robot also needs to know where the chassis should stop and in what direction it should face the target, in order to facilitate the mechanical arm's subsequent movements.

For this, the team proposed MAP (Operational Anchor Pose) to clearly define suitable reference positions and orientations for subsequent operations as navigation learning targets.

More importantly, MAP supervision data can be massively automatically generated.

The team uses HM3D, InteriorGS, and proprietary large-scale scene data, combined with scene geometry information, visual language models, and automatic path planning, to generate reference docking poses and approach trajectories for different operational targets.

Each trajectory not only includes visual observations and navigation actions, but also records target bounding boxes, MAP image projections, and other geometric supervision information.

MAP-Data automatic generation process.

The final MAP-Data contains over 1.5 million trajectories and approximately 7,500 hours of data.

This means that robots do not need to fully rely on real mobile operation demonstrations to learn how to move to a suitable position for manipulation.

But solving 'where to stop' does not yet mean solving 'how to learn together'.

Mixed-Stream: Navigation and Operation Separated, Model Capabilities Shared

The visual information, action space, and control methods required for navigation and operation have significant differences.

Directly concatenating the two types of actions together may increase the difficulty of unified learning; separately training two models makes it difficult to fully share capabilities.

Fysiverse-UniWAM adopts a different approach: Two data streams are independent, with a shared core backbone network.

Architecture of Fysiverse-UniWAM Mixed-Stream.

The navigation stream specializes in processing environmental observations and chassis actions, while the operation stream specializes in multi-view observations and robotic arm actions.

The two data streams have their own input encoders and action output heads, yet share a unified video DiT and action DiT backbone network, achieving collaborative modeling of visual dynamics and action generation through Mixture-of-Transformers (MoT) mechanism.

More importantly, Navigation data and operation data do not need to come from the same complete mobile operation trajectory.

During training, the two types of data can be independently sampled and batched for input, jointly updating the shared backbone; during inference, the two data streams can both work independently and execute in parallel.

This allows massive navigation data and fixed workstation operation data to jointly participate in unified model training, reducing the reliance on complete mobile operation demonstration data.

In addition, the model introduces image plane auxiliary supervision: the navigation stream predicts the target bounding box and MAP projection position, while the operation stream predicts the 2D projection trajectory of the robotic arm end-effector, further strengthening the connection between visual information and action prediction.

Visualization of Navigation MAP Prediction and Operation End-Effector Trajectory Prediction

At this point, the technical logic of UniWAM becomes clear:

MAP solves where the robot should stop, Mixed-Stream solves how navigation and operation can learn together.

Combining the two, the originally relatively independent navigation, mobile operation, and fixed workstation operation can be collaboratively modeled within a unified world action model.

How effective is this approach from benchmark testing to zero-shot transfer?

To verify this design, the team built MAP-Bench, which includes 15,000 evaluation trajectories, and compared it with models such as π₀.₅, GR00T N1.7, FastWAM, Uni-NaVid, and InternVLA-N1.

The results show that UniWAM-Scaled achieves the lowest docking error in short, medium, and long-distance tasks, with an overall average position error of 9.85 centimeters, and an orientation error of 2.30°.

Compared to the strongest external baseline, the position error is reduced by 30.1%, and the orientation error is reduced by 44.0%.

Table 4 in the technical report shows the performance comparison of various models in MAP-Bench

Moreover, the advantage is not entirely due to additional data.

Without introducing MAP-Data, UniWAM-Base already achieved results better than external baselines; adding MAP-Data further improved docking accuracy.

Architecture comparison experiments also show that Mixed-Stream, compared to unified action Token, independent action experts, and sequence concatenation, can improve the success rate of mobile operations and docking accuracy.

More interestingly, there is zero-shot true robot transfer.

The team used only MAP-Data to train UniWAM-Sim, without adding real robot demonstration data, and then directly deployed it to two real robot platforms.

Ultimately, the model achieved a position error of 18.7 centimeters and an orientation error of 6.34°, demonstrating the potential of transferring from generative scene data to real robots.

Table 8 in the technical report shows the zero-shot true robot transfer results of UniWAM-Sim

This further indicates that improving robot capabilities does not have to rely solely on expanding the scale of real robot pre-training.

Unified robot capabilities, not only through expanding pre-training scale

What is truly worth paying attention to in Fysiverse-UniWAM is not only the leading performance in navigation and mobile operation metrics.

More importantly, it demonstrates a different path for building robot capabilities.

Through MAP, navigation is truly oriented towards subsequent operations; through Mixed-Stream, dispersed navigation and operation data can jointly learn within a unified world action model.

This does not mean that large-scale robot pre-training is no longer important, but it shows that it may not be the only prerequisite for building a unified robot capability.

From precise navigation to mobile grasping, to dual-arm collaboration and fine manipulation, Fysiverse-UniWAM demonstrates the breadth of a unified model's capabilities through 24 real robot tests, also proving the potential of this technical route.

When embodied intelligence is no longer just about the scale of robot data, how to truly integrate different capabilities may be just as important as the path to general-purpose robots.

Finally, here are the project and open-source resources:

Project Homepage and Demonstration

https://fysics-ai.github.io/uniwam.github.io/ 

Technical Report

https://arxiv.org/abs/2609.39388 

GitHub Code

https://github.com/Fysics-AI/UniWAM 

Hugging Face Navigation Weight

https://huggingface.co/Fysics-AI/UniWAM 

MAP-Data Dataset

https://huggingface.co/datasets/Fysics-AI/MAP-Data 

MAP-Bench Data and Evaluation Trajectory

https://huggingface.co/datasets/Fysics-AI/MAP-Bench