Sinovation Tests GPT-6-Astra and Opens Sources SenseNova-RoboRSI: What's Next for Embodied Intelligence?

A robot failed to complete a task continuously5times, would you judge it“unable to do”?

SenseTime encountered an unexpected result during testing: GPT-6-Astra In the simulation evaluation, RoboDojo tasks were originally all9A task, originally all5times the original, resulting in10times all successful.5Some

were originally due to us pressing the stop button too early.“unable to do”But if we keep waiting, what is the cost? In the extended budget test,

the average time for stacking bowls was about minutes, with an estimated inference cost of131dollars; inserting a pen into a pen holder took an average of19dollars.263dollars; inserting a pen into a pen holder took an average of55The task was finally completed, but such a robot is still far from daily use. More interestingly, solving a problem doesn't always require continuous retries. In a real-world flower arrangement experiment,

when the bottle mouth was not visible, it moved the idle robotic arm and used the camera on the other hand to help see, then completed the insertion. It turned the second hand into an GPT-6-Astra were unable to see the bottle mouth, so it moved the idle mechanical arm, used the camera on the other hand to help itself see, and then completed the insertion. It transformed its second hand into a tool to help itself complete the task.“to help complete the task.”.

Which failures are worth giving another chance? Which difficulties require a different approach? And which problems, even with a sufficiently smart model, are hard to solve through repeated attempts?

SenseTime attempted to answer these questions through12simulation evaluations and8robot systems research, and simultaneously open-sourced the SenseNova-RoboRSI framework and27B embodied models, reducing the task completion cost to GPT-6-Astra the 1/50, and allowing the community to retest these findings and continue exploring more efficient solutions.

I. Exploring the upper limit of capability: some failures are due to stopping too early

In our research on the 12 simulator rankings, covering desktop robotic arms, humanoid robots, wheel arms, and dexterous hands, we found that by removing two limitations, we can further explore the GPT-6-Astra upper limit of capability:

Figure 1 By removing two limitations, the GPT-6-Astra upper limit of embodied capability is significantly increased

After removing the above two limitations, GPT-6-Astra further approached the current capability boundary in multiple evaluations. In the LIBERO series (relatively simple desktop robotic arm operations) and the SIMPLE (humanoid robots performing mobility and operational tasks), its performance has already fully saturated; in the recently focused RoboDojo tasks, the success rate also reached 81.9%.

Figure 2 In-context Learning in the middle, GPT-6-Astra what kind of“cheat sheet”?

ICL was prepared for itself. The process also revealed how the model thinks about an embodied problem. Before executing a task, GPT-6-Astra would study the images, robot states, and action trajectories from the training demonstrations, and organize useful information into task notes, reference actions, and code tools; during testing, it would combine on-site observations to call and adjust these materials. Usually, each task only needs to be prepared once, and then it can be used for multiple tests of the same task. Figure 2 summarizes the six capabilities formed during the ICL process.

Figure 3 The performance improvement shown by step growth across all rankings and the estimated performance upper limit for infinite steps

Figure 3 shows the performance changes after increasing the number of steps and the estimated performance upper limit under infinite steps. Under the assumption that the number of steps approaches infinity, GPT-6-Astra has already reached or approached performance saturation on a large number of rankings. This means that the current community rankings have, to some extent, been GPT-6-Astra pushed closer or even touched the capability ceiling; the research focus needs to shift from merely pursuing ranking scores to whether the capability can be transferred, whether the cost can be reduced, and whether the task can be reliably completed in real-world environments.

II. Bottlenecks for practical application: capable, but not affordable

GPT-6-Astra The demonstrated capabilities are surprising, but the ranking scores approaching the upper limit does not necessarily mean that embodied intelligence has solved the practical application problem. Combining experiments, we found that at least three types of bottlenecks need to be overcome.

Figure 4 GPT-6-Astra The average Token cost per task across all rankings (simple conversion can be considered as 1M Tokens = 1 USD) and time consumption

III. Fast and slow collaboration, the key is closed-loop

Therefore, a key question is: Can we use a Harness system to let System 2 slow-thinking model (GPT-6-Astra) and System 1 fast-thinking model (?0.5) work together, while keeping the performance as high as possible while controlling the cost?

Figure 5 Taking GPT-6-Astra As a reference, different collaboration methods are presented in the“performance—cost”quadrants. Experimental results show that how to assign tasks between models and whether to check the execution results significantly affect system performance:

Figure 5 : GPT-6-Astra In RoboDojo different System 1&2 collaboration methods and action interfaces under Token costs and average scores comparison

Overall, the second quadrant representing“low cost, high performance”area still has few methods that can enter. How to design a that can control the cost while maintaining high performance Harness has become an important research direction for the next step.

IV. From simulators to real-world experiments

Excellent performance on the simulator rankings does not necessarily mean that the capability can be transferred to real robots. To further observe this gap, we conducted tests on 8 robot models, covering different combinations such as single-arm, dual-arm, wheel arms, humanoid robots, grippers, and dexterous hands.

Real-world experiments further show that the challenges faced by real robots cannot be fully covered by the simulator rankings:

Therefore, even if GPT-6-Astra performs well on the simulator rankings, there is still a significant distance to stable and reliable real-world deployment. Evaluations need to be closer to real-world needs and also need to consider environmental changes, time costs, and execution consequences.

V. Embodied evaluation standards for the future

The above experiment not only demonstrates the capabilities of existing models, but also highlights the gap between current evaluations and real-world usage. To promote embodied intelligence to true implementation, evaluations should not only focus on whether actions are successful, but also examine whether robots can understand human true needs, accumulate experience through interaction, and complete tasks in continuously changing environments. We believe that future embodied benchmarks should at least follow three principles:

Based on these principles, we built a platform that supports physical interaction,“sample room”. Robots can operate furniture, items, and appliances within it; even if the model is still thinking, processes such as water flow and heating will continue to occur.

In this sample room, GPT-6-Astra exposes several representative issues:

These cases show that the challenges of embodied intelligence are not only“understanding the environment and generating actions”, but also coordinating with the surrounding environment and implicit responsibilities, understanding the spatial relationships of objects and the body, and updating judgments in dynamic worlds. We hope that future evaluations can more comprehensively reflect these capabilities and drive models from benchmark performance to real-world tasks.

Six, Open-Source Practice: SenseTime SenseNova-RoboRSI, results can be reproduced with one click

Based on the above research, SenseTime open-sourced SenseNova-RoboRSI Harness, and launched an open-source, visual–language–action models, and world– SenseNova-RoboRSI-27B-Preview. The former helps researchers reproduce evaluations, compare system designs, and conduct secondary development; the latter attempts to consolidate verified embodied experiences into the model, reducing inference costs while improving task execution efficiency. Its cost to complete embodied tasks is only the cost of GPT-6-Astra the 1/50.

SenseNova-RoboRSI various Benchmark and their runtime dependencies are encapsulated as independent Docker images, and through unified HTTP API provides environment interaction and native scoring services, achieving decoupling between the evaluation environment and Agent models and strategies. Researchers do not need to manually deploy complex simulation environments; they only need to prepare the corresponding configuration files to complete end-to-end evaluation and result reproduction with one click. This lowers the threshold for reproducing experiments and facilitates the community in verifying results, comparing different models and Harness designs under a unified environment.

Beyond the evaluation framework, we also attempt to further consolidate the verified experiences from experiments into the model itself. SenseNova-RoboRSI-27B-Preview Through multimodal supervision iteration, the model internalizes the capabilities of target understanding, action decision-making, and feedback correction into its parameters. In multiple Benchmark tasks, the model achieved the following results:

Figure1 SenseNova-RoboRSI-27B-Preview Performance on multiple embodied Benchmark tasks

In addition to Benchmark data, the model also demonstrates corresponding capabilities in specific operations: it can actively observe details, reason about spatial relationships, and adjust strategies based on execution feedback. The following three clips show how the model combines observation, reasoning, and action to solve practical operational challenges:

Figure2 SenseNova-RoboRSI-27B-Preview Embodied Capability Demonstration: Active Perception, Autonomous Planning, and Spatial Reasoning

SenseNova-RoboRSI-27B-Preview Reduce the cost of single inference with a smaller model size, and by internalizing embodied capabilities into the model, reduce redundant inference and trial-and-error, bringing the task completion cost down to GPT-6-Astra the 1/50.

From GPT-6-Astra the system test to identify capability boundaries and implementation bottlenecks, and through RoboRSI open and reproducible evaluation infrastructure, and by 27B model exploring more cost-effective execution paths, SenseTime hopes to connect problem discovery, experimental verification, and solution iteration. We look forward to working with the community to drive embodied research from chasing benchmark scores to continuous exploration of real task completion capabilities, execution efficiency, and reliability.

(LeFeng Net (Official Account: LeFeng Net)）