GPT-6-Astra Embodied Capabilities Face Major Test. Traditional Benchmarks Score Perfectly, Real Robots Hit Walls

Editor | Du Wei

When OpenAI released GPT-6-Astra last month, it called it the ``model with the highest global intelligence and most aligned with human intent.'' The embodied community was surprised to find that the debate between VLA and WAM routes was decisively outmaneuvered by the general-purpose large model through a text modality ``dimensional strike,'' prompting everyone to wonder:

What is the ultimate limit of embodied capabilities for GPT-6-Astra? What is the most worthwhile next step for the embodied community?

Recently, Sinovation Robotics collaborated with Daxiao Robotics, Ropedia, Nanyang Technological University in Singapore, and the University of Tokyo to publish a paper titled ``Has GPT-6-Astra Achieved Embodied Intelligence?'' in which they provided a systematic analysis covering 12 simulation evaluations and 8 real robot platforms.

They also analyzed the collaborative approach between GPT-6-Astra and the underlying robot control system, and built a prototype evaluation system that is closer to real household needs.

Technical report link: https://www.sensenova.ai/has-gpt-6-astra-achieved-embodied-intelligence

Interestingly, the same GPT-6-Astra, after simply removing some restrictions, achieved full scores or nearly full scores in several traditional embodied intelligence evaluations: it reached 100% on the LIBERO-Long++ benchmark for long-horizon compositional task evaluation, 99.3% on LIBERO-Pro which tests operational robustness and generalization ability, 98.8% on the humanoid robot benchmark SIMPLE, and 81.9% on RoboDojo. However, when this model directs real robots to perform some everyday tasks, it still faces difficulties.

``Putting the pen into the pen holder, a task that takes a person just a few seconds, takes over 4 hours and consumes 55 dollars.``

'Mingming judged the eggs were cooked, but didn't turn off the heat in time, finally burning the eggs; after opening the fridge to put down the milk, he turned back to busy moving fruits, leaving the fridge door open for several minutes.'

The operation of inserting the plug was tested across multiple real devices 11 times, and it never succeeded once.

High scores and mistakes placed together, the obvious contrast reveals a key issue: how far away is a general-purpose large model capable of planning, observing, and correcting actions from being reliable in doing real work?

The report's judgment is: GPT-6-Astra has already crossed the threshold of embodied capabilities to a large extent, but there is still a significant gap before it can be practically deployed.. The issues of cost and time revealed in the experiments, along with those unexpected mistakes, precisely indicate where the next biggest challenge lies.

Notably, after the release of GPT-5 last year, the same team also published an evaluation article on its spatial intelligence titled Has GPT-5 Achieved Spatial Intelligence , and then released the SenseNova-SI series models. This suggests that this evaluation seems more like a preview .

Putting on a more powerful brain, execution cost is a major challenge

Currently, the field of robotic operation widely adopts specialized trained control strategies, as well as technical approaches such as vision-language-action models (VLA). These models typically train using real or simulated robotic operation data, allowing robots to perform actions such as grasping, moving, and placing based on visual information and task instructions.

The emergence of GPT-6-Astra provides another possibility for embodied intelligence. General-purpose large models can combine visual information, robot states, and task requirements to directly generate operational instructions for subsequent execution and adjustment. The tasks it can perform are also beginning to span different robot forms.

In order to systematically study this capability, SenseNova from SenseTime and its joint research team selected 12 representative evaluations, covering five scenarios: desktop operations, dexterous operations, humanoid robot mobility operations, home long-range tasks, and visual-language navigation. During the evaluations, GPT-6-Astra used the same evaluation approach, reading observed information in different environments, generating robot actions, and continuing to make decisions based on the execution results.

The results are indeed very impressive. In addition to the LIBERO-Long++, LIBERO-Pro, and SIMPLE mentioned above, it achieved 98.9% on the ultra-long-range robotic operation benchmark LIBERO-Ultra-Long. For more complex dexterous operations, GPT-6-Astra achieved 87.3% success rate on DexJoCo and 81.9% on RoboDojo. This indicates that GPT-6-Astra achieved or exceeded the best results of previously publicly available methods in most evaluations.

Figure 1: Comparison of the 12 evaluations in the report with previously public results.

Behind these high scores, the research team designed two key conditions:

The first is in-context learning (ICL) : Before the evaluation, GPT-6-Astra can view successful demonstrations from the training set that are allowed to be used, extracting task descriptions, action trajectories, visual references, auxiliary code, and failure recovery methods to form a knowledge package. The knowledge package is frozen before the formal test, and the model parameters remain unchanged, with no new experiences exchanged between test tasks.

This is equivalent to letting the model first study existing operational data, then enter the exam. When a robot encounters the current scenario, it can retrieve similar demonstration actions and adjust them based on the position of the object in front of it. It utilizes task experience but does not learn these actions through retraining the model weights.

Figure 2: ICL training paradigm.

The second condition is increasing the number of execution steps. Traditional embodied evaluations usually limit the maximum number of actions a robot can take. Once the limit is reached, even if the task is only a few actions away from completion, it will be judged as a failure.

The research team allowed the model to use up to 10 times the standard execution budget (1×). When a robot fails to grasp an object, the object shifts, or the operation does not meet expectations, it can continue to observe the environment, adjust the operational plan, and retry.

This adjustment yielded immediate results. For example, on the RoboDojo benchmark, the success rate increased from 27.6% under the standard budget to 81.9% under the 10× budget. Bench2Dex also increased from 25.6% to 61.5%. In tasks that require repeated adjustment of grasping positions, bimanual coordination, and contact precision, GPT-6-Astra has a certain ability to recover from failure.

Figure 4: The task success rate of GPT-6-Astra changes with the increase in allowed execution steps.

However, such improvements did not occur in all benchmarks. On the home long-range benchmark BEHAVIOR, the success rate was only 37.0%. On the long-range navigation benchmark LHPR-VLN, even with an expanded execution budget, the success rate was only 28.9%. It can be seen that, for these complex scenarios that require continuous management of multiple subtasks, the help provided by increasing the number of operations is relatively limited.

Even if errors can be corrected, it may still get worse

In the failure cases compiled by SenseNova from SenseTime and its joint research team, one scenario is particularly representative: The robot needs to hang a cup on a hook. GPT-6-Astra observed via the head camera, found that the cup handle seemed to have aligned with the hook, and thus judged the operation as successful. However, from another camera perspective, the cup was actually not hung on the hook and had fallen onto the table. The model then attempted to re-grasp, but mistakenly closed the mechanical gripper and interpreted it as having contacted the cup, leading to subsequent operational errors.

A similar situation occurred in the puzzle task. GPT-6-Astra judged that the four puzzle pieces had formed the target shape, but in reality, some positions were still misaligned. Since it could not identify where the problem was, it continuously adjusted the puzzle positions, yet ended up messing up the part that was originally close to completion.

These two cases illustrate that even if the model can understand the task goal, it may still fail to accurately determine whether the physical operation was truly successful. In language or software tasks, many results can be verified through text, program output, or test feedback. However, in robot operations, just because an object appears to be placed in the correct position does not mean it is stably contacted, supported, aligned, or fixed in place. Robots need to determine the real contact and positional relationships between objects, but for GPT-6-Astra, this is challenging. More troubling is that some errors in the physical world are difficult to rectify after they occur.

In the liquid pouring task of RoboDojo, GPT-6-Astra accidentally knocked over the cup, spilling liquid onto the table. The model then tried to steady the cup and continue pouring, but the spilled portion could not be recovered through simple retries. In the assembly task of DexJoCo, a part slid from the robot's hand to the ground. GPT-6-Astra judged the part to be out of the operable range and stopped subsequent actions.

This explains why increasing the number of execution steps does not necessarily solve all problems. If a robot just slightly misaligned the grasp position, it can re-observe and adjust the gripper. However, if the object is already damaged, liquid has already spilled, or the part has fallen to an unreachable position, more operational attempts may not be helpful.

Figure 6: (a), (b), and (c) correspond to misjudgment of physical success conditions, actions with difficult-to-reverse consequences, and the completed state of long tasks being disrupted.

Long tasks add another layer of complexity. When storing a laptop and headphones, GPT-6-Astra had already placed the laptop and hung the headphones on the stand. However, when it returned to adjust the headphone position, it accidentally knocked the headphones down. By the end, the headphones were never re-hung. Such failures are easy to overlook. Once a step is completed, the environment does not remain unchanged. If previously completed conditions are disrupted, subsequent tasks may be affected. This means that as models become increasingly skilled at planning and error correction, verification and recovery during the physical execution process become new challenges.

They turned their attention to

Some evaluations, although almost solved, still have enormous time and Token costs. LIBERO-Long++ has achieved a success rate of 97.5% under standard budgets, but each test round still consumes an average of 1,000,000 Tokens, with inference and tool usage taking close to an hour. For the longer LIBERO-Ultra-Long, the average Token consumption reaches about 9,000,000, with the time close to two hours. In the extended budget test of RoboDojo, a single round can take even over three hours.

These times are not all spent on robot movement; a large amount of time is used for model inference, checking execution results, and calling tools. The greater the task difficulty, the more significant the cost of repeated thinking and attempts.

Figure 5: (a) Average token usage per task, (b) Average time per task.

Since increasing the number of attempts brings high time and Token costs, SenseNova and its joint research team began exploring reducing unnecessary model calls, making each inference and error correction more effective. To this end, they focused on Harness, which is the execution system connecting the model, environment observation, and robot control interface.

On the RoboDojo benchmark, they let GPT-6-Astra act as the System 2 responsible for reasoning and action review, and let the π0.5 VLA, fine-tuned on this evaluation task, act as the System 1 proposing actions, and compared different collaboration methods.

In open-loop collaboration, GPT-6-Astra provided sub-goals, VLA generated and executed actions, with a success rate of only 5.83%. Closed-loop collaboration would convert the candidate actions proposed by VLA into end-effector movements, allowing GPT-6-Astra to decide to accept, modify, or reject them before execution. When reviewing 4 candidate actions each time, the success rate increased to 25.83%. Further combined with ICL, under the configuration of reviewing 2 candidate actions each time, closed-loop collaboration achieved a success rate of 54.17%, but the Token consumption reached about 19,335,000, which is about 6.9 times that of the closed-loop configuration without ICL and reviewing 4 actions each time.

Figure 7: Comparison of Token costs and success rates of different System 1/System 2 collaboration strategies and action interfaces on the RoboDojo benchmark.

In addition to the collaboration between models and VLA, Harness also needs to address the question of how to convert the decisions of large models into actions that robots can accurately execute.

The research team found in the experiment where the GPT-6-Astra directly controls robots that specifying the joint angles of the robotic arm for the model resulted in a success rate of 9.72%; changing to specify the position and orientation of the end-effector of the robotic arm increased the success rate to 15.83%; further using relative direction and movement distance to describe actions increased the success rate to 18.06%. ``Letting the large model handle the judgment of where the robotic arm should move, and leaving the specific joint movements to the lower-level controller results in better performance.``

Another factor affecting success rate and cost is the frequency of model invocation. Re-invoking the model at each step, although convenient for timely adjustment, has a high cost. Continuously executing fixed multi-step actions may miss environmental changes. They let the model choose to execute 1 to 4 steps at once, the success rate increased from 15.83% to 20.00%, and token consumption dropped from 2,560,000 to 1,146,000.

These experiments all point to an important trade-off in Harness design: large models participate in judgment and error correction when necessary, while minimizing unnecessary invocations. How the execution system arranges these two factors will directly affect the robot's success rate and token cost.

Eight real devices platforms, capable of grabbing and moving, with precise operations hitting walls

High simulation evaluation scores do not mean the robot will enter a real physical environment. To verify GPT-6-Astra's actual capabilities, SenseNova from SenseTime and its joint research team deployed it to 8 real robot platforms, covering five different forms, including Franka 3, Franka Panda, etc., single-arm robotic arms, dual-arm Piper robots, the Tianji M6S equipped with dexterous hands, wheeled mobile operation robots, and Unitree G1 humanoid robots.

Unlike the previous simulation experiments, these real robot experiments did not provide GPT-6-Astra with successful task demonstrations, nor did they perform additional training on the robot. The model received structured information written by humans, including robot interfaces, joint limitations, camera positions, safety constraints, and operation hints on some platforms. Execution was measured by real time, usually limited to 60 minutes, and on-site operators were responsible for safety supervision.

The results showed that under these conditions, the same GPT-6-Astra could indeed control different forms of robots to complete a series of operations: in rigid object grasping and placing tasks, 18 tests succeeded 17 times. On Unitree G1, GPT-6-Astra completed 2 out of 3 mechanical arm operations through the underlying control interface, as well as all 3 turning, walking, and stopping tasks.

However, Once it involves fine contact and long-range tasks, the success rate drops significantly. In the fabric folding task on Franka 3, 4 attempts succeeded 2 times. In the plug-in tasks on three robot platforms, a total of 11 attempts all failed. In the long-range navigation and search task on AgileX Tracer, 23 tests succeeded only 9 times.

Figure 8: Zero-shot real robot control: different robot forms from simple to complex.

These results reflect that the ability of a robot to grasp rigid objects does not mean it can handle flexible object operations, precise contact, and long-range autonomous tasks. But an unexpected phenomenon also emerged in the real robot experiments.

In the flower-arranging task, the robot needs to grasp the flower stem and insert the flower into the vase. When the single-arm Franka 3 attempted the task, the wrist camera could only see the flower stem gripped by the gripper, but could not see the vase opening below. Due to the lack of a critical viewpoint, the robot ultimately failed. When switching to the dual-arm Piper, the situation changed. GPT-6-Astra controlled one mechanical arm to grasp the flower stem, while actively moving the other idle mechanical arm to gradually bring its wrist camera closer to the vase opening. Once the model obtained a clear view, it smoothly completed the flower-arranging operation.

This action demonstrates the ability of 'active vision'. GPT-6-Astra can actively change perspectives without task demonstrations, providing a valuable case for studying how a general model can understand and utilize a robot's own perceptual capabilities.

Figure 9: What GPT-6-Astra can do depends on what it can see.

The real world has exposed problems that simulation environments struggle to fully reflect. First is Security. In some operations, robotic arms triggered safety shutdowns due to contact with the desk or joint collisions, causing tasks to be directly interrupted. Second is mechanical load. During repeated grasping attempts on the Tianji M6S, some finger joints of the dexterous hand stalled due to load issues, with temperatures rising to 62 degrees Celsius, interrupting the operation.

The model's decision-making speed is also an issue. In experiments with the Unitree G1, GPT-6-Astra spent about 90% of its runtime on decision-making, with a single operation experiment possibly taking over an hour. Robots wait for the model to generate the next action, hardware remains occupied, and time costs, equipment risks, and manual supervision costs accumulate continuously.

For robot manufacturers, converting the model's judgments into accurate physical actions and maintaining stability across different tasks and environments has become a key challenge for product deployment.

Learning from failure, robot systems begin to self-improve

When GPT-6-Astra achieved scores close to full marks in multiple embodied evaluations, the SenseNova from SenseTime and its joint research team began to re-examine a question: to what extent do these high scores reflect the robot's capabilities in real-life scenarios?

Current embodied evaluations mostly focus on clear task goals, while real-life tasks are often not that simple. Suppose a user tells a robot: ``Help me prepare breakfast.'' If the user needs to leave in 15 minutes, then preparing eggs and bread within 10 minutes may be more valuable than spending an hour making a lavish breakfast. The robot also needs to remember to turn off the stove and close the refrigerator door, and not let the kitchen become a mess just because it is busy cooking.

According to some existing evaluation rules, as long as the food reaches the specified doneness, even if it is later burned, it may still satisfy the success condition. They used the cooking task of BEHAVIOR-1K as an example to point out the limitations of this evaluation method.

Figure 10: Comparison between traditional embodied evaluation and the new evaluation method. The former mainly checks whether the task reaches the preset goal, while the latter further considers completion time, actual effects, and safety requirements that must be followed during the operation.

They believe that future embodied evaluations need to make some adjustments. Tasks should be more aligned with human actual needs, allowing robots to decide on their own how to complete them, and considering the value of the final result to the user. Tests should not default to providing pre-existing success demonstrations, allowing robots to explore on their own in unfamiliar environments. At the same time, reasoning time should also be included in the evaluation.

To verify these ideas, the research team used NVIDIA Isaac Sim to build an interactive single-bedroom home environment, equipped with appliances such as a refrigerator, sink, stove, and air fryer, simulating object contact, liquid flow, and heating processes. GPT-6-Astra observed the environment through the camera and robot status information of the simulated robot AgiBot G2, attempting to complete tasks such as preparing breakfast, tidying the table, and organizing items without task demonstrations.

GPT-6-Astra revealed many issues (see Figure 12). While tidying the table, the robotic arm knocked over other items, even pushing over the monitor. When organizing food, it placed milk in the refrigerator and then turned to handle the fruit, leaving the refrigerator door open. Facing an unfamiliar faucet, the robot repeatedly pushed the handle that should have rotated, failing to close the water in time, leading to water spilling into the sink. When frying eggs, the model judged the eggs to be cooked and decided to turn off the heat, but it took about 90 seconds before turning off the stove. At this point, the eggs were already burned, and the residual heat continued to cook them.

These home experiments are currently only small-scale explorations and cannot represent the performance of robots after actual deployment. However, from the eggs being cooked and then burned, to forgetting to close the refrigerator door after putting milk in, it shows that only looking at whether the task is completed can easily miss problems that occurred during execution.

Exactly these issues provide clues for further improving the robot system. If every failure leaves a complete execution record, researchers have the opportunity to find out at which step the problem occurred and test different improvement approaches.

Learning from failures, the robot system begins self-improvement

Looking back at this study, we can find a phenomenon that runs through the entire research. GPT-6-Astra's performance in robot tasks is positively correlated with the degree of utilization of existing experience and the coordination between the model and the execution system. Providing task demonstrations, adjusting action interfaces, and adding feedback checks may significantly change the success rate without updating the model weights.

This opens up space for another research direction: how to let robots accumulate experience from past failures and avoid making the same mistakes in the next execution? Around this direction, SenseNova-RoboRSI research was conducted by SenseTime, trying to let Harness improve based on task feedback. The core idea is: to build a unified Agent operation and evaluation framework for robot tasks.

Project address: https://github.com/OpenSenseNova/SenseNova-RoboRSI

Specifically, developers can connect different models, robot control tools, skill modules, and execution environments to the same runtime system, and observe the entire execution process through actual task feedback. Here, Harness is responsible for managing which information the model receives, when to call tools, how to generate actions, and how to continue processing after a failure.

On this basis, Harness also needs to have continuous improvement capabilities. For example, after a robot fails in a grasping task, the system analyzes the cause using the execution trajectory and feedback information, and adjusts the action generation method, planning strategy, feedback checks, and memory organization accordingly, and then returns to the corresponding evaluation task for verification. Effective improvements are retained in subsequent versions, and the entire process does not require updating the base model weights, which corresponds to the idea of recursive self-improvement (RSI).

SenseTime demonstrated preliminary results in this direction in its previously published RoboDojo-Sim experiment. Compared to the GPT-6-Astra baseline, The average score of SenseNova-RoboRSI increased from 28.97 to 60.08, and the success rate increased from 22.48% to 54.17%.

In addition to RoboDojo-Sim, SenseNova-RoboRSI also achieved good results in other embodied evaluations. In the LIBERO-Pro evaluation, its success rate reached 94.50%, higher than the RPent method that also uses GPT-6-Astra (92.63%). In the target kitchen environment of RoboCasa365, combining task memory achieved a success rate of 64.60%, higher than methods such as Paimon-0 (58.10%).

SenseNova-RoboRSI-27B-Preview further explored the effectiveness of smaller-scale models combined with Harness through RSI, with preliminary evaluation results that are equally impressive.

On RoboDojo, the 27B model achieved an average score of 26.13 through VLA-assisted execution, 2.84 points lower than the GPT-6-Astra baseline. On LIBERO-Pro, the success rates of VLA-assisted execution and VLM direct control reached 95.0% and 96.1%, respectively, both ranking second, just behind GPT-6-Astra. On RoboCasa365, after RSI training, the model's success rate increased from 26.0% to 46.5%, with the average output Token reduced by 73.3%, achieving dual improvements in task performance and output Token efficiency.

Next, SenseNova-RoboRSI will continue to optimize around the issues exposed by the GPT-6-Astra experiment, including transferring operational experience to other objects or robot platforms, using accumulated cases to quickly select appropriate operational strategies, reducing unnecessary large model calls, and allocating reasoning resources to critical decision-making stages.

GPT-6-Astra proved the capability to perform complex operations across different robot forms. RoboRSI focuses on the self-evolution of the execution system: by analyzing failures and improving Harness, making robots perform better in the next attempt. However, this approach is currently mainly validated in simulation environments, and whether the benefits can be extended to real robots remains to be tested in the future.

Evaluation results are still far from reliable execution in the real world. Users won't accept a robot that runs well on tests but forgets to turn off the stove when making breakfast. For embodied intelligence to truly enter everyday life, it depends on whether robots can reliably do things well without leaving a mess behind.