LLM4Robot-Bench: How Far Can GPT-6 Astra Go in Robotic Manipulation? A Hierarchical, Paired Sim-Real Benchmark
LLM4Robot-Bench: How Far Can GPT-6 Astra Go in Robotic Manipulation? A Hierarchical, Paired Sim-Real Benchmark
This paper introduces LLM4Robot-Bench, a hierarchical benchmark with 16 household tasks evaluated in matched simulated and real environments. GPT-6 Astra zero-shot achieves 80.0% success at L1 (contact selection/alignment), 47.6% at L2 (force regulation), and 0.0% at L3 (coordinated reconfiguration) in real-world trials, with simulation rates of 66.7%, 47.6%, and 0.0% respectively; ten tasks show identical success counts across environments while six differ by one trial.
The study evaluates GPT-6 Astra zero-shot on a paired Sim-Real benchmark for contact-rich manipulation, showing real-world success rates dropping from 80.0% at L1 to 47.6% at L2 and 0.0% at L3 across 16 household tasks.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S3
Reported numbers
Reported success rate
80%
Reported zero-shot result; the excerpt does not specify an evaluation protocol.
Reported trials
View original evidence
ach task is instantiated in physically and visually matched simulated and real-world environments, with shared task definitions and success criteria. We evaluate GPT-6 Astra zero-shot with three trials per task in each environment. Real-world success rates decline from 80.0% at L1 to 47.6% at L2 and 0.0% at L3, with corresponding simulation rates of 66.7%, 47.6%, and 0.0%. Ten of the 16 tasks have identical success c
Open source S3Reported success rate
47.6%
Reported zero-shot result; the excerpt does not specify an evaluation protocol.
Reported trials
View original evidence
ach task is instantiated in physically and visually matched simulated and real-world environments, with shared task definitions and success criteria. We evaluate GPT-6 Astra zero-shot with three trials per task in each environment. Real-world success rates decline from 80.0% at L1 to 47.6% at L2 and 0.0% at L3, with corresponding simulation rates of 66.7%, 47.6%, and 0.0%. Ten of the 16 tasks have identical success c
Open source S3Reported success rate
0%
Reported zero-shot result; the excerpt does not specify an evaluation protocol.
Reported trials
View original evidence
ach task is instantiated in physically and visually matched simulated and real-world environments, with shared task definitions and success criteria. We evaluate GPT-6 Astra zero-shot with three trials per task in each environment. Real-world success rates decline from 80.0% at L1 to 47.6% at L2 and 0.0% at L3, with corresponding simulation rates of 66.7%, 47.6%, and 0.0%. Ten of the 16 tasks have identical success c
Open source S3
- paper: not_reported
What remains unknown
Not established in the collected evidence: Control, Data origin.
Reported performance applies to the described task. It does not establish general autonomy or deployment readiness.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
ach task is instantiated in physically and visually matched simulated and real-world environments, with shared task definitions and success criteria. We evaluate GPT-6 Astra zero-shot with three trials per task in each environment. Real-world success rates decline from 80.0% at L1 to 47.6% at L2 and 0.0% at L3, with corresponding simulation rates of 66.7%, 47.6%, and 0.0%. Ten of the 16 tasks have identical success c
Open source S3
Source：arXiv Robotics — research abstracts · arxiv.org