CoEnv: Code-Driven Exploration in Real-to-Sim Digital Twins for Multi-Arm Manipulation
Overview
This paper presents CoEnv, a framework that reconstructs physical scenes into executable digital twins using multi-view RGBD data. It enables code-driven exploration where an agent writes, debugs, and refines complete multi-arm programs within the simulation before transferring verified episodes to hardware. On five real-world tasks involving up to three heterogeneous robots, the system achieves 70% average success compared to 50% for a baseline without the code agent. Additionally, policies fine-tuned on agent-generated rollouts reached high success rates on two tasks without requiring human demonstrations.
Generated from attributed reports · Updated 1 hours ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · improvement factor: 3.8 other · Basis not reported
“code agent (where a VLM must choose every primitive step by step), outperforms two prior VLM-planning frameworks, and produces verified episodes at 3.8-11.7x the ablated rate. Because every verified rollout is also a demonstration, exploration leaves reusable experience behind: a $\pi_{0.5}$ policy fine-tuned purely on 200 agent-generated rollouts reaches 100% and 70% real-robot success on two tasks, with no human d”
Exact source · revision 1Source owner not reported
Reported quantity · rollouts count: 200 other · Basis not reported
“code agent (where a VLM must choose every primitive step by step), outperforms two prior VLM-planning frameworks, and produces verified episodes at 3.8-11.7x the ablated rate. Because every verified rollout is also a demonstration, exploration leaves reusable experience behind: a $\pi_{0.5}$ policy fine-tuned purely on 200 agent-generated rollouts reaches 100% and 70% real-robot success on two tasks, with no human d”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalCoEnv: Code-Driven Exploration in Real-to-Sim Digital Twins for Multi-Arm Manipulation
This paper presents CoEnv, a framework that reconstructs physical scenes into executable digital twins using multi-view RGBD data. It enables code-driven exploration where an agent writes, debugs, and refines complete multi-arm programs within the simulation before transferring verified episodes to hardware. On five real-world tasks involving up to three heterogeneous robots, the system achieves 70% average success compared to 50% for a baseline without the code agent. Additionally, policies fine-tuned on agent-generated rollouts reached high success rates on two tasks without requiring human demonstrations.
Event coverage history
There is not enough continuous observation data to show a trend.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).