CoEnv: Code-Driven Exploration in Real-to-Sim Digital Twins for Multi-Arm Manipulation
CoEnv: Code-Driven Exploration in Real-to-Sim Digital Twins for Multi-Arm Manipulation
This paper presents CoEnv, a framework that reconstructs physical scenes into executable digital twins using multi-view RGBD data. It enables code-driven exploration where an agent writes, debugs, and refines complete multi-arm programs within the simulation before transferring verified episodes to hardware. On five real-world tasks involving up to three heterogeneous robots, the system achieves 70% average success compared to 50% for a baseline without the code agent. Additionally, policies fine-tuned on agent-generated rollouts reached high success rates on two tasks without requiring human demonstrations.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
improvement factor
3.8–11.7
View original evidence
code agent (where a VLM must choose every primitive step by step), outperforms two prior VLM-planning frameworks, and produces verified episodes at 3.8-11.7x the ablated rate. Because every verified rollout is also a demonstration, exploration leaves reusable experience behind: a $\pi_{0.5}$ policy fine-tuned purely on 200 agent-generated rollouts reaches 100% and 70% real-robot success on two tasks, with no human d
Open source S5rollouts count
200
View original evidence
code agent (where a VLM must choose every primitive step by step), outperforms two prior VLM-planning frameworks, and produces verified episodes at 3.8-11.7x the ablated rate. Because every verified rollout is also a demonstration, exploration leaves reusable experience behind: a $\pi_{0.5}$ policy fine-tuned purely on 200 agent-generated rollouts reaches 100% and 70% real-robot success on two tasks, with no human d
Open source S5
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2604.05484v2 Announce Type: replace Abstract: General-purpose coding agents can now write, debug, and refine entire robot programs, but only if they have somewhere to practice. Physical hardware is a poor practice ground: exploratory rollouts are slow to reset and, when several arms share a workspace, unsafe. Generic simulation benchmarks are safe but do not match the geometry, assets, or viewpoints of the work
Open source S2
am, running it in the twin, and refining it against grounded verification (privileged state checks, checkpoint predicates on risky steps, and adaptive viewpoints against inter-arm occlusion). Verified programs transfer to hardware through collision-checked deployment. On five real-world multi-arm tasks with up to three heterogeneous robots, CoEx reaches 70% average success, against 50% for the same system without the
Open source S4
code agent (where a VLM must choose every primitive step by step), outperforms two prior VLM-planning frameworks, and produces verified episodes at 3.8-11.7x the ablated rate. Because every verified rollout is also a demonstration, exploration leaves reusable experience behind: a $\pi_{0.5}$ policy fine-tuned purely on 200 agent-generated rollouts reaches 100% and 70% real-robot success on two tasks, with no human d
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org