RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations
RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations
This research presents RoboTalk, a synthetic data-generation pipeline and dataset for training small vision-language models (VLMs) on multi-robot communication and coordination. The dataset includes 7,950 multimodal trajectories across 53 mobile-manipulation kitchen tasks, featuring leader-follower protocols, tool calls, and natural language communication. Fine-tuning open-source models on this dataset achieves 77% success on novel tasks, a significant improvement over untuned models.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Data origin
- Robot trajectoriesOpen source S3
Reported numbers
trajectories
7,950
Trajectory count
Reported success rate
≈2%
What remains unknown
Not established in the collected evidence: Environment, Control.
Reported performance applies to the described task. It does not establish general autonomy or deployment readiness.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
tion and skill-level action selection from multimodal demonstrations remains underexplored for small vision-language models (VLMs) intended for on-device deployment. To address this gap, we introduce RoboTalk, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate. The dataset includes a lead
Open source S3
er-follower planning protocol, tool calls (perception, manipulation, navigation, and communication), rationale traces, and diversified natural-language communication. Fine-tuning open-source models on our dataset can reach 77% success on novel held-out tasks, a significant improvement over the untuned open source models, which had a success rate of around ~2%.
Open source S4
Implications for data suppliers
RoboSignal interpretation and collection questions, not statements of buyer demand.
- Match the buyer’s robot embodiment, action space, sensors and timestamp requirements before planning collection.
- Compare the reported units and scope before using these quantities in a budget. Recording hours, sensor-hours and trajectories are different measures.
- Validate demand and acceptance criteria with a buyer before scaling. Publication, popularity and a research result do not establish a purchase commitment.
Source：arXiv Robotics — research abstracts · arxiv.org