RoboFollow: A Benchmark for Evaluating Language Understanding in Robot Control
机器人明明听懂了指令，为什么还是干了别的事
A research team from Shanghai Jiao Tong University identifies a critical flaw in current robot control models: they rely on visual shortcuts rather than true language understanding. To address this, they introduce RoboFollow, a benchmark with high-scene-entropy scenarios that force models to depend on language for task resolution. The study shows that even top models fail under slight variations in scene layout or instruction phrasing, indicating a deeper issue in training paradigms.
Source: Chinese robotics — Research news discovery · Read original article ↗
English translation unavailable. Choose the original article to read it.
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
scene entropy
3.782
View original evidence
研究团队专门计算了这个数据集的场景熵，达到了3.782比特，作为对比，业内常用的LIBERO基准里几个子任务集合的场景熵平均只有0.88比特。
Open source S14training tasks
75
scene entropy
0.88
View original evidence
研究团队专门计算了这个数据集的场景熵，达到了3.782比特，作为对比，业内常用的LIBERO基准里几个子任务集合的场景熵平均只有0.88比特。
Open source S14
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
整个数据集一共设计了75种训练任务标签，归入六组场景配置，总共产生了3750条训练演示轨迹。
Open source S13
研究团队专门计算了这个数据集的场景熵，达到了3.782比特，作为对比，业内常用的LIBERO基准里几个子任务集合的场景熵平均只有0.88比特。
Open source S14
以π0.5这个模型为例，它在场景1（空间关系任务）的L0测试里拿到了99.1%的意图分数,近乎满分。
Open source S22
但是切换到L1，也就是仅仅调换了物体位置、指令一个字没变的情况下，这个分数直接跌到45.5%，几乎腰斩。
Open source S23
研究团队挑选了九个当前最有代表性的机器人控制模型进行测试，涵盖了VLA和另一类叫做世界动作模型（ WAM ）的方法。
Open source S21
它的同门师兄π0表现得更极端，L0是98.2%,L1直接归零，变成0.0%。
Open source S24
Source：Chinese robotics — Research news discovery · news.qq.com