The Relative Value of Demonstration Data: Execution Time Limits and Operator Mixtures in Behavior Cloning
The Relative Value of Demonstration Data: Execution Time Limits and Operator Mixtures in Behavior Cloning
This research investigates how the quality of teleoperator demonstrations affects behavior cloning performance. By analyzing datasets from 'Better' and 'Worse' labeled operators on simulated Square and Can tasks, the authors measure data multiples required to achieve equivalent success rates. Results show that within a 25-second time limit, training with only 'Worse' demonstrations requires 1.55 times as many samples as using only 'Better' ones, primarily due to late task completion. Even when accounting for execution time differences, mixed datasets require approximately 13% more demonstrations than predicted by simple linear weighting. The study concludes that switching to lower-quality operators is economically viable only if their hourly pay is less than one-third of better operators, considering the additional time required to record their demonstrations.
The study quantifies the relative value of demonstration data from teleoperators with varying skill levels on simulated tasks, finding that lower-quality demonstrations require significantly more samples to match success rates due to longer execution times and policy completion delays.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S2
- Control
- TeleoperatedOpen source S2
Reported numbers
demonstration ratio (Worse vs Better)
1.55
View original evidence
nd Can tasks, we train behavior-cloning policies on demonstrations from the two teleoperators robomimic labels Better and the two it labels Worse, varying dataset size and the share of Worse demonstrations. On Square, within robomimic's 25-second time limit, an all-Worse training set needs 1.55 times as many demonstrations as an all-Better set. Most of this difference is late completion: policies trained on Worse dem
Open source S3demonstration ratio (half-Better half-Worse vs all-Better)
1.22
View original evidence
onstrations finish later, as the Worse teleoperators do, and with a 100-second limit the multiple falls to 1.11. On Can, most of the difference remains at 100 seconds. If each Worse demonstration were worth 1/1.55 of a Better one, as in an all-Worse set, a half-Better, half-Worse set on Square would need 1.22 times as many demonstrations as an all-Better set. In fact, it needs 13% more than that, and this excess pers
Open source S4time limit (Square)
25
View original evidence
nd Can tasks, we train behavior-cloning policies on demonstrations from the two teleoperators robomimic labels Better and the two it labels Worse, varying dataset size and the share of Worse demonstrations. On Square, within robomimic's 25-second time limit, an all-Worse training set needs 1.55 times as many demonstrations as an all-Better set. Most of this difference is late completion: policies trained on Worse dem
Open source S3time limit range
20–200 seconds
View original evidence
ists at every time limit from 20 to 200 seconds. The excess remains when training draws demonstrations uniformly instead of in proportion to their length. On Square, within the 25-second limit, switching to these Worse teleoperators is cheaper at equal success only if they earn less than a third of the Better teleoperators' hourly pay, because Worse demonstrations also take longer to record.
Open source S5
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2611.11947v1 Announce Type: new Abstract: Robot demonstration datasets combine demonstrations from teleoperators of uneven experience, so collection teams need to know how many demonstrations from one group match those from another. We measure this with a data multiple: the ratio of the numbers of demonstrations two groups need for their policies to reach the same success rate. On robomimic's simulated Square a
Open source S2
nd Can tasks, we train behavior-cloning policies on demonstrations from the two teleoperators robomimic labels Better and the two it labels Worse, varying dataset size and the share of Worse demonstrations. On Square, within robomimic's 25-second time limit, an all-Worse training set needs 1.55 times as many demonstrations as an all-Better set. Most of this difference is late completion: policies trained on Worse dem
Open source S3
onstrations finish later, as the Worse teleoperators do, and with a 100-second limit the multiple falls to 1.11. On Can, most of the difference remains at 100 seconds. If each Worse demonstration were worth 1/1.55 of a Better one, as in an all-Worse set, a half-Better, half-Worse set on Square would need 1.22 times as many demonstrations as an all-Better set. In fact, it needs 13% more than that, and this excess pers
Open source S4
ists at every time limit from 20 to 200 seconds. The excess remains when training draws demonstrations uniformly instead of in proportion to their length. On Square, within the 25-second limit, switching to these Worse teleoperators is cheaper at equal success only if they earn less than a third of the Better teleoperators' hourly pay, because Worse demonstrations also take longer to record.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org