The Relative Value of Demonstration Data: Execution Time Limits and Operator Mixtures in Behavior Cloning
Overview
This research investigates how the quality of teleoperator demonstrations affects behavior cloning performance. By analyzing datasets from 'Better' and 'Worse' labeled operators on simulated Square and Can tasks, the authors measure data multiples required to achieve equivalent success rates. Results show that within a 25-second time limit, training with only 'Worse' demonstrations requires 1.55 times as many samples as using only 'Better' ones, primarily due to late task completion. Even when accounting for execution time differences, mixed datasets require approximately 13% more demonstrations than predicted by simple linear weighting. The study concludes that switching to lower-quality operators is economically viable only if their hourly pay is less than one-third of better operators, considering the additional time required to record their demonstrations.
Generated from attributed reports · Updated 7 hours ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Environment: simulation
“arXiv:2611.11947v1 Announce Type: new Abstract: Robot demonstration datasets combine demonstrations from teleoperators of uneven experience, so collection teams need to know how many demonstrations from one group match those from another. We measure this with a data multiple: the ratio of the numbers of demonstrations two groups need for their policies to reach the same success rate. On robomimic's simulated Square a”
Exact source · revision 1Source owner not reported
Control: teleoperated
“arXiv:2611.11947v1 Announce Type: new Abstract: Robot demonstration datasets combine demonstrations from teleoperators of uneven experience, so collection teams need to know how many demonstrations from one group match those from another. We measure this with a data multiple: the ratio of the numbers of demonstrations two groups need for their policies to reach the same success rate. On robomimic's simulated Square a”
Exact source · revision 1Source owner not reported
Reported quantity · demonstration ratio (Worse vs Better): 1.55 other · Basis not reported
“nd Can tasks, we train behavior-cloning policies on demonstrations from the two teleoperators robomimic labels Better and the two it labels Worse, varying dataset size and the share of Worse demonstrations. On Square, within robomimic's 25-second time limit, an all-Worse training set needs 1.55 times as many demonstrations as an all-Better set. Most of this difference is late completion: policies trained on Worse dem”
Exact source · revision 1Source owner not reported
Reported quantity · demonstration ratio (half-Better half-Worse vs all-Better): 1.22 other · Basis not reported
“onstrations finish later, as the Worse teleoperators do, and with a 100-second limit the multiple falls to 1.11. On Can, most of the difference remains at 100 seconds. If each Worse demonstration were worth 1/1.55 of a Better one, as in an all-Worse set, a half-Better, half-Worse set on Square would need 1.22 times as many demonstrations as an all-Better set. In fact, it needs 13% more than that, and this excess pers”
Exact source · revision 1Source owner not reported
Reported quantity · time limit (Square): 25 other · Basis not reported
“nd Can tasks, we train behavior-cloning policies on demonstrations from the two teleoperators robomimic labels Better and the two it labels Worse, varying dataset size and the share of Worse demonstrations. On Square, within robomimic's 25-second time limit, an all-Worse training set needs 1.55 times as many demonstrations as an all-Better set. Most of this difference is late completion: policies trained on Worse dem”
Exact source · revision 1Source owner not reported
Reported quantity · time limit range: 20 other · Basis not reported
“ists at every time limit from 20 to 200 seconds. The excess remains when training draws demonstrations uniformly instead of in proportion to their length. On Square, within the 25-second limit, switching to these Worse teleoperators is cheaper at equal success only if they earn less than a third of the Better teleoperators' hourly pay, because Worse demonstrations also take longer to record.”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalThe Relative Value of Demonstration Data: Execution Time Limits and Operator Mixtures in Behavior Cloning
This research investigates how the quality of teleoperator demonstrations affects behavior cloning performance. By analyzing datasets from 'Better' and 'Worse' labeled operators on simulated Square and Can tasks, the authors measure data multiples required to achieve equivalent success rates. Results show that within a 25-second time limit, training with only 'Worse' demonstrations requires 1.55 times as many samples as using only 'Better' ones, primarily due to late task completion. Even when accounting for execution time differences, mixed datasets require approximately 13% more demonstrations than predicted by simple linear weighting. The study concludes that switching to lower-quality operators is economically viable only if their hourly pay is less than one-third of better operators, considering the additional time required to record their demonstrations.
Event coverage history
There is not enough continuous observation data to show a trend.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).