Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm
Overview
This research investigates the correlation between held-out validation loss and closed-loop success rates for four imitation-learning policies (ACT, Diffusion Policy, SmolVLA, and pi_0.5) trained on an SO-101 robotic arm. By evaluating 82 checkpoints across 930 bench rollouts and analyzing 700 intermediate checkpoints from 70 training runs, the authors find that while action error correlates with success within individual policies, no single offline metric consistently predicts real-world performance across different architectures or training seeds. The study highlights that common failure modes, such as the arm failing to leave its rest pose, are absent in held-out datasets and recommends reporting initiation failures separately from overall success rates.
Generated from attributed reports · Updated 2 hours ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · rollouts: 930 other · reported trials · Differing source assertions
“82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearm”
Exact source · revision 1Source owner not reported
Reported quantity · checkpoints: 82 other · Basis not reported
“82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearm”
Exact source · revision 1Source owner not reported
Reported quantity · training runs: 70 other · Basis not reported
“82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearm”
Exact source · revision 1Source owner not reported
Reported quantity · rollouts: 110 other · reported trials · Differing source assertions
“an $\rho$ from -0.23 to -0.88), but no more strongly than the number of training episodes alone, and not at all among seeds trained on the same data. Across policies the ranking is inverted ($\rho$ = +0.90 at 200 episodes): the policy with the lowest hold-out error succeeded in 3 of 110 rollouts. The most common failure, the arm never leaving its rest pose, ended 96% of rollouts for Diffusion Policy trained on untrim”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalDoes Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm
This research investigates the correlation between held-out validation loss and closed-loop success rates for four imitation-learning policies (ACT, Diffusion Policy, SmolVLA, and pi_0.5) trained on an SO-101 robotic arm. By evaluating 82 checkpoints across 930 bench rollouts and analyzing 700 intermediate checkpoints from 70 training runs, the authors find that while action error correlates with success within individual policies, no single offline metric consistently predicts real-world performance across different architectures or training seeds. The study highlights that common failure modes, such as the arm failing to leave its rest pose, are absent in held-out datasets and recommends reporting initiation failures separately from overall success rates.
Event coverage history
There is not enough continuous observation data to show a trend.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).