Evaluating Compound Robustness in Vision-Language-Action Policies
Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies
Study shows single-axis tests fail to capture compound robustness. LIBERO-CTRL benchmark reveals emergent failures and compensated successes. Aggregate success rates miss critical outcome changes.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies
Open source S1
red from single-axis evaluations. We introduce LIBERO-CTRL, a six-axis benchmark that pairs each initial state across single-axis conditions and a matched simultaneous condition. This design reveals two opposing outcome changes that aggregate success rates cannot distinguish: emergent failures, where all single-axis rollouts succeed but the simultaneous rollout fails, and compensated successes, where at least one sin
Open source S3
gle-axis rollout fails but the simultaneous rollout succeeds. Because one transition decreases compound success while the other increases it, they can cancel, making aggregate compound performance appear consistent with single-axis measurements even when individual outcomes differ substantially. These opposing transitions can largely cancel in aggregate: even when the difference between the two transition rates is no
Open source S4
t statistically distinguishable from zero, as many as 29.0% of matched initial states still change outcome. Across six policies and three severity levels, such outcome changes reach 34.5% in the most affected condition. The relative prevalence of the two transitions varies across policies and severities, while the transition rates remain similar under independent re-evaluation of stochastic policies. Compound robustn
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org