Perturbot: Breaking Shortcut Priors in Vision-Language-Action Policies
This paper identifies modality shortcuts in vision-language-action (VLA) policies, where policies rely on superficial cues like visual, lexical, or motor signals rather than task-relevant evidence. The authors propose Perturbot, a method that introduces task-preserving wrist-view perturbations, enriched instructions with decision-relevant captions, and relabeled trajectory segments to reduce reliance on these shortcuts. They also introduce GroundFscore, an offline score to evaluate how much a policy depends on shortcuts. The work provides a framework for training and evaluating VLA policies that disentangle decisions from shortcut priors while preserving responsiveness to task-relevant evidence.