Perturbot: Breaking Shortcut Priors in Vision-Language-Action Policies

Mingyu Liu  Chonghao Sima  Tianjian Feng  Hanqing Wang
Cong Chen  Hao Chen  Chunhua Shen
Zhejiang University Shanghai Innovation Institute University of Hong Kong HKUST(GZ) github.com/aim-uofa/PerturBot

Abstract

A vision–language–action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot, which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.

¹¹footnotetext: Equal contribution. ^†Corresponding authors.

1 Introduction

“Success is a lousy teacher.”

— Bill Gates, The Road Ahead (1995)

Vision–language–action (VLA) policies can now perform complex manipulation tasks (Zitkovich et al., 2023; Kim et al., 2025; Ghosh et al., 2024; Black et al., 2025b), yet a simple change in view can still divert them (Figure 1). An object held near the wrist camera can displace the instructed target; changing “pick up the bowl” to “push the bowl” may still elicit a grasp; and a gripper that closes on nothing may lift anyway. Task success cannot tell whether the policy uses the scene, instruction, and observed outcome to decide what to do, or merely replays an action when a familiar cue appears.

We call such dependencies modality shortcuts, a multimodal instance of shortcut learning (Geirhos et al., 2020) closely related to causal confusion in imitation learning (de Haan et al., 2019; Wen et al., 2020). A modality shortcut is not an unimportant input but an easily decoded cue that stands in for the evidence a decision requires. Such cues arise from the regularity of successful demonstrations: the target dominates the wrist view as the expert reaches for it, each object name may be paired with a single operation, and gripper closure is reliably followed by a lift. Behavior cloning rewards agreement with the demonstrated action, not the evidence used to predict it, so nothing in the objective stops the policy from relying on the cue.

This matters for scaling. Larger and more diverse demonstration corpora are central to generalist robot learning (Open X-Embodiment Collaboration et al., 2023; Khazatsky et al., 2024; Lin et al., 2025), but if new trajectories preserve the same correlations, the shortcut rule and the intended rule keep prescribing the same actions, and more data cannot distinguish them. Higher success on familiar tasks then need not mean more reliable target selection, instruction following, or outcome assessment: the policy grows more proficient where shortcut and evidence agree yet stays brittle where they disagree, which makes modality shortcuts a bottleneck for scaling.

Addressing this requires changing what the training data demand, not weakening an input modality: the evidence a correct decision needs should be easier to use, and shortcuts should be insufficient on their own (Section 3.1). Perturbot does so with three training-time data interventions and no extra model or forward pass at deployment: task-preserving wrist-view perturbations add plausible distractors without changing the target or invalidating the recorded action; decision-relevant captions make object attributes, spatial relations, and requested operations explicit, each supported by the recorded behavior; and caption-grounded trajectory expansion adds random-motion and failed-execution segments relabeled with the behavior they contain, which add behavioral support rather than recovery skills, since relabeling cannot create an unrecorded correction. Perturbot thus changes what is scaled: as data grow, it expands not only repetitions of familiar behavior but also the range of decisions that depend on the scene, instruction, and observed outcome.

Whether scaling achieves this cannot be read from task success: when shortcut and evidence agree, policies relying on either can succeed, and robustness to distractors can come from simply ignoring vision. We therefore propose GroundFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. It pairs each input with edits that should leave the action unchanged, such as an off-path distractor, and edits that should change it in a known way, such as naming a different object; a high score requires both, so ignoring an input does not help. Because it needs no robot rollouts, GroundFscore can be computed for every checkpoint as data grow: task success shows whether a policy improves, and GroundFscore whether it improves for the right reason.

With a π0.5 backbone, we evaluate all eight component combinations on 50 real-robot General PnP cases and cover broader skills in RoboTwin 2.0 (Chen et al., 2025b); ablations test whether success and evidence use rise together as data grow and whether GroundFscore predicts online failures. Our contributions are threefold:

1. 1.

   We identify modality shortcuts in VLA learning, unifying salience capture, noun lock-in, and motor inertia, and explain why scaling demonstrations that preserve these correlations need not remove them.

2. 2.

   We propose Perturbot, three independently ablatable data interventions that keep action supervision valid, change what is scaled, and leave inference unchanged.

3. 3.

   We introduce GroundFscore, an offline score that, unlike task success, indicates whether a policy scales healthily and cannot be raised by ignoring useful evidence.

2 Related work

Scaling VLA policies. Vision–language–action (VLA) policies map observations and instructions to actions; they are trained on large robot datasets and often initialized from pretrained vision–language models (Brohan et al., 2023; Zitkovich et al., 2023; Kim et al., 2025; Ghosh et al., 2024; Black et al., 2025b; Black et al., 2025a; Wang et al., 2025; Liu et al., 2025; Liu et al., 2026b). Their progress has followed the growth of demonstration data across tasks and embodiments (Open X-Embodiment Collaboration et al., 2023; Khazatsky et al., 2024; Walke et al., 2023) and studies of how data diversity and composition shape transfer (Lin et al., 2025; Hejna et al., 2024; Zha et al., 2025), while evaluation has expanded from nominal simulation benchmarks (Liu et al., 2023; Mees et al., 2022; Li et al., 2024) to perturbed scenes and instructions (Fei et al., 2025; Zhou et al., 2025; Zhao et al., 2026). Success alone cannot show whether a policy improves by using task evidence or by using a shortcut more consistently. GroundFscore measures shortcut reliance offline; unlike action sensitivity to task-preserving perturbations, which has been used to select demonstrations (D’urso et al., 2026), it also requires the action to change when task-relevant evidence changes.

Shortcut learning. Neural networks often rely on features that are predictive in training but fail under distribution shift, as studied under shortcut learning, simplicity bias, and dataset bias (Geirhos et al., 2020; Shah et al., 2020; Torralba and Efros, 2011). Visual question answering models can answer from language priors without grounding the evidence in the image (Goyal et al., 2017; Agrawal et al., 2018; Cadène et al., 2019), and in imitation learning, causal confusion and the copycat problem show that matching expert actions need not identify the evidence that should control behavior (de Haan et al., 2019; Wen et al., 2020; Park et al., 2021). Recent VLA studies find that policies can underuse language or let vision override instructions (Xu et al., 2025; Lian et al., 2026; Fang et al., 2026), and diagnose or regularize how modalities are used (Shi et al., 2026; Xu et al., 2026; Jia et al., 2026; Lin et al., 2026). These accounts usually examine one modality at a time; we tie each shortcut to the decision it affects and trace all three to the regularity of successful demonstrations.

Data interventions for robust policies. Data-centric methods improve robustness by varying appearance through augmentation and domain randomization (Tobin et al., 2017; Laskin et al., 2020; Bharadhwaj et al., 2023), synthesizing or counterfactually editing demonstrations (Mandlekar et al., 2023; Ameperosa et al., 2025), enriching or relabeling instructions (Kim et al., 2026; Glossop et al., 2025), and widening behavioral coverage with corrective, perturbed, or unstructured data (Ross et al., 2011; Laskey et al., 2017; Lynch et al., 2020; Hu et al., 2025; Liu et al., 2026a). Perturbative training has likewise reduced language-prior hallucinations in vision–language models (Chen et al., 2025a). Each of these interventions is typically designed for one axis of variation, such as appearance, geometry, language, or coverage; Perturbot instead targets the cue that substitutes for evidence in a given decision, while keeping every action label valid and leaving inference unchanged.

3 Method

We first model modality shortcuts (Section 3.1), then describe the three data components of Perturbot (Section 3.2) and how GroundFscore measures the resulting evidence use (Section 3.3).

3.1 Modeling modality shortcuts

We study three failures, one per decision a manipulation policy makes. In salience capture (what to manipulate), an object near the wrist camera can divert a reach from the instructed target; in noun lock-in (which semantics apply), a policy that has pushed other objects but only picked up the bowl may grasp it when asked to push it; in motor inertia (how to move), a gripper that closed on nothing may lift anyway. In each, a cue that is reliable in successful demonstrations stands in for the required evidence: wrist-view salience for the instruction, the noun for the verb, and the close–lift sequence for visual grasp feedback.

Consider a policy πθ​(a∣v,ℓ,q) mapping camera observations v (wrist views included), an instruction ℓ, and motor context q (proprioception, past actions) to an action or action chunk a. For a task decision d, a shortcut s, and task-relevant evidence e, Bayes’ rule under the data distribution p separates a shortcut-based prediction from an evidence update, whose expectation splits again:

where I⁡(⋅,⋅) is mutual information and ℛ=I⁡(d;s)+I⁡(d;e)−I⁡(d,s,e) is the redundancy, the information about d that s duplicates (nonnegative in the redundant regime we consider). A modality shortcut is the remainder I⁡(d;e∣s) vanishing on successful demonstrations, for one of two reasons. Under collinearity, ℛ≈I⁡(d,e): in salience capture the target fills the wrist view because the operator reaches for it, and in noun lock-in a noun always comes with one verb. Under degeneracy, the decision never varies, H⁡(d)≈0: in motor inertia nothing follows gripper closure but a lift. More demonstrations of the same kind preserve both, so scaling alone need not remove the shortcut.

Equation 1 thus suggests two moves with different costs. Lowering ℛ, as a decoy inserted off the grasp path does, makes the shortcut insufficient on its own without changing the expert action, so the labels come free. Raising I⁡(d,e), as captions naming decision-relevant attributes do, makes the evidence easier to use, but under degeneracy I⁡(d,e)≤H⁡(d) stays near zero until a missing branch, such as reopening after an empty grasp, is recorded; relabeling cannot supply it. These are goals for the training data, not a training loss or a guarantee of learned evidence use.

3.2 Perturbot

Perturbot trains on two sources: task demonstrations, which supervise real-robot General PnP and, separately, the 50 RoboTwin 2.0 tasks (Section 4), and recorded random-motion and failed-execution trajectories, screened and relabeled as auxiliary data rather than extra tasks. Its three components build training examples from both while keeping every action label valid (Figure 1): V targets the redundancy ℛ, C the evidence term I⁡(d,e), and R adds recorded segments in which d varies.

The visual component V applies task-preserving wrist-view perturbations TV: a plausible distractor of varying identity and placement is inserted off the grasp path, without hiding the target or changing contact geometry, so local salience changes while the task, motor context, and recorded action stay valid, and the edit itself cannot become a task cue. An edit that would change the target, collision constraints, or required grasp needs a new action and is rejected. The intended lesson is that a salient object need not be the instructed one, yet wrist evidence should still guide the motion; noise, masking, and camera removal are controls, not substitutes.

The caption component C applies decision-relevant caption enrichment TC and leaves observations and actions unchanged. A vision–language model splits each recording into atomic skills (grasp, lift, move, put down) and describes each by the acting hand and whether it opens or closes, the operation, the object with just enough attributes to single it out, and its position, initial state, and grasp point (Section E.1): the grasp in pick up the mug becomes the left hand closes and grasps the upright blue mug left of the tray by its body. The aim is decision-relevant detail, not length; a different verb appears only when the data contain that action, since rewriting a grasp as a push does not create a push demonstration. Both sources mix brief and detailed instructions under the same rules, so caption style does not reveal the source, and the deployed policy takes ordinary instructions.

The trajectory component R applies caption-grounded trajectory expansion TR to recorded random motion and failed executions. These are divided into segments with synchronized images, motor context, and actions; unusable segments are discarded, and each retained one gets a brief instruction for what it actually does, plus a detailed version from C. A failed attempt to pick up a bowl may contain a valid segment move the empty gripper left and open it, whose label describes this local behavior, not the original goal and not a failure that has yet to happen. R thus widens language-conditioned state–action coverage but does not by itself teach recovery, which requires recorded corrective continuations evaluated under the original instruction.

All three components train with the backbone’s own action loss ℒact (flow matching for π0.5). With policy input o=(v,ℓ,q), empirical distributions 𝒟demo of task demonstrations and 𝒟extra of relabeled auxiliary segments, and an auxiliary fraction ρ∈[0,1], we train on

where v~ is the original or perturbed visual input, ℓ~ is a brief or detailed instruction for the same behavior, and the expectation covers augmentation and caption sampling. Setting v~=v, ℓ~=ℓ, or ρ=0 disables V, C, or R; R without C still uses accurate brief instructions. All rates are fixed within a run and selected on validation data. Because V, C, and R need not act only on the visual, lexical, or motor shortcut family (cV,cL,cA), every configuration is scored on all three (Appendices B and F).

3.3 GroundFscore

Task success cannot separate a policy that reads the shortcut from one that reads the evidence, since both succeed on the demonstration distribution. GroundFscore breaks this agreement with paired edits for each shortcut family c∈{cV,cL,cA} (Figure 2a): a null edit e∈ℰc0 alters the input without changing the correct action, and a causal edit e∈ℰc1 changes the correct action by a known amount Δ​ae⋆. With a⋆ the reference action at the unedited observation o,

where the spurious response Rc measures how much the policy moves when it should not, and the causal sensitivity Sc is the fraction of the demanded change that its response realizes (1 for exact compliance, 0 for none). Clipping both to [0,1], we combine them as

an F-score with precision 1−Rc (not moving when nothing should move) and recall Sc (moving when something should). As in HalFscore (Chen et al., 2025a), the harmonic mean is high only when both terms are: a policy that ignores the input has Rc=0 but also Sc=0, and scores zero.

GroundFscore needs only paired forward passes, not rollouts. We compute it on 100 held-out General PnP observations, each given null and causal edits of all three families: language edits rewrite the instruction, visual edits are made by image editing, and motor edits substitute recorded alternatives. For each causal edit, Δ​ae⋆ is the difference between the action chunks of two expert recordings from the same state, one under the original and one under the edited condition. Actions are full predicted chunks in the normalized action space of π0.5, and the edited and unedited passes share the flow-matching noise, so sampling noise does not count as a response; ∥a⋆∥ is floored at a tenth of its median, and causal edits with ∥Δ​ae⋆∥ below a fifth of the median are discarded. The score is a diagnostic, not a certificate of task competence; Section 5.4 tests its association with online failures.

4 Experiments

4.1 Protocol

In real-robot General PnP, the robot places the instructed object in the instructed container. Pick-and-place underlies most longer tasks yet is far from trivial here: each layout draws six objects and three containers from 100 items, so combinations rarely repeat and evaluation layouts are unseen in training. Distractors in every scene invite salience capture, instructions require grounding the named object and container, and any grasp can close on nothing, the setting of motor inertia. Short, uniform demonstrations also make data scaling affordable (Section 5.1); RoboTwin 2.0 covers other skills (Section A.3).

The 50 evaluation layouts, generated once by rule-based random placement, are restored before each rollout, so every policy runs the same cases once (Figure 3, Section A.2).

All experiments use π0.5 (Black et al., 2025a); conditions share initialization, optimizer, training steps, and batch size, so each comparison varies only the training data, with auxiliary examples replacing a fixed fraction of each batch. Recordings and scene groups are split before segmentation, captioning, or augmentation, validation selects the fixed rates, and the primary result uses brief instructions (Appendix A).

4.2 General Pick and Place

Table 1 covers all eight combinations of V, C, and R, so each component appears alone and as a leave-one-out. An off-path wrist distractor probes salience capture (SC) and a grasp scripted to close on nothing probes motor inertia (MI); verb-change probes for noun lock-in run in RoboTwin 2.0 (Section F.2). The controls test generic regularization (noise and dropout at the V budget), data quantity (twice the demonstrations), discarding evidence (camera removal), and inference-time correction (negative guidance; Appendix A).

Each component raises success and mainly the GroundFscore of its target family: V lifts GFV from 0.28 to 0.51 and cuts SC from 68% to 36%, C lifts GFL from 0.24 to 0.47, and R lifts GFA from 0.19 to 0.44 and cuts MI from 82% to 44%. The gains compound: every pair beats both of its members, and the full method doubles baseline success to 84% with the best readout in all three families, while removing any one component costs 16–20 points. The other controls stay within 6 points of the baseline, and camera removal attains the lowest SC only by losing 16 points of success and lowering GFV to 0.16.

4.3 Generality across skills

RoboTwin 2.0 (Chen et al., 2025b) tests whether Perturbot generalizes beyond General PnP. Its 50 bimanual tasks span skills such as handing over a block, opening a microwave, hammering, and hanging a mug, and its domain randomization varies textures, lighting, clutter, tabletop height, and instruction wording. We follow the two settings of Cai et al. (2026b): Full trains on clean and randomized demonstrations, whereas Clean2Random trains only on clean ones, so randomized scenes are unseen; both evaluate on clean and randomized scenes. Clean2Random is the more direct test of modality shortcuts, since randomized clutter adds salient cues that clean training never contradicted. R uses simulator rollouts relabeled as in the real-robot study (Section A.3), and all variants share one budget; published results, from different architectures and recipes, give context only.

Perturbot improves the π0.5 baseline in every column of Table 2, most where shortcuts should matter: on randomized scenes in Clean2Random it gains 12.2 points, against 2.7 on clean scenes, the best randomized-scene result among the listed systems. Removing V costs most on randomized scenes in both settings (8.4 and 2.6 points), consistent with its role against salient distractors.

5 Ablation study

We test whether Perturbot’s gains require new rather than more data, are mechanism-specific, and preserve control, and whether GroundFscore predicts online failures. Unless noted, ablations use real-robot General PnP with π0.5 (Figure 4; full tables in Appendix F).

5.1 Data scaling

The default remedy for a brittle policy is more demonstrations, yet Section 3.1 predicts that same-kind ones preserve the collinearity and degeneracy behind shortcuts, so scaling tests whether the data must change. We train the π0.5 baseline and Perturbot on nested subsets of 125, 250, 500 (1×), and all 1,000 General PnP demonstrations at fixed optimizer steps, three seeds each, and report each curve’s least-squares slope per doubling (Table 4).

Perturbot scales faster, gaining 16.2 points of success per doubling against 6.0 for fine-tuning; at 1/4× it already matches fine-tuning at 2×, with eight times fewer demonstrations. The gains also differ in kind: from 1/4× to 2×, fine-tuning adds 18 points while GF¯ moves by only 0.04 and both failure rates stay high; extra demonstrations add proficiency but not reliance on task evidence, the unhealthy regime GroundFscore is meant to expose. Under Perturbot, success and GF¯ rise together, by 48 points and 0.30, and both failure rates fall at every budget. Success and GF¯ correlate at Spearman ρ=0.93 for Perturbot but 0.41 for fine-tuning, whose GF¯ barely varies, so seed noise dominates its ranking.

5.2 Component specificity

Every component adds training data, so its gain could come from more samples or generic regularization rather than from the shortcut it targets (Equation 1). Each is therefore compared with controls that match its budget but remove its mechanism (Table 4): for V, noise, cropping, and dropout, or the same distractors placed anywhere in the wrist view, including on the grasp path; for C, paraphrases or length-matched text that does not bear on the decision; for R, equal-duration successful demonstrations or either auxiliary source alone. Only the mechanism moves the targeted readout. V reaches a GFV of 0.51 against at most 0.38, and random placement drops success below the baseline, as on-path decoys invalidate the recorded action. Paraphrases and length-matched text raise GFL by at most 0.06, against 0.23 for C. Extra successful demonstrations leave MI near the baseline, whereas random and failed segments each lower it and together reach 44%.

5.3 Controllability

A policy can pass shortcut probes by ignoring an input, as removing the wrist camera does for salience capture (Table 1), but such robustness discards evidence the task needs. Table 6 therefore pairs changes that should not alter the action with changes that should: a distractor should not redirect a reach, yet the same object should be selected when it is instructed, and lifting should depend on the observed grasp outcome. Perturbot picks the distractor in 20% against 68% for fine-tuning, yet still selects the instructed object in 90%; without a wrist camera, a policy avoids the distractor more often but finds the instructed object in only 48%. Perturbot also lifts secured objects nearly as often as fine-tuning (92% vs. 94%) but lifts after empty grasps in 24% rather than 82%.

5.4 Validity of GroundFscore

GroundFscore is meant to replace costly shortcut-targeted rollouts with offline forward passes, which is only useful if it predicts what those rollouts reveal. Across the twelve conditions in Table 1 and three seeds, we rank-correlate GroundFscore on the 100 paired observations of Section 3.3 with the negative online failure rate on shortcut-targeted trials, and ablate the score itself: stability alone (1−R) rewards ignoring an input, and responsiveness alone (S) rewards moving under any change. GroundFscore predicts online failures best (ρ=0.72), and its advantage of 0.41 over task success has a 95% cluster-bootstrap interval of [0.12, 0.68] (Table 13). Neither term suffices alone: stability reaches 0.44 but ranks the no-wrist policy above the baseline, and responsiveness reaches only 0.38.

6 Limitations

Perturbot needs detailed captions and relabeled segments annotated to match the recorded behavior, and random and failed trajectories in addition to demonstrations; relabeling cannot create behaviors that were never recorded, such as recovery. Real-robot rollouts offer a ready source of such trajectories: successful trajectories tend to resemble one another, while each failure goes wrong in its own way.

7 Conclusion

VLA policies can succeed on familiar tasks while relying on modality shortcuts: visual, lexical, or motor cues that successful demonstrations make sufficient to predict the expert’s action, and that more demonstrations of the same kind keep predictive. Perturbot changes what the training data demand through task-preserving wrist-view perturbations, decision-relevant captions, and relabeled random and failed segments, without changing inference, and GroundFscore complements task success by diagnosing, offline, whether a policy improves through task evidence or through shortcuts. Together, they point to a different aim for scaling robot data: expanding not only repetitions of familiar behavior but also the range of decisions that depend on the scene, instruction, and observed outcome.

AI use statement

In this work, we used generative AI tools to polish the writing for readability, including rephrasing sentences and reorganizing sentence order and logical flow. All AI-assisted text was reviewed and verified by the authors, who take full responsibility for the final content of this work, including all text, claims, and artifacts.

Reproducibility statement

We will open-source the data, the trained checkpoints, and the models upon publication. The main modeling relation is Equation 1, and the GroundFscore equations and probes are specified in Section 3.3. Appendix A describes the real-robot platform and evaluation protocols, Appendix B the training data and its collection, Appendix C the π0.5 configuration, Appendix D the wrist-view perturbation, and Section E.1 the prompt used to generate detailed captions; the common action-training objective is Equation 2.

References

• A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971–4980. External Links: Link Cited by: §2.
• Ameperosa et al. (2025) E. Ameperosa, J. A. Collins, M. Jain, and A. Garg RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations. In IEEE International Conference on Robotics and Automation, External Links: Link Cited by: §2.
• Bharadhwaj et al. (2023) H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking. External Links: 2309.01918, Link Cited by: §2.
• Bi et al. (2026) H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, H. Zhao, H. Liu, Z. Su, L. Ma, H. Su, and J. Zhu Motus: A Unified Latent Action World Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 35101–35113. External Links: Link Cited by: Table 7.
• Black et al. (2025a) K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky π0.5: A vision-language-action model with open-world generalization. In Proceedings of the 9th Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research, Vol. 305, pp. 17–40. Cited by: Table 7, Appendix C, §2, §4.1, Table 2.
• Black et al. (2025b) K. Black, N. Brown, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, L. Smith, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky π0: A Vision-Language-Action Flow Model for General Robot Control. In Robotics: Science and Systems XXI, Vol. 21. External Links: Document, Link Cited by: Table 7, §1, §2, Table 2.
• Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, P. R. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. H. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich RT-1: Robotics Transformer for Real-World Control at Scale. In Robotics: Science and Systems XIX, Vol. 19. External Links: Document, Link Cited by: §2.
• Cadène et al. (2019) R. Cadène, C. Dancette, H. Ben-Younes, M. Cord, and D. Parikh RUBi: reducing unimodal biases for visual question answering. In Advances in Neural Information Processing Systems (NeurIPS), pp. 839–850. Cited by: §2.
• Cai et al. (2026a) J. Cai, L. Ling, S. Chu, Z. Liu, J. Kang, Z. Liang, W. Xu, Y. Mao, W. Zhang, X. Yang, R. Ying, R. Zheng, and Y. Mu AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing. External Links: 2606.09811, Link Cited by: Table 7.
• Cai et al. (2026b) J. Cai, Y. Mu, G. Yang, Z. Cao, Z. Tu, X. Gao, K. Li, X. Zhan, L. Yang, Y. Zhu, H. Ma, M. Zhou, Q. Yu, Y. Xue, L. He, Y. Yao, Y. Zhu, L. Ling, B. Jiang, H. Guo, X. Zhu, B. Zhou, B. Zhao, T. Xue, C. Shen, and W. Zhang InternW0: A Foundational Physical World Model for Efficient Real-World Interactions. External Links: 2609.27656, Link Cited by: §A.3, Table 7, Table 7, §4.3, Table 2.
• Chen et al. (2025a) C. Chen, M. Liu, C. Jing, Y. Zhou, F. Rao, H. Chen, B. Zhang, and C. Shen PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training. In International Conference on Learning Representations, External Links: Link Cited by: §2, §3.3.
• Chen et al. (2026) R. Chen, Y. Yang, Z. Tang, D. Huo, T. Lin, H. Wu, H. Liu, Y. Chen, L. Zheng, B. Yuan, T. Li, M. Wang, D. Qi, B. Hu, W. Mei, Y. Xuan, H. Yang, Y. Zhu, M. Xu, Z. Ma, and X. Chang ABot-M0.5: Unified Mobility-and-Manipulation World Action Model. External Links: 2607.00678, Link Cited by: Table 7.
• Chen et al. (2025b) T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H. Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088. External Links: 2506.18088, Link Cited by: §1, §4.3.
• de Haan et al. (2019) P. de Haan, D. Jayaraman, and S. Levine Causal Confusion in Imitation Learning. In Advances in Neural Information Processing Systems, pp. 11693–11704. External Links: Link Cited by: §1, §2.
• D’urso et al. (2026) G. D’urso, K. Roy, N. Lawrance, and B. Tidd It’s Not Just More Demos: Counterfactual Action Sensitivity Coverage for Data-Efficient Robust Robot Imitation. In RSS 2026 Workshop on Data-Centric Robotics: What Data Do Robots Really Need?, External Links: Link Cited by: §2.
• Fang et al. (2026) Y. Fang, Y. Feng, D. Jing, J. Liu, Y. Yang, Z. Wei, D. Szafir, and M. Ding When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs. External Links: 2602.17659, Link Cited by: §2.
• Fei et al. (2025) S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. External Links: 2510.13626, Link Cited by: §2.
• Geirhos et al. (2020) R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann Shortcut learning in deep neural networks. Nature Machine Intelligence 2 (11), pp. 665–673. External Links: Document, Link Cited by: §1, §2.
• Ghosh et al. (2024) D. Ghosh, H. R. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, Y. L. Tan, L. Y. Chen, Q. Vuong, T. Xiao, P. R. Sanketi, D. Sadigh, C. Finn, and S. Levine Octo: An Open-Source Generalist Robot Policy. In Robotics: Science and Systems XX, Vol. 20. External Links: Document, Link Cited by: §1, §2.
• GigaBrain Team et al. (2026) GigaBrain Team, A. Ye, A. Sun, C. Jin, C. Cheng, C. Shi, D. Shang, D. Zhang, G. Huang, G. Wang, G. Ding, G. Li, H. Li, H. Zhong, H. Lu, J. Qin, J. Mao, J. Zhu, J. Lv, J. Cui, J. Xie, J. Bao, K. Liu, L. Yuan, L. Long, L. Feng, M. Yu, P. Li, P. Yi, Q. Li, Q. Zhang, Q. Li, Q. Hu, R. Zhang, S. Sun, S. Sun, S. Duan, T. Chen, T. Liu, W. Ke, W. Xue, X. Wang, X. Tian, X. Liu, X. Chen, Y. Wang, Y. Wang, Y. Zeng, Y. Li, Y. Nie, Y. Li, Y. Liu, Y. Feng, Y. Wang, Y. Ye, Z. Liu, Z. He, Z. Yang, and Z. Zhu GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture. External Links: 2608.15875, Link Cited by: Table 7.
• Glossop et al. (2025) C. Glossop, W. Chen, A. Bhorkar, D. Shah, and S. Levine CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models. External Links: 2508.13446, Link Cited by: §2.
• Goyal et al. (2017) Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh Making the V in VQA matter: elevating the role of image understanding in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6325–6334. External Links: Document Cited by: §2.
• Guo et al. (2026) J. Guo, Q. Li, P. Li, Z. Chen, N. Sun, Y. Su, H. Wang, Y. Zhang, X. Li, and H. Liu Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising. External Links: 2604.26694, Link Cited by: Table 7, Table 2.
• Hejna et al. (2024) J. Hejna, C. Bhateja, Y. Jiang, K. Pertsch, and D. Sadigh Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning. External Links: 2408.14037, Link Cited by: §2.
• Hu et al. (2025) Z. Hu, R. Wu, N. Enock, J. Li, R. Kadakia, Z. Erickson, and A. Kumar RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction. External Links: 2509.07953, Link Cited by: §2.
• Jia et al. (2026) X. Jia, B. Yang, Z. Ge, X. Nie, Y. Zhou, C. Fan, Y. Li, Y. Chai, C. Jing, Z. Liang, Q. Bu, H. Cao, C. Wu, Q. Li, Z. Yang, C. Zhang, H. Li, Z. Wu, J. Yan, and Y. Jiang GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization. In Robotics: Science and Systems XXII, Vol. 22. External Links: Link Cited by: §2.
• Khazatsky et al. (2024) A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le, Y. Li, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martín, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn DROID: a large-scale in-the-wild robot manipulation dataset. In Proceedings of Robotics: Science and Systems, External Links: Document Cited by: §1, §2.
• Kim et al. (2026) B. Kim, R. Wang, D. Acuna, J. Jung, A. Trevithick, B. Cui, Y. Choi, and P. Ammanabrolu How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning. External Links: 2605.17077, Link Cited by: §2.
• Kim et al. (2025) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn OpenVLA: An Open-Source Vision-Language-Action Model. In Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 270, pp. 2679–2713. External Links: Link Cited by: §1, §2.
• Laskey et al. (2017) M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg DART: Noise Injection for Robust Imitation Learning. In Conference on Robot Learning, pp. 143–156. External Links: Link Cited by: §2.
• Laskin et al. (2020) M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.
• Li et al. (2026a) F. Li, W. Song, H. Zhao, J. Wang, P. Ding, D. Wang, L. Zeng, and H. Li Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model. In International Conference on Learning Representations, External Links: Link Cited by: Table 7, Table 2.
• Li et al. (2026b) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu Causal World Modeling for Robot Control. External Links: 2601.21998, Link Cited by: Table 7.
• Li et al. (2024) X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao Evaluating real-world robot manipulation policies in simulation. In Proceedings of the 8th Conference on Robot Learning (CoRL), pp. 3705–3728. Cited by: §2.
• Lian et al. (2026) S. Lian, B. Yu, X. Lin, L. T. Yang, Z. Shen, C. Wu, Y. Miao, C. Huang, and K. Chen LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries. In International Conference on Machine Learning, External Links: Link Cited by: §2.
• Lin et al. (2025) F. Lin, Y. Hu, P. Sheng, C. Wen, J. You, and Y. Gao Data Scaling Laws in Imitation Learning for Robotic Manipulation. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2.
• Lin et al. (2026) J. Lin, S. Shailesh, Z. Luo, and J. Duan Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models. External Links: 2609.12641, Link Cited by: §2.
• Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Cited by: §2.
• Liu et al. (2025) M. Liu, Z. Huang, X. Lin, M. Zhu, C. Zhao, Y. Wang, H. Zhu, H. Chen, and C. Shen GAE: unleashing physical potential of vlm with generalizable action expert. In Forty-third International Conference on Machine Learning, Cited by: §2.
• Liu et al. (2026a) M. Liu, Z. Li, J. Shu, H. Wang, Y. Chao, H. Chen, and C. Shen Perfect demo makes poor teacher: learning robust alignment from critical motion segments. arXiv preprint arXiv:2606.15587. Cited by: §2.
• Liu et al. (2026b) M. Liu, J. Shu, H. Chen, Z. Li, C. Zhao, J. Yang, S. Gao, H. Chen, and C. Shen Stamo: unsupervised learning of generalizable robot motion from compact state representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 35014–35024. Cited by: §2.
• Lynch et al. (2020) C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet Learning Latent Plans from Play. In Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 100, pp. 1113–1132. External Links: Link Cited by: §2.
• Mandlekar et al. (2023) A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations. In Conference on Robot Learning, pp. 1820–1864. External Links: Link Cited by: §2.
• Mees et al. (2022) O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard CALVIN: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters 7 (3), pp. 7327–7334. External Links: Document Cited by: §2.
• Open X-Embodiment Collaboration et al. (2023) Open X-Embodiment Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang, H. Su, H. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Vakil, J. Bohg, J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang, J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. ". Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, M. Z. Irshad, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. ". Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Martín-Martín, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, V. Guizilini, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin Open X-Embodiment: Robotic Learning Datasets and RT-X Models. External Links: 2310.08864, Link Cited by: §1, §2.
• Park et al. (2021) J. Park, Y. Seo, C. Liu, L. Zhao, T. Qin, J. Shin, and T. Liu Object-aware regularization for addressing causal confusion in imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3029–3042. Cited by: §2.
• Ross et al. (2011) S. Ross, G. Gordon, and D. Bagnell A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 627–635. External Links: Link Cited by: §2.
• Shah et al. (2020) H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.
• Shi et al. (2026) H. Shi, X. Ren, Y. Zhang, Q. Zhang, J. Hu, H. Shan, H. Dong, J. Lu, Y. Chen, Y. Zhang, Y. Dai, and X. Ju VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing. External Links: 2605.30117, Link Cited by: §2.
• Tobin et al. (2017) J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30. External Links: Document, Link Cited by: §2.
• Torralba and Efros (2011) A. Torralba and A. A. Efros Unbiased look at dataset bias. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1521–1528. External Links: Document Cited by: §2.
• Walke et al. (2023) H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, A. Lee, K. Fang, C. Finn, and S. Levine BridgeData V2: a dataset for robot learning at scale. In Proceedings of the 7th Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research, Vol. 229, pp. 1723–1736. Cited by: §2.
• Wang et al. (2025) Y. Wang, H. Zhu, M. Liu, J. Yang, H. Fang, and T. He Vq-vla: improving vision-language-action models via scaling vector-quantized action tokenizers. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11089–11099. Cited by: §2.
• Wang et al. (2026) Y. Wang, S. Huang, M. Li, C. Zhang, J. Liang, W. Jin, Y. Chen, X. Chi, D. Zhou, Q. Yu, Y. Wang, Y. Rui, S. Yao, Z. Yuan, Z. Shen, K. Zhu, Z. Zhu, N. Gao, X. Chi, G. He, S. Zhang, H. Dong, L. Shao, and H. Zhao OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. External Links: 2609.07398, Link Cited by: Table 7.
• Wen et al. (2020) C. Wen, J. Lin, T. Darrell, D. Jayaraman, and Y. Gao Fighting Copycat Agents in Behavioral Cloning from Observation Histories. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §2.
• Xu et al. (2026) H. Xu, S. Zheng, H. Luo, W. Zhang, Z. Xi, and Z. Lu Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models. External Links: 2604.18000, Link Cited by: §2.
• Xu et al. (2025) K. Xu, Z. Zhu, A. Chen, S. Zhao, Q. Huang, Y. Yang, H. Lu, R. Xiong, M. Tomizuka, and Y. Wang Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy. External Links: 2512.11218, Link Cited by: §2.
• Yang et al. (2026a) L. Yang, W. Song, X. Wang, P. Sheng, Z. Fang, Z. Zhou, J. He, H. Yan, J. Chen, N. Sun, Q. Sun, P. Wang, L. Liu, Y. Wang, Y. Gao, F. Dayoub, and H. Li 4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields. External Links: 2608.08023, Link Cited by: Table 7, Table 2.
• Yang et al. (2026b) Y. Yang, S. Zeng, T. Lin, X. Chang, D. Qi, J. Xiao, H. Liu, R. Chen, Y. Chen, D. Huo, F. Xiong, X. Wei, Z. Ma, and M. Xu ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learning. External Links: 2602.11236, Link Cited by: Table 7, Table 2.
• Yuan et al. (2026) T. Yuan, Z. Dong, Y. Liu, and H. Zhao Fast-WAM: Do World Action Models Need Test-time Future Imagination?. External Links: 2603.16666, Link Cited by: Table 7.
• Zha et al. (2025) L. Zha, A. Badithela, M. Zhang, J. Lidard, J. Bao, E. Zhou, D. Snyder, A. Z. Ren, D. Shah, and A. Majumdar Guiding Data Collection via Factored Scaling Curves. External Links: 2505.07728, Link Cited by: §2.
• Zhao et al. (2026) M. Zhao, Z. Li, C. Huang, M. Ma, H. Jiang, Y. Jin, X. Wang, Y. Du, X. Lin, T. Ding, H. Xie, J. Jiang, C. Yu, K. Zhang, L. Huang, L. Liu, T. Lin, and Z. Su InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation. External Links: 2608.22990, Link Cited by: §2.
• Zheng et al. (2026) J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, T. Wang, Y. Zhang, J. Liu, and X. Zhan X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model. In International Conference on Learning Representations, External Links: Link Cited by: Table 7, Table 2.
• Zhou et al. (2025) X. Zhou, Y. Xu, G. Tie, Y. Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun LIBERO-PRO: towards robust and fair evaluation of vision-language-action models beyond memorization. External Links: 2510.03827 Cited by: §2.
• Zitkovich et al. (2023) B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu, S. Levine, L. Lee, T. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, B. Ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, C. Finn, K. A. Dubey, D. Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, J. Carbajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. In Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 229, pp. 2165–2183. External Links: Link Cited by: §1, §2.

Appendix

Appendix A Experimental setup

A.1 Real-robot platform

All real-robot data (task demonstrations, random-motion and failed-execution recordings, and evaluation rollouts) are collected on the dual-arm AgileX PiPER X platform in Figure 5. Two 6-DoF arms, each with a two-finger parallel gripper, are mounted side by side facing the tabletop. A RealSense D435 head camera is mounted on the central vertical post, and a RealSense wrist camera is mounted on each arm. All three streams are recorded at 30 FPS and 640×360 pixels. The head camera supplies the third-person view vthird and the wrist cameras supply vwrist; the three views are resized to the policy input resolution (Table 9) and form the visual input v. The motor context q is the proprioceptive state of both arms, and the action a commands both arms and their grippers. Demonstrations and auxiliary recordings are collected by teleoperation.

A.2 General PnP evaluation

The real-robot study evaluates only General PnP: six objects and three containers on the tabletop, with instructions such as place the red apple in the basket (Figure 3). Placement rules are fixed before evaluation. Each of the 50 cases specifies the object identities, target, destination, and instruction; object and container positions are drawn at random in a common tabletop frame, and each object’s orientation follows its placement rule. Conflicts are resolved by fixed rules (a distractor overlapping the target is moved rather than the target, and an object–container conflict moves the container), and the adjusted coordinates are stored. The 50 layouts are generated once and restored before every rollout, so all policies and training seeds face the same cases.

Each checkpoint executes every case once. We report success over the 50 cases, the success of the five blocks G1–G5 (cases 1–10, …, 41–50), and the mean 0–5 progress score. The primary evaluation uses brief instructions; a second pass replays the same 50 cases with detailed instructions (Section F.3). The salience-capture, motor-inertia, target-selection, and grasp-outcome probes use separate trial lists and add no physical-robot task; each list has 50 trials per policy. The intervals in Table 1 are 95% percentile bootstrap intervals from 10,000 resamples of the evaluation units of one checkpoint (cases, probe trials, or GroundFscore observations), reported as half-widths.

A.3 RoboTwin 2.0

RoboTwin 2.0 is evaluated on its 50-task suite in the two settings of Cai et al. (2026b). Full trains on the official demo_clean and demo_randomized demonstrations, whereas Clean2Random trains only on demo_clean; both evaluate on held-out clean and randomized trials, so Clean2Random tests unseen randomized scenes. Within a setting, all methods share the simulator randomization, demonstration pool, initial-state seeds, and trial budget, and the simulator’s domain randomization is not counted as V. Verb-change probes for noun lock-in use only task pairs that admit both operations from matched states with valid demonstrations (Section F.2). Table 7 lists all published results reported by Cai et al. (2026b).

Auxiliary trajectories.

In simulation, R uses rollouts instead of teleoperated recordings. Random-motion trajectories start from a task’s initial state and execute scripted, randomly sampled arm motions that sweep the workspace, as in the real-robot protocol. Failed trajectories come from the simulator’s scripted expert with injected perturbations that make a grasp or placement go wrong, such as offset grasp poses, early gripper release, or shifted placement targets; only rollouts that the success check marks as failed are kept, so they contain the near misses, empty grasps, and misplacements that successful demonstrations lack. Because the simulator records exact states and actions, both kinds are segmented, captioned, and relabeled with the same rules as the real recordings (Appendix B). In Clean2Random, rollouts use clean scenes only, so no randomized scene enters training.

Appendix B Training data

Table 8 summarizes the real-robot training data. All of it is collected on the General PnP setup: task demonstrations supervise the task, and the auxiliary recordings supply R, never complete the task, and are not evaluation cases.

B.1 General PnP demonstrations

We collect 1,000 teleoperated demonstrations of the task, each with its brief instruction. All main conditions train on a fixed subset of 500 (1×); the 1/4× and 1/2× budgets are nested subsets of it, the 2× budget and the 2× demonstrations control use all 1,000, and the equal-duration successful demonstrations of the R control come from the 500 outside the 1× subset. Items come from a pool of 100 objects and containers (Figure 3); each layout places nine of them, six objects and three containers as in evaluation, at random positions, and the 50 evaluation layouts are never used for training. Recordings and scene groups are split into training, validation, and evaluation before any segmentation, captioning, or augmentation, and all derivatives of a recording stay in its split.

B.2 Random-motion and failed-execution recordings

The auxiliary recordings are collected to produce varied contacts and contact failures rather than completed tasks: the arm approaches objects, touches them, and occasionally touches the wrong one, pushes in the wrong direction, or grasps nothing. Each recording lasts 5–10 s and contains three to six distinct motions, without long pauses and without completing a task. Each uses a layout of nine items from the same pool as the demonstrations, with a randomly varied tablecloth color.

Random motion (200 recordings).

With the items of a layout randomly placed, the operator moves the arm through three to six consecutive motions (left–right, forward–backward, up–down, wrist rotation, and gripper opening or closing), covering the workspace without idling, reaching its edge, or repeating one motion, and then resets. Each recording must show clear motion with changes of direction and height. These recordings expose how commanded actions move the robot.

Failed execution (200 recordings).

Each recording brings the gripper within 2–5 cm above or 1–3 cm beside an object, makes one contact attempt (pressing down, pushing horizontally, or closing on part of the object), follows with one variation (pushing again, changing direction, lifting slightly and setting down, or rotating the wrist), and leaves. Operators deliberately make small errors: near misses, off-center or wrong-direction pushes, empty grasps, grasps that slip or are released after a short lift, and pushes that barely move the object. They act quickly, as in play rather than task execution. The recordings are balanced across three outcomes: a deliberate miss, a light touch, and a clear interaction, for example tipping a bowl. They expose the consequences of contact, including the empty grasps and failed pushes that successful demonstrations never show.

B.3 Segmentation, relabeling, and mixture

Each auxiliary recording is assigned to a split and then divided into atomic-skill segments with synchronized images, motor context, and actions (Appendix E); unusable segments are discarded. Each retained segment receives a correct brief instruction for its local behavior and a detailed caption. The original goal of a failed attempt is never kept as its label, an action chunk must stay inside the segment its instruction describes, and measured states and recorded actions are never overwritten. Because the recordings contain no corrective continuations, R widens state–action coverage but does not demonstrate recovery under the original goal.

Training samples task demonstrations with probability 1−ρ and auxiliary segments with probability ρ (Equation 2); the wrist-perturbation probability pV and the detailed-caption probability pC apply to both sources, so caption style does not reveal the source. The three rates are fixed within a run and selected on validation data. Disabling V, C, or R sets pV, pC, or ρ to zero, and R without C still uses correct brief instructions. Auxiliary segments replace a fraction of each batch rather than adding optimizer updates.

Appendix C π0.5 training

We fine-tune π0.5 (Black et al., 2025a) with the configuration in Table 9, shared by all conditions; only the data construction varies. The policy receives the head and both wrist views, the instruction, and the dual-arm proprioceptive state, and is trained with its flow-matching action loss to predict chunks of H=16 actions, about one second at the 15 Hz control rate. All modules are fine-tuned. Every condition and data-scaling budget uses the same optimizer steps, each run takes about four days on eight GPUs, and the enabled values of pV, pC, and ρ are shared across the factorial comparisons.

Appendix D Wrist-view perturbation

V edits wrist-view images with GPT-Image-2.5 (gpt-image-2.5-flare),¹¹ 1 OpenAI, https://openai.com/index/introducing-chatgpt-images-2-5/. an image generation and editing model, to add plausible distractor objects. An edit is kept only if the instructed target, the relevant spatial relations, the grasp path, and the contact evidence remain readable and no new collision is implied; otherwise it is rejected rather than given the original label. Only the wrist image changes: the head view, motor state, instruction, and action label stay as recorded. Distractor identity, appearance, and placement vary independently of the task, and the training distractors are disjoint from the decoys used by the GroundFscore probes (Section 3.3). Generic corruption (noise, cropping, and dropout at the same rate) and wrist-camera removal are separate controls, and the salience-capture probes use physical distractors, so a policy cannot pass them by detecting editing artifacts.

Appendix E Detailed-caption annotation

Each behavior has a brief instruction and a detailed caption. Detailed captions are generated offline by a vision–language model with the prompt in Section E.1: the model receives the timestamped frames of an entire recording, splits it into contiguous atomic-skill segments, and describes each segment by the acting hand and whether it opens or closes, the operation, the position, the object’s state, its distinguishing attributes, the object, and the grasp point. Attributes such as color or position are added only when needed to single out the object, and duplicates are distinguished by counting from one side, so captions carry decision-relevant detail rather than length. Each description uses only features visible in the frames and is checked against the images and action record; uncertain object identities, motion directions, or contact descriptions are corrected or excluded. A different verb is used only when the recorded action matches it, and the same rules apply to demonstrations and auxiliary recordings.

E.1 Annotation prompt

Appendix F Additional ablation results

F.1 Sensitivity to the sampling rates

We vary the wrist-perturbation probability pV, the detailed-caption probability pC, and the auxiliary fraction ρ one at a time, holding the other two at their validation-selected values (Table 10). Level 0 disables the component and coincides with the corresponding leave-one-out row of Table 1, and 1× is the full method. Reporting success together with GroundFscore and the two failure rates tests whether stronger perturbation buys resistance to shortcuts at the cost of useful conditioning; Figure 4c plots the change in success from the selected setting against each rate.

F.2 Verb-change probes for noun lock-in

Real-robot General PnP has a single operation, so it cannot test whether a noun overrides a changed verb. We therefore probe noun lock-in in RoboTwin 2.0 under the Full setting, using 5 task pairs whose shared object admits both operations from the same initial state and whose scripted expert produces a valid demonstration of each, such as pressing versus moving the stapler. A probe restores the initial state of one task in a pair on a clean scene and issues the brief instruction of the other, so the scene and the noun favor the familiar operation while only the verb specifies the correct one. Each pair is probed in both directions with 10 trials per direction, giving 100 trials per policy. A trial counts as followed if the policy completes the instructed operation, as lock-in if it performs the operation of the restored task, and as other otherwise.

Fine-tuning performs the familiar operation in 52% of probes and the instructed one in only 36% (Table 12). Perturbot reverses this, and removing C restores most of the lock-in (43%), whereas removing V or R raises it by at most 7 points, consistent with C making the requested operation explicit. Because C never introduces a verb absent from the data, the gain reflects grounding verbs the demonstrations already contain.

F.3 Detailed-instruction evaluation

The primary evaluation uses brief instructions, as a deployed policy would receive. A second pass replays the same 50 cases with a detailed instruction for each, written under the rules of Appendix E from the stored case specification and checked against the initial scene; the rest of the protocol is unchanged (Table 12). Without C, detailed instructions are out of distribution and cost 6–10 points; with C, they add 2–6 points. Perturbot therefore does not depend on detailed instructions at test time: its brief-instruction success exceeds every other configuration under either instruction style.

F.4 Confidence intervals for the validity of GroundFscore

The 36 checkpoints of Table 6 (twelve conditions, three seeds) share evaluation cases and training recipes, so they are not independent. We therefore report 95% percentile intervals from 10,000 two-level cluster bootstrap resamples: each resample draws conditions and seeds with replacement and, within each checkpoint, resamples its evaluation units (cases, probe trials, and the 100 GroundFscore observations). All predictors share the same draws, so their differences are paired (Table 13). The lower end of the GroundFscore interval lies above the point estimate of every other predictor, and its paired advantage excludes zero against each of them; the interval for task success includes zero.