SWAM: Scaling World Action Models with Cross-Modal Alignment
SWAM: Scaling World Action Models with Cross-Modal Alignment
This research paper introduces SWAM, a world action model that jointly predicts video and actions conditioned on language. The authors address the challenge of maintaining language-video alignment while introducing action learning by curating approximately 47,000 hours of diverse data including general videos, egocentric videos, UMI demonstrations, and robot demonstrations. They systematically study how different supervision strategies and model architectures affect performance. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. The resulting model achieves 88.10% success on Easy (Clean-to-Clean) and 61.56% on Hard (Clean-to-Random) conditions in simulation, and 40.00% success across three real-world tasks compared to 23.33% for the strongest baseline. The authors release the curated data, training code, and model weights.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
Reported duration
≈47,000 hours
View original evidence
design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic
Open source S3Reported success rate
88.1%
Reported trials
View original evidence
control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three rea
Open source S5Reported success rate
61.56%
Reported trials
View original evidence
control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three rea
Open source S5
Dataset details reported by the source
- Modalities
- video Open source S3action Open source S3
- Embodiments
- general videos Open source S3egocentric videos Open source S3UMI demonstrations Open source S3robot demonstrations Open source S3
- paper: not_reported
- code: Reported pending Open source S6
- weights: Reported pending Open source S6
- code: not_reported
What remains unknown
Not established in the collected evidence: Environment, Control, Data origin.
Dataset licensing and commercial-use rights are not established here.
Reported performance applies to the described task. It does not establish general autonomy or deployment readiness.
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic
Open source S3
control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three rea
Open source S5
SWAM: Scaling World Action Models with Cross-Modal Alignment
Open source S1
arXiv:2611.12870v1 Announce Type: cross Abstract: World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision
Open source S2
l-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
Open source S6
Implications for data suppliers
RoboSignal interpretation and collection questions, not statements of buyer demand.
- Confirm the required data type and collection setting with the buyer; this source does not establish a complete collection specification.
- Compare the reported units and scope before using these quantities in a budget. Recording hours, sensor-hours and trajectories are different measures.
- Check the exact dataset version, license, modalities and task coverage against the intended use.
- Validate demand and acceptance criteria with a buyer before scaling. Publication, popularity and a research result do not establish a purchase commitment.
Source：arXiv Robotics — research abstracts · arxiv.org